{"entities":[{"id":"vct-benchmark-paper-2025","name":"Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark (arXiv:2504.16137)","subtype":"Preprint","description":"Version 2 of 29 April 2025, as described on the arXiv abstract page. Dates from the arXiv abstract page: submitted 21 April 2025 (v1), last revised 29 April 2025 (v2).","sourceIds":["source-ab-vct-abs"]},{"id":"arxiv","name":"arXiv"},{"id":"bio-misuse-capability-evaluation","name":"Biological misuse capability evaluation"}],"claims":[{"id":"ab-ev-vct-benchmark-measures","statement":"The Virology Capabilities Test paper states that it presents a large language model benchmark measuring the capability to troubleshoot complex virology laboratory protocols, built from the inputs of dozens of expert virologists.","derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-ab-ev-vct-benchmark-measures-1"]},{"id":"ab-ev-vct-benchmark-result","statement":"The Virology Capabilities Test paper states that expert virologists with internet access scored an average of 22.1% on the benchmark's questions within their own sub-areas of expertise, while the most performant model it tested, OpenAI's o3, reached 43.8% accuracy and outperformed 94% of those experts within their sub-areas.","derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-ab-ev-vct-benchmark-result-1"],"attributes":{"evaluatedSystem":"The models the paper tested, of which it names OpenAI's o3 as the most performant","evaluation":{"what":"Performance on the benchmark's multimodal questions, compared with expert virologists in their own sub-areas","how":"A 322-question multimodal benchmark scored against a human expert baseline with internet access","result":"Expert average of 22.1% within sub-areas of expertise; the most performant model reached 43.8% accuracy, outperforming 94% of experts within their sub-areas","caveats":"The paper states that the capability measured is inherently dual-use and proposes that it be taken into existing frameworks for dual-use technologies; it does not state that benchmark performance measures real-world capability."}}}],"sources":[{"id":"source-ab-vct-abs","title":"arXiv abstract page for arXiv:2504.16137 (Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark)","authors":["arXiv (Cornell University)"],"publisher":"arXiv (Cornell University)","publicationDate":null,"version":"Retrieved for the candidate pack on 2026-10-28 (SHA-256 3ad6043cc84942ab180140c67ad4e128b37899ee1e4fc957d8f80d93a80d4898). Abstract page as served on 2026-10-28, listing v1 of 21 April 2025 and v2 of 29 April 2025 (the current version). Only the abstract page was read; the full paper was not retrieved, so no record rests on its body. Retrieved again at admission on 2026-10-30 (SHA-256 cc6f6960f2503c7789a77800e49cac2de3af708305ca7f232a5322507a12bc80). Its wording can change; earlier wording is not authenticated.","evidenceUrl":"https://arxiv.org/abs/2504.16137","language":"en"}],"evidence":[{"id":"evidence-ab-ev-vct-benchmark-measures-1","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-ab-ev-vct-benchmark-result-1","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1252,1261,1266],"claims":[3380,3381],"sources":[1091],"evidence":[4643,4644]},"version":"1lg47t7"}