Claim · 2025-04-29 · revision · Stated in source · Supported

The Virology Capabilities Test paper states that expert virologists with internet access scored an average of 22.1% on the benchmark's questions within their own sub-areas of expertise, while the most performant model it tested, OpenAI's o3, reached 43.8% accuracy and outperformed 94% of those experts within their sub-areas.

One recorded claim. The claim panel gives its qualification, the source passages with their place in each source, and how it was checked.

What it connects

Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark (arXiv:2504.16137) → states evaluation result → The paper states that expert virologists with internet access scored an average of 22.1% on questions in their own sub-areas, while the most performant model tested, OpenAI's o3, reached 43.8% accuracy, outperforming 94% of the expert virologists within their sub-areas.

Where it appears

No collection page holds this claim. It is listed with its records in the explorer.

Claim and evidence

Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark (arXiv:2504.16137)states evaluation result ↓The paper states that expert virologists with internet access scored an average of 22.1% on questions in their own sub-areas, while the most performant model tested, OpenAI's o3, reached 43.8% accuracy, outperforming 94% of the expert virologists within their sub-areas.

Machine-checked, not reviewed by an editorStated in sourceSupported

Supporting evidence: primary · 1 source

Subject version: Version 2 of 29 April 2025, as described on the arXiv abstract page · Published 2025-04-21 · Effective date not established · Revised 2025-04-29
Date: 2025-04-29 · revision
Evaluated system: The models the paper tested, of which it names OpenAI's o3 as the most performant
What the report measured: Performance on the benchmark's multimodal questions, compared with expert virologists in their own sub-areas
By what kind of method: A 322-question multimodal benchmark scored against a human expert baseline with internet access
The result, in the report’s words: Expert average of 22.1% within sub-areas of expertise; the most performant model reached 43.8% accuracy, outperforming 94% of experts within their sub-areas
The caveats the report gives: The paper states that the capability measured is inherently dual-use and proposes that it be taken into existing frameworks for dual-use technologies; it does not state that benchmark performance measures real-world capability.

Scope and limitsA benchmark result as the paper states it, for the models and expert comparison group it describes at that date. Scores on this benchmark are not a measure of real-world capability or risk, and nothing is generalised to other models or later versions. Only the abstract page was read, so the figures are the paper's own summary. The figures and the comparison are recorded; no question, subject matter or procedure is recorded. Paraphrase is used because the abstract page renders the numbers in LaTeX markup.

The evidence 01

primary sourceSupports the claim

arXiv abstract page for arXiv:2504.16137 (Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark)

arXiv (Cornell University) · Publication date unknown

Cite the original source

Cite the original document, not Selekoa. Check these details against it.

Paraphrase

The abstract states that expert virologists with access to the internet score an average of 22.1% on questions in their own sub-areas of expertise, and that the most performant model tested, OpenAI's o3, reaches 43.8% accuracy, outperforming 94% of expert virologists even within their sub-areas of specialisation.

Location in source Abstract, third and fourth sentences

How it was read Retrieved directly from the publisher

Machine-checked · 2026-10-30

Source details and check notes

Authors: arXiv (Cornell University)
Version: Retrieved for the candidate pack on 2026-10-28 (SHA-256 3ad6043cc84942ab180140c67ad4e128b37899ee1e4fc957d8f80d93a80d4898). Abstract page as served on 2026-10-28, listing v1 of 21 April 2025 and v2 of 29 April 2025 (the current version). Only the abstract page was read; the full paper was not retrieved, so no record rests on its body. Retrieved again at admission on 2026-10-30 (SHA-256 cc6f6960f2503c7789a77800e49cac2de3af708305ca7f232a5322507a12bc80). Its wording can change; earlier wording is not authenticated.
Canonical address: Open the canonical source ↗
Accessed: 2026-10-28
Language: English (en)
Attribution: arXiv (Cornell University), “arXiv abstract page for arXiv:2504.16137 (Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark)”.
Reuse: No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution.
Check notes: Candidate pack passage, AI and biosecurity (research/ai-bio). The paraphrase is Selekoa’s, written from the text retrieved for the candidate pack on 2026-10-28; it is not a quotation and was not machine-matched. Explicit paraphrase: the abstract page renders the percentages as LaTeX, so a verbatim quotation would not match the retrieved text. A cross-family machine check (2026-10-30, item 28) re-read the record’s sources and found full support. No human editorial review.

Status and meaning

Editorial status. Not reviewed by an editor. An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.

Relation. A report states the result of an evaluation it ran or commissioned, in its own words, with what was measured, by what kind of method, and the caveats the report itself gives (schema 2.22). The record is the report’s statement, not a verified finding: it says nothing about another system, a later version or real-world risk, and a null or limiting result is not evidence that no effect exists.

Date. As stated in version 2 of the preprint; the dates of the model evaluations are not given on the abstract page.

Record ID ab-ev-vct-benchmark-result