Claim and evidence Close ×
The joint pre-deployment evaluation summary for OpenAI's o1 states that US AISI found the model's performance on a set of multiple-choice biology research task questions to be generally comparable to the best-performing reference models tested across an array of question sets. Copy claim link
Copy claim reference
Machine-checked, not reviewed by an editor Stated in source Supported
Supporting evidence: primary · 1 source; Context: primary · 1 source
Date 2024-12-18 · publication Evaluated system o1 as made available for pre-deployment testing, compared with o1-preview, GPT-4o and two versions of Claude 3.5 Sonnet What the report measured The model's relative biological capabilities on practical research tasks, including how it could be misused By what kind of method Question answering on multiple-choice research task questions, with an evaluation method that also augments the model with access to bioinformatic tools The result, in the report’s words Performance generally comparable to the best-performing reference models tested; the summary states that performance increased when the model was augmented with additional tooling The caveats the report gives The summary states that the biological findings are from US AISI's testing only, that the tested version had tool-calling and formatting problems addressed by adapted agent designs, that smaller differences may fall within the margins of error, and that the report makes no claims about the performance of other versions of o1. Scope and limits The result is the institutes' own, for the version tested and the reference models named. The summary states that the biological findings rest on US AISI's testing only, that the version tested had performance issues the evaluators worked around with scaffolding, and that the report makes no claims about the performance of other versions of o1. Only the summary's category-level wording is recorded; the tasks and tools are not described.
The evidence 02 primary source Supports the claim
Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024) ↗ National Institute of Standards and Technology · 2024-12-18
Archived copy, captured 2024-12-19: Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024) ↗
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “Overall, US AISI found that on a set of multiple-choice biology research task questions, the o1 model achieves performance that is generally comparable to best-performing reference models tested across an array of question sets.” Location in source 'Biological Capabilities' section, key findings, first bullet
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors National Institute of Standards and Technology (U.S. AI Safety Institute) Version Retrieved on 2026-10-27 (SHA-256 b2604d18a75f4b2ea52c3a6f70df54230fb62971a7bdbff96f1b15fb29ea4315). The cross-family check of 2026-10-28 opened the Internet Archive capture linked here and matched the quoted wording in it, the release-period expression (decision 9); a capture shows only what the archive retrieved that day. Canonical address Open the canonical source ↗ Accessed 2026-10-27 Language English (en) Archive copy Open archived copy, captured 2024-12-19: Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024) ↗ . Captured 1 day after the recorded publication date. An archive copy shows only what the archive retrieved on 2024-12-19. It does not prove what the page said earlier, and it can differ from the copy checked on 2026-10-27.Attribution National Institute of Standards and Technology (U.S. AI Safety Institute), “Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024)”, 2024-12-18. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (34 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval (SHA-256 8f38bef33205a708d2246601bb0c39020aa5ed69c4b396a44284f7aa70799ece; the bytes differ from the pack’s retrieval). Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-30, item 26) re-read the record’s sources and found full support. No human editorial review. primary source Context only
Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024) ↗ National Institute of Standards and Technology · 2024-12-18
Archived copy, captured 2024-12-19: Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024) ↗
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “This report makes no claims about the performance of other versions of o1.” Location in source 'Overview of the Joint Safety Research & Testing Exercise', paragraph on performance issues
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors National Institute of Standards and Technology (U.S. AI Safety Institute) Version Retrieved on 2026-10-27 (SHA-256 b2604d18a75f4b2ea52c3a6f70df54230fb62971a7bdbff96f1b15fb29ea4315). The cross-family check of 2026-10-28 opened the Internet Archive capture linked here and matched the quoted wording in it, the release-period expression (decision 9); a capture shows only what the archive retrieved that day. Canonical address Open the canonical source ↗ Accessed 2026-10-27 Language English (en) Archive copy Open archived copy, captured 2024-12-19: Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024) ↗ . Captured 1 day after the recorded publication date. An archive copy shows only what the archive retrieved on 2024-12-19. It does not prove what the page said earlier, and it can differ from the copy checked on 2026-10-27.Attribution National Institute of Standards and Technology (U.S. AI Safety Institute), “Pre-Deployment Evaluation of OpenAI’s o1 Model (December 18, 2024)”, 2024-12-18. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (13 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval (SHA-256 8f38bef33205a708d2246601bb0c39020aa5ed69c4b396a44284f7aa70799ece; the bytes differ from the pack’s retrieval). Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-30, item 26) re-read the record’s sources and found full support. No human editorial review. Status and meaning Editorial status. Not reviewed by an editor . An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.
Relation. A report states the result of an evaluation it ran or commissioned, in its own words, with what was measured, by what kind of method, and the caveats the report itself gives (schema 2.22). The record is the report’s statement, not a verified finding: it says nothing about another system, a later version or real-world risk, and a null or limiting result is not evidence that no effect exists.
Date. The finding as published on 18 December 2024, for the version of o1 to which the institutes had pre-deployment access. The expression checked is the copy served on 2026-10-28, whose footer reads 'Released December 18, 2024, Updated February 4, 2025'; what the page was changed in on that date is not stated, and the record keeps the release date, not the update date.
Record ID ab-ev-usaisi-o1-bio-result