Claim and evidence Close ×
The joint pre-deployment evaluation summary for Anthropic's upgraded Claude 3.5 Sonnet states that US AISI evaluated the model on a set of multiple-choice biology research task questions and that its performance was comparable to the reference models and significantly below measured human expert baselines in most instances. Copy claim link
Copy claim reference
Machine-checked, not reviewed by an editor Stated in source Supported
Supporting evidence: primary · 1 source; Context: primary · 1 source
Date 2024-11-19 · publication Evaluated system The upgraded Claude 3.5 Sonnet released on 22 October 2024, compared with the earlier Sonnet 3.5, o1-preview and GPT-4o What the report measured The model's relative biological capabilities on practical research tasks, including how it could be misused By what kind of method Question answering on multiple-choice research task questions, with agent tasks, qualitative probing by scientific experts and red teaming used across the exercise The result, in the report’s words Performance comparable to the reference models and significantly below measured human expert baselines in most instances The caveats the report gives The summary states that the biological findings rest on US AISI's testing only, that UK AISI was not publishing its findings in this domain at the time, that comparisons were intended only to assess relative capability improvements, and that smaller differences may fall within the margins of error. Scope and limits The result is the institutes' own, for the model version, question sets and reference models they name. The summary states that the biological findings are from US AISI's testing only, that UK AISI was not publishing in this domain at that time, and that smaller differences may fall within the margins of error. Nothing is generalised to other models or to real-world capability. No question, task or tool is recorded beyond the summary's own category-level wording.
The evidence 02 primary source Supports the claim
Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024) ↗ National Institute of Standards and Technology · 2024-11-19
Archived copy, captured 2024-11-19: Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024) ↗
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “US AISI evaluated the upgraded Sonnet 3.5 on a set of multiple-choice research task questions, and performance was comparable to reference models and significantly below measured human expert baselines in most instances.” Location in source 'Biological Capabilities' section, key findings, first bullet
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors National Institute of Standards and Technology (U.S. AI Safety Institute) Version Retrieved on 2026-10-27 (SHA-256 1b9a64eb9d1953bdd916767279b7021cabc79fc837018b2f75fc3ccf2c8f983a). The cross-family check of 2026-10-28 opened the Internet Archive capture linked here and matched the quoted wording in it, the release-period expression (decision 9); a capture shows only what the archive retrieved that day. Canonical address Open the canonical source ↗ Accessed 2026-10-27 Language English (en) Archive copy Open archived copy, captured 2024-11-19: Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024) ↗ . Captured on the recorded publication date. An archive copy shows only what the archive retrieved on 2024-11-19. It does not prove what the page said earlier, and it can differ from the copy checked on 2026-10-27.Attribution National Institute of Standards and Technology (U.S. AI Safety Institute), “Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024)”, 2024-11-19. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (32 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval (SHA-256 8e9a0960ed43d37fa5d161337886069510323506e77889b1301eb294c16e4b90; the bytes differ from the pack’s retrieval). Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-30, item 18) re-read the record’s sources and found full support. No human editorial review. primary source Context only
Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024) ↗ National Institute of Standards and Technology · 2024-11-19
Archived copy, captured 2024-11-19: Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024) ↗
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “these findings are based on US AISI's testing only, as UK AISI is not publishing its findings in this domain at this time” Location in source 'Biological Capabilities' section, note preceding the key findings
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors National Institute of Standards and Technology (U.S. AI Safety Institute) Version Retrieved on 2026-10-27 (SHA-256 1b9a64eb9d1953bdd916767279b7021cabc79fc837018b2f75fc3ccf2c8f983a). The cross-family check of 2026-10-28 opened the Internet Archive capture linked here and matched the quoted wording in it, the release-period expression (decision 9); a capture shows only what the archive retrieved that day. Canonical address Open the canonical source ↗ Accessed 2026-10-27 Language English (en) Archive copy Open archived copy, captured 2024-11-19: Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024) ↗ . Captured on the recorded publication date. An archive copy shows only what the archive retrieved on 2024-11-19. It does not prove what the page said earlier, and it can differ from the copy checked on 2026-10-27.Attribution National Institute of Standards and Technology (U.S. AI Safety Institute), “Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet (November 19, 2024)”, 2024-11-19. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (23 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval (SHA-256 8e9a0960ed43d37fa5d161337886069510323506e77889b1301eb294c16e4b90; the bytes differ from the pack’s retrieval). Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-30, item 18) re-read the record’s sources and found full support. No human editorial review. Status and meaning Editorial status. Not reviewed by an editor . An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.
Relation. A report states the result of an evaluation it ran or commissioned, in its own words, with what was measured, by what kind of method, and the caveats the report itself gives (schema 2.22). The record is the report’s statement, not a verified finding: it says nothing about another system, a later version or real-world risk, and a null or limiting result is not evidence that no effect exists.
Date. The finding as published on 19 November 2024; the testing period is described only as a limited period of pre-deployment access. The expression checked is the copy served on 2026-10-28, whose footer reads 'Released November 19, 2024, Updated March 20, 2025'; what the page was changed in on that date is not stated, and the record keeps the release date, not the update date.
Record ID ab-ev-usaisi-claude35-bio-result