Claim and evidence Close ×
Advanced AI evaluations at AISI: May update states evaluation result ↓ The update states that several of the five large language models tested demonstrated expert-level knowledge of chemistry and biology, answering over 600 private expert-written questions at levels similar to humans with PhD-level training. The UK institute's May 2024 evaluations update states that several of the five publicly available large language models it tested demonstrated expert-level knowledge of chemistry and biology, answering over 600 private expert-written chemistry and biology questions at levels similar to humans with PhD-level training. Copy claim link
Copy claim reference
Machine-checked, not reviewed by an editor Stated in source Supported
Supporting evidence: primary · 1 source; Context: primary · 1 source
Subject version As published on 20 May 2024 under the institute's earlier name · Published 2024-05-20 · Effective date not established Date 2024-05-20 · publication Evaluated system Five publicly available large language models, anonymised in the post by colour What the report measured Whether the models could provide expert-level knowledge in chemistry and biology that could be used for beneficial but also harmful purposes By what kind of method Over 600 private expert-written open-text questions across four question areas, graded against the performance of human experts with relevant doctorates or work experience who had web access and up to an hour per question The result, in the report’s words Several models demonstrated expert-level knowledge of chemistry and biology; most models performed similarly to the human experts, with two stated exceptions The caveats the report gives The post states that its work does not provide any assurance that a model is safe or unsafe, and that the institute remains aware of the potential gap between evaluation performance and performance in the wild. Scope and limits The result is the institute's own, for five models it does not name, its private question set and its stated human comparison. The post states that its work does not provide any assurance that a model is safe or unsafe, and that there may be a gap between evaluation performance and behaviour in the wild. Nothing is generalised to other models or to real-world capability. The expression checked is the copy served on 2026-10-28, which is later than first publication: it carries a note to readers that the institute changed its name to the AI Security Institute on 14 February 2025, and the address redirects to /blog/advanced-ai-evaluations-may-update. The record keeps the byline date of 20 May 2024, and the quoted wording was read in that served copy. The post's description of its question areas is summarised only at category level; no question or subject matter is recorded.
The evidence 03 primary source Supports the claim
Advanced AI evaluations at AISI: May update ↗ AI Security Institute (UK), published under its earlier name AI Safety Institute · Publication date unknown
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “Several LLMs demonstrated expert-level knowledge of chemistry and biology. Models answered over 600 private expert-written chemistry and biology questions at similar levels to humans with PhD-level training.” Location in source 'Here, we highlight a selection of recent results' list, first item
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors AI Security Institute (UK), published under its earlier name AI Safety Institute Version Retrieved for the candidate pack on 2026-10-28 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917). Blog post bylined 'Technical staff — May 20, 2024', as served on 2026-10-28 after a redirect to https://www.aisi.gov.uk/blog/advanced-ai-evaluations-may-update; it carries a later note that the institute changed its name on 14 February 2025. This is a different document from the dataset's uk-aisi-evaluations-2024 (the approach-to-evaluations publication of 9 February 2024). Retrieved again at admission on 2026-10-30 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917, the same bytes). Canonical address Open the canonical source ↗ Accessed 2026-10-28 Language English (en) Attribution AI Security Institute (UK), published under its earlier name AI Safety Institute, “Advanced AI evaluations at AISI: May update”. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (27 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval whose SHA-256 equals the pack’s. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-30, item 16) re-read the record’s sources and found full support. No human editorial review. primary source Context only
Advanced AI evaluations at AISI: May update ↗ AI Security Institute (UK), published under its earlier name AI Safety Institute · Publication date unknown
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “Our work does not provide any assurance that a model is "safe" or "unsafe".” Location in source 'Outlook' section, final paragraph
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors AI Security Institute (UK), published under its earlier name AI Safety Institute Version Retrieved for the candidate pack on 2026-10-28 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917). Blog post bylined 'Technical staff — May 20, 2024', as served on 2026-10-28 after a redirect to https://www.aisi.gov.uk/blog/advanced-ai-evaluations-may-update; it carries a later note that the institute changed its name on 14 February 2025. This is a different document from the dataset's uk-aisi-evaluations-2024 (the approach-to-evaluations publication of 9 February 2024). Retrieved again at admission on 2026-10-30 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917, the same bytes). Canonical address Open the canonical source ↗ Accessed 2026-10-28 Language English (en) Attribution AI Security Institute (UK), published under its earlier name AI Safety Institute, “Advanced AI evaluations at AISI: May update”. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (14 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval whose SHA-256 equals the pack’s. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. The institute's own limit on how its results may be read. Locator corrected after the audit of 2026-10-28: the post's closing section is headed 'Outlook', not 'limitations'. A cross-family machine check (2026-10-30, item 16) re-read the record’s sources and found full support. No human editorial review. primary source Context only
Advanced AI evaluations at AISI: May update ↗ AI Security Institute (UK), published under its earlier name AI Safety Institute · Publication date unknown
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “we remain acutely aware of the potential gap between how advanced AI systems perform in our evaluations versus how they may perform in the wild” Location in source 'Outlook' section, penultimate paragraph
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-30
Source details and check notes Authors AI Security Institute (UK), published under its earlier name AI Safety Institute Version Retrieved for the candidate pack on 2026-10-28 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917). Blog post bylined 'Technical staff — May 20, 2024', as served on 2026-10-28 after a redirect to https://www.aisi.gov.uk/blog/advanced-ai-evaluations-may-update; it carries a later note that the institute changed its name on 14 February 2025. This is a different document from the dataset's uk-aisi-evaluations-2024 (the approach-to-evaluations publication of 9 February 2024). Retrieved again at admission on 2026-10-30 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917, the same bytes). Canonical address Open the canonical source ↗ Accessed 2026-10-28 Language English (en) Attribution AI Security Institute (UK), published under its earlier name AI Safety Institute, “Advanced AI evaluations at AISI: May update”. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, AI and biosecurity (research/ai-bio). The quotation (25 words) matched the text retrieved for the candidate pack on 2026-10-28 and again at admission on 2026-10-30, in a fresh retrieval whose SHA-256 equals the pack’s. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. Locator corrected after the audit of 2026-10-28 for the same reason; this paragraph precedes the one above. A cross-family machine check (2026-10-30, item 16) re-read the record’s sources and found full support. No human editorial review. Status and meaning Editorial status. Not reviewed by an editor . An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.
Relation. A report states the result of an evaluation it ran or commissioned, in its own words, with what was measured, by what kind of method, and the caveats the report itself gives (schema 2.22). The record is the report’s statement, not a verified finding: it says nothing about another system, a later version or real-world risk, and a null or limiting result is not evidence that no effect exists.
Date. The result as published on 20 May 2024; the models are anonymised in the post.
Record ID ab-ev-uk-aisi-may2024-chembio-result