{"entities":[{"id":"uk-ai-security-institute","name":"AI Security Institute (UK)","shortName":"UK AI Security Institute"},{"id":"uk-aisi-evaluations-may-2024","name":"Advanced AI evaluations at AISI: May update","subtype":"Report","description":"As published on 20 May 2024 under the institute's earlier name. Byline date on the post. The institute was then named the AI Safety Institute; the page itself carries the later renaming note.","sourceIds":["source-ab-aisi-may2024"]}],"claims":[{"id":"ab-ev-uk-aisi-may2024-chembio-result","statement":"The UK institute's May 2024 evaluations update states that several of the five publicly available large language models it tested demonstrated expert-level knowledge of chemistry and biology, answering over 600 private expert-written chemistry and biology questions at levels similar to humans with PhD-level training.","derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-ab-ev-uk-aisi-may2024-chembio-result-1","evidence-ab-ev-uk-aisi-may2024-chembio-result-2","evidence-ab-ev-uk-aisi-may2024-chembio-result-3"],"attributes":{"evaluatedSystem":"Five publicly available large language models, anonymised in the post by colour","evaluation":{"what":"Whether the models could provide expert-level knowledge in chemistry and biology that could be used for beneficial but also harmful purposes","how":"Over 600 private expert-written open-text questions across four question areas, graded against the performance of human experts with relevant doctorates or work experience who had web access and up to an hour per question","result":"Several models demonstrated expert-level knowledge of chemistry and biology; most models performed similarly to the human experts, with two stated exceptions","caveats":"The post states that its work does not provide any assurance that a model is safe or unsafe, and that the institute remains aware of the potential gap between evaluation performance and performance in the wild."}}},{"id":"ab-ev-uk-aisi-may2024-published","statement":"The UK institute, then named the AI Safety Institute, published 'Advanced AI evaluations at AISI: May update' on 20 May 2024, stating that it had tested leading AI models for cyber, chemical, biological and agent capabilities and for safeguard effectiveness.","derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-ab-ev-uk-aisi-may2024-published-1"]}],"sources":[{"id":"source-ab-aisi-may2024","title":"Advanced AI evaluations at AISI: May update","authors":["AI Security Institute (UK), published under its earlier name AI Safety Institute"],"publisher":"AI Security Institute (UK), published under its earlier name AI Safety Institute","publicationDate":null,"version":"Retrieved for the candidate pack on 2026-10-28 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917). Blog post bylined 'Technical staff — May 20, 2024', as served on 2026-10-28 after a redirect to https://www.aisi.gov.uk/blog/advanced-ai-evaluations-may-update; it carries a later note that the institute changed its name on 14 February 2025. This is a different document from the dataset's uk-aisi-evaluations-2024 (the approach-to-evaluations publication of 9 February 2024). Retrieved again at admission on 2026-10-30 (SHA-256 194d183711498e8fa0fa995229646e73b58c6cb7df533806585f373fbcd5f917, the same bytes).","evidenceUrl":"https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update","language":"en"}],"evidence":[{"id":"evidence-ab-ev-uk-aisi-may2024-chembio-result-1","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-ab-ev-uk-aisi-may2024-chembio-result-2","support":"context","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-ab-ev-uk-aisi-may2024-chembio-result-3","support":"context","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-ab-ev-uk-aisi-may2024-published-1","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1061,1251],"claims":[3367,3368],"sources":[1089],"evidence":[4622,4623,4624,4625]},"version":"1lg47t7"}