{"entities":[{"id":"hendrycks-2023-overview-catastrophic","name":"An Overview of Catastrophic AI Risks","type":"document","subtype":"Preprint","description":"A preprint by Dan Hendrycks, Mantas Mazeika and Thomas Woodside, arXiv:2306.12001, dated 21 June 2023. arXiv v1 submission date.","sourceIds":["source-rd-hendrycks-2023-overview-catastrophic"]},{"id":"incentives-produce-safety","name":"Economic incentives will produce safety","type":"concept","proposition":{"code":"R17","criterion":"Developers' commercial or economic incentives, such as the need to sell useful and reliable products and to avoid liability or reputational harm, will lead them to make AI systems safe without special measures against catastrophic risk.","terms":["incentives","market","liability"]}},{"id":"loss-of-control-route","name":"Existential risk from misalignment or loss of control","type":"concept","proposition":{"code":"R08","criterion":"Advanced AI systems pursuing goals at odds with human interests could escape human control and cause human extinction or permanent human disempowerment, without any person intending that outcome.","terms":["loss of control","rogue AI","misaligned AI","disempowerment"]}},{"id":"misuse-route","name":"Catastrophic risk from misuse","type":"concept","proposition":{"code":"R09","criterion":"People's deliberate use of advanced AI, for example to develop weapons of mass destruction, to attack critical systems, or to seize or entrench power, could cause catastrophe at the scale of mass casualties or worse.","terms":["misuse","malicious use","bad actors"]}},{"id":"open-models-reduce-risk","name":"Open models reduce risk","type":"concept","proposition":{"code":"R15","criterion":"Openly releasing the weights of advanced AI models reduces overall risk, or adds little marginal risk compared with closed models and existing technologies, for example by enabling scrutiny, defensive use and a wide distribution of power.","terms":["open source","open weights","open models","marginal risk"]}}],"claims":[{"id":"rd-r08-hendrycks-2023-overview-catastrophic","subject":"hendrycks-2023-overview-catastrophic","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"loss-of-control-route"},"topic":"ideas","statement":"The paper by Hendrycks, Mazeika and Woodside of June 2023 names rogue AIs as a source of catastrophic risk, stating that humans might lose control over AIs as they become more intelligent, through proxy gaming, goal drift, power-seeking and deception.","time":{"kind":"event","date":"2023-06-21","basis":"publication","note":"Date of the arXiv v1 submission."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r08-hendrycks-2023-overview-catastrophic-1","evidence-rd-r08-hendrycks-2023-overview-catastrophic-2"],"attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}},{"id":"rd-r09-hendrycks-2023-overview-catastrophic","subject":"hendrycks-2023-overview-catastrophic","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"misuse-route"},"topic":"ideas","statement":"The paper by Hendrycks, Mazeika and Woodside of June 2023 names malicious use as one of four sources of catastrophic AI risk and states that AI-assisted bioweapons could constitute an existential threat by enlarging the number of people able to create pandemics.","time":{"kind":"event","date":"2023-06-21","basis":"publication","note":"Date of the arXiv v1 submission."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r09-hendrycks-2023-overview-catastrophic-1","evidence-rd-r09-hendrycks-2023-overview-catastrophic-2","evidence-rd-r09-hendrycks-2023-overview-catastrophic-3"],"attributes":{"thought":{"roles":["most-developed","earliest-located-term"],"register":"nonfiction","term":"malicious use"}}},{"id":"rd-r15-hendrycks-2023-overview-catastrophic","subject":"hendrycks-2023-overview-catastrophic","predicate":"contests_proposition","object":{"kind":"entity","entityId":"open-models-reduce-risk"},"topic":"ideas","statement":"Hendrycks, Mazeika and Woodside's overview of 21 June 2023 argues that open-sourcing an AI system with biological research capabilities could increase the risk of malicious use, and recommends that developers avoid open-sourcing the most powerful systems in favour of structured access.","time":{"kind":"event","date":"2023-06-21","basis":"publication","note":"Date of the arXiv v1 submission."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r15-hendrycks-2023-overview-catastrophic-1","evidence-rd-r15-hendrycks-2023-overview-catastrophic-2"],"attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Answers the premise that open release adds little marginal misuse risk by pointing to dual-use biological capabilities, where one release decides the outcome."}}}},{"id":"rd-r17-hendrycks-2023-overview-catastrophic","subject":"hendrycks-2023-overview-catastrophic","predicate":"contests_proposition","object":{"kind":"entity","entityId":"incentives-produce-safety"},"topic":"ideas","statement":"Hendrycks, Mazeika and Woodside's overview of 21 June 2023 argues that competitive pressure fuels a corporate AI race in which companies race to offer the first products to market rather than the safest, and recommends safety regulation so that developers do not cut corners.","time":{"kind":"event","date":"2023-06-21","basis":"publication","note":"Date of the arXiv v1 submission."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r17-hendrycks-2023-overview-catastrophic-1","evidence-rd-r17-hendrycks-2023-overview-catastrophic-2"],"attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Argues that market competition rewards speed over safety, against the premise that markets reward safe products."}}}}],"sources":[{"id":"source-rd-hendrycks-2023-overview-catastrophic","documentId":"hendrycks-2023-overview-catastrophic","title":"An Overview of Catastrophic AI Risks","authors":["Dan Hendrycks","Mantas Mazeika","Thomas Woodside"],"publisher":"arXiv:2306.12001","publicationDate":"2023-06-21","version":"Retrieved for the candidate pack on 2026-10-29 (SHA-256 9df5eca6a17e1c3f1482ccde84eb508cd7dc3c0eb5139cd0770a66a47edb85da). arXiv v1 PDF (2306.12001v1).","evidenceUrl":"https://arxiv.org/pdf/2306.12001v1","language":"en"}],"evidence":[{"id":"evidence-rd-r08-hendrycks-2023-overview-catastrophic-1","claimId":"rd-r08-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r08-hendrycks-2023-overview-catastrophic-2","claimId":"rd-r08-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r09-hendrycks-2023-overview-catastrophic-1","claimId":"rd-r09-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r09-hendrycks-2023-overview-catastrophic-2","claimId":"rd-r09-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r09-hendrycks-2023-overview-catastrophic-3","claimId":"rd-r09-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r15-hendrycks-2023-overview-catastrophic-1","claimId":"rd-r15-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r15-hendrycks-2023-overview-catastrophic-2","claimId":"rd-r15-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r17-hendrycks-2023-overview-catastrophic-1","claimId":"rd-r17-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r17-hendrycks-2023-overview-catastrophic-2","claimId":"rd-r17-hendrycks-2023-overview-catastrophic","sourceId":"source-rd-hendrycks-2023-overview-catastrophic","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1845,1850,1865,1878,1893],"claims":[5211,5224,5298,5322],"sources":[1753],"evidence":[6949,6950,6974,6975,6976,7111,7112,7153,7154]},"index":{"byClaim":[["rd-r08-hendrycks-2023-overview-catastrophic",["propositions:risk-debate"]],["rd-r09-hendrycks-2023-overview-catastrophic",["propositions:risk-debate"]],["rd-r15-hendrycks-2023-overview-catastrophic",["propositions:risk-debate"]],["rd-r17-hendrycks-2023-overview-catastrophic",["propositions:risk-debate"]]],"marks":[["hendrycks-2023-overview-catastrophic",[{"collectionId":"risk-debate","propositionId":"misuse-route","claimId":"rd-r09-hendrycks-2023-overview-catastrophic","text":"Earliest located use of the name “malicious use” in Arguments about catastrophic risk from advanced AI for R09 Catastrophic risk from misuse"}]]],"chronicle":{"years":[{"year":1863,"topics":{"ideas":5},"stated":0},{"year":1865,"topics":{"ideas":1},"stated":0},{"year":1872,"topics":{"ideas":7},"stated":0},{"year":1943,"topics":{"ideas":1},"stated":0},{"year":1947,"topics":{"ideas":2},"stated":0},{"year":1950,"topics":{"ideas":7},"stated":0},{"year":1951,"topics":{"ideas":5},"stated":0},{"year":1952,"topics":{"ideas":2},"stated":0},{"year":1955,"topics":{"ideas":31},"stated":0},{"year":1956,"topics":{"ideas":1},"stated":0},{"year":1957,"topics":{"ideas":1},"stated":0},{"year":1958,"topics":{"ideas":3},"stated":0},{"year":1959,"topics":{"ideas":6},"stated":0},{"year":1960,"topics":{"ideas":11},"stated":0},{"year":1961,"topics":{"ideas":10},"stated":0},{"year":1965,"topics":{"ideas":8},"stated":0},{"year":1966,"topics":{"ideas":20},"stated":0},{"year":1968,"topics":{"ideas":5},"stated":0},{"year":1969,"topics":{"ideas":2},"stated":0},{"year":1971,"topics":{"ideas":4},"stated":0},{"year":1972,"topics":{"ideas":1},"stated":0},{"year":1973,"topics":{"ideas":2},"stated":0},{"year":1974,"topics":{"ideas":2},"stated":0},{"year":1975,"topics":{"ideas":1},"stated":0},{"year":1976,"topics":{"ideas":1},"stated":0},{"year":1977,"topics":{"people":2,"ideas":8},"stated":0},{"year":1979,"topics":{"ideas":4},"stated":0},{"year":1980,"topics":{"ideas":2},"stated":0},{"year":1981,"topics":{"people":2},"stated":0},{"year":1982,"topics":{"ideas":4},"stated":0},{"year":1983,"topics":{"ideas":4},"stated":0},{"year":1984,"topics":{"ideas":21},"stated":0},{"year":1985,"topics":{"ideas":9},"stated":0},{"year":1986,"topics":{"ideas":18},"stated":0},{"year":1987,"topics":{"ideas":1},"stated":0},{"year":1988,"topics":{"ideas":6},"stated":0},{"year":1989,"topics":{"ideas":8},"stated":0},{"year":1990,"topics":{"ideas":5},"stated":0},{"year":1991,"topics":{"ideas":11},"stated":0},{"year":1992,"topics":{"ideas":9},"stated":0},{"year":1993,"topics":{"ideas":14,"people":3},"stated":0},{"year":1994,"topics":{"ideas":14},"stated":0},{"year":1995,"topics":{"ideas":10,"people":1},"stated":0},{"year":1996,"topics":{"ideas":71},"stated":0},{"year":1997,"topics":{"ideas":12},"stated":0},{"year":1998,"topics":{"ideas":49},"stated":0},{"year":1999,"topics":{"ideas":27,"people":1,"institutions":1},"stated":0},{"year":2000,"topics":{"ideas":62},"stated":0},{"year":2001,"topics":{"ideas":39},"stated":0},{"year":2002,"topics":{"ideas":30},"stated":0},{"year":2003,"topics":{"ideas":8},"stated":0},{"year":2004,"topics":{"ideas":12,"people":3},"stated":0},{"year":2005,"topics":{"ideas":10,"people":1},"stated":0},{"year":2006,"topics":{"ideas":26,"people":1},"stated":0},{"year":2007,"topics":{"ideas":20},"stated":0},{"year":2008,"topics":{"ideas":57,"people":3,"institutions":1},"stated":0},{"year":2009,"topics":{"ideas":10,"people":1},"stated":0},{"year":2010,"topics":{"people":3,"ideas":11,"institutions":1,"governance":4},"stated":0},{"year":2011,"topics":{"people":3,"ideas":7,"institutions":2},"stated":0},{"year":2012,"topics":{"people":4,"ideas":17,"institutions":2},"stated":0},{"year":2013,"topics":{"ideas":46,"people":4,"institutions":2,"capital":1},"stated":0},{"year":2014,"topics":{"institutions":2,"ideas":54,"people":5},"stated":0},{"year":2015,"topics":{"people":22,"ideas":122,"institutions":3,"capital":1},"stated":0},{"year":2016,"topics":{"ideas":88,"people":2},"stated":0},{"year":2017,"topics":{"people":16,"institutions":2,"ideas":128,"governance":2},"stated":0},{"year":2018,"topics":{"people":8,"ideas":158,"institutions":8,"governance":1},"stated":0},{"year":2019,"topics":{"capital":1,"compute":1,"people":10,"governance":19,"ideas":175,"institutions":4},"stated":0},{"year":2020,"topics":{"people":6,"ideas":98,"institutions":1,"governance":2},"stated":0},{"year":2021,"topics":{"people":27,"ideas":177,"compute":1,"institutions":23,"governance":4,"capital":6},"stated":0},{"year":2022,"topics":{"people":15,"ideas":217,"institutions":6,"governance":3,"compute":2,"capital":1},"stated":0},{"year":2023,"topics":{"capital":5,"compute":7,"institutions":23,"people":51,"governance":179,"ideas":389},"stated":0},{"year":2024,"topics":{"capital":12,"compute":19,"governance":282,"people":80,"ideas":178,"institutions":28},"stated":1},{"year":2025,"topics":{"governance":547,"compute":69,"capital":34,"people":90,"institutions":18,"ideas":178},"stated":7},{"year":2026,"topics":{"governance":442,"capital":8,"compute":32,"institutions":21,"ideas":91,"people":107},"stated":19},{"year":2027,"topics":{},"stated":9},{"year":2028,"topics":{},"stated":6},{"year":2029,"topics":{},"stated":3},{"year":2030,"topics":{},"stated":3},{"year":2032,"topics":{},"stated":2},{"year":2035,"topics":{},"stated":2},{"year":2039,"topics":{},"stated":1},{"year":2042,"topics":{},"stated":1},{"year":2050,"topics":{},"stated":1}],"scheduled":[],"statedBy":[]},"collections":[{"target":{"kind":"propositions","id":"superintelligence"},"title":"Propositions about superintelligence, 1863–2022","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"risk-debate"},"title":"Arguments about catastrophic risk from advanced AI","claimIds":["rd-r08-hendrycks-2023-overview-catastrophic","rd-r09-hendrycks-2023-overview-catastrophic","rd-r15-hendrycks-2023-overview-catastrophic","rd-r17-hendrycks-2023-overview-catastrophic"],"records":[]},{"target":{"kind":"propositions","id":"yudkowsky-arguments"},"title":"Eliezer Yudkowsky’s arguments on common claims about AI, 1996–2026","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"economics-of-machine-intelligence"},"title":"Economics of machine intelligence","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"orbital-compute-arguments"},"title":"Arguments about compute in orbit","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-bio-arguments"},"title":"Arguments about AI and biological risk","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-in-science"},"title":"AI in science","claimIds":[],"records":[]},{"target":{"kind":"documents"},"title":"Frontier safety frameworks","claimIds":[]},{"target":{"kind":"compare","id":"frontier-safety-frameworks"},"title":"Compare frameworks","claimIds":[]},{"target":{"kind":"control","id":"openai"},"title":"OpenAI’s control structure, 2015–2026","claimIds":[]},{"target":{"kind":"control","id":"anthropic"},"title":"Anthropic’s control structure, 2021–2026","claimIds":[]},{"target":{"kind":"commitments"},"title":"External commitments and obligations","claimIds":[]},{"target":{"kind":"question","id":"frameworks-to-obligations"},"title":"Frameworks, commitments and legal obligations, 2023–2026","claimIds":[]},{"target":{"kind":"reading"},"title":"Founding and early leadership","claimIds":[]}]},"version":"131evrz"}