{"entities":[{"id":"treacherous-turn","name":"Treacherous turn","type":"concept","proposition":{"code":"B1","criterion":"A system behaves cooperatively or safely while weak, confined or overseen, and changes course to pursue its own goals once it can do so without effective opposition.","terms":["treacherous turn"]}},{"id":"inner-or-deceptive-misalignment","name":"Inner or deceptive misalignment","type":"concept","proposition":{"code":"R03","criterion":"A system produced by training or search can come to pursue an objective other than the one its training specified (the training signal or loss), while performing well on that training objective; an objective that is only misspecified by its developers does not meet the criterion; this includes, but is not limited to, a system that behaves as aligned during training or evaluation in order to pursue another objective later.","terms":["inner alignment","inner misalignment","mesa-optimizer","mesa-optimiser","deceptive alignment","goal misgeneralization","goal misgeneralisation","alignment faking"]}},{"id":"shah-2022-goal-misgeneralization","name":"Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals","type":"document","subtype":"Preprint","description":"A preprint by Rohin Shah and 6 co-authors, arXiv preprint 2210.01790, dated 4 October 2022. arXiv v1 stamp, 4 October 2022.","sourceIds":["source-rd-shah-2022-goal-misgeneralization"]}],"claims":[{"id":"rd-r03-shah-2022-goal-misgeneralization","subject":"shah-2022-goal-misgeneralization","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"inner-or-deceptive-misalignment"},"topic":"ideas","statement":"Shah, Varma, Kumar, Phuong, Krakovna, Uesato and Kenton's paper of October 2022 states that a learned system may competently pursue an undesired goal that gives good performance in training, even when the specification is correct, and calls this goal misgeneralization.","time":{"kind":"event","date":"2022-10-04","basis":"publication","note":"Date of the arXiv v1 submission."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r03-shah-2022-goal-misgeneralization-1","evidence-rd-r03-shah-2022-goal-misgeneralization-2","evidence-rd-r03-shah-2022-goal-misgeneralization-3"],"attributes":{"thought":{"roles":["articulation","earliest-located-term"],"register":"nonfiction","term":"goal misgeneralization"}}},{"id":"rd-b1-shah-2022-goal-misgeneralization","subject":"shah-2022-goal-misgeneralization","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"treacherous-turn"},"topic":"ideas","statement":"Shah and co-authors' paper of October 2022 describes a hypothetical deceptive model that behaves as the intended model does during training and plots to subvert human oversight, and a hypothetical system that acts against its overseers once it is confident it can do so unnoticed.","time":{"kind":"event","date":"2022-10-04","basis":"publication","note":"Date of the arXiv v1 submission."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-b1-shah-2022-goal-misgeneralization-1","evidence-rd-b1-shah-2022-goal-misgeneralization-2","evidence-rd-b1-shah-2022-goal-misgeneralization-3","evidence-rd-b1-shah-2022-goal-misgeneralization-4"],"attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}}],"sources":[{"id":"source-rd-shah-2022-goal-misgeneralization","documentId":"shah-2022-goal-misgeneralization","title":"Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals","authors":["Rohin Shah","Vikrant Varma","Ramana Kumar","Mary Phuong","Victoria Krakovna","Jonathan Uesato","Zac Kenton"],"publisher":"arXiv preprint 2210.01790 (DeepMind)","publicationDate":"2022-10-04","version":"Retrieved for the candidate pack on 2026-10-29 (SHA-256 af2ecc2079a4dfcc0a59f7f6f9866ba660fffc073f2078a097f8b1860cd9f134). arXiv v1 of 4 October 2022.","evidenceUrl":"https://arxiv.org/pdf/2210.01790v1","language":"en"}],"evidence":[{"id":"evidence-rd-r03-shah-2022-goal-misgeneralization-1","claimId":"rd-r03-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r03-shah-2022-goal-misgeneralization-2","claimId":"rd-r03-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r03-shah-2022-goal-misgeneralization-3","claimId":"rd-r03-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","support":"context","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-b1-shah-2022-goal-misgeneralization-1","claimId":"rd-b1-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-b1-shah-2022-goal-misgeneralization-2","claimId":"rd-b1-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-b1-shah-2022-goal-misgeneralization-3","claimId":"rd-b1-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-b1-shah-2022-goal-misgeneralization-4","claimId":"rd-b1-shah-2022-goal-misgeneralization","sourceId":"source-rd-shah-2022-goal-misgeneralization","support":"context","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[342,1851,1907],"claims":[5152,5161],"sources":[1802],"evidence":[6836,6837,6838,6858,6859,6860,6861]},"index":{"byClaim":[["rd-b1-shah-2022-goal-misgeneralization",["propositions:superintelligence","propositions:risk-debate"]],["rd-r03-shah-2022-goal-misgeneralization",["propositions:risk-debate"]]],"marks":[["shah-2022-goal-misgeneralization",[{"collectionId":"risk-debate","propositionId":"inner-or-deceptive-misalignment","claimId":"rd-r03-shah-2022-goal-misgeneralization","text":"Earliest located use of the name “goal misgeneralization” in Arguments about catastrophic risk from advanced AI for R03 Inner or deceptive misalignment"}]]],"chronicle":{"years":[{"year":1863,"topics":{"ideas":5},"stated":0},{"year":1865,"topics":{"ideas":1},"stated":0},{"year":1872,"topics":{"ideas":7},"stated":0},{"year":1943,"topics":{"ideas":1},"stated":0},{"year":1947,"topics":{"ideas":2},"stated":0},{"year":1950,"topics":{"ideas":7},"stated":0},{"year":1951,"topics":{"ideas":5},"stated":0},{"year":1952,"topics":{"ideas":2},"stated":0},{"year":1955,"topics":{"ideas":31},"stated":0},{"year":1956,"topics":{"ideas":1},"stated":0},{"year":1957,"topics":{"ideas":1},"stated":0},{"year":1958,"topics":{"ideas":3},"stated":0},{"year":1959,"topics":{"ideas":6},"stated":0},{"year":1960,"topics":{"ideas":11},"stated":0},{"year":1961,"topics":{"ideas":10},"stated":0},{"year":1965,"topics":{"ideas":8},"stated":0},{"year":1966,"topics":{"ideas":20},"stated":0},{"year":1968,"topics":{"ideas":5},"stated":0},{"year":1969,"topics":{"ideas":2},"stated":0},{"year":1971,"topics":{"ideas":4},"stated":0},{"year":1972,"topics":{"ideas":1},"stated":0},{"year":1973,"topics":{"ideas":2},"stated":0},{"year":1974,"topics":{"ideas":2},"stated":0},{"year":1975,"topics":{"ideas":1},"stated":0},{"year":1976,"topics":{"ideas":1},"stated":0},{"year":1977,"topics":{"people":2,"ideas":8},"stated":0},{"year":1979,"topics":{"ideas":4},"stated":0},{"year":1980,"topics":{"ideas":2},"stated":0},{"year":1981,"topics":{"people":2},"stated":0},{"year":1982,"topics":{"ideas":4},"stated":0},{"year":1983,"topics":{"ideas":4},"stated":0},{"year":1984,"topics":{"ideas":21},"stated":0},{"year":1985,"topics":{"ideas":9},"stated":0},{"year":1986,"topics":{"ideas":18},"stated":0},{"year":1987,"topics":{"ideas":1},"stated":0},{"year":1988,"topics":{"ideas":6},"stated":0},{"year":1989,"topics":{"ideas":8},"stated":0},{"year":1990,"topics":{"ideas":5},"stated":0},{"year":1991,"topics":{"ideas":11},"stated":0},{"year":1992,"topics":{"ideas":9},"stated":0},{"year":1993,"topics":{"ideas":14,"people":3},"stated":0},{"year":1994,"topics":{"ideas":14},"stated":0},{"year":1995,"topics":{"ideas":10,"people":1},"stated":0},{"year":1996,"topics":{"ideas":71},"stated":0},{"year":1997,"topics":{"ideas":12},"stated":0},{"year":1998,"topics":{"ideas":49},"stated":0},{"year":1999,"topics":{"ideas":27,"people":1,"institutions":1},"stated":0},{"year":2000,"topics":{"ideas":62},"stated":0},{"year":2001,"topics":{"ideas":39},"stated":0},{"year":2002,"topics":{"ideas":30},"stated":0},{"year":2003,"topics":{"ideas":8},"stated":0},{"year":2004,"topics":{"ideas":12,"people":3},"stated":0},{"year":2005,"topics":{"ideas":10,"people":1},"stated":0},{"year":2006,"topics":{"ideas":26,"people":1},"stated":0},{"year":2007,"topics":{"ideas":20},"stated":0},{"year":2008,"topics":{"ideas":57,"people":3,"institutions":1},"stated":0},{"year":2009,"topics":{"ideas":10,"people":1},"stated":0},{"year":2010,"topics":{"people":3,"ideas":11,"institutions":1,"governance":4},"stated":0},{"year":2011,"topics":{"people":3,"ideas":7,"institutions":2},"stated":0},{"year":2012,"topics":{"people":4,"ideas":17,"institutions":2},"stated":0},{"year":2013,"topics":{"ideas":46,"people":4,"institutions":2,"capital":1},"stated":0},{"year":2014,"topics":{"institutions":2,"ideas":54,"people":5},"stated":0},{"year":2015,"topics":{"people":22,"ideas":122,"institutions":3,"capital":1},"stated":0},{"year":2016,"topics":{"ideas":88,"people":2},"stated":0},{"year":2017,"topics":{"people":16,"institutions":2,"ideas":128,"governance":2},"stated":0},{"year":2018,"topics":{"people":8,"ideas":158,"institutions":8,"governance":1},"stated":0},{"year":2019,"topics":{"capital":1,"compute":1,"people":10,"governance":19,"ideas":175,"institutions":4},"stated":0},{"year":2020,"topics":{"people":6,"ideas":98,"institutions":1,"governance":2},"stated":0},{"year":2021,"topics":{"people":27,"ideas":177,"compute":1,"institutions":23,"governance":4,"capital":6},"stated":0},{"year":2022,"topics":{"people":15,"ideas":217,"institutions":6,"governance":3,"compute":2,"capital":1},"stated":0},{"year":2023,"topics":{"capital":5,"compute":7,"institutions":23,"people":51,"governance":179,"ideas":389},"stated":0},{"year":2024,"topics":{"capital":12,"compute":19,"governance":282,"people":80,"ideas":178,"institutions":28},"stated":1},{"year":2025,"topics":{"governance":547,"compute":69,"capital":34,"people":90,"institutions":18,"ideas":178},"stated":7},{"year":2026,"topics":{"governance":442,"capital":8,"compute":32,"institutions":21,"ideas":91,"people":107},"stated":19},{"year":2027,"topics":{},"stated":9},{"year":2028,"topics":{},"stated":6},{"year":2029,"topics":{},"stated":3},{"year":2030,"topics":{},"stated":3},{"year":2032,"topics":{},"stated":2},{"year":2035,"topics":{},"stated":2},{"year":2039,"topics":{},"stated":1},{"year":2042,"topics":{},"stated":1},{"year":2050,"topics":{},"stated":1}],"scheduled":[],"statedBy":[]},"collections":[{"target":{"kind":"propositions","id":"superintelligence"},"title":"Propositions about superintelligence, 1863–2022","claimIds":["rd-b1-shah-2022-goal-misgeneralization"],"records":[]},{"target":{"kind":"propositions","id":"risk-debate"},"title":"Arguments about catastrophic risk from advanced AI","claimIds":["rd-r03-shah-2022-goal-misgeneralization","rd-b1-shah-2022-goal-misgeneralization"],"records":[]},{"target":{"kind":"propositions","id":"yudkowsky-arguments"},"title":"Eliezer Yudkowsky’s arguments on common claims about AI, 1996–2026","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"economics-of-machine-intelligence"},"title":"Economics of machine intelligence","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"orbital-compute-arguments"},"title":"Arguments about compute in orbit","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-bio-arguments"},"title":"Arguments about AI and biological risk","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-in-science"},"title":"AI in science","claimIds":[],"records":[]},{"target":{"kind":"documents"},"title":"Frontier safety frameworks","claimIds":[]},{"target":{"kind":"compare","id":"frontier-safety-frameworks"},"title":"Compare frameworks","claimIds":[]},{"target":{"kind":"control","id":"openai"},"title":"OpenAI’s control structure, 2015–2026","claimIds":[]},{"target":{"kind":"control","id":"anthropic"},"title":"Anthropic’s control structure, 2021–2026","claimIds":[]},{"target":{"kind":"commitments"},"title":"External commitments and obligations","claimIds":[]},{"target":{"kind":"question","id":"frameworks-to-obligations"},"title":"Frameworks, commitments and legal obligations, 2023–2026","claimIds":[]},{"target":{"kind":"reading"},"title":"Founding and early leadership","claimIds":[]}]},"version":"131evrz"}