{"entities":[{"id":"alignment-is-hard","name":"Difficulty of alignment","type":"concept","proposition":{"code":"R06","criterion":"Making an advanced AI system reliably pursue the goals its developers intend (aligning it) is a hard, unsolved technical problem that will not be solved by default in the ordinary course of development.","terms":["alignment problem","control problem","value alignment"]}},{"id":"inner-or-deceptive-misalignment","name":"Inner or deceptive misalignment","type":"concept","proposition":{"code":"R03","criterion":"A system produced by training or search can come to pursue an objective other than the one its training specified (the training signal or loss), while performing well on that training objective; an objective that is only misspecified by its developers does not meet the criterion; this includes, but is not limited to, a system that behaves as aligned during training or evaluation in order to pursue another objective later.","terms":["inner alignment","inner misalignment","mesa-optimizer","mesa-optimiser","deceptive alignment","goal misgeneralization","goal misgeneralisation","alignment faking"]}},{"id":"soares-2017-ensuring-positive-outcome","name":"Ensuring smarter-than-human intelligence has a positive outcome","type":"document","subtype":"Lecture or talk","description":"A lecture or talk by Nate Soares, Machine Intelligence Research Institute blog, dated 12 April 2017. post date on intelligence.org; the date of the talk at Google is not given.","sourceIds":["source-rd-soares-2017-ensuring-positive-outcome"]}],"claims":[{"id":"rd-r03-soares-2017-ensuring-positive-outcome","subject":"soares-2017-ensuring-positive-outcome","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"inner-or-deceptive-misalignment"},"topic":"ideas","statement":"Soares's talk transcript posted in April 2017 states that a powerful optimiser produced by gradient descent should by default be expected to have a goal that strongly correlates with its training objective in the training environment but sharply diverges from it in new environments.","time":{"kind":"event","date":"2017-04-12","basis":"publication","note":"Date of posting."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r03-soares-2017-ensuring-positive-outcome-1","evidence-rd-r03-soares-2017-ensuring-positive-outcome-2"],"attributes":{"thought":{"roles":["earliest-located"],"register":"nonfiction"}}},{"id":"rd-r06-soares-2017-ensuring-positive-outcome","subject":"soares-2017-ensuring-positive-outcome","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"alignment-is-hard"},"topic":"ideas","statement":"Soares's talk transcript posted in April 2017 states as one of four propositions that aligning advanced AI systems is hard to solve and that AI systems will not naturally end up sharing human objectives.","time":{"kind":"event","date":"2017-04-12","basis":"publication","note":"Date of posting."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r06-soares-2017-ensuring-positive-outcome-1","evidence-rd-r06-soares-2017-ensuring-positive-outcome-2"],"attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}}],"sources":[{"id":"source-rd-soares-2017-ensuring-positive-outcome","documentId":"soares-2017-ensuring-positive-outcome","title":"Ensuring smarter-than-human intelligence has a positive outcome","authors":["Nate Soares"],"publisher":"Machine Intelligence Research Institute blog (intelligence.org)","publicationDate":"2017-04-12","version":"Retrieved for the candidate pack on 2026-10-29 (SHA-256 37fbc1df05cfffa12b9313f40eaf121fa05933de2928b131568d4df637137a05). Modified transcript posted 12 April 2017 of a talk given at Google; current page as served on 2026-10-29; the page prints question marks where mathematical symbols stood. The cross-family check retrieved it again on 2026-10-31; the bytes differ, and the check compared the quoted passages.","evidenceUrl":"https://intelligence.org/2017/04/12/ensuring/","language":"en"}],"evidence":[{"id":"evidence-rd-r03-soares-2017-ensuring-positive-outcome-1","claimId":"rd-r03-soares-2017-ensuring-positive-outcome","sourceId":"source-rd-soares-2017-ensuring-positive-outcome","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r03-soares-2017-ensuring-positive-outcome-2","claimId":"rd-r03-soares-2017-ensuring-positive-outcome","sourceId":"source-rd-soares-2017-ensuring-positive-outcome","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r06-soares-2017-ensuring-positive-outcome-1","claimId":"rd-r06-soares-2017-ensuring-positive-outcome","sourceId":"source-rd-soares-2017-ensuring-positive-outcome","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r06-soares-2017-ensuring-positive-outcome-2","claimId":"rd-r06-soares-2017-ensuring-positive-outcome","sourceId":"source-rd-soares-2017-ensuring-positive-outcome","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1776,1851,1910],"claims":[5149,5187],"sources":[1805],"evidence":[6830,6831,6906,6907]},"index":{"byClaim":[["rd-r03-soares-2017-ensuring-positive-outcome",["propositions:risk-debate"]],["rd-r06-soares-2017-ensuring-positive-outcome",["propositions:risk-debate"]]],"marks":[["soares-2017-ensuring-positive-outcome",[{"collectionId":"risk-debate","propositionId":"inner-or-deceptive-misalignment","claimId":"rd-r03-soares-2017-ensuring-positive-outcome","text":"Earliest located in Arguments about catastrophic risk from advanced AI for R03 Inner or deceptive misalignment"}]]],"chronicle":{"years":[{"year":1863,"topics":{"ideas":5},"stated":0},{"year":1865,"topics":{"ideas":1},"stated":0},{"year":1872,"topics":{"ideas":7},"stated":0},{"year":1943,"topics":{"ideas":1},"stated":0},{"year":1947,"topics":{"ideas":2},"stated":0},{"year":1950,"topics":{"ideas":7},"stated":0},{"year":1951,"topics":{"ideas":5},"stated":0},{"year":1952,"topics":{"ideas":2},"stated":0},{"year":1955,"topics":{"ideas":31},"stated":0},{"year":1956,"topics":{"ideas":1},"stated":0},{"year":1957,"topics":{"ideas":1},"stated":0},{"year":1958,"topics":{"ideas":3},"stated":0},{"year":1959,"topics":{"ideas":6},"stated":0},{"year":1960,"topics":{"ideas":11},"stated":0},{"year":1961,"topics":{"ideas":10},"stated":0},{"year":1965,"topics":{"ideas":8},"stated":0},{"year":1966,"topics":{"ideas":20},"stated":0},{"year":1968,"topics":{"ideas":5},"stated":0},{"year":1969,"topics":{"ideas":2},"stated":0},{"year":1971,"topics":{"ideas":4},"stated":0},{"year":1972,"topics":{"ideas":1},"stated":0},{"year":1973,"topics":{"ideas":2},"stated":0},{"year":1974,"topics":{"ideas":2},"stated":0},{"year":1975,"topics":{"ideas":1},"stated":0},{"year":1976,"topics":{"ideas":1},"stated":0},{"year":1977,"topics":{"people":2,"ideas":8},"stated":0},{"year":1979,"topics":{"ideas":4},"stated":0},{"year":1980,"topics":{"ideas":2},"stated":0},{"year":1981,"topics":{"people":2},"stated":0},{"year":1982,"topics":{"ideas":4},"stated":0},{"year":1983,"topics":{"ideas":4},"stated":0},{"year":1984,"topics":{"ideas":21},"stated":0},{"year":1985,"topics":{"ideas":9},"stated":0},{"year":1986,"topics":{"ideas":18},"stated":0},{"year":1987,"topics":{"ideas":1},"stated":0},{"year":1988,"topics":{"ideas":6},"stated":0},{"year":1989,"topics":{"ideas":8},"stated":0},{"year":1990,"topics":{"ideas":5},"stated":0},{"year":1991,"topics":{"ideas":11},"stated":0},{"year":1992,"topics":{"ideas":9},"stated":0},{"year":1993,"topics":{"ideas":14,"people":3},"stated":0},{"year":1994,"topics":{"ideas":14},"stated":0},{"year":1995,"topics":{"ideas":10,"people":1},"stated":0},{"year":1996,"topics":{"ideas":71},"stated":0},{"year":1997,"topics":{"ideas":12},"stated":0},{"year":1998,"topics":{"ideas":49},"stated":0},{"year":1999,"topics":{"ideas":27,"people":1,"institutions":1},"stated":0},{"year":2000,"topics":{"ideas":62},"stated":0},{"year":2001,"topics":{"ideas":39},"stated":0},{"year":2002,"topics":{"ideas":30},"stated":0},{"year":2003,"topics":{"ideas":8},"stated":0},{"year":2004,"topics":{"ideas":12,"people":3},"stated":0},{"year":2005,"topics":{"ideas":10,"people":1},"stated":0},{"year":2006,"topics":{"ideas":26,"people":1},"stated":0},{"year":2007,"topics":{"ideas":20},"stated":0},{"year":2008,"topics":{"ideas":57,"people":3,"institutions":1},"stated":0},{"year":2009,"topics":{"ideas":10,"people":1},"stated":0},{"year":2010,"topics":{"people":3,"ideas":11,"institutions":1,"governance":4},"stated":0},{"year":2011,"topics":{"people":3,"ideas":7,"institutions":2},"stated":0},{"year":2012,"topics":{"people":4,"ideas":17,"institutions":2},"stated":0},{"year":2013,"topics":{"ideas":46,"people":4,"institutions":2,"capital":1},"stated":0},{"year":2014,"topics":{"institutions":2,"ideas":54,"people":5},"stated":0},{"year":2015,"topics":{"people":22,"ideas":122,"institutions":3,"capital":1},"stated":0},{"year":2016,"topics":{"ideas":88,"people":2},"stated":0},{"year":2017,"topics":{"people":16,"institutions":2,"ideas":128,"governance":2},"stated":0},{"year":2018,"topics":{"people":8,"ideas":158,"institutions":8,"governance":1},"stated":0},{"year":2019,"topics":{"capital":1,"compute":1,"people":10,"governance":19,"ideas":175,"institutions":4},"stated":0},{"year":2020,"topics":{"people":6,"ideas":98,"institutions":1,"governance":2},"stated":0},{"year":2021,"topics":{"people":27,"ideas":177,"compute":1,"institutions":23,"governance":4,"capital":6},"stated":0},{"year":2022,"topics":{"people":15,"ideas":217,"institutions":6,"governance":3,"compute":2,"capital":1},"stated":0},{"year":2023,"topics":{"capital":5,"compute":7,"institutions":23,"people":51,"governance":179,"ideas":389},"stated":0},{"year":2024,"topics":{"capital":12,"compute":19,"governance":282,"people":80,"ideas":178,"institutions":28},"stated":1},{"year":2025,"topics":{"governance":547,"compute":69,"capital":34,"people":90,"institutions":18,"ideas":178},"stated":7},{"year":2026,"topics":{"governance":442,"capital":8,"compute":32,"institutions":21,"ideas":91,"people":107},"stated":19},{"year":2027,"topics":{},"stated":9},{"year":2028,"topics":{},"stated":6},{"year":2029,"topics":{},"stated":3},{"year":2030,"topics":{},"stated":3},{"year":2032,"topics":{},"stated":2},{"year":2035,"topics":{},"stated":2},{"year":2039,"topics":{},"stated":1},{"year":2042,"topics":{},"stated":1},{"year":2050,"topics":{},"stated":1}],"scheduled":[],"statedBy":[]},"collections":[{"target":{"kind":"propositions","id":"superintelligence"},"title":"Propositions about superintelligence, 1863–2022","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"risk-debate"},"title":"Arguments about catastrophic risk from advanced AI","claimIds":["rd-r03-soares-2017-ensuring-positive-outcome","rd-r06-soares-2017-ensuring-positive-outcome"],"records":[]},{"target":{"kind":"propositions","id":"yudkowsky-arguments"},"title":"Eliezer Yudkowsky’s arguments on common claims about AI, 1996–2026","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"economics-of-machine-intelligence"},"title":"Economics of machine intelligence","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"orbital-compute-arguments"},"title":"Arguments about compute in orbit","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-bio-arguments"},"title":"Arguments about AI and biological risk","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-in-science"},"title":"AI in science","claimIds":[],"records":[]},{"target":{"kind":"documents"},"title":"Frontier safety frameworks","claimIds":[]},{"target":{"kind":"compare","id":"frontier-safety-frameworks"},"title":"Compare frameworks","claimIds":[]},{"target":{"kind":"control","id":"openai"},"title":"OpenAI’s control structure, 2015–2026","claimIds":[]},{"target":{"kind":"control","id":"anthropic"},"title":"Anthropic’s control structure, 2021–2026","claimIds":[]},{"target":{"kind":"commitments"},"title":"External commitments and obligations","claimIds":[]},{"target":{"kind":"question","id":"frameworks-to-obligations"},"title":"Frameworks, commitments and legal obligations, 2023–2026","claimIds":[]},{"target":{"kind":"reading"},"title":"Founding and early leadership","claimIds":[]}]},"version":"131evrz"}