{"entities":[{"id":"bengio-2019-instrumental-convergence-thread","name":"Yoshua Bengio's comments in the Facebook thread on instrumental convergence (2019)","shortName":"Bengio, comments in the Facebook thread on instrumental convergence (2019)","type":"document"},{"id":"control-by-design","name":"Advanced AI can be kept under control by design","type":"concept","proposition":{"code":"R10b","criterion":"People can build advanced AI systems with objectives, safeguards or oversight that keep them under human control, so that loss of control is avoidable in ordinary engineering practice.","terms":[]}},{"id":"instrumental-convergence","name":"Instrumental convergence","type":"concept","proposition":{"code":"R02","criterion":"Agents with a wide range of final goals would tend to pursue the same intermediate goals, such as self-preservation, keeping their goals unchanged, acquiring resources or power, or improving themselves, because these serve almost any final goal.","terms":["instrumental convergence","convergent instrumental","basic AI drives","power-seeking"]}},{"id":"lecun-2019-instrumental-convergence-thread","name":"Yann LeCun's comments in the Facebook thread on instrumental convergence (2019)","type":"document","subtype":"Comment","description":"A comment by Yann LeCun, dated 4 October 2019. Known from the LessWrong copy of 4 October 2019; the comments themselves are undated.","sourceIds":["source-rd-pace-2019-instrumental-convergence-debate"]},{"id":"loss-of-control-route","name":"Existential risk from misalignment or loss of control","type":"concept","proposition":{"code":"R08","criterion":"Advanced AI systems pursuing goals at odds with human interests could escape human control and cause human extinction or permanent human disempowerment, without any person intending that outcome.","terms":["loss of control","rogue AI","misaligned AI","disempowerment"]}},{"id":"russell-2019-instrumental-convergence-thread","name":"Stuart Russell's comments in the Facebook thread on instrumental convergence (2019)","shortName":"Russell, comments in the Facebook thread on instrumental convergence (2019)","type":"document"}],"claims":[{"id":"rd-r02-lecun-2019-instrumental-convergence-thread","subject":"lecun-2019-instrumental-convergence-thread","predicate":"contests_proposition","object":{"kind":"entity","entityId":"instrumental-convergence"},"topic":"ideas","statement":"LeCun's comments in the 2019 thread argue that the instrumental drives Russell describes are avoided by safeguard terms in a system's objective, and that instrumental subgoals are much weaker drives of behaviour than hardwired objectives.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r02-lecun-2019-instrumental-convergence-thread-1","evidence-rd-r02-lecun-2019-instrumental-convergence-thread-2"],"attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."},"engagement":{"kind":"engages-premise","rationale":"Gives reasons against the premise that instrumental subgoals would drive a built system's behaviour: safeguard terms can be added to the objective, and instrumental subgoals are weaker than hardwired objectives."}}}},{"id":"rd-r08-lecun-2019-instrumental-convergence-thread","subject":"lecun-2019-instrumental-convergence-thread","predicate":"contests_proposition","object":{"kind":"entity","entityId":"loss-of-control-route"},"topic":"ideas","statement":"LeCun's comments in the 2019 thread state that he disagrees that there is a high risk of accidentally building existential threats to humanity, holding that such threats have to be designed as such.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r08-lecun-2019-instrumental-convergence-thread-1"],"attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."},"engagement":{"kind":"contrary-only","rationale":"States the contrary conclusion (existential threats must be designed) without a further argument in this comment; the reasons are given in his other comments on R02 and R10b."}}}},{"id":"rd-r10b-lecun-2019-instrumental-convergence-thread","subject":"lecun-2019-instrumental-convergence-thread","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"control-by-design"},"topic":"ideas","statement":"LeCun's comments in the 2019 thread state that it will be much easier to control the behaviour of autonomous AI systems than that of humans and human organisations, because their intrinsic objective function can be modified directly.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r10b-lecun-2019-instrumental-convergence-thread-1"],"attributes":{"thought":{"roles":["articulation"],"register":"nonfiction","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."}}}},{"id":"rd-reply-lecun-2019-thread-to-russell-2019-thread","subject":"lecun-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"russell-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"LeCun's comment in the 2019 thread replies to Russell's coffee-fetching argument, which Bengio had asked him to answer.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-lecun-2019-thread-to-russell-2019-thread-1"]},{"id":"rd-reply-bengio-2019-thread-to-lecun-2019-thread","subject":"bengio-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"lecun-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"Bengio's comment in the 2019 thread replies to LeCun's points on designing objectives.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-bengio-2019-thread-to-lecun-2019-thread-1"]},{"id":"rd-reply-lecun-2019-thread-to-bengio-2019-thread","subject":"lecun-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"bengio-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"LeCun's comment in the 2019 thread replies to Bengio's summary of the debate.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-lecun-2019-thread-to-bengio-2019-thread-1"]}],"sources":[{"id":"source-rd-pace-2019-instrumental-convergence-debate","documentId":"pace-2019-instrumental-convergence-debate","title":"Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More","canonicalUrl":"https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell"}],"evidence":[{"id":"evidence-rd-r02-lecun-2019-instrumental-convergence-thread-1","claimId":"rd-r02-lecun-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r02-lecun-2019-instrumental-convergence-thread-2","claimId":"rd-r02-lecun-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r08-lecun-2019-instrumental-convergence-thread-1","claimId":"rd-r08-lecun-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r10b-lecun-2019-instrumental-convergence-thread-1","claimId":"rd-r10b-lecun-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-lecun-2019-thread-to-russell-2019-thread-1","claimId":"rd-reply-lecun-2019-thread-to-russell-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-bengio-2019-thread-to-lecun-2019-thread-1","claimId":"rd-reply-bengio-2019-thread-to-lecun-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-lecun-2019-thread-to-bengio-2019-thread-1","claimId":"rd-reply-lecun-2019-thread-to-bengio-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1791,1812,1852,1862,1865,1905],"claims":[5137,5206,5242,5342,5345,5346],"sources":[1792],"evidence":[6813,6814,6940,7005,7181,7184,7185]},"index":{"byClaim":[["rd-r02-lecun-2019-instrumental-convergence-thread",["propositions:risk-debate"]],["rd-r08-lecun-2019-instrumental-convergence-thread",["propositions:risk-debate"]],["rd-r10b-lecun-2019-instrumental-convergence-thread",["propositions:risk-debate"]]],"marks":[["russell-2019-instrumental-convergence-thread",[{"collectionId":"risk-debate","propositionId":"alignment-is-hard","claimId":"rd-r06-russell-2019-instrumental-convergence-thread","text":"Earliest located use of the name “alignment problem” in Arguments about catastrophic risk from advanced AI for R06 Difficulty of alignment"}]]],"chronicle":{"years":[{"year":1863,"topics":{"ideas":5},"stated":0},{"year":1865,"topics":{"ideas":1},"stated":0},{"year":1872,"topics":{"ideas":7},"stated":0},{"year":1943,"topics":{"ideas":1},"stated":0},{"year":1947,"topics":{"ideas":2},"stated":0},{"year":1950,"topics":{"ideas":7},"stated":0},{"year":1951,"topics":{"ideas":5},"stated":0},{"year":1952,"topics":{"ideas":2},"stated":0},{"year":1955,"topics":{"ideas":31},"stated":0},{"year":1956,"topics":{"ideas":1},"stated":0},{"year":1957,"topics":{"ideas":1},"stated":0},{"year":1958,"topics":{"ideas":3},"stated":0},{"year":1959,"topics":{"ideas":6},"stated":0},{"year":1960,"topics":{"ideas":11},"stated":0},{"year":1961,"topics":{"ideas":10},"stated":0},{"year":1965,"topics":{"ideas":8},"stated":0},{"year":1966,"topics":{"ideas":20},"stated":0},{"year":1968,"topics":{"ideas":5},"stated":0},{"year":1969,"topics":{"ideas":2},"stated":0},{"year":1971,"topics":{"ideas":4},"stated":0},{"year":1972,"topics":{"ideas":1},"stated":0},{"year":1973,"topics":{"ideas":2},"stated":0},{"year":1974,"topics":{"ideas":2},"stated":0},{"year":1975,"topics":{"ideas":1},"stated":0},{"year":1976,"topics":{"ideas":1},"stated":0},{"year":1977,"topics":{"people":2,"ideas":8},"stated":0},{"year":1979,"topics":{"ideas":4},"stated":0},{"year":1980,"topics":{"ideas":2},"stated":0},{"year":1981,"topics":{"people":2},"stated":0},{"year":1982,"topics":{"ideas":4},"stated":0},{"year":1983,"topics":{"ideas":4},"stated":0},{"year":1984,"topics":{"ideas":21},"stated":0},{"year":1985,"topics":{"ideas":9},"stated":0},{"year":1986,"topics":{"ideas":18},"stated":0},{"year":1987,"topics":{"ideas":1},"stated":0},{"year":1988,"topics":{"ideas":6},"stated":0},{"year":1989,"topics":{"ideas":8},"stated":0},{"year":1990,"topics":{"ideas":5},"stated":0},{"year":1991,"topics":{"ideas":11},"stated":0},{"year":1992,"topics":{"ideas":9},"stated":0},{"year":1993,"topics":{"ideas":14,"people":3},"stated":0},{"year":1994,"topics":{"ideas":14},"stated":0},{"year":1995,"topics":{"ideas":10,"people":1},"stated":0},{"year":1996,"topics":{"ideas":71},"stated":0},{"year":1997,"topics":{"ideas":12},"stated":0},{"year":1998,"topics":{"ideas":49},"stated":0},{"year":1999,"topics":{"ideas":27,"people":1,"institutions":1},"stated":0},{"year":2000,"topics":{"ideas":62},"stated":0},{"year":2001,"topics":{"ideas":39},"stated":0},{"year":2002,"topics":{"ideas":30},"stated":0},{"year":2003,"topics":{"ideas":8},"stated":0},{"year":2004,"topics":{"ideas":12,"people":3},"stated":0},{"year":2005,"topics":{"ideas":10,"people":1},"stated":0},{"year":2006,"topics":{"ideas":26,"people":1},"stated":0},{"year":2007,"topics":{"ideas":20},"stated":0},{"year":2008,"topics":{"ideas":57,"people":3,"institutions":1},"stated":0},{"year":2009,"topics":{"ideas":10,"people":1},"stated":0},{"year":2010,"topics":{"people":3,"ideas":11,"institutions":1,"governance":4},"stated":0},{"year":2011,"topics":{"people":3,"ideas":7,"institutions":2},"stated":0},{"year":2012,"topics":{"people":4,"ideas":17,"institutions":2},"stated":0},{"year":2013,"topics":{"ideas":46,"people":4,"institutions":2,"capital":1},"stated":0},{"year":2014,"topics":{"institutions":2,"ideas":54,"people":5},"stated":0},{"year":2015,"topics":{"people":22,"ideas":122,"institutions":3,"capital":1},"stated":0},{"year":2016,"topics":{"ideas":88,"people":2},"stated":0},{"year":2017,"topics":{"people":16,"institutions":2,"ideas":128,"governance":2},"stated":0},{"year":2018,"topics":{"people":8,"ideas":158,"institutions":8,"governance":1},"stated":0},{"year":2019,"topics":{"capital":1,"compute":1,"people":10,"governance":19,"ideas":175,"institutions":4},"stated":0},{"year":2020,"topics":{"people":6,"ideas":98,"institutions":1,"governance":2},"stated":0},{"year":2021,"topics":{"people":27,"ideas":177,"compute":1,"institutions":23,"governance":4,"capital":6},"stated":0},{"year":2022,"topics":{"people":15,"ideas":217,"institutions":6,"governance":3,"compute":2,"capital":1},"stated":0},{"year":2023,"topics":{"capital":5,"compute":7,"institutions":23,"people":51,"governance":179,"ideas":389},"stated":0},{"year":2024,"topics":{"capital":12,"compute":19,"governance":282,"people":80,"ideas":178,"institutions":28},"stated":1},{"year":2025,"topics":{"governance":547,"compute":69,"capital":34,"people":90,"institutions":18,"ideas":178},"stated":7},{"year":2026,"topics":{"governance":442,"capital":8,"compute":32,"institutions":21,"ideas":91,"people":107},"stated":19},{"year":2027,"topics":{},"stated":9},{"year":2028,"topics":{},"stated":6},{"year":2029,"topics":{},"stated":3},{"year":2030,"topics":{},"stated":3},{"year":2032,"topics":{},"stated":2},{"year":2035,"topics":{},"stated":2},{"year":2039,"topics":{},"stated":1},{"year":2042,"topics":{},"stated":1},{"year":2050,"topics":{},"stated":1}],"scheduled":[],"statedBy":[]},"collections":[{"target":{"kind":"propositions","id":"superintelligence"},"title":"Propositions about superintelligence, 1863–2022","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"risk-debate"},"title":"Arguments about catastrophic risk from advanced AI","claimIds":["rd-r02-lecun-2019-instrumental-convergence-thread","rd-r08-lecun-2019-instrumental-convergence-thread","rd-r10b-lecun-2019-instrumental-convergence-thread"],"records":[]},{"target":{"kind":"propositions","id":"yudkowsky-arguments"},"title":"Eliezer Yudkowsky’s arguments on common claims about AI, 1996–2026","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"economics-of-machine-intelligence"},"title":"Economics of machine intelligence","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"orbital-compute-arguments"},"title":"Arguments about compute in orbit","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-bio-arguments"},"title":"Arguments about AI and biological risk","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-in-science"},"title":"AI in science","claimIds":[],"records":[]},{"target":{"kind":"documents"},"title":"Frontier safety frameworks","claimIds":[]},{"target":{"kind":"compare","id":"frontier-safety-frameworks"},"title":"Compare frameworks","claimIds":[]},{"target":{"kind":"control","id":"openai"},"title":"OpenAI’s control structure, 2015–2026","claimIds":[]},{"target":{"kind":"control","id":"anthropic"},"title":"Anthropic’s control structure, 2021–2026","claimIds":[]},{"target":{"kind":"commitments"},"title":"External commitments and obligations","claimIds":[]},{"target":{"kind":"question","id":"frameworks-to-obligations"},"title":"Frameworks, commitments and legal obligations, 2023–2026","claimIds":[]},{"target":{"kind":"reading"},"title":"Founding and early leadership","claimIds":[]}]},"version":"131evrz"}