{"entities":[{"id":"instrumental-convergence","name":"Instrumental convergence","type":"concept","proposition":{"code":"R02","criterion":"Agents with a wide range of final goals would tend to pursue the same intermediate goals, such as self-preservation, keeping their goals unchanged, acquiring resources or power, or improving themselves, because these serve almost any final goal.","terms":["instrumental convergence","convergent instrumental","basic AI drives","power-seeking"]}},{"id":"present-harms-priority","name":"Present harms deserve priority","type":"concept","proposition":{"code":"R12","criterion":"Documented present harms of AI systems (for example discrimination, exploitation of workers, surveillance, misinformation or concentration of power) deserve priority over hypothetical future catastrophic risks, or attention to the latter diverts attention from the former.","terms":["present harms","current harms","existing harms"]}},{"id":"russell-2019-instrumental-convergence-thread","name":"Stuart Russell's comments in the Facebook thread on instrumental convergence (2019)","shortName":"Russell, comments in the Facebook thread on instrumental convergence (2019)","type":"document"},{"id":"zador-2019-instrumental-convergence-thread","name":"Anthony Zador's comments in the Facebook thread on instrumental convergence (2019)","type":"document","subtype":"Comment","description":"A comment by Anthony Zador, dated 4 October 2019. Known from the LessWrong copy of 4 October 2019; the comments themselves are undated.","sourceIds":["source-rd-pace-2019-instrumental-convergence-debate"]}],"claims":[{"id":"rd-r02-zador-2019-instrumental-convergence-thread","subject":"zador-2019-instrumental-convergence-thread","predicate":"contests_proposition","object":{"kind":"entity","entityId":"instrumental-convergence"},"topic":"ideas","statement":"Zador's comments in the 2019 thread argue that for a robot that is not omnipotent, killing humans carries the risk of retaliation, so that it is unlikely to be an optimal strategy in most scenarios.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r02-zador-2019-instrumental-convergence-thread-1"],"attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."},"engagement":{"kind":"engages-premise","rationale":"Gives a reason against the premise that eliminating a threat is the optimal subgoal: for a non-omnipotent agent it invites retaliation, and other strategies serve survival."}}}},{"id":"rd-r12-zador-2019-instrumental-convergence-thread","subject":"zador-2019-instrumental-convergence-thread","predicate":"bears_on_proposition","object":{"kind":"entity","entityId":"present-harms-priority"},"topic":"ideas","statement":"Zador's comment in the 2019 thread states that focusing on killer robots that dominate humans is a distraction from much more serious issues, which he presents as the unintended societal consequences that improvements in AI will have, by analogy with books, food production technology and social media.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r12-zador-2019-instrumental-convergence-thread-1"],"attributes":{"thought":{"roles":["context"],"register":"nonfiction","missing":"Documented present harms of AI systems: the serious issues are unforeseen future consequences of AI, illustrated by other technologies.","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."}}}},{"id":"rd-reply-zador-2019-thread-to-russell-2019-thread","subject":"zador-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"russell-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"Zador's comment in the 2019 thread replies to Russell's argument, naming him.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-zador-2019-thread-to-russell-2019-thread-1"]},{"id":"rd-reply-russell-2019-thread-to-zador-2019-thread","subject":"russell-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"zador-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"Russell's comment in the 2019 thread replies to Zador's.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-russell-2019-thread-to-zador-2019-thread-1"]}],"sources":[{"id":"source-rd-pace-2019-instrumental-convergence-debate","documentId":"pace-2019-instrumental-convergence-debate","title":"Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More","canonicalUrl":"https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell"}],"evidence":[{"id":"evidence-rd-r02-zador-2019-instrumental-convergence-thread-1","claimId":"rd-r02-zador-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r12-zador-2019-instrumental-convergence-thread-1","claimId":"rd-r12-zador-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-zador-2019-thread-to-russell-2019-thread-1","claimId":"rd-reply-zador-2019-thread-to-russell-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-russell-2019-thread-to-zador-2019-thread-1","claimId":"rd-reply-russell-2019-thread-to-zador-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1852,1900,1905,1926],"claims":[5139,5256,5343,5344],"sources":[1792],"evidence":[6816,7028,7182,7183]},"index":{"byClaim":[["rd-r02-zador-2019-instrumental-convergence-thread",["propositions:risk-debate"]],["rd-r12-zador-2019-instrumental-convergence-thread",["propositions:risk-debate"]]],"marks":[["russell-2019-instrumental-convergence-thread",[{"collectionId":"risk-debate","propositionId":"alignment-is-hard","claimId":"rd-r06-russell-2019-instrumental-convergence-thread","text":"Earliest located use of the name “alignment problem” in Arguments about catastrophic risk from advanced AI for R06 Difficulty of alignment"}]]],"chronicle":{"years":[{"year":1863,"topics":{"ideas":5},"stated":0},{"year":1865,"topics":{"ideas":1},"stated":0},{"year":1872,"topics":{"ideas":7},"stated":0},{"year":1943,"topics":{"ideas":1},"stated":0},{"year":1947,"topics":{"ideas":2},"stated":0},{"year":1950,"topics":{"ideas":7},"stated":0},{"year":1951,"topics":{"ideas":5},"stated":0},{"year":1952,"topics":{"ideas":2},"stated":0},{"year":1955,"topics":{"ideas":31},"stated":0},{"year":1956,"topics":{"ideas":1},"stated":0},{"year":1957,"topics":{"ideas":1},"stated":0},{"year":1958,"topics":{"ideas":3},"stated":0},{"year":1959,"topics":{"ideas":6},"stated":0},{"year":1960,"topics":{"ideas":11},"stated":0},{"year":1961,"topics":{"ideas":10},"stated":0},{"year":1965,"topics":{"ideas":8},"stated":0},{"year":1966,"topics":{"ideas":20},"stated":0},{"year":1968,"topics":{"ideas":5},"stated":0},{"year":1969,"topics":{"ideas":2},"stated":0},{"year":1971,"topics":{"ideas":4},"stated":0},{"year":1972,"topics":{"ideas":1},"stated":0},{"year":1973,"topics":{"ideas":2},"stated":0},{"year":1974,"topics":{"ideas":2},"stated":0},{"year":1975,"topics":{"ideas":1},"stated":0},{"year":1976,"topics":{"ideas":1},"stated":0},{"year":1977,"topics":{"people":2,"ideas":8},"stated":0},{"year":1979,"topics":{"ideas":4},"stated":0},{"year":1980,"topics":{"ideas":2},"stated":0},{"year":1981,"topics":{"people":2},"stated":0},{"year":1982,"topics":{"ideas":4},"stated":0},{"year":1983,"topics":{"ideas":4},"stated":0},{"year":1984,"topics":{"ideas":21},"stated":0},{"year":1985,"topics":{"ideas":9},"stated":0},{"year":1986,"topics":{"ideas":18},"stated":0},{"year":1987,"topics":{"ideas":1},"stated":0},{"year":1988,"topics":{"ideas":6},"stated":0},{"year":1989,"topics":{"ideas":8},"stated":0},{"year":1990,"topics":{"ideas":5},"stated":0},{"year":1991,"topics":{"ideas":11},"stated":0},{"year":1992,"topics":{"ideas":9},"stated":0},{"year":1993,"topics":{"ideas":14,"people":3},"stated":0},{"year":1994,"topics":{"ideas":14},"stated":0},{"year":1995,"topics":{"ideas":10,"people":1},"stated":0},{"year":1996,"topics":{"ideas":71},"stated":0},{"year":1997,"topics":{"ideas":12},"stated":0},{"year":1998,"topics":{"ideas":49},"stated":0},{"year":1999,"topics":{"ideas":27,"people":1,"institutions":1},"stated":0},{"year":2000,"topics":{"ideas":62},"stated":0},{"year":2001,"topics":{"ideas":39},"stated":0},{"year":2002,"topics":{"ideas":30},"stated":0},{"year":2003,"topics":{"ideas":8},"stated":0},{"year":2004,"topics":{"ideas":12,"people":3},"stated":0},{"year":2005,"topics":{"ideas":10,"people":1},"stated":0},{"year":2006,"topics":{"ideas":26,"people":1},"stated":0},{"year":2007,"topics":{"ideas":20},"stated":0},{"year":2008,"topics":{"ideas":57,"people":3,"institutions":1},"stated":0},{"year":2009,"topics":{"ideas":10,"people":1},"stated":0},{"year":2010,"topics":{"people":3,"ideas":11,"institutions":1,"governance":4},"stated":0},{"year":2011,"topics":{"people":3,"ideas":7,"institutions":2},"stated":0},{"year":2012,"topics":{"people":4,"ideas":17,"institutions":2},"stated":0},{"year":2013,"topics":{"ideas":46,"people":4,"institutions":2,"capital":1},"stated":0},{"year":2014,"topics":{"institutions":2,"ideas":54,"people":5},"stated":0},{"year":2015,"topics":{"people":22,"ideas":122,"institutions":3,"capital":1},"stated":0},{"year":2016,"topics":{"ideas":88,"people":2},"stated":0},{"year":2017,"topics":{"people":16,"institutions":2,"ideas":128,"governance":2},"stated":0},{"year":2018,"topics":{"people":8,"ideas":158,"institutions":8,"governance":1},"stated":0},{"year":2019,"topics":{"capital":1,"compute":1,"people":10,"governance":19,"ideas":175,"institutions":4},"stated":0},{"year":2020,"topics":{"people":6,"ideas":98,"institutions":1,"governance":2},"stated":0},{"year":2021,"topics":{"people":27,"ideas":177,"compute":1,"institutions":23,"governance":4,"capital":6},"stated":0},{"year":2022,"topics":{"people":15,"ideas":217,"institutions":6,"governance":3,"compute":2,"capital":1},"stated":0},{"year":2023,"topics":{"capital":5,"compute":7,"institutions":23,"people":51,"governance":179,"ideas":389},"stated":0},{"year":2024,"topics":{"capital":12,"compute":19,"governance":282,"people":80,"ideas":178,"institutions":28},"stated":1},{"year":2025,"topics":{"governance":547,"compute":69,"capital":34,"people":90,"institutions":18,"ideas":178},"stated":7},{"year":2026,"topics":{"governance":442,"capital":8,"compute":32,"institutions":21,"ideas":91,"people":107},"stated":19},{"year":2027,"topics":{},"stated":9},{"year":2028,"topics":{},"stated":6},{"year":2029,"topics":{},"stated":3},{"year":2030,"topics":{},"stated":3},{"year":2032,"topics":{},"stated":2},{"year":2035,"topics":{},"stated":2},{"year":2039,"topics":{},"stated":1},{"year":2042,"topics":{},"stated":1},{"year":2050,"topics":{},"stated":1}],"scheduled":[],"statedBy":[]},"collections":[{"target":{"kind":"propositions","id":"superintelligence"},"title":"Propositions about superintelligence, 1863–2022","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"risk-debate"},"title":"Arguments about catastrophic risk from advanced AI","claimIds":["rd-r02-zador-2019-instrumental-convergence-thread","rd-r12-zador-2019-instrumental-convergence-thread"],"records":[]},{"target":{"kind":"propositions","id":"yudkowsky-arguments"},"title":"Eliezer Yudkowsky’s arguments on common claims about AI, 1996–2026","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"economics-of-machine-intelligence"},"title":"Economics of machine intelligence","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"orbital-compute-arguments"},"title":"Arguments about compute in orbit","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-bio-arguments"},"title":"Arguments about AI and biological risk","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-in-science"},"title":"AI in science","claimIds":[],"records":[]},{"target":{"kind":"documents"},"title":"Frontier safety frameworks","claimIds":[]},{"target":{"kind":"compare","id":"frontier-safety-frameworks"},"title":"Compare frameworks","claimIds":[]},{"target":{"kind":"control","id":"openai"},"title":"OpenAI’s control structure, 2015–2026","claimIds":[]},{"target":{"kind":"control","id":"anthropic"},"title":"Anthropic’s control structure, 2021–2026","claimIds":[]},{"target":{"kind":"commitments"},"title":"External commitments and obligations","claimIds":[]},{"target":{"kind":"question","id":"frameworks-to-obligations"},"title":"Frameworks, commitments and legal obligations, 2023–2026","claimIds":[]},{"target":{"kind":"reading"},"title":"Founding and early leadership","claimIds":[]}]},"version":"131evrz"}