{"entities":[{"id":"bengio-2019-instrumental-convergence-thread","name":"Yoshua Bengio's comments in the Facebook thread on instrumental convergence (2019)","type":"document","subtype":"Comment","description":"A comment by Yoshua Bengio, dated 4 October 2019. Known from the LessWrong copy of 4 October 2019; the comments themselves are undated.","sourceIds":["source-rd-pace-2019-instrumental-convergence-debate"]},{"id":"control-by-design","name":"Advanced AI can be kept under control by design","type":"concept","proposition":{"code":"R10b","criterion":"People can build advanced AI systems with objectives, safeguards or oversight that keep them under human control, so that loss of control is avoidable in ordinary engineering practice.","terms":[]}},{"id":"lecun-2019-instrumental-convergence-thread","name":"Yann LeCun's comments in the Facebook thread on instrumental convergence (2019)","shortName":"LeCun, comments in the Facebook thread on instrumental convergence (2019)","type":"document"},{"id":"misuse-route","name":"Catastrophic risk from misuse","type":"concept","proposition":{"code":"R09","criterion":"People's deliberate use of advanced AI, for example to develop weapons of mass destruction, to attack critical systems, or to seize or entrench power, could cause catastrophe at the scale of mass casualties or worse.","terms":["misuse","malicious use","bad actors"]}}],"claims":[{"id":"rd-r09-bengio-2019-instrumental-convergence-thread","subject":"bengio-2019-instrumental-convergence-thread","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"misuse-route"},"topic":"ideas","statement":"Bengio's comments in the 2019 thread state that even if rogue AI from value misalignment can be mitigated, a single person acting greedily for power could unleash dangerous AIs, and that the misalignment of powerful people or companies with the common good can lead to catastrophic outcomes.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r09-bengio-2019-instrumental-convergence-thread-1"],"attributes":{"thought":{"roles":["articulation"],"register":"nonfiction","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."}}}},{"id":"rd-r10b-bengio-2019-instrumental-convergence-thread","subject":"bengio-2019-instrumental-convergence-thread","predicate":"contests_proposition","object":{"kind":"entity","entityId":"control-by-design"},"topic":"ideas","statement":"Bengio's comments in the 2019 thread agree that machines' objectives can be designed and trained, and state that AI systems surpassing human intellectual power could foil attempts at setting objectives that avoid harm, which calls for more research on value alignment.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-r10b-bengio-2019-instrumental-convergence-thread-1"],"attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"date":"2019-10-04","note":"The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared."},"engagement":{"kind":"engages-premise","rationale":"Gives a reason against the premise that designing a system's objective keeps it controllable: a system more capable than its designers could foil the objectives set for it."}}}},{"id":"rd-reply-bengio-2019-thread-to-lecun-2019-thread","subject":"bengio-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"lecun-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"Bengio's comment in the 2019 thread replies to LeCun's points on designing objectives.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-bengio-2019-thread-to-lecun-2019-thread-1"]},{"id":"rd-reply-lecun-2019-thread-to-bengio-2019-thread","subject":"lecun-2019-instrumental-convergence-thread","predicate":"cites","object":{"kind":"entity","entityId":"bengio-2019-instrumental-convergence-thread"},"topic":"ideas","statement":"LeCun's comment in the 2019 thread replies to Bengio's summary of the debate.","time":{"kind":"event","date":"2019-10-04","basis":"publication","note":"Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments."},"derivation":"explicit","resolution":"supported","editorialState":"seed","evidenceIds":["evidence-rd-reply-lecun-2019-thread-to-bengio-2019-thread-1"]}],"sources":[{"id":"source-rd-pace-2019-instrumental-convergence-debate","documentId":"pace-2019-instrumental-convergence-debate","title":"Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More","canonicalUrl":"https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell"}],"evidence":[{"id":"evidence-rd-r09-bengio-2019-instrumental-convergence-thread-1","claimId":"rd-r09-bengio-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-r10b-bengio-2019-instrumental-convergence-thread-1","claimId":"rd-r10b-bengio-2019-instrumental-convergence-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-bengio-2019-thread-to-lecun-2019-thread-1","claimId":"rd-reply-bengio-2019-thread-to-lecun-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"},{"id":"evidence-rd-reply-lecun-2019-thread-to-bengio-2019-thread-1","claimId":"rd-reply-lecun-2019-thread-to-bengio-2019-thread","sourceId":"source-rd-pace-2019-instrumental-convergence-debate","sourceRole":"primary","support":"supports","verification":"machine_checked","retrievalRoute":"direct"}],"at":{"entities":[1791,1812,1862,1878],"claims":[5221,5241,5345,5346],"sources":[1792],"evidence":[6968,7004,7184,7185]},"index":{"byClaim":[["rd-r09-bengio-2019-instrumental-convergence-thread",["propositions:risk-debate"]],["rd-r10b-bengio-2019-instrumental-convergence-thread",["propositions:risk-debate"]]],"marks":[],"chronicle":{"years":[{"year":1863,"topics":{"ideas":5},"stated":0},{"year":1865,"topics":{"ideas":1},"stated":0},{"year":1872,"topics":{"ideas":7},"stated":0},{"year":1943,"topics":{"ideas":1},"stated":0},{"year":1947,"topics":{"ideas":2},"stated":0},{"year":1950,"topics":{"ideas":7},"stated":0},{"year":1951,"topics":{"ideas":5},"stated":0},{"year":1952,"topics":{"ideas":2},"stated":0},{"year":1955,"topics":{"ideas":31},"stated":0},{"year":1956,"topics":{"ideas":1},"stated":0},{"year":1957,"topics":{"ideas":1},"stated":0},{"year":1958,"topics":{"ideas":3},"stated":0},{"year":1959,"topics":{"ideas":6},"stated":0},{"year":1960,"topics":{"ideas":11},"stated":0},{"year":1961,"topics":{"ideas":10},"stated":0},{"year":1965,"topics":{"ideas":8},"stated":0},{"year":1966,"topics":{"ideas":20},"stated":0},{"year":1968,"topics":{"ideas":5},"stated":0},{"year":1969,"topics":{"ideas":2},"stated":0},{"year":1971,"topics":{"ideas":4},"stated":0},{"year":1972,"topics":{"ideas":1},"stated":0},{"year":1973,"topics":{"ideas":2},"stated":0},{"year":1974,"topics":{"ideas":2},"stated":0},{"year":1975,"topics":{"ideas":1},"stated":0},{"year":1976,"topics":{"ideas":1},"stated":0},{"year":1977,"topics":{"people":2,"ideas":8},"stated":0},{"year":1979,"topics":{"ideas":4},"stated":0},{"year":1980,"topics":{"ideas":2},"stated":0},{"year":1981,"topics":{"people":2},"stated":0},{"year":1982,"topics":{"ideas":4},"stated":0},{"year":1983,"topics":{"ideas":4},"stated":0},{"year":1984,"topics":{"ideas":21},"stated":0},{"year":1985,"topics":{"ideas":9},"stated":0},{"year":1986,"topics":{"ideas":18},"stated":0},{"year":1987,"topics":{"ideas":1},"stated":0},{"year":1988,"topics":{"ideas":6},"stated":0},{"year":1989,"topics":{"ideas":8},"stated":0},{"year":1990,"topics":{"ideas":5},"stated":0},{"year":1991,"topics":{"ideas":11},"stated":0},{"year":1992,"topics":{"ideas":9},"stated":0},{"year":1993,"topics":{"ideas":14,"people":3},"stated":0},{"year":1994,"topics":{"ideas":14},"stated":0},{"year":1995,"topics":{"ideas":10,"people":1},"stated":0},{"year":1996,"topics":{"ideas":71},"stated":0},{"year":1997,"topics":{"ideas":12},"stated":0},{"year":1998,"topics":{"ideas":49},"stated":0},{"year":1999,"topics":{"ideas":27,"people":1,"institutions":1},"stated":0},{"year":2000,"topics":{"ideas":62},"stated":0},{"year":2001,"topics":{"ideas":39},"stated":0},{"year":2002,"topics":{"ideas":30},"stated":0},{"year":2003,"topics":{"ideas":8},"stated":0},{"year":2004,"topics":{"ideas":12,"people":3},"stated":0},{"year":2005,"topics":{"ideas":10,"people":1},"stated":0},{"year":2006,"topics":{"ideas":26,"people":1},"stated":0},{"year":2007,"topics":{"ideas":20},"stated":0},{"year":2008,"topics":{"ideas":57,"people":3,"institutions":1},"stated":0},{"year":2009,"topics":{"ideas":10,"people":1},"stated":0},{"year":2010,"topics":{"people":3,"ideas":11,"institutions":1,"governance":4},"stated":0},{"year":2011,"topics":{"people":3,"ideas":7,"institutions":2},"stated":0},{"year":2012,"topics":{"people":4,"ideas":17,"institutions":2},"stated":0},{"year":2013,"topics":{"ideas":46,"people":4,"institutions":2,"capital":1},"stated":0},{"year":2014,"topics":{"institutions":2,"ideas":54,"people":5},"stated":0},{"year":2015,"topics":{"people":22,"ideas":122,"institutions":3,"capital":1},"stated":0},{"year":2016,"topics":{"ideas":88,"people":2},"stated":0},{"year":2017,"topics":{"people":16,"institutions":2,"ideas":128,"governance":2},"stated":0},{"year":2018,"topics":{"people":8,"ideas":158,"institutions":8,"governance":1},"stated":0},{"year":2019,"topics":{"capital":1,"compute":1,"people":10,"governance":19,"ideas":175,"institutions":4},"stated":0},{"year":2020,"topics":{"people":6,"ideas":98,"institutions":1,"governance":2},"stated":0},{"year":2021,"topics":{"people":27,"ideas":177,"compute":1,"institutions":23,"governance":4,"capital":6},"stated":0},{"year":2022,"topics":{"people":15,"ideas":217,"institutions":6,"governance":3,"compute":2,"capital":1},"stated":0},{"year":2023,"topics":{"capital":5,"compute":7,"institutions":23,"people":51,"governance":179,"ideas":389},"stated":0},{"year":2024,"topics":{"capital":12,"compute":19,"governance":282,"people":80,"ideas":178,"institutions":28},"stated":1},{"year":2025,"topics":{"governance":547,"compute":69,"capital":34,"people":90,"institutions":18,"ideas":178},"stated":7},{"year":2026,"topics":{"governance":442,"capital":8,"compute":32,"institutions":21,"ideas":91,"people":107},"stated":19},{"year":2027,"topics":{},"stated":9},{"year":2028,"topics":{},"stated":6},{"year":2029,"topics":{},"stated":3},{"year":2030,"topics":{},"stated":3},{"year":2032,"topics":{},"stated":2},{"year":2035,"topics":{},"stated":2},{"year":2039,"topics":{},"stated":1},{"year":2042,"topics":{},"stated":1},{"year":2050,"topics":{},"stated":1}],"scheduled":[],"statedBy":[]},"collections":[{"target":{"kind":"propositions","id":"superintelligence"},"title":"Propositions about superintelligence, 1863–2022","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"risk-debate"},"title":"Arguments about catastrophic risk from advanced AI","claimIds":["rd-r09-bengio-2019-instrumental-convergence-thread","rd-r10b-bengio-2019-instrumental-convergence-thread"],"records":[]},{"target":{"kind":"propositions","id":"yudkowsky-arguments"},"title":"Eliezer Yudkowsky’s arguments on common claims about AI, 1996–2026","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"economics-of-machine-intelligence"},"title":"Economics of machine intelligence","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"orbital-compute-arguments"},"title":"Arguments about compute in orbit","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-bio-arguments"},"title":"Arguments about AI and biological risk","claimIds":[],"records":[]},{"target":{"kind":"propositions","id":"ai-in-science"},"title":"AI in science","claimIds":[],"records":[]},{"target":{"kind":"documents"},"title":"Frontier safety frameworks","claimIds":[]},{"target":{"kind":"compare","id":"frontier-safety-frameworks"},"title":"Compare frameworks","claimIds":[]},{"target":{"kind":"control","id":"openai"},"title":"OpenAI’s control structure, 2015–2026","claimIds":[]},{"target":{"kind":"control","id":"anthropic"},"title":"Anthropic’s control structure, 2021–2026","claimIds":[]},{"target":{"kind":"commitments"},"title":"External commitments and obligations","claimIds":[]},{"target":{"kind":"question","id":"frameworks-to-obligations"},"title":"Frameworks, commitments and legal obligations, 2023–2026","claimIds":[]},{"target":{"kind":"reading"},"title":"Founding and early leadership","claimIds":[]}]},"version":"131evrz"}