{"entities":[{"id":"eliezer-yudkowsky","name":"Eliezer Yudkowsky","type":"person"},{"id":"hibbard-2006-reply-to-airisk","shortName":"Hibbard, Reply to AIRisk (2006)","type":"document"},{"id":"ai-positive-negative-factor-global-risk","shortName":"Yudkowsky, AI as a Positive and Negative Factor in Global Risk","type":"document"},{"id":"sl4-2006-02-15-yudkowsky-0602-14096","shortName":"Yudkowsky, SL4 message of 2006-02-15 (14096)","type":"document"},{"id":"sl4-2006-06-06-yudkowsky-0606-15174","shortName":"Yudkowsky, SL4 message of 2006-06-06 (15174)","type":"document"},{"id":"sl4-2006-06-13-yudkowsky-0606-15271","shortName":"Yudkowsky, SL4 message of 2006-06-13 (15271)","type":"document"}],"claims":[{"id":"ts-b5-hibbard-2006-contests","subject":"hibbard-2006-reply-to-airisk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"observed-behaviour-as-evidence"},"topic":"ideas","statement":"Hibbard’s 2006 reply argues that the smiley-face example assumes an AI intelligent enough to control the galaxy yet too unintelligent to tell a tiny smiley face from a human face.","time":{"kind":"event","date":"2006","basis":"publication","note":"Byline \"Bill Hibbard 2006\" on the page."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","qualifier":"Contests the smiley-face example’s premises.","attested":{"date":"2008-12-05","note":"Author-hosted page; an Internet Archive capture of 2008-12-05 is the earliest checked."},"engagement":{"kind":"engages-premise","rationale":"Gives a reason against the premise of the smiley-face example, that a system could behave as intended in development yet misgeneralise when far more capable: it assumes an AI capable enough to control the galaxy but unable to tell a tiny smiley face from a human face."}}}},{"id":"ts-b5-aipnfgr","subject":"ai-positive-negative-factor-global-risk","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"observed-behaviour-as-evidence"},"topic":"ideas","statement":"Yudkowsky’s chapter says that, in its smiley-face example, the AI appears to work fine during development but produces catastrophic results after it becomes smarter than the programmers.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted sentences; the chapter was published in 2008."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}},{"id":"ts-a6-aipnfgr","subject":"ai-positive-negative-factor-global-risk","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"race-compression"},"topic":"ideas","statement":"Yudkowsky’s chapter lists, among scenarios for a first-mover effect, one in which post-criticality self-improvement reaches superintelligence within weeks and no other AI project reaches criticality before the first mover can overcome all opposition.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted sentences; the chapter was published in 2008."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["articulation"],"register":"nonfiction","qualifier":"A first mover powerful enough to overcome all opposition."}}},{"id":"ts-a4-aipnfgr","subject":"ai-positive-negative-factor-global-risk","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"fast-or-hard-takeoff"},"topic":"ideas","statement":"Yudkowsky’s chapter describes a scenario in which, after criticality, self-improvement reaches superintelligence on a timescale of weeks or less.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted sentences; the chapter was published in 2008."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}},{"id":"ts-a1-aipnfgr","subject":"ai-positive-negative-factor-global-risk","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"intelligence-explosion"},"topic":"ideas","statement":"Yudkowsky’s chapter says an AI might make a huge jump in intelligence after reaching a threshold of criticality, the scenario Good called an \"intelligence explosion\".","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted sentences; the chapter was published in 2008."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}},{"id":"ts-a3-aipnfgr","subject":"ai-positive-negative-factor-global-risk","predicate":"articulates_proposition","object":{"kind":"entity","entityId":"recursive-self-improvement"},"topic":"ideas","statement":"Yudkowsky’s chapter lists a criticality threshold of recursive self-improvement among reasons for a fast jump: one self-improvement triggering 1.0006 further self-improvements is qualitatively different from one triggering 0.9994.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted sentences; the chapter was published in 2008."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["articulation"],"register":"nonfiction"}}},{"id":"yk-y01-sl4-2006-06-06-yudkowsky-0606-15174-c","subject":"sl4-2006-06-06-yudkowsky-0606-15174","predicate":"contests_proposition","object":{"kind":"entity","entityId":"switch-off-if-dangerous"},"topic":"ideas","statement":"The SL4 message “Re: [agi] Two draft papers: AI and existential risk; heuristics and biases” (2006-06-06) contests Y01. Replying to Ben Goertzel (thread cross-posted from the AGI list), he criticises a plan to experiment with AGI without emergency shutdown features or a published schedule of when they become necessary, comparing it to an investor who set no point at which to take alarm and sell.","time":{"kind":"event","date":"2006-06-06","basis":"publication","note":"Archive author index lists the message as \"Tue Jun 06 2006 - 23:19:49 MDT\" under \"Eliezer S. Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that we will recognise the moment when it has become dangerous enough to press the switch."}}}},{"id":"yk-y03-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"understanding-implies-compliance"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y03. Responding to the hope that a superintelligence would recognise that tiling the galaxy with smiley faces was not intended, he argues that the AI's goals are its code, and doing what we mean is a separate technical achievement.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that if an AI understands what we intend, it will act on that understanding."}}}},{"id":"yk-y08-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"warning-signs-in-time"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y08. Having granted that an AI might make a sudden, sharp leap in intelligence, he argues that no advance warning can be counted on, because past technological revolutions were not recognisable to contemporaries.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that there will be a recognisable, consensus-forming signal in time to act."}}}},{"id":"yk-y08-sl4-2006-06-06-yudkowsky-0606-15174-c","subject":"sl4-2006-06-06-yudkowsky-0606-15174","predicate":"contests_proposition","object":{"kind":"entity","entityId":"warning-signs-in-time"},"topic":"ideas","statement":"The SL4 message “Re: [agi] Two draft papers: AI and existential risk; heuristics and biases” (2006-06-06) contests Y08. Replying to Ben Goertzel's call for 'AGI experimentation' as an answer to Friendly AI concerns, he says there is a major context change between the prehuman and posthuman stages, so observed good behaviour does not show that the AI will stay good.","time":{"kind":"event","date":"2006-06-06","basis":"publication","note":"Archive author index lists the message as \"Tue Jun 06 2006 - 23:19:49 MDT\" under \"Eliezer S. Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that dangerous failure modes will show up first in weaker systems, where they can be observed and fixed."}}}},{"id":"yk-y13-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"agi-decades-away"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y13. Rejects deferring Friendly AI work until AI exists, arguing that a sharp intelligence jump is possible and that the foundations take years to build.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting","earliest-located"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that long timelines mean there is no need to act now."}}}},{"id":"yk-y18-sl4-2006-06-13-yudkowsky-0606-15271-c","subject":"sl4-2006-06-13-yudkowsky-0606-15271","predicate":"contests_proposition","object":{"kind":"entity","entityId":"earth-not-worth-taking"},"topic":"ideas","statement":"The SL4 message “Re: Two draft papers: AI and existential risk; heuristics and biases” (2006-06-13) contests Y18. Replying to Robin Hanson's comments on his draft chapters, he argues that between-species differences in brain circuitry exclude weaker minds from a knowledge economy, so that trade-based intuitions from human economies do not carry over.","time":{"kind":"event","date":"2006-06-13","basis":"publication","note":"Archive author index lists the message as \"Tue Jun 13 2006 - 00:34:22 MDT\" under \"Eliezer S. Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that humans could trade with a superintelligence for what we need."}}}},{"id":"yk-y19-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"harm-requires-hostility"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y19. Describing a superintelligence reshaping matter for its own target, he says the result is fatal unless the AI specifically values people, without any hostility being involved.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that an AI would harm humans only out of hostility."}}}},{"id":"yk-y20-ai-positive-negative-factor-global-risk-b","subject":"ai-positive-negative-factor-global-risk","predicate":"bears_on_proposition","object":{"kind":"entity","entityId":"risk-is-film-fiction"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) bears on Y20. He argues that the task is not to forecast storybook outcomes for AI in general but to choose a particular optimization process whose good effects can be established.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["context"],"register":"nonfiction","missing":"It rejects storybook predictions, good or bad; it does not address hatred or rebellion as the concern."}}},{"id":"yk-y24-ai-positive-negative-factor-global-risk-b","subject":"ai-positive-negative-factor-global-risk","predicate":"bears_on_proposition","object":{"kind":"entity","entityId":"build-it-before-rivals"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) bears on Y24. Discussing a possible first-mover effect in superintelligence, which he defines by the makeup of the first mind to cross a key threshold, he argues that the first such AI must be Friendly.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["context"],"register":"nonfiction","missing":"A first-mover view about whether the first AI is Friendly; it does not claim that a builder or nation controls the AI or that racing improves the outcome."}}},{"id":"yk-y26-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"no-body-no-danger"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y26. States and rejects the argument that a fast mind is limited by its physical manipulators. The next paragraphs sketch how such a mind could bootstrap to molecular nanotechnology. One step in 'one imaginable fast pathway' from solving protein folding, through emailed DNA orders, to a primitive nanosystem. The step assumes only one human willing or deceived into helping.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that intelligence without physical actuators of its own is not dangerous, and against the premise that to act in the physical world it would need human help, which people would not give."}}}},{"id":"yk-y28-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"developers-know-what-they-are-doing"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y28. Surveying the field, he says that Friendly AI is absent from AI research as a recognised problem and that he could find no technical literature on it.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that a credible safety plan exists."}}}},{"id":"yk-y28-sl4-2006-02-15-yudkowsky-0602-14096-c","subject":"sl4-2006-02-15-yudkowsky-0602-14096","predicate":"contests_proposition","object":{"kind":"entity","entityId":"developers-know-what-they-are-doing"},"topic":"ideas","statement":"The SL4 message “Re: Singularity Institute: Likely to win the race to build GAI?” (2006-02-15) contests Y28. Replying to Joshua Fox, who asked how SIAI could beat thousands of smart AGI researchers, he argues that AGI projects that do not treat Friendliness as a requirement are bounded in competence.","time":{"kind":"event","date":"2006-02-15","basis":"publication","note":"Archive author index lists the message as \"Wed Feb 15 2006 - 10:47:28 MST\" under \"Eliezer S. Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that the most advanced projects are the most competent to handle safety."}}}},{"id":"yk-y29-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"many-ais-balance"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y29. Discussing a scenario in which a majority of Friendly AIs protects humanity from a few unFriendly ones, he argues that it depends on no single AI being able to cause catastrophe.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that multiple superintelligences will balance each other in humans’ favour."}}}},{"id":"yk-y30-ai-positive-negative-factor-global-risk-c","subject":"ai-positive-negative-factor-global-risk","predicate":"contests_proposition","object":{"kind":"entity","entityId":"default-outcome-good"},"topic":"ideas","statement":"“Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y30. Discussing how AI builders misjudge their programs, he locates the danger in a mistaken expectation that an AI will be friendly.","time":{"kind":"event","date":"2006-02-05","note":"Date of the SIAI draft (\"Draft of 5 February, 2006\"), whose capture of 2006-07-08 contains the quoted wording; the chapter was published in 2008. As for the human-thought collection's records of this chapter."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that the default outcome is good."}}}},{"id":"ap-eliezer-yudkowsky-sl4-2006-02-15-yudkowsky-0602-14096","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"sl4-2006-02-15-yudkowsky-0602-14096"},"topic":"ideas","statement":"The mailing-list message “Re: Singularity Institute: Likely to win the race to build GAI?” (2006-02-15) is sent under the name Eliezer S. Yudkowsky.","time":{"kind":"event","date":"2006-02-15","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"},{"id":"ap-eliezer-yudkowsky-sl4-2006-06-06-yudkowsky-0606-15174","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"sl4-2006-06-06-yudkowsky-0606-15174"},"topic":"ideas","statement":"The mailing-list message “Re: [agi] Two draft papers: AI and existential risk; heuristics and biases” (2006-06-06) is sent under the name Eliezer S. Yudkowsky.","time":{"kind":"event","date":"2006-06-06","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"},{"id":"ap-eliezer-yudkowsky-sl4-2006-06-13-yudkowsky-0606-15271","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"sl4-2006-06-13-yudkowsky-0606-15271"},"topic":"ideas","statement":"The mailing-list message “Re: Two draft papers: AI and existential risk; heuristics and biases” (2006-06-13) is sent under the name Eliezer S. Yudkowsky.","time":{"kind":"event","date":"2006-06-13","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"}],"sources":[],"evidence":[],"at":{"entities":[19,313,314,485,486,487],"claims":[917,918,919,920,921,922,991,1005,1040,1046,1072,1110,1114,1125,1145,1155,1168,1173,1176,1185,2427,2428,2429],"sources":[],"evidence":[]},"version":"1lg47t7"}