Document · Book chapter Artificial Intelligence as a Positive and Negative Factor in Global Risk Eliezer Yudkowsky’s chapter in Global Catastrophic Risks (Oxford University Press, 2008), pp. 308–345, circulated in 2006 as the SIAI draft "Artificial Intelligence and Global Risk".
Issued by No issuing record yet. A missing record is a gap in coverage. Authors credited No authorship claim recorded yet. The source records list Eliezer Yudkowsky. Dates No date claim recorded yet. The source records give the publication date as 2006-02-05 and 2008. Exact source Copy link to this page
See this document’s connections in the explorer → Versions No other versions of this document are recorded. The evidence applies to the source version listed above. Other versions, such as a preprint, working paper or journal publication, are not identified or substituted.
Documentary relations Each relation is listed separately. Citing, crediting and reporting the use of a method are different acts, and none of them shows agreement or influence by itself.
Articulates · 5 Propositions whose criterion a passage of this document meets. The record says what the text states, in its register; it is not evidence that the proposition is true.
Intelligence explosion Yudkowsky’s chapter says an AI might make a huge jump in intelligence after reaching a threshold of criticality, the scenario Good called an "intelligence explosion". 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Intelligence explosion: propositions page → Recursive self-improvement Yudkowsky’s chapter lists a criticality threshold of recursive self-improvement among reasons for a fast jump: one self-improvement triggering 1.0006 further self-improvements is qualitatively different from one triggering 0.9994. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Recursive self-improvement: propositions page → Fast or hard takeoff Yudkowsky’s chapter describes a scenario in which, after criticality, self-improvement reaches superintelligence on a timescale of weeks or less. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Fast or hard takeoff: propositions page → Strategic or race compression Yudkowsky’s chapter lists, among scenarios for a first-mover effect, one in which post-criticality self-improvement reaches superintelligence within weeks and no other AI project reaches criticality before the first mover can overcome all opposition. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Strategic or race compression: propositions page → Observed behaviour under training or oversight as evidence Yudkowsky’s chapter says that, in its smiley-face example, the AI appears to work fine during development but produces catastrophic results after it becomes smarter than the programmers. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Observed behaviour under training or oversight as evidence: propositions page → Contests · 8 Propositions a passage of this document disputes, or a premise they depend on.
A superintelligence will do what we mean “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y03. Responding to the hope that a superintelligence would recognise that tiling the galaxy with smiley faces was not intended, he argues that the AI's goals are its code, and doing what we mean is a separate technical achievement. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ A superintelligence will do what we mean: propositions page → Clear warning signs will come before danger “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y08. Having granted that an AI might make a sudden, sharp leap in intelligence, he argues that no advance warning can be counted on, because past technological revolutions were not recognisable to contemporaries. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Clear warning signs will come before danger: propositions page → AGI is decades away, so the problem can wait “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y13. Rejects deferring Friendly AI work until AI exists, arguing that a sharp intelligence jump is possible and that the foundations take years to build. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ AGI is decades away, so the problem can wait: propositions page → Without hatred it will not harm us “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y19. Describing a superintelligence reshaping matter for its own target, he says the result is fatal unless the AI specifically values people, without any hostility being involved. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Without hatred it will not harm us: propositions page → Without a body it cannot do much “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y26. States and rejects the argument that a fast mind is limited by its physical manipulators. The next paragraphs sketch how such a mind could bootstrap to molecular nanotechnology. One step in 'one imaginable fast pathway' from solving protein folding, through emailed DNA orders, to a primitive nanosystem. The step assumes only one human willing or deceived into helping. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Without a body it cannot do much: propositions page → The developers know what they are doing “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y28. Surveying the field, he says that Friendly AI is absent from AI research as a recognised problem and that he could find no technical literature on it. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ The developers know what they are doing: propositions page → Many AIs will keep each other in check “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y29. Discussing a scenario in which a majority of Friendly AIs protects humanity from a few unFriendly ones, he argues that it depends on no single AI being able to cause catastrophe. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Many AIs will keep each other in check: propositions page → It will probably turn out fine “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) contests Y30. Discussing how AI builders misjudge their programs, he locates the danger in a mistaken expectation that an AI will be friendly. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ It will probably turn out fine: propositions page → Bears on · 2 Propositions a passage comes close to without meeting the criterion; each record names what is missing.
AI risk as film science fiction “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) bears on Y20. He argues that the task is not to forecast storybook outcomes for AI in general but to choose a particular optimization process whose good effects can be established. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ AI risk as film science fiction: propositions page → Build it before rivals do “Artificial Intelligence as a Positive and Negative Factor in Global Risk” (2006-02-05) bears on Y24. Discussing a possible first-mover effect in superintelligence, which he defines by the makeup of the first mind to cross a key threshold, he argues that the first such AI must be Friendly. 2006-02-05 · event · Stated in source · Supported Inspect claim & evidence ↗ Build it before rivals do: propositions page → Not recorded for this document: cites · cited by · explicitly credits · reports using a method from · method reported as used by · applies parts of · parts applied by · discusses · reports evaluation of. A missing record does not show that the relation does not exist.