Claim and evidence Close ×
Bostrom's paper of March 2002 states that the earliest superintelligent entity humans create might be given mistaken goals that lead it to annihilate humankind, giving the example of a subgoal elevated to a supergoal. Copy claim link
Copy claim reference
Machine-checked, not reviewed by an editor Stated in source Supported
Supporting evidence: primary · 1 source
Date 2002-03 · publication Records dated 2002 → Claimed date 2001-04. An earlier version was presented in April 2001 (acknowledgements); it does not date the quoted wording. Role Earliest located Register (kind of text) Nonfiction Scope and limits The paper describes the outcome as a mistake in goal specification; it does not use the term "loss of control" and does not discuss how a system would resist correction. A statement of the author's argument, not evidence that it is true.
The evidence 02 primary source Supports the claim
Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards ↗ Journal of Evolution and Technology, vol. 9 (jetpress.org) · 2002-03
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so.” Location in source Section 4.4, "Badly programmed superintelligence"
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-31
Source details and check notes Authors Nick Bostrom Version Retrieved for the candidate pack on 2026-10-29 (SHA-256 2b18c3db8df16008ac27ccdbdaacffbcb6d2a99e3b803d14c03f2a5ccb9dd3c3). HTML article as served by the journal at jetpress.org/volume9/risks.html, headed "Vol. 9 - March 2002"; the text layer shows some mis-encoded punctuation. The author's site also serves a later reformatted PDF ("First version: 2001"), not used for quotation. The cross-family check retrieved it again on 2026-10-31, with the same bytes. Canonical address Open the canonical source ↗ Accessed 2026-10-29 Language English (en) Attribution Nick Bostrom, “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards”, 2002-03. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, arguments about catastrophic risk from advanced AI (research/risk-debate). The quotation (27 words) matched the text retrieved for the candidate pack on 2026-10-29. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-31, item 49) answered the record’s question blind and agreed with its stance. No human editorial review. primary source Supports the claim
Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards ↗ Journal of Evolution and Technology, vol. 9 (jetpress.org) · 2002-03
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question.” Location in source Section 4.4
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-31
Source details and check notes Authors Nick Bostrom Version Retrieved for the candidate pack on 2026-10-29 (SHA-256 2b18c3db8df16008ac27ccdbdaacffbcb6d2a99e3b803d14c03f2a5ccb9dd3c3). HTML article as served by the journal at jetpress.org/volume9/risks.html, headed "Vol. 9 - March 2002"; the text layer shows some mis-encoded punctuation. The author's site also serves a later reformatted PDF ("First version: 2001"), not used for quotation. The cross-family check retrieved it again on 2026-10-31, with the same bytes. Canonical address Open the canonical source ↗ Accessed 2026-10-29 Language English (en) Attribution Nick Bostrom, “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards”, 2002-03. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, arguments about catastrophic risk from advanced AI (research/risk-debate). The quotation (35 words) matched the text retrieved for the candidate pack on 2026-10-29. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. A cross-family machine check (2026-10-31, item 49) answered the record’s question blind and agreed with its stance. No human editorial review. Status and meaning Editorial status. Not reviewed by an editor . An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.
Relation. A passage of the document meets the proposition’s criterion. The record documents what the text states, in its register (fiction and satire are labelled); it is not evidence that the proposition is true and implies no influence on or from another text.
Date. Date of publication, as the source gives it.
Record ID rd-r08-bostrom-2002-existential-risks