Claim · 2002-05 · publication · Stated in source · Supported

Goertzel's essay of May 2002 argues that a proposed design for a friendly AI goal system is very unlikely to produce a value system robust enough to survive repeated, drastic self-modifications aimed at increasing intelligence.

One recorded claim. The claim panel gives its qualification, the source passages with their place in each source, and how it was checked.

What it connects

Thoughts on AI Morality → articulates → Difficulty of alignment

Where it appears

Arguments about catastrophic risk from advanced AI →

Claim and evidence

Thoughts on AI Moralityarticulates ↓Difficulty of alignment

Machine-checked, not reviewed by an editorStated in sourceSupported

Supporting evidence: primary · 1 source

Date: 2002-05 · publication Records dated 2002 →
Role: Earliest located
Register (kind of text): Nonfiction

Scope and limitsThe essay states the difficulty as an objection to one proposed design (Yudkowsky's), not as an impossibility result. Earliest located among the texts the searches found, after a cross-family check located it; it is not evidence of coinage.

The evidence 01

primary sourceSupports the claim

Thoughts on AI Morality

Dynamical Psychology (goertzel.org) · 2002-05

Cite the original source

Cite the original document, not Selekoa. Check these details against it.

Source excerpt

“very unlikely to succeed at producing an AGI value system that is robust enough to survive successive, drastic intelligence-increase-directed self-modifications.”

Location in source Section 1, discussion of Yudkowsky's proposed architecture

How it was read Retrieved directly from the publisher

Machine-checked · 2026-10-31

Source details and check notes

Authors: Ben Goertzel
Version: HTML page served at goertzel.org, headed "Ben Goertzel, May, 2002"; the file's own metadata records its creation on 5 May 2002. The wording is the current file's; earlier bytes are not authenticated. Retrieved at admission on 2026-10-31 (SHA-256 d27d65bd6f2a6fbb45e8aceb28c27020f069686436df5826e84bc68d59dcd66e).
Canonical address: Open the canonical source ↗
Accessed: 2026-10-31
Language: English (en)
Attribution: Ben Goertzel, “Thoughts on AI Morality”, 2002-05.
Reuse: No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution.
Check notes: Added at the admission of the risk-debate pack (research/risk-debate). The quotation (20 words) matched a fresh retrieval of 2026-10-31 (SHA-256 d27d65bd6f2a6fbb45e8aceb28c27020f069686436df5826e84bc68d59dcd66e). Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms. The cross-family check of 2026-10-31 proposed the record and read this passage. No human editorial review.

Status and meaning

Editorial status. Not reviewed by an editor. An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.

Relation. A passage of the document meets the proposition’s criterion. The record documents what the text states, in its register (fiction and satire are labelled); it is not evidence that the proposition is true and implies no influence on or from another text.

Date. Dated May 2002 on its page.

Record ID rd-r06-goertzel-2002-thoughts-on-ai-morality