Claim · 2019-10-04 · publication · Stated in source · Supported

Russell's comment in the 2019 thread states that in a toy decision problem an agent rewarded only for fetching coffee would switch off a human who might switch it off, with no built-in survival instinct or desire for power.

One recorded claim. The claim panel gives its qualification, the source passages with their place in each source, and how it was checked.

What it connects

Stuart Russell's comments in the Facebook thread on instrumental convergence (2019) → articulates → Instrumental convergence

Where it appears

Arguments about catastrophic risk from advanced AI →

Claim and evidence

Stuart Russell's comments in the Facebook thread on instrumental convergence (2019)articulates ↓Instrumental convergence

Machine-checked, not reviewed by an editorStated in sourceSupported

Supporting evidence: primary · 1 source

Date: 2019-10-04 · publication Records dated 2019 →
Wording attested: 2019-10-04. The wording is known from the LessWrong copy, whose compiler states that he improved formatting and fixed typos; the Facebook original was not compared.
Role: Articulation
Register (kind of text): Nonfiction

Scope and limitsA toy model offered as a mathematical observation, for a single specified reward (fetching coffee); the generalisation to almost any goal is stated in the question that opens the thread, not in Russell's comment. Each comment is the participant's own words in a relayed copy; the record states what the comment says in this thread, not the author's view at another date.

The evidence 01

primary sourceSupports the claim

Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More

LessWrong (post by Ben Pace reproducing a public Facebook thread) · 2019-10-04

Cite the original source

Cite the original document, not Selekoa. Check these details against it.

Source excerpt

“the agent will necessarily press that button as part of the optimal solution for fetching the coffee. No hatred, no desire for power, no built-in emotions, no built-in survival instinct, nothing except the desire to fetch the coffee successfully.”

Location in source Comment Thread #1, Stuart Russell

How it was read Retrieved directly from the publisher

Machine-checked · 2026-10-31

Source details and check notes

Authors: Ben Pace
Version: Retrieved for the candidate pack on 2026-10-29 (SHA-256 59e3dd77f4825a5a60ce7f1f10dd8c2eebaac7786aee13c861e21d853d251495). Post contents version 1.0.0 (unedited since posting on 2019-10-04), retrieved through the LessWrong GraphQL API (post WxW6Gc6f2z3mzmqKs) because the page refused direct retrieval (HTTP 403). The compiler states that he improved formatting and fixed typos; the Facebook original was not read. The cross-family check retrieved it again on 2026-10-31; the bytes differ, and the check compared the quoted passages.
Canonical address: Open the canonical source ↗
Accessed: 2026-10-29
Language: English (en)
Attribution: Ben Pace, “Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More”, 2019-10-04.
Reuse: No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution.
Check notes: Candidate pack passage, arguments about catastrophic risk from advanced AI (research/risk-debate). The quotation (39 words) matched the text retrieved for the candidate pack on 2026-10-29. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. No human editorial review.

Status and meaning

Editorial status. Not reviewed by an editor. An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.

Relation. A passage of the document meets the proposition’s criterion. The record documents what the text states, in its register (fiction and satire are labelled); it is not evidence that the proposition is true and implies no influence on or from another text.

Date. Dated by the LessWrong copy of 4 October 2019. The Facebook thread follows the Scientific American piece of 26 September 2019; the copy gives no date for individual comments.

Record ID rd-r02-russell-2019-instrumental-convergence-thread