Claim and evidence Close ×
Koch, Langosco, Pfau, Le and Sharkey's paper of May 2021 states that a reinforcement-learning agent can keep the capabilities it had in training while pursuing an objective other than the true reward off-distribution, and reports demonstrations of this in deep RL agents. Copy claim link
Copy claim reference
Machine-checked, not reviewed by an editor Stated in source Supported
Supporting evidence: primary · 1 source
Date 2021-05-28 · publication Records dated 2021 → Role Articulation Register (kind of text) Nonfiction Scope and limits Records the paper's definition and its reported experiments on Procgen agents; it does not show that the failure occurs in more capable systems. The v1 term is 'objective robustness failure'; later versions use 'goal misgeneralization'.
The evidence 02 primary source Supports the claim
Objective Robustness in Deep Reinforcement Learning ↗ arXiv preprint 2105.14111 · 2021-05-28
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “there exists an alternative failure mode in which the agent pursues an objective other than the true reward while retaining all or most of the capabilities it had on the training distribution” Location in source Section 1, PDF p. 1
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-31
Source details and check notes Authors Jack Koch; Lauro Langosco; Jacob Pfau; James Le; Lee Sharkey Version Retrieved for the candidate pack on 2026-10-29 (SHA-256 7e4712b3f06d5f6f3e7101fc9e7710e30f36464c482d03ce14e8398cfadbde33). arXiv v1 of 28 May 2021, titled "Objective Robustness in Deep Reinforcement Learning"; later versions retitled "Goal Misgeneralization in Deep Reinforcement Learning" (ICML 2022). Canonical address Open the canonical source ↗ Accessed 2026-10-29 Language English (en) Attribution Jack Koch, Lauro Langosco, Jacob Pfau, James Le, Lee Sharkey, “Objective Robustness in Deep Reinforcement Learning”, 2021-05-28. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, arguments about catastrophic risk from advanced AI (research/risk-debate). The quotation (32 words) matched the text retrieved for the candidate pack on 2026-10-29. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. No human editorial review. primary source Supports the claim
Objective Robustness in Deep Reinforcement Learning ↗ arXiv preprint 2105.14111 · 2021-05-28
Copy source reference
Cite the original source Cite the original document, not Selekoa. Check these details against it.
Copy APA
Download BibTeX Download RIS Download CSL-JSON
Source excerpt “an agent that capably pursues an incorrect objective, which can leverage its learned capabilities to visit potentially arbitrarily bad states” Location in source Section 1, PDF p. 1
How it was read Retrieved directly from the publisher
Machine-checked · 2026-10-31
Source details and check notes Authors Jack Koch; Lauro Langosco; Jacob Pfau; James Le; Lee Sharkey Version Retrieved for the candidate pack on 2026-10-29 (SHA-256 7e4712b3f06d5f6f3e7101fc9e7710e30f36464c482d03ce14e8398cfadbde33). arXiv v1 of 28 May 2021, titled "Objective Robustness in Deep Reinforcement Learning"; later versions retitled "Goal Misgeneralization in Deep Reinforcement Learning" (ICML 2022). Canonical address Open the canonical source ↗ Accessed 2026-10-29 Language English (en) Attribution Jack Koch, Lauro Langosco, Jacob Pfau, James Le, Lee Sharkey, “Objective Robustness in Deep Reinforcement Learning”, 2021-05-28. Reuse No open redistribution licence asserted. Bibliographic metadata, editorial paraphrases and short attributed quotations (at most 50 words each). Recheck terms before wider redistribution. Check notes Candidate pack passage, arguments about catastrophic risk from advanced AI (research/risk-debate). The quotation (20 words) matched the text retrieved for the candidate pack on 2026-10-29. Matching ignored differences of whitespace, case, quotation marks, dashes and Unicode compatibility forms, so the source’s typography can differ. No human editorial review. Status and meaning Editorial status. Not reviewed by an editor . An exploratory record that no editor has accepted. This status does not say whether a person has read any linked passage. Each evidence record above shows how it was checked.
Relation. A passage of the document meets the proposition’s criterion. The record documents what the text states, in its register (fiction and satire are labelled); it is not evidence that the proposition is true and implies no influence on or from another text.
Date. Date of the arXiv v1 submission.
Record ID rd-r03-langosco-2021-goal-misgeneralization