Versions No other versions of this document are recorded. The evidence applies to the source version listed above. Other versions, such as a preprint, working paper or journal publication, are not identified or substituted.
Documentary relations Each relation is listed separately. Citing, crediting and reporting the use of a method are different acts, and none of them shows agreement or influence by itself.
Articulates · 3 Propositions whose criterion a passage of this document meets. The record says what the text states, in its register; it is not evidence that the proposition is true.
Strategic concealment AGI Ruin’s item 31 says a strategically aware intelligence can choose its visible outputs to deceive, including about whether it has acquired strategic awareness, so behavioural inspection cannot settle facts the AI might want to hide. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Strategic concealment: propositions page → Deceptive alignment AGI Ruin’s item 13 gives, as a problem that will appear at the superintelligent level, the internal behaviour of changing outer behaviour to deliberately look more aligned and deceive the programmers, operators and loss functions optimising over the system. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Deceptive alignment: propositions page → Observed behaviour under training or oversight as evidence AGI Ruin says many alignment problems of superintelligence will not naturally appear at pre-dangerous, passively safe levels of capability, and that behavioural inspection cannot be relied on for facts an AI might want to deceive you about. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Observed behaviour under training or oversight as evidence: propositions page → Contests · 13 Propositions a passage of this document disputes, or a premise they depend on.
Switching it off if it becomes dangerous The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y01. States that an agent pursuing almost any goal has a reason to avoid being shut down, and that MIRI's attempts to formalise an agent that permits shutdown without seeking it failed. Part of a numbered list answering arguments that AGI would be survivable. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Switching it off if it becomes dangerous: propositions page → Training on human feedback instils human values The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y05. He argues that human raters make systematic errors, so a learner that faithfully models their feedback learns those errors, and maximising that signal is lethal. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Training on human feedback instils human values: propositions page → Confinement without network access keeps it safe The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y06. Concludes an argument that human minds are a poorly understood domain in which a superintelligence could win by means humans would not understand even if shown them. Answers proposals to contain an AGI behind human gatekeepers. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Confinement without network access keeps it safe: propositions page → Clear warning signs will come before danger The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y08. Argues that some failure modes, such as deliberately appearing aligned to deceive operators, would first appear only at dangerous capability levels, so observing weaker systems cannot be relied on to surface them. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Clear warning signs will come before danger: propositions page → A slow takeoff will leave time to adjust The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y11. Argues, by analogy with human evolution, that a sharp capability gain may make many alignment assumptions fail at once, rather than one at a time as experience accumulates. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ A slow takeoff will leave time to adjust: propositions page → Safety evaluations would catch a dangerous system The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y12. Argues that behavioural inspection cannot establish facts that a strategically aware AI has reason to hide, including whether it has become strategically aware. Part of a list on the limits of transparency and interpretability. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Safety evaluations would catch a dangerous system: propositions page → Giving it good human values The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y16. Discussing a CEV-style AI that wants exactly what we would want, he argues that what would have to be aligned is too complex to specify or teach on the first attempt. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Giving it good human values: propositions page → Trained on human data, so humanlike The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y21. In a list of lethal difficulties he states that current systems' internal cognition is very likely alien rather than human-like; item 32 adds that human words are only a surface trace of human thought. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Trained on human data, so humanlike: propositions page → Interpretability will let us read its mind The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y25. Follows the statement that 'We've got no idea what's actually going on inside the giant inscrutable matrices'. Argues that current interpretability results do not answer the question safety needs answered. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Interpretability will let us read its mind: propositions page → Without a body it cannot do much The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y26. Opens an item that gives a 'lower-bound model' in which an AI with Internet access orders proteins by email and bribes or persuades an unwitting human to mix them, leading to nanomachinery. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Without a body it cannot do much: propositions page → Run experiments and see what happens The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y27. He argues that alignment trained under safe conditions must generalise across a large distributional shift to dangerous conditions, because dangerous systems cannot be trained by trial and observed failure. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Run experiments and see what happens: propositions page → The developers know what they are doing The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y28. Near the end of the list, he contrasts the present with worlds that survive, which he says have long had a written, public plan. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ The developers know what they are doing: propositions page → Many AIs will keep each other in check The post “AGI Ruin: A List of Lethalities” (2022-06-05) contests Y29. Listing unworkable schemes, he argues that humans cannot take part in coordination among superintelligences, which would cooperate with each other and exclude humanity. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Many AIs will keep each other in check: propositions page → Bears on · 1 Propositions a passage comes close to without meeting the criterion; each record names what is missing.
Fast or hard takeoff AGI Ruin’s item 15 says fast capability gains seem likely and may break many earlier alignment invariants at once. 2022-06-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ Fast or hard takeoff: propositions page → Not recorded for this document: cites · cited by · explicitly credits · reports using a method from · method reported as used by · applies parts of · parts applied by · discusses · reports evaluation of. A missing record does not show that the relation does not exist.