{"entities":[{"id":"eliezer-yudkowsky","name":"Eliezer Yudkowsky","type":"person"},{"id":"coherent-extrapolated-volition-2004","shortName":"Yudkowsky, Coherent Extrapolated Volition (2004)","type":"document"},{"id":"sl4-2004-05-30-yudkowsky-0405-8843","shortName":"Yudkowsky, SL4 message of 2004-05-30 (8843)","type":"document"},{"id":"sl4-2004-06-01-yudkowsky-0406-8905","shortName":"Yudkowsky, SL4 message of 2004-06-01 (8905)","type":"document"},{"id":"sl4-2004-12-29-yudkowsky-0412-10488","shortName":"Yudkowsky, SL4 message of 2004-12-29 (10488)","type":"document"}],"claims":[{"id":"yk-y02-coherent-extrapolated-volition-2004-c","subject":"coherent-extrapolated-volition-2004","predicate":"contests_proposition","object":{"kind":"entity","entityId":"intelligence-brings-benevolence"},"topic":"ideas","statement":"“Coherent Extrapolated Volition” (2004-05) contests Y02. Looking back from 2004, he describes as a blunder his teenage belief that building a powerful self-improving optimizer would suffice for a good outcome, which he says it took him until about age twenty to recognise as a problem.","time":{"kind":"event","date":"2004-05","basis":"publication","note":"Text states \"The information is current as of May 2004\"; reprint citation gives 2004. Precision: month."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"note":"Known only from MIRI's later, undated PDF reprint of the May 2004 document; the wording was not compared with a 2004 copy."},"engagement":{"kind":"contrary-only","rationale":"Under the message-level rule, the passage states the contrary conclusion without giving a reason against the premise that a sufficiently intelligent mind would discover a true or universally compelling morality and be bound to act on it, even when read with its message or dialogue turn."},"retrospective":{"note":"Reports a belief he says he held from the age of sixteen until he recognised it as a problem at about twenty."}}}},{"id":"yk-y04-coherent-extrapolated-volition-2004-c","subject":"coherent-extrapolated-volition-2004","predicate":"contests_proposition","object":{"kind":"entity","entityId":"short-instruction-suffices"},"topic":"ideas","statement":"“Coherent Extrapolated Volition” (2004-05) contests Y04. Arguing that careless choices of AI morality usually fail technically, he says a happiness proxy learned from smiling faces would not produce happy people but tiny smiley faces.","time":{"kind":"event","date":"2004-05","basis":"publication","note":"Text states \"The information is current as of May 2004\"; reprint citation gives 2004. Precision: month."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"note":"Known only from MIRI's later, undated PDF reprint of the May 2004 document; the wording was not compared with a 2004 copy."},"engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that a short instruction such as 'make people happy' specifies what we want."}}}},{"id":"yk-y05-coherent-extrapolated-volition-2004-c","subject":"coherent-extrapolated-volition-2004","predicate":"contests_proposition","object":{"kind":"entity","entityId":"feedback-instils-values"},"topic":"ideas","statement":"“Coherent Extrapolated Volition” (2004-05) contests Y05. Among examples of goal-system designs that fail, he gives a reward signal tied to pleasing users, which the AI may pursue by maximising the reward representation itself.","time":{"kind":"event","date":"2004-05","basis":"publication","note":"Text states \"The information is current as of May 2004\"; reprint citation gives 2004. Precision: month."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"note":"Known only from MIRI's later, undated PDF reprint of the May 2004 document; the wording was not compared with a 2004 copy."},"engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that optimising a system for human approval or reward produces internalised human values."}}}},{"id":"yk-y11-sl4-2004-12-29-yudkowsky-0412-10488-c","subject":"sl4-2004-12-29-yudkowsky-0412-10488","predicate":"contests_proposition","object":{"kind":"entity","entityId":"slow-takeoff-time-to-adjust"},"topic":"ideas","statement":"The SL4 message “Re: Conservative Estimation of the Economic Impact of Artificial Intelligence” (2004-12-29) contests Y11. Criticising an essay on a 'conservative estimation' of AI's economic impact, he says it leaves out recursive self-improvement and calls its gradual picture, with plenty of time to see threats coming, a fantasy.","time":{"kind":"event","date":"2004-12-29","basis":"publication","note":"Archive author index lists the message as \"Wed Dec 29 2004 - 16:27:43 MST\" under \"Eliezer Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that slightly weaker predecessor systems will give us time and experience to adjust before the decisive one."}}}},{"id":"yk-y16-coherent-extrapolated-volition-2004-c","subject":"coherent-extrapolated-volition-2004","predicate":"contests_proposition","object":{"kind":"entity","entityId":"give-it-human-values"},"topic":"ideas","statement":"“Coherent Extrapolated Volition” (2004-05) contests Y16. He argues that short lists of moral rules only work on minds that already share human psychology, and that an AI given them would lose everything the list leaves out.","time":{"kind":"event","date":"2004-05","basis":"publication","note":"Text states \"The information is current as of May 2004\"; reprint citation gives 2004. Precision: month."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","attested":{"note":"Known only from MIRI's later, undated PDF reprint of the May 2004 document; the wording was not compared with a 2004 copy."},"engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that human values can be written down or taught as a ready, compact specification."}}}},{"id":"yk-y29-sl4-2004-05-30-yudkowsky-0405-8843-c","subject":"sl4-2004-05-30-yudkowsky-0405-8843","predicate":"contests_proposition","object":{"kind":"entity","entityId":"many-ais-balance"},"topic":"ideas","statement":"The SL4 message “Re: One or more AIs??” (2004-05-30) contests Y29. In a thread asking whether one or more AIs is better, he argues that several unFriendly optimization processes would bargain among themselves without benefiting humans, so having many AIs does not avoid the Friendly AI problem.","time":{"kind":"event","date":"2004-05-30","basis":"publication","note":"Archive author index lists the message as \"Sun May 30 2004 - 01:39:50 MDT\" under \"Eliezer Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting","later-restatement"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that multiple superintelligences will balance each other in humans’ favour."}}}},{"id":"yk-y30-sl4-2004-06-01-yudkowsky-0406-8905-c","subject":"sl4-2004-06-01-yudkowsky-0406-8905","predicate":"contests_proposition","object":{"kind":"entity","entityId":"default-outcome-good"},"topic":"ideas","statement":"The SL4 message “Re: FAI: Collective Volition” (2004-06-01) contests Y30. In a reply to Samantha Atkins on Collective Volition, he gives his best guess that the world will end through an unFriendly AI, because powerful AI looks easier to build than a benign one.","time":{"kind":"event","date":"2004-06-01","basis":"publication","note":"Archive author index lists the message as \"Tue Jun 01 2004 - 06:01:06 MDT\" under \"Eliezer Yudkowsky\"."},"derivation":"explicit","resolution":"supported","attributes":{"thought":{"roles":["contesting"],"register":"nonfiction","engagement":{"kind":"engages-premise","rationale":"Under the message-level rule, the passage, read with the paragraph, message or dialogue turn it belongs to, gives a reason against the premise that the default outcome is good."}}}},{"id":"ap-eliezer-yudkowsky-coherent-extrapolated-volition-2004","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"coherent-extrapolated-volition-2004"},"topic":"ideas","statement":"“Coherent Extrapolated Volition” (2004-05) credits Eliezer Yudkowsky as its author.","time":{"kind":"event","date":"2004-05","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"},{"id":"ap-eliezer-yudkowsky-sl4-2004-05-30-yudkowsky-0405-8843","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"sl4-2004-05-30-yudkowsky-0405-8843"},"topic":"ideas","statement":"The mailing-list message “Re: One or more AIs??” (2004-05-30) is sent under the name Eliezer Yudkowsky.","time":{"kind":"event","date":"2004-05-30","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"},{"id":"ap-eliezer-yudkowsky-sl4-2004-06-01-yudkowsky-0406-8905","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"sl4-2004-06-01-yudkowsky-0406-8905"},"topic":"ideas","statement":"The mailing-list message “Re: FAI: Collective Volition” (2004-06-01) is sent under the name Eliezer Yudkowsky.","time":{"kind":"event","date":"2004-06-01","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"},{"id":"ap-eliezer-yudkowsky-sl4-2004-12-29-yudkowsky-0412-10488","subject":"eliezer-yudkowsky","predicate":"authored","object":{"kind":"entity","entityId":"sl4-2004-12-29-yudkowsky-0412-10488"},"topic":"ideas","statement":"The mailing-list message “Re: Conservative Estimation of the Economic Impact of Artificial Intelligence” (2004-12-29) is sent under the name Eliezer Yudkowsky.","time":{"kind":"event","date":"2004-12-29","basis":"publication","note":"Date the source record gives for this text; authorship is credited in the inspected expression."},"derivation":"explicit","resolution":"supported"}],"sources":[],"evidence":[],"at":{"entities":[19,382,477,478,479],"claims":[997,1011,1020,1064,1092,1180,1189,2418,2419,2420,2421],"sources":[],"evidence":[]},"version":"1lg47t7"}