“The possibility of sharp jumps in intelligence also implies a higher standard for Friendly AI techniques. The technique cannot assume the programmers’ ability to monitor the AI against its will, rewrite the AI against its will, bring to bear the threat of superior military force; nor may the algorithm assume that the programmers control a «reward button» which a smarter AI could wrest from the programmers; et cetera. Indeed no one should be making these assumptions to begin with.”
“The possibility of sharp jumps in intelligence also implies a higher standard for Friendly AI techniques. The technique cannot assume the programmers’ ability to monitor the AI against its will, rewrite the AI against its will, bring to bear the threat of superior military force; nor may the algorithm assume that the programmers control a «reward button» which a smarter AI could wrest from the programmers; et cetera. Indeed no one should be making these assumptions to begin with.”
EY, AI as a pos neg factor, 2006ish