One of the most annoying arguments when discussing AI is the perennial “But if the AI is so smart, why won’t it figure out the right thing to do anyway?” It’s often the ultimate curiosity stopper.
How is this a curiosity stopper? It’s a good question, as is evidenced by your trying to find an answer to it.
“Smart” implicitly entails “knows the true beliefs”, whereas it doesn’t entail “has the right goals”.
It doesn’t exclude having the right goals, either. You could engineer something whose self-improvement was restricted from affecting its goals. But if that is dangerous, why would you?
Well, the difference is that building an AI without figuring out where goals come from gives you a dangerous AI, while building an AI without figuring out where beliefs come from gives you a highly-optimized compiler that wants to save humanity.
It’s a curiosity stopper in the sense that people don’t worry any more about risks from AI when they assume that intelligence correlates with doing the right thing, and that superintelligence would do the right thing all the time.
Stuart is trying to answer a different question, which is “Given that we think that’s probably false, what are some good examples that help people to see its falsity?”
How is this a curiosity stopper? It’s a good question, as is evidenced by your trying to find an answer to it.
“What should we have the AI’s goals be?”
“Eh, just make it self-improve, once it’s smart it can figure out the right goals.”
How’s that any different from:
“What should we have the AI’s beliefs be?”
“Eh, just make it self-improve, once it’s smart it can figure out the true beliefs.”
It’s not very different. They are both different from:
“The AI will acquire accurate beliefs by using a well understood epistemology to process its observations, as it is explicitly designed to do.”
“Smart” implicitly entails “knows the true beliefs”, whereas it doesn’t entail “has the right goals”.
It doesn’t exclude having the right goals, either. You could engineer something whose self-improvement was restricted from affecting its goals. But if that is dangerous, why would you?
Well, the difference is that building an AI without figuring out where goals come from gives you a dangerous AI, while building an AI without figuring out where beliefs come from gives you a highly-optimized compiler that wants to save humanity.
factual beliefs != moral beliefs
And the methods for investigating them are very different.
It’s a curiosity stopper in the sense that people don’t worry any more about risks from AI when they assume that intelligence correlates with doing the right thing, and that superintelligence would do the right thing all the time.
Stuart is trying to answer a different question, which is “Given that we think that’s probably false, what are some good examples that help people to see its falsity?”