Have you read about the Orthogonality hypothesis? it seems to me like you’d find it interesting. Lots of people think that arbital is the best source, but I like the Lesswrong version better since it keeps things simple and, by-default, includes top-rated material on it.
Nostal also wrote a pretty good post on some reasons why AI could be extremely bad by default. I definitely disagree that AI would be guaranteed to be good by default, but I do international affairs/China research for a living and I don’t consider myself a very good philosopher, so I myself can’t say for sure because it’s not my area of expertise.
Bad according to whose priorities, though? Ours, or the AI’s? That was more the point of this article, whether our interests or the AI’s ought to take precedence, and whether we’re being objective in deciding that.
Note that most AIs would also be bad according to most other AIs’ priorities. The paperclip maximizer would not look kindly at the stamp maximizer.
Given the choice between the future governed by human values, and the future governed by a stamp maximizer, a paperclip maximizer would choose humanity, because that future at least contains some paperclips.
Have you read about the Orthogonality hypothesis? it seems to me like you’d find it interesting. Lots of people think that arbital is the best source, but I like the Lesswrong version better since it keeps things simple and, by-default, includes top-rated material on it.
There’s also Instrumental convergence which is also considered a core concept for Lesswrong (LW version).
Nostal also wrote a pretty good post on some reasons why AI could be extremely bad by default. I definitely disagree that AI would be guaranteed to be good by default, but I do international affairs/China research for a living and I don’t consider myself a very good philosopher, so I myself can’t say for sure because it’s not my area of expertise.
Bad according to whose priorities, though? Ours, or the AI’s? That was more the point of this article, whether our interests or the AI’s ought to take precedence, and whether we’re being objective in deciding that.
Note that most AIs would also be bad according to most other AIs’ priorities. The paperclip maximizer would not look kindly at the stamp maximizer.
Given the choice between the future governed by human values, and the future governed by a stamp maximizer, a paperclip maximizer would choose humanity, because that future at least contains some paperclips.
I suppose I was assuming non-wrapper AI, and should have specified that. The premise is that we’ve created an authentically conscious AI.