Yes, a full-blown expected utility maximizer, with a utility-function enclosing goals with detailed enough optimization parameters to make useful utility calculations about real-world causal relationships relative to its own self-perception. I think that before something like that becomes possible, some other less sophisticated intelligence will have already been employed as a tool to do, or solve, something that destroys pretty much everything. An AI that can solve bio or nanotech problems should be much easier to design than one that can destroy the world as a side-effect of unbounded self-improvement. And only the latter category is subject to friendliness research.
I don’t doubt that the kind of intelligence you have in mind has certain drives that cause it to take certain actions. But that’s not what I would call “basic AI drives”, rather a certain kind of sophisticated AGI design that is unlikely to come into existence as a result of ignorance or unpredictable implications of a design someone stumbled upon by luck.
Well, okay. Restricting the validity of the “AI drives” to general AIs seems reasonable. After all, the local traffic lights have not started trying to refine their goals, and PageRank has not yet started manipulating society to eliminate threats to its well-being.
I think that before something like that becomes possible, some other less sophisticated intelligence will have already been employed as a tool to do, or solve, something that destroys pretty much everything.
DOOM!
An AI that can solve bio or nanotech problems should be much easier to design than one that can destroy the world as a side-effect of unbounded self-improvement.
Probably true.
And only the latter category is subject to friendliness research.
Hmm. Possibly some research avenues are more promising than others—but this sounds a bit broad.
Yes, a full-blown expected utility maximizer, with a utility-function enclosing goals with detailed enough optimization parameters to make useful utility calculations about real-world causal relationships relative to its own self-perception. I think that before something like that becomes possible, some other less sophisticated intelligence will have already been employed as a tool to do, or solve, something that destroys pretty much everything. An AI that can solve bio or nanotech problems should be much easier to design than one that can destroy the world as a side-effect of unbounded self-improvement. And only the latter category is subject to friendliness research.
I don’t doubt that the kind of intelligence you have in mind has certain drives that cause it to take certain actions. But that’s not what I would call “basic AI drives”, rather a certain kind of sophisticated AGI design that is unlikely to come into existence as a result of ignorance or unpredictable implications of a design someone stumbled upon by luck.
Well, okay. Restricting the validity of the “AI drives” to general AIs seems reasonable. After all, the local traffic lights have not started trying to refine their goals, and PageRank has not yet started manipulating society to eliminate threats to its well-being.
So far as we know.
DOOM!
Probably true.
Hmm. Possibly some research avenues are more promising than others—but this sounds a bit broad.