The goal is not to “make an AI friendly” (non-lethal), it’s to make a Friendly AI. That is, not to make some powerful agent that doesn’t kill you (and does something useful), but make an agent that can be trusted with autonomously building the future. For example, a merely non-lethal AI won’t help with preventing UFAI risks.
So it’s possible that some kind of Oracle AI can be built, but so what? And the risk of unknown unknowns remains, so it’s probably a bad idea even if it looks provably safe.
With FAI, you have a commensurate reason to take the risk.
Sure, but if the Oracle AI is used as a stepping stone towards FAI, then you also have a reason to take the risk.
I guess you could argue that the risk of Oracle + Friendly AI is higher than just going straight for FAI, but you can’t be sure how much the FAI risk could be mitigated by the Oracle AI (or any other type of not-so-powerful / constrained / narrow-domain AI). At least it doesn’t seem obvious to me.
To the extent you should expect it to be useful. It’s not clear in what way it can even in principle help with specifying morality. (See also this thread.)
Assume you have a working halting oracle. Now what? (Actually you could get inside to have infinite time to think about the problem.)
I think he means Oracle as in general powerful question-answer, not as in a halting oracle. A halting oracle could be used to answer many mathematical questions (like the aforementioned Riemann Hypothesis) though.
I know he doesn’t mean a halting oracle. A halting oracle is a well-specified superpower that can do more than real Oracles. The thought experiment I described considers an upper bound on usefulness of Oracles.
I figure we will build experts and forecasters before both oracles and full machine intelligence. That will be good—since forecasters will help to give us foresight—which we badly need.
Generally speaking, replacing the brain’s functions one-at-a-time seems more desirable than replacing them all-at-once. It is likely to result in a more gradual shift, and a smoother transfer—with a reduced chance of the baton getting dropped during the switch over.
If we get a working Oracle AI, couldn’t we just ask it how to build an FAI. I just don’t think this is of much use since the Oracle route doesn’t really seem much easier than the FAI route.
I just don’t think this is of much use since the Oracle route doesn’t really seem much easier than the FAI route.
Experts and general forecasters are easier to build than general intelligent agents—or so I argue in my section on Machine Forecasting Implications. That is before we even get to constraints on how we want them to behave.
If we get a working Oracle AI, couldn’t we just ask it how to build an FAI.
At a given tech level, if you trying to use an use an general oracle on its own to create a general intelligence would probably produce a less intelligent agent than could be produced by other means, using a broader set of tools. An oracle might well be able to help, though.
(1) There is a way to make an AI that is useful and provably not-unfriendly
(2) This requires a subset of the breakthroughs required for a true FAI
(3) It can be used to provide extra leverage towards building a FAI (i.e. using it to generate prestige and funds for hiring and training the best brains available. How? Start by solving protein folding or something.)
Then this safe & useful AI should certainly be a milestone on the way towards FAI.
Just barely possible, but any such system is also a recipe for destroying the universe, if mixed in slightly different proportions. Which on the net makes the plan wrong (destroy-the-universe wrong).
The goal is not to “make an AI friendly” (non-lethal), it’s to make a Friendly AI. That is, not to make some powerful agent that doesn’t kill you (and does something useful), but make an agent that can be trusted with autonomously building the future. For example, a merely non-lethal AI won’t help with preventing UFAI risks.
So it’s possible that some kind of Oracle AI can be built, but so what? And the risk of unknown unknowns remains, so it’s probably a bad idea even if it looks provably safe.
Doesn’t this also apply to provably friendly Friendly AI? Perhaps even more so, given that it is a project of higher complexity.
With FAI, you have a commensurate reason to take the risk.
Sure, but if the Oracle AI is used as a stepping stone towards FAI, then you also have a reason to take the risk.
I guess you could argue that the risk of Oracle + Friendly AI is higher than just going straight for FAI, but you can’t be sure how much the FAI risk could be mitigated by the Oracle AI (or any other type of not-so-powerful / constrained / narrow-domain AI). At least it doesn’t seem obvious to me.
To the extent you should expect it to be useful. It’s not clear in what way it can even in principle help with specifying morality. (See also this thread.)
Assume you have a working halting oracle. Now what? (Actually you could get inside to have infinite time to think about the problem.)
I think he means Oracle as in general powerful question-answer, not as in a halting oracle. A halting oracle could be used to answer many mathematical questions (like the aforementioned Riemann Hypothesis) though.
I know he doesn’t mean a halting oracle. A halting oracle is a well-specified superpower that can do more than real Oracles. The thought experiment I described considers an upper bound on usefulness of Oracles.
I figure we will build experts and forecasters before both oracles and full machine intelligence. That will be good—since forecasters will help to give us foresight—which we badly need.
Generally speaking, replacing the brain’s functions one-at-a-time seems more desirable than replacing them all-at-once. It is likely to result in a more gradual shift, and a smoother transfer—with a reduced chance of the baton getting dropped during the switch over.
If we get a working Oracle AI, couldn’t we just ask it how to build an FAI. I just don’t think this is of much use since the Oracle route doesn’t really seem much easier than the FAI route.
No, it won’t know what you mean. Even you don’t know what you mean, which is part of the problem.
Experts and general forecasters are easier to build than general intelligent agents—or so I argue in my section on Machine Forecasting Implications. That is before we even get to constraints on how we want them to behave.
At a given tech level, if you trying to use an use an general oracle on its own to create a general intelligence would probably produce a less intelligent agent than could be produced by other means, using a broader set of tools. An oracle might well be able to help, though.
If:
(1) There is a way to make an AI that is useful and provably not-unfriendly
(2) This requires a subset of the breakthroughs required for a true FAI
(3) It can be used to provide extra leverage towards building a FAI (i.e. using it to generate prestige and funds for hiring and training the best brains available. How? Start by solving protein folding or something.)
Then this safe & useful AI should certainly be a milestone on the way towards FAI.
Just barely possible, but any such system is also a recipe for destroying the universe, if mixed in slightly different proportions. Which on the net makes the plan wrong (destroy-the-universe wrong).
I just don’t think that this assertion has been adequately backed up.