I think this is at bottom a restatement of “determining the right goals with sufficient rigor to program it into an AI is hard; ensuring that these goals are stable under recursive self-modification is also hard.” If I’m right, then don’t worry; we already know it’s hard. Worry, if you like, about how to do it anyway.
In a bit more detail:
the most promising developments have been through imitating the human brain, and we have no reason to believe that the human brain (or any other brain for that matter) can be guaranteed to have a primary directive. One could argue that evolution has given us our prime directives: to ensure our own continued existence, to reproduce and to cooperate with each other; but there are many people who are suicidal, who have no interest in reproducing and who violently rebel against society (for example psychopaths).
Evolution did a bad job. Humans were never given a single primary drive; we have many. If our desires were simple, AI would be easier, but they are not. So evolution isn’t a good example here. Also, I’m not sure of your assertion that the best advances in AI so far came from mimicking the brain. The brain can tell us useful stuff as an example of various kinds of program (belief-former, decision-maker, etc.) but I don’t think we’ve been mimicking it directly. As for machine learning, yes there are pitfalls in using that to come up with the goal function, at least if you can’t look over the resulting goal function before you make it the goal of an optimizer. And making a potential superintelligence with a goal of finding [the thing you want to use as a goal function] might not be a good idea either.
I never claimed that evolution did a good job, but I would argue that it gave us a primary directive; to further the human species. All of our desires are part of our programming; they should perfectly align with desires which would optimize the primary goal, but they don’t. Simply put, mistakes were made. As the most effective way of developing optimizing programs we have seen is through machine learning, which is very similar to evolution; we should be very careful of the desires of any singleton created by this method.
I’m not sure of your assertion that the best advances in AI so far came from mimicking the brain.
Mimicking the human brain is fundamental to most AI research; on DeepMind’s website, they say that they employ computational neuroscientists and companies such as IBM are very interested in whole brain emulation.
I never claimed that evolution did a good job, but I would argue that it gave us a primary directive; to further the human species.
No, it didn’t. That’s why I linked “Adaptation Executers, not Fitness Maximizers”. Evolution didn’t even “try to” give us a primary directive; it just increased the frequency of anything that worked on the margin. But I agree that we shouldn’t rely on machine learning to find the right utility function.
Only a pantheist would claim that evolution is a personal being, and so it can’t “try to” do anything. It is, however, a directed process, serving to favor individuals that can better further the species.
But I agree that we shouldn’t rely on machine learning to find the right utility function.
How would you suggest we find the right utility function without using machine learning?
How would you suggest we find the right utility function without using machine learning?
How would you find the right utility function using machine learning? With machine learning you have to have some way of classifying examples as good vs bad. That classifier itself is equivalent to the FAI problem.
The point I am making is that machine learning, though not provably safe, is the most effective way we can imagine of making the utility function. It’s very likely that many AI’s are going to be created by this method, and if the failure rate is anywhere near as high as that for humans, this could be very serious indeed. Some misguided person may attempt to create an FAI using machine learning and then we may have the situation in the H+ article
That wasn’t what I claimed, I proposed that the current, most promising methods of producing an FAI are far too likely to produce a UFAI to be considered safe
Why do you think the whole website is obsessed with provably-friendly AI?
The whole point of MIRI is that pretty much every superintelligence that is anything other than provably safe is going to be unfriendly! This site is littered with examples of how terribly almost-friendly AI would go wrong! We don’t consider current methods “too likely” to produce a UFAI, we think they’re almost certainly going to produce UFAI! (Conditional on creating a superintelligence at all, of course).
So as much as I hate asking this question because it’s alienating, have you read the sequences?
Mimicking the human brain is fundamental to most AI research; on DeepMind’s website, they say that they employ computational neuroscientists and companies such as IBM are very interested in whole brain emulation.
Mimicking the human brain is an obscure branch of AI. Most AI projects, and certainly the successful ones you’ve heard about, are at best inspired by stripped down models of specific isolated aspects of human thought, if they take any inspiration from the human brain at all.
DeepMind for example is reinforcement learning on top of modern machine learning. Machine learning may make use of neural networks, but beware of the name: neural networks only casually resemble the biological structure from which they take their name. DeepMind doesn’t work anything like the human brain, nor does Watson, Deep Blue, or self driving cars.
Learn a bit about practical AI and neuroscience and you’d be surprise how little they have in common.
It can easily be argued that evolution did a good job, not a bad job, by not giving us a “primary directive.” The reason AI is dangerous is precisely because it might have such a directive; being an “optimizer” is precisely the reason that one fears that AI might destroy the world. So if anything, kingmaker is correct to think that since human beings are like this, it is at least theoretically possible that AI’s will be like this, and that they will not destroy the world for similar reasons.
If we had a simple primary directive, we would be fully satisfied by having a machine accomplish it for us, and it would be much easier to get a machine that would do it.
I think this is at bottom a restatement of “determining the right goals with sufficient rigor to program it into an AI is hard; ensuring that these goals are stable under recursive self-modification is also hard.” If I’m right, then don’t worry; we already know it’s hard. Worry, if you like, about how to do it anyway.
In a bit more detail:
Evolution did a bad job. Humans were never given a single primary drive; we have many. If our desires were simple, AI would be easier, but they are not. So evolution isn’t a good example here. Also, I’m not sure of your assertion that the best advances in AI so far came from mimicking the brain. The brain can tell us useful stuff as an example of various kinds of program (belief-former, decision-maker, etc.) but I don’t think we’ve been mimicking it directly. As for machine learning, yes there are pitfalls in using that to come up with the goal function, at least if you can’t look over the resulting goal function before you make it the goal of an optimizer. And making a potential superintelligence with a goal of finding [the thing you want to use as a goal function] might not be a good idea either.
I never claimed that evolution did a good job, but I would argue that it gave us a primary directive; to further the human species. All of our desires are part of our programming; they should perfectly align with desires which would optimize the primary goal, but they don’t. Simply put, mistakes were made. As the most effective way of developing optimizing programs we have seen is through machine learning, which is very similar to evolution; we should be very careful of the desires of any singleton created by this method.
Mimicking the human brain is fundamental to most AI research; on DeepMind’s website, they say that they employ computational neuroscientists and companies such as IBM are very interested in whole brain emulation.
No, it didn’t. That’s why I linked “Adaptation Executers, not Fitness Maximizers”. Evolution didn’t even “try to” give us a primary directive; it just increased the frequency of anything that worked on the margin. But I agree that we shouldn’t rely on machine learning to find the right utility function.
Only a pantheist would claim that evolution is a personal being, and so it can’t “try to” do anything. It is, however, a directed process, serving to favor individuals that can better further the species.
How would you suggest we find the right utility function without using machine learning?
How would you find the right utility function using machine learning? With machine learning you have to have some way of classifying examples as good vs bad. That classifier itself is equivalent to the FAI problem.
If I find out, you’ll be one of the first to know.
The point I am making is that machine learning, though not provably safe, is the most effective way we can imagine of making the utility function. It’s very likely that many AI’s are going to be created by this method, and if the failure rate is anywhere near as high as that for humans, this could be very serious indeed. Some misguided person may attempt to create an FAI using machine learning and then we may have the situation in the H+ article
Congratulations! You’ve figured out that UFAI is a threat!
That wasn’t what I claimed, I proposed that the current, most promising methods of producing an FAI are far too likely to produce a UFAI to be considered safe
Why do you think the whole website is obsessed with provably-friendly AI? The whole point of MIRI is that pretty much every superintelligence that is anything other than provably safe is going to be unfriendly! This site is littered with examples of how terribly almost-friendly AI would go wrong! We don’t consider current methods “too likely” to produce a UFAI, we think they’re almost certainly going to produce UFAI! (Conditional on creating a superintelligence at all, of course).
So as much as I hate asking this question because it’s alienating, have you read the sequences?
Mimicking the human brain is an obscure branch of AI. Most AI projects, and certainly the successful ones you’ve heard about, are at best inspired by stripped down models of specific isolated aspects of human thought, if they take any inspiration from the human brain at all.
DeepMind for example is reinforcement learning on top of modern machine learning. Machine learning may make use of neural networks, but beware of the name: neural networks only casually resemble the biological structure from which they take their name. DeepMind doesn’t work anything like the human brain, nor does Watson, Deep Blue, or self driving cars.
Learn a bit about practical AI and neuroscience and you’d be surprise how little they have in common.
Plus a sense of boredom. Those may add up to a good thing, since humans are unlikely to paperclip, ie focus on one thing obsessively.
It can easily be argued that evolution did a good job, not a bad job, by not giving us a “primary directive.” The reason AI is dangerous is precisely because it might have such a directive; being an “optimizer” is precisely the reason that one fears that AI might destroy the world. So if anything, kingmaker is correct to think that since human beings are like this, it is at least theoretically possible that AI’s will be like this, and that they will not destroy the world for similar reasons.
If we had a simple primary directive, we would be fully satisfied by having a machine accomplish it for us, and it would be much easier to get a machine that would do it.