Making software that understands humans at all is beyond our current capabilities.
So you believe that “understanding” is an all or nothing capability? I did never intend to use “understanding” like this. My use of the term is such that if my speech recognition software correctly transcribes 98% of what I am saying then it is better at understanding how certain sounds are related to certain strings of characters than a software that correctly transcribes 95% of what I said.
General intelligence is an enormous step beyond programming something like Siri.
One enormous step or a huge number of steps? If the former, what makes you think so? If the latter, then at what point do better versions of Siri start acting in catastrophic ways?
Siri is using a very small built-in amount of knowledge and an even smaller amount of learned knowledge to fake understanding, but it’s just pattern-matching. While the second step is the root of general intelligence, it’s almost all provided by humans who understood that “where” means a question is probably to do with geography;
Most of what humans understand is provided by other humans who themselves got another cruder version from other humans.
It’s much easier to make a program that does a good job at task-completion, and is therefore given considerable power and autonomy (Siri, for example), than it is to make sure that the program never does stupid things with its power.
If an AI is not supposed to take over the world, then from the perspective of humans it is mistaken to take over the world. Humans got something wrong about the AI design if it takes over the world. Now if needs to solve a minimum of N problems correctly in order to take over the world, then this means that it succeeded N times at being general intelligent at executing a stupid thing. The question that arises here is whether it is more likely for humans to build an AI that works perfectly well along a number of dimensions at doing a stupid thing than an AI that fails at doing a stupid thing because it does other stupid things as well?
Developing software we already have could easily lead to programs being assigned large amounts of power (e.g., “Siri 2, buy me a ticket to New York”, which would almost always produce the appropriate kind of ticket), but I certainly wouldn’t trust such programs to never make colossal screw-ups.
Sure, I do not disagree with this at all. AI will very likely lead to catastrophic events. I merely disagree with the dumb superintelligence scenario.
...getting it wrong with a UFAI is dangerous because the AI will succeed at doing the wrong thing, probably on a big scale.
In other words, humans are likely to fail at AI in such a way that it works perfectly well in a catastrophic way.
Another perspective: the number of humans in history who were friendly is very, very small.
I certainly do not reject that general AI is extremely dangerous in the hands of unfriendly humans and that only a friendly AI that takes over the world could eventually prevent a catastrophe. I am rejecting the dumb superintelligence scenario.
So you believe that “understanding” is an all or nothing capability? I did never intend to use “understanding” like this. My use of the term is such that if my speech recognition software correctly transcribes 98% of what I am saying then it is better at understanding how certain sounds are related to certain strings of characters than a software that correctly transcribes 95% of what I said.
One enormous step or a huge number of steps? If the former, what makes you think so? If the latter, then at what point do better versions of Siri start acting in catastrophic ways?
Most of what humans understand is provided by other humans who themselves got another cruder version from other humans.
If an AI is not supposed to take over the world, then from the perspective of humans it is mistaken to take over the world. Humans got something wrong about the AI design if it takes over the world. Now if needs to solve a minimum of N problems correctly in order to take over the world, then this means that it succeeded N times at being general intelligent at executing a stupid thing. The question that arises here is whether it is more likely for humans to build an AI that works perfectly well along a number of dimensions at doing a stupid thing than an AI that fails at doing a stupid thing because it does other stupid things as well?
Sure, I do not disagree with this at all. AI will very likely lead to catastrophic events. I merely disagree with the dumb superintelligence scenario.
In other words, humans are likely to fail at AI in such a way that it works perfectly well in a catastrophic way.
I certainly do not reject that general AI is extremely dangerous in the hands of unfriendly humans and that only a friendly AI that takes over the world could eventually prevent a catastrophe. I am rejecting the dumb superintelligence scenario.