XiXiDu, I get the impression you’ve never coded anything. Is that accurate?
Present-day software is better than previous software generations at understanding and doing what humans mean.
Increasing the intelligence of Google Maps will enable it to satisfy human intentions by parsing less specific commands.
Present-day everyday software (e.g. Google Maps, Siri) is better at doing what humans mean. It is not better at understanding humans. Learning programs like the one that runs PARO appear to be good at understanding humans, but are actually following a very simple utility function (in the decision sense, not the experiental sense); they change their behaviour in response to programmed cues, generally by doing more/less of actions associated with those cues (example: PARO “likes” being stroked and will do more of things that tend to preceed stroking). In each case of a program that improves itself, it has a simple thing it “wants” to optimise and makes changes according to how well it seems to be doing.
Making software that understands humans at all is beyond our current capabilities. Theory of mind, the ability to recognise agents and see them as having desires of their own, is something we have no idea how to produce; we don’t even know how humans have it. General intelligence is an enormous step beyond programming something like Siri. Siri is “just” interpreting vocal commands as text (which requires no general intelligence), matching that to a list of question structures (which requires no general intelligence; Siri does not have to understand what the word “where” means to know that Google Maps may be useful for that type of question) and delegating to Web services, with a layer of learning code to produce more of the results you liked (i.e., that made you stop asking related questions) in the past. Siri is using a very small built-in amount of knowledge and an even smaller amount of learned knowledge to fake understanding, but it’s just pattern-matching. While the second step is the root of general intelligence, it’s almost all provided by humans who understood that “where” means a question is probably to do with geography; Siri’s ability to improve this step is virtually nonexistent.
catastrophically worse than all previous generations at doing what humans mean
The more powerful something is, the more dangerous it is. A very stupid adult is much more dangerous than a very intelligent child because adults are allowed to drive cars. Driving a car requires very little intelligence and no general intelligence whatsoever (we already have robots that can do a pretty good job), but can go catastrophically wrong very easily. Holding an intelligent conversation requires huge amounts of specialised intelligence and often requires general intelligence, but nothing a four-year-old says is likely to kill people.
It’s much easier to make a program that does a good job at task-completion, and is therefore given considerable power and autonomy (Siri, for example), than it is to make sure that the program never does stupid things with its power. Developing software we already have could easily lead to programs being assigned large amounts of power (e.g., “Siri 2, buy me a ticket to New York”, which would almost always produce the appropriate kind of ticket), but I certainly wouldn’t trust such programs to never make colossal screw-ups. (Siri 2 will only tell you that you can’t afford a ticket if a human programmer thought that might be important, because Siri 2 does not care that you need to buy groceries, because it does not understand that you exist.)
I hope I have convinced you that present software only fakes understanding and that developing it will not produce software that can do better than an intelligent human with the same resources. Siri 2 will not be more than a very useful tool, and neither will Siri 5. Software does not stop caring because it has never cared.
It is very easy (relatively speaking) to produce code that can fake understanding and act like it cares about your objectives, because this merely requires a good outline of the sort of things the code is likely to be wanted for. (This is the second stage of Siri outlined above, where Siri refers to a list saying that “where” means that Google Maps is probably the best service to outsource to.) Making code that does more of the things that get good results is also very easy.
Making code that actually cares requires outlining exactly what the code is really and truly wanted to do. You can’t delegate this step by saying “Learn what I care about and then satisfy me” because that’s just changing what you want the code to do. It might or might not be easier than saying “This is what I care about, satisfy me”, but at some stage you have to say what you want done exactly right or the code will do something else. (Currently getting it wrong is pretty safe because computers have little autonomy and very little general intelligence, so they mostly do nothing much; getting it wrong with a UFAI is dangerous because the AI will succeed at doing the wrong thing, probably on a big scale.) This is the only kind of code you can trust to program itself and to have significant power, because it’s the only kind that will modify itself right.
You can’t progress Siri into an FAI, no matter how much you know about producing general intelligence. You need to know either Meaning-in-General, Preferences-in-General or exactly what Human Prefernces are, or you won’t get what you hoped for.
Another perspective: the number of humans in history who were friendly is very, very small. The number of humans who are something resembling capital-F Friendly is virtually nil. Why should “an AI created by humans to care” be Friendly, or even friendly? Unless friendliness or Friendliness is your specific goal, you’ll probably produce software that is friendly-to-the-maker (or maybe Friendly-to-the-maker, if making Friendly code really is as easy as you seem to think). Who would you trust with a superintelligence that did exactly what they said? Who would you trust with a superintelligence that did exactly what they really wanted, not what they said? I wouldn’t trust my mother with either, and she’s certainly highly intelligent and has my best interests at heart. I’d need a fair amount of convincing to trust me with either. Most humans couldn’t program AIs that care because most humans don’t care themselves, let alone know how to express it.
Making software that understands humans at all is beyond our current capabilities.
So you believe that “understanding” is an all or nothing capability? I did never intend to use “understanding” like this. My use of the term is such that if my speech recognition software correctly transcribes 98% of what I am saying then it is better at understanding how certain sounds are related to certain strings of characters than a software that correctly transcribes 95% of what I said.
General intelligence is an enormous step beyond programming something like Siri.
One enormous step or a huge number of steps? If the former, what makes you think so? If the latter, then at what point do better versions of Siri start acting in catastrophic ways?
Siri is using a very small built-in amount of knowledge and an even smaller amount of learned knowledge to fake understanding, but it’s just pattern-matching. While the second step is the root of general intelligence, it’s almost all provided by humans who understood that “where” means a question is probably to do with geography;
Most of what humans understand is provided by other humans who themselves got another cruder version from other humans.
It’s much easier to make a program that does a good job at task-completion, and is therefore given considerable power and autonomy (Siri, for example), than it is to make sure that the program never does stupid things with its power.
If an AI is not supposed to take over the world, then from the perspective of humans it is mistaken to take over the world. Humans got something wrong about the AI design if it takes over the world. Now if needs to solve a minimum of N problems correctly in order to take over the world, then this means that it succeeded N times at being general intelligent at executing a stupid thing. The question that arises here is whether it is more likely for humans to build an AI that works perfectly well along a number of dimensions at doing a stupid thing than an AI that fails at doing a stupid thing because it does other stupid things as well?
Developing software we already have could easily lead to programs being assigned large amounts of power (e.g., “Siri 2, buy me a ticket to New York”, which would almost always produce the appropriate kind of ticket), but I certainly wouldn’t trust such programs to never make colossal screw-ups.
Sure, I do not disagree with this at all. AI will very likely lead to catastrophic events. I merely disagree with the dumb superintelligence scenario.
...getting it wrong with a UFAI is dangerous because the AI will succeed at doing the wrong thing, probably on a big scale.
In other words, humans are likely to fail at AI in such a way that it works perfectly well in a catastrophic way.
Another perspective: the number of humans in history who were friendly is very, very small.
I certainly do not reject that general AI is extremely dangerous in the hands of unfriendly humans and that only a friendly AI that takes over the world could eventually prevent a catastrophe. I am rejecting the dumb superintelligence scenario.
XiXiDu, I get the impression you’ve never coded anything. Is that accurate?
Present-day everyday software (e.g. Google Maps, Siri) is better at doing what humans mean. It is not better at understanding humans. Learning programs like the one that runs PARO appear to be good at understanding humans, but are actually following a very simple utility function (in the decision sense, not the experiental sense); they change their behaviour in response to programmed cues, generally by doing more/less of actions associated with those cues (example: PARO “likes” being stroked and will do more of things that tend to preceed stroking). In each case of a program that improves itself, it has a simple thing it “wants” to optimise and makes changes according to how well it seems to be doing.
Making software that understands humans at all is beyond our current capabilities. Theory of mind, the ability to recognise agents and see them as having desires of their own, is something we have no idea how to produce; we don’t even know how humans have it. General intelligence is an enormous step beyond programming something like Siri. Siri is “just” interpreting vocal commands as text (which requires no general intelligence), matching that to a list of question structures (which requires no general intelligence; Siri does not have to understand what the word “where” means to know that Google Maps may be useful for that type of question) and delegating to Web services, with a layer of learning code to produce more of the results you liked (i.e., that made you stop asking related questions) in the past. Siri is using a very small built-in amount of knowledge and an even smaller amount of learned knowledge to fake understanding, but it’s just pattern-matching. While the second step is the root of general intelligence, it’s almost all provided by humans who understood that “where” means a question is probably to do with geography; Siri’s ability to improve this step is virtually nonexistent.
The more powerful something is, the more dangerous it is. A very stupid adult is much more dangerous than a very intelligent child because adults are allowed to drive cars. Driving a car requires very little intelligence and no general intelligence whatsoever (we already have robots that can do a pretty good job), but can go catastrophically wrong very easily. Holding an intelligent conversation requires huge amounts of specialised intelligence and often requires general intelligence, but nothing a four-year-old says is likely to kill people.
It’s much easier to make a program that does a good job at task-completion, and is therefore given considerable power and autonomy (Siri, for example), than it is to make sure that the program never does stupid things with its power. Developing software we already have could easily lead to programs being assigned large amounts of power (e.g., “Siri 2, buy me a ticket to New York”, which would almost always produce the appropriate kind of ticket), but I certainly wouldn’t trust such programs to never make colossal screw-ups. (Siri 2 will only tell you that you can’t afford a ticket if a human programmer thought that might be important, because Siri 2 does not care that you need to buy groceries, because it does not understand that you exist.)
I hope I have convinced you that present software only fakes understanding and that developing it will not produce software that can do better than an intelligent human with the same resources. Siri 2 will not be more than a very useful tool, and neither will Siri 5. Software does not stop caring because it has never cared.
It is very easy (relatively speaking) to produce code that can fake understanding and act like it cares about your objectives, because this merely requires a good outline of the sort of things the code is likely to be wanted for. (This is the second stage of Siri outlined above, where Siri refers to a list saying that “where” means that Google Maps is probably the best service to outsource to.) Making code that does more of the things that get good results is also very easy.
Making code that actually cares requires outlining exactly what the code is really and truly wanted to do. You can’t delegate this step by saying “Learn what I care about and then satisfy me” because that’s just changing what you want the code to do. It might or might not be easier than saying “This is what I care about, satisfy me”, but at some stage you have to say what you want done exactly right or the code will do something else. (Currently getting it wrong is pretty safe because computers have little autonomy and very little general intelligence, so they mostly do nothing much; getting it wrong with a UFAI is dangerous because the AI will succeed at doing the wrong thing, probably on a big scale.) This is the only kind of code you can trust to program itself and to have significant power, because it’s the only kind that will modify itself right.
You can’t progress Siri into an FAI, no matter how much you know about producing general intelligence. You need to know either Meaning-in-General, Preferences-in-General or exactly what Human Prefernces are, or you won’t get what you hoped for.
Another perspective: the number of humans in history who were friendly is very, very small. The number of humans who are something resembling capital-F Friendly is virtually nil. Why should “an AI created by humans to care” be Friendly, or even friendly? Unless friendliness or Friendliness is your specific goal, you’ll probably produce software that is friendly-to-the-maker (or maybe Friendly-to-the-maker, if making Friendly code really is as easy as you seem to think). Who would you trust with a superintelligence that did exactly what they said? Who would you trust with a superintelligence that did exactly what they really wanted, not what they said? I wouldn’t trust my mother with either, and she’s certainly highly intelligent and has my best interests at heart. I’d need a fair amount of convincing to trust me with either. Most humans couldn’t program AIs that care because most humans don’t care themselves, let alone know how to express it.
So you believe that “understanding” is an all or nothing capability? I did never intend to use “understanding” like this. My use of the term is such that if my speech recognition software correctly transcribes 98% of what I am saying then it is better at understanding how certain sounds are related to certain strings of characters than a software that correctly transcribes 95% of what I said.
One enormous step or a huge number of steps? If the former, what makes you think so? If the latter, then at what point do better versions of Siri start acting in catastrophic ways?
Most of what humans understand is provided by other humans who themselves got another cruder version from other humans.
If an AI is not supposed to take over the world, then from the perspective of humans it is mistaken to take over the world. Humans got something wrong about the AI design if it takes over the world. Now if needs to solve a minimum of N problems correctly in order to take over the world, then this means that it succeeded N times at being general intelligent at executing a stupid thing. The question that arises here is whether it is more likely for humans to build an AI that works perfectly well along a number of dimensions at doing a stupid thing than an AI that fails at doing a stupid thing because it does other stupid things as well?
Sure, I do not disagree with this at all. AI will very likely lead to catastrophic events. I merely disagree with the dumb superintelligence scenario.
In other words, humans are likely to fail at AI in such a way that it works perfectly well in a catastrophic way.
I certainly do not reject that general AI is extremely dangerous in the hands of unfriendly humans and that only a friendly AI that takes over the world could eventually prevent a catastrophe. I am rejecting the dumb superintelligence scenario.