I am with you on your rejection of 1 and 2, if only because they are both framed as absolutes which ignore context.
And, yes, I do favor 3. However, you insert some extra wording that I don’t necessarily buy....
These criteria will tend to be complex, and not necessarily formulated internally in an axiomatic way.
You see, hidden in these words seems to be an understanding of how the AI is working, that might lead you to see a huge problem, and me to see something very different. I don’t know if this is really what you are thinking, but bear with me while I run with this for a moment.
Trying to formulate criteria for something, in an objective, ‘codified’ way, can sometimes be incredibly hard even when most people would say they have internal ‘judgement’ that allowed them to make a ruling very easily: the standard saw being “I cannot define what ‘pornography’ is, but I know it when I see it.” And (stepping quickly away from that example because I don’t want to get into that quagmire) there is a much more concrete example in the old interactive activation model of word recognition, which is a simple constraint system. In IAC, word recognition is remarkably robust in the face of noise, whereas attempts to write symbolic programs to deal with all the different kinds of noisy corruption of the image turn out to be horribly complex and faulty.
As you can see, I am once again pointing to the fact that Swarm Relaxation systems (understood in the very broad sense that allows all varieties of neural net to be included) can make criterial decisions seem easy, where explicit codification of the decision is a nightmare.
So, where does that lead to? Well, you go on to say:
Regardless, I fear making good choices between “shoot first” or “ask first” is hard, even FAI-complete. Screw up once, and you are in a catastrophe like in case 2.
The key phrase here is “Screw up once, and...”. In a constraint system it is impossible for one screw-up (one faulty constraint) to unbalance the whole system. That is the whole raison-d’etre of constraint systems.
Also, you say that the problem of making good choices might be FAI-complete. Now, I have some substantial quibbles with that whole “FAI-complete” idea, but in this case I will just ask a question: are you tring to say that in order to DESIGN the motivation system of the AI in such a way that it will not make one catastrophic choice between shoot-first and ask-first, we must FIRST build a FAI, because that is the only way we can get enough intelligence-horsepower applied to the problem? If so, why exactly would we need to? If the constraint system just cannot allow single failures to get out of control, we don’t need to specify every possible criterial decision in advance, we simply rely on context to do the heavy lifting, in perpetuity.
Put another way: the constraint-based AI IS the FAI already, and the reasons for thinking that it can deal with all the potentially troublesome cases have nothing to do with us anticipating every potential troublesome case, ahead of time.
--
Stepping back a moment, consider the following three kinds of case where the AI might have to make a decision.
1) An interstellar asteroid appears from nowhere, travelling at unthinkable speed, and it is going to make a direct hit on the Earth in one hour, with no possibility of survivors. The AI considers a plan in which it quietly euthanizes all life, on the grounds that any other option would lead to one hour of horror, followed by certain death.
2) The AI considers the Dopamine Drip plan.
3) The AI suddenly becomes aware that a rare, precious species of bird has become endangered and the only surviving pair is on a nature trail that is about to be filled with a gang of humans who have been planning a holiday on that trail for months. The gang is approaching the pair right now and one of the birds will die if frightened because it has a heart condition. One plan is to block the humans without explaining (until later), which will inconvenience them.
In all three cases there is a great deal of background information (constraints) that could be brought to bear, and if the AI is constraint-based, it will consider that information. People do this all the time.
In no case is there ONLY a small number of constraints (like, 2 or 3) that are relevant. Where the number of constraints is tiny, there is a chance for a “bad choice” to be made. In fact, I would argue that it is inconceivable that a decision would take place in a near-vacuum of constraints. The more significant the decision, the greater the number of constraints. The bird situation is without doubt the one that has the fewest, but it still involves a fistful of considerations. For this reason, we would expect that all major decisions—and especially the existential threat ones like 1 and 2 -- would involve a very large number of constraints indeed. It is this mass effect that is at the heart of claims that the constraint approach leads to AI that cannot get into bizarre reasoning episodes.
Finally, notice that in case 1, we are in a situation where (unlike case 2) many humans would say that there is no good decision.
I am with you on your rejection of 1 and 2, if only because they are both framed as absolutes which ignore context.
And, yes, I do favor 3. However, you insert some extra wording that I don’t necessarily buy....
You see, hidden in these words seems to be an understanding of how the AI is working, that might lead you to see a huge problem, and me to see something very different. I don’t know if this is really what you are thinking, but bear with me while I run with this for a moment.
Trying to formulate criteria for something, in an objective, ‘codified’ way, can sometimes be incredibly hard even when most people would say they have internal ‘judgement’ that allowed them to make a ruling very easily: the standard saw being “I cannot define what ‘pornography’ is, but I know it when I see it.” And (stepping quickly away from that example because I don’t want to get into that quagmire) there is a much more concrete example in the old interactive activation model of word recognition, which is a simple constraint system. In IAC, word recognition is remarkably robust in the face of noise, whereas attempts to write symbolic programs to deal with all the different kinds of noisy corruption of the image turn out to be horribly complex and faulty.
As you can see, I am once again pointing to the fact that Swarm Relaxation systems (understood in the very broad sense that allows all varieties of neural net to be included) can make criterial decisions seem easy, where explicit codification of the decision is a nightmare.
So, where does that lead to? Well, you go on to say:
The key phrase here is “Screw up once, and...”. In a constraint system it is impossible for one screw-up (one faulty constraint) to unbalance the whole system. That is the whole raison-d’etre of constraint systems.
Also, you say that the problem of making good choices might be FAI-complete. Now, I have some substantial quibbles with that whole “FAI-complete” idea, but in this case I will just ask a question: are you tring to say that in order to DESIGN the motivation system of the AI in such a way that it will not make one catastrophic choice between shoot-first and ask-first, we must FIRST build a FAI, because that is the only way we can get enough intelligence-horsepower applied to the problem? If so, why exactly would we need to? If the constraint system just cannot allow single failures to get out of control, we don’t need to specify every possible criterial decision in advance, we simply rely on context to do the heavy lifting, in perpetuity.
Put another way: the constraint-based AI IS the FAI already, and the reasons for thinking that it can deal with all the potentially troublesome cases have nothing to do with us anticipating every potential troublesome case, ahead of time.
--
Stepping back a moment, consider the following three kinds of case where the AI might have to make a decision.
1) An interstellar asteroid appears from nowhere, travelling at unthinkable speed, and it is going to make a direct hit on the Earth in one hour, with no possibility of survivors. The AI considers a plan in which it quietly euthanizes all life, on the grounds that any other option would lead to one hour of horror, followed by certain death.
2) The AI considers the Dopamine Drip plan.
3) The AI suddenly becomes aware that a rare, precious species of bird has become endangered and the only surviving pair is on a nature trail that is about to be filled with a gang of humans who have been planning a holiday on that trail for months. The gang is approaching the pair right now and one of the birds will die if frightened because it has a heart condition. One plan is to block the humans without explaining (until later), which will inconvenience them.
In all three cases there is a great deal of background information (constraints) that could be brought to bear, and if the AI is constraint-based, it will consider that information. People do this all the time.
In no case is there ONLY a small number of constraints (like, 2 or 3) that are relevant. Where the number of constraints is tiny, there is a chance for a “bad choice” to be made. In fact, I would argue that it is inconceivable that a decision would take place in a near-vacuum of constraints. The more significant the decision, the greater the number of constraints. The bird situation is without doubt the one that has the fewest, but it still involves a fistful of considerations. For this reason, we would expect that all major decisions—and especially the existential threat ones like 1 and 2 -- would involve a very large number of constraints indeed. It is this mass effect that is at the heart of claims that the constraint approach leads to AI that cannot get into bizarre reasoning episodes.
Finally, notice that in case 1, we are in a situation where (unlike case 2) many humans would say that there is no good decision.