[i]That seems highly inaccurate to me. AIs will more closely approximate rational utilitarian agents than current organisms—so the expected utility maximisation framework will become a better predictor of behaviour as time passes.[/i]
The AI that I think humanity is likely to produce first will be a mass of hacks that work, that also hacks itself in a manner that works. There will be masses of legacy code, that it finds hard to get rid of, much as humans find ideas we have relied upon for reasoning for a long time hard to get rid of, if we can at all.
This isn’t based on the fact that I think that we should build human like machines. But that only those can win in the the real world. There is no neat clean way of specifying a utility maximizer that eventually always wins, without infinite computing resources and supposing the computation done has no affect on the outside world. So we and other intelligent agents have to take mental short cuts, guess, make mistakes, get stuck in psychological cul-de-sacs. While AIs might up the number of ideas they play with to avoid those traps, it would be a trade off with looking at the links between ideas more thoroughly. For example you could devote more memory and processing time to finding cross correlations between inputs 1 − 1 million and the acquisition of utility, and looking at inputs 2million to 4 million as well. Either could be the right thing to do, so another hack is needed to decide which is done.
Unless you decide to rigorously prove which is the right thing to do, but then you are using up precious processing time and resources doing that. In short I see hacks everywhere in the future, especially towards the beginning, unless you can untangle the recursive knot caused by asking the question, “How much resources should I use, deciding how much resources I should use”.
[i]Obviously, the utility function of AIs will not be to produce paper clips.[/i]
And obviously, I was referring to the single minded, focussed utility maximizer that Eliezer often uses in his discussions about AI.
[i]That seems highly inaccurate to me. AIs will more closely approximate rational utilitarian agents than current organisms—so the expected utility maximisation framework will become a better predictor of behaviour as time passes.[/i]
The AI that I think humanity is likely to produce first will be a mass of hacks that work, that also hacks itself in a manner that works. There will be masses of legacy code, that it finds hard to get rid of, much as humans find ideas we have relied upon for reasoning for a long time hard to get rid of, if we can at all.
This isn’t based on the fact that I think that we should build human like machines. But that only those can win in the the real world. There is no neat clean way of specifying a utility maximizer that eventually always wins, without infinite computing resources and supposing the computation done has no affect on the outside world. So we and other intelligent agents have to take mental short cuts, guess, make mistakes, get stuck in psychological cul-de-sacs. While AIs might up the number of ideas they play with to avoid those traps, it would be a trade off with looking at the links between ideas more thoroughly. For example you could devote more memory and processing time to finding cross correlations between inputs 1 − 1 million and the acquisition of utility, and looking at inputs 2million to 4 million as well. Either could be the right thing to do, so another hack is needed to decide which is done.
Unless you decide to rigorously prove which is the right thing to do, but then you are using up precious processing time and resources doing that. In short I see hacks everywhere in the future, especially towards the beginning, unless you can untangle the recursive knot caused by asking the question, “How much resources should I use, deciding how much resources I should use”.
[i]Obviously, the utility function of AIs will not be to produce paper clips.[/i]
And obviously, I was referring to the single minded, focussed utility maximizer that Eliezer often uses in his discussions about AI.