I suspect that it hides more assumptions about the nature of intelligence than we can necessarily make at this time.
At the present moment, we are the only general intelligences around, and we don’t seem to have terminal goals as such. As biological bodies, we are constrained by evolutionary processes, and there are many ways in which human behavior actually is reducible to offspring maximization (social status games, etc.). But it doesn’t appear to be a ‘utility function’, so much as a series of strong tendencies in the face of specific stimuli. Using novel approaches like superstimuli, it’s just as easy to make an impulse’s reproductive utility drop sharply. So we have habits constrained by evolutionary forces, but not algorithmic utility in the paper clipper sense.
There is no such thing as a general intelligence with a ‘goal’ (as Bostrom defines it). There may be at some point, but it’s not real yet. And we do have non-general intelligences with goals, that’s an easy weekend coding project. But before we declare that a GI could accept any goal regardless of its strength, we should at least check to make sure that a GI can have a goal at all.
Potential source of misunderstanding: we do have stated ‘terminal goals’, sometimes. But these goals do not function in the same way that a paperclipper utility function maximizes paperclips- there are a very weird set of obstacles, which this site generally deals with under headings like ‘akrasia’ or ‘superstimulus’. Asking a human about their ‘terminal goal’ is roughly equivalent to the question ‘what would you want, if you could want anything?’ It’s a form of emulation.
But these goals do not function in the same way that a paperclipper utility function maximizes paperclips
Sure, because humans are not utility maximizers.
The question, however, is whether terminal goals exist. A possible point of confusion is that I think of humans as having multiple, inconsistent terminal goals.
I suspect that it hides more assumptions about the nature of intelligence than we can necessarily make at this time.
At the present moment, we are the only general intelligences around, and we don’t seem to have terminal goals as such. As biological bodies, we are constrained by evolutionary processes, and there are many ways in which human behavior actually is reducible to offspring maximization (social status games, etc.). But it doesn’t appear to be a ‘utility function’, so much as a series of strong tendencies in the face of specific stimuli. Using novel approaches like superstimuli, it’s just as easy to make an impulse’s reproductive utility drop sharply. So we have habits constrained by evolutionary forces, but not algorithmic utility in the paper clipper sense.
There is no such thing as a general intelligence with a ‘goal’ (as Bostrom defines it). There may be at some point, but it’s not real yet. And we do have non-general intelligences with goals, that’s an easy weekend coding project. But before we declare that a GI could accept any goal regardless of its strength, we should at least check to make sure that a GI can have a goal at all.
Huh? Why not?
Potential source of misunderstanding: we do have stated ‘terminal goals’, sometimes. But these goals do not function in the same way that a paperclipper utility function maximizes paperclips- there are a very weird set of obstacles, which this site generally deals with under headings like ‘akrasia’ or ‘superstimulus’. Asking a human about their ‘terminal goal’ is roughly equivalent to the question ‘what would you want, if you could want anything?’ It’s a form of emulation.
Sure, because humans are not utility maximizers.
The question, however, is whether terminal goals exist. A possible point of confusion is that I think of humans as having multiple, inconsistent terminal goals.
Here’s an example of a terminal goal: to survive.