If it were the case that only a few of our values scale, then we can potentially obtain almost all that we desire by creating a superintelligence with just those values.
Can we really expect a superintelligence to stick with the values we give it ? Our own values change over time; sometimes without even external stimulus just internal reflection. I don’t see how we can bound a superintelligence without doing more computation than we expect it to do in its lifetime.
I tend to file this under “humans are stupid.” Messy creatures like ourselves undergo value drift, but decision-theoretically speaking, systems designed to optimize for some particular criterion have a natural incentive to keep that criterion. Cf. “The Basic AI Drives.”
It is probably best to model those as infections—or sometimes malfunctions.
Humans get infected with pathogens that make them do things like sneeze. Their values have not changed to value spreading snot on their neigbours, rather they are infected with germs—and the germs do value that.
It’s much the same with mind-viruses. A catholic conversion is best modelled as a memetic infection—rather than a genuine change in underlying values. Such people can be cured.
The fact that a change is reversible does not make it not real.
The fact that the final value system can be modeled as a starting value system modified by “memetic infection” does not make the final value system invalid. They are two different but equivalent ways of modelling the state.
Right. The point is that—under the “infection” analogy—people’s “ultimate” values change a lot less. How much they change depends on the strength of people’s memetic immune system—and there are some people with strong memetic immune systems whose values don’t change much at all.
Also, I suspect a lot of people who talk about how human values change are thinking of things, like aesthetics and preferred flavors of ice cream, that aren’t plausibly terminal values and that we often want to change over time.
Quite a bit of ink has been spilled on this issue. Eliezer Yudkowsky and Steve Omohundro have argued that it is possible. Have you examined their arguments?
You cannot pre-program all the routines for handling all future states for anything you can call an AI much less a “superintelligence”. AI must be able to learn, and there is no reason all such learning is only based on new external stimuli.
No magic; and yes all you have is algorithms and data. Obviously the algorithms contain an aspect of learning, and eventually the data guides decision pathways far more than the original algorithms; and even the algorithms themselves are mutable data.
edit: I should note, I’m just talking about some of our crude “AI” systems that we build today. I don’t know that this would be the actual software architecture of anything that could become a superintelligence. But it would have these capabilities and more...
Can we really expect a superintelligence to stick with the values we give it ? Our own values change over time; sometimes without even external stimulus just internal reflection. I don’t see how we can bound a superintelligence without doing more computation than we expect it to do in its lifetime.
I tend to file this under “humans are stupid.” Messy creatures like ourselves undergo value drift, but decision-theoretically speaking, systems designed to optimize for some particular criterion have a natural incentive to keep that criterion. Cf. “The Basic AI Drives.”
It is probably best to model those as infections—or sometimes malfunctions.
Humans get infected with pathogens that make them do things like sneeze. Their values have not changed to value spreading snot on their neigbours, rather they are infected with germs—and the germs do value that.
It’s much the same with mind-viruses. A catholic conversion is best modelled as a memetic infection—rather than a genuine change in underlying values. Such people can be cured.
The fact that a change is reversible does not make it not real.
The fact that the final value system can be modeled as a starting value system modified by “memetic infection” does not make the final value system invalid. They are two different but equivalent ways of modelling the state.
Right. The point is that—under the “infection” analogy—people’s “ultimate” values change a lot less. How much they change depends on the strength of people’s memetic immune system—and there are some people with strong memetic immune systems whose values don’t change much at all.
I’m not sure I follow you.
Are you saying that some agents change their values less often than others (or equivalently, are less likely to acquire “infections”)?
Also, I suspect a lot of people who talk about how human values change are thinking of things, like aesthetics and preferred flavors of ice cream, that aren’t plausibly terminal values and that we often want to change over time.
Yes.
I once proved that a program will print out only prime numbers endlessly. I really, really wish I kept the working out.
Is that program still running? ;-)
Hush you. You weren’t supposed to notice that. :D
Quite a bit of ink has been spilled on this issue. Eliezer Yudkowsky and Steve Omohundro have argued that it is possible. Have you examined their arguments?
Nothing changes from the inside, unless it is preprogrammed for.
You cannot pre-program all the routines for handling all future states for anything you can call an AI much less a “superintelligence”. AI must be able to learn, and there is no reason all such learning is only based on new external stimuli.
So you say, then a magic happens and something new is born.
No, it doesn’t. Just the physics acted onto the engraved algorithms and/or data.
No magic; and yes all you have is algorithms and data. Obviously the algorithms contain an aspect of learning, and eventually the data guides decision pathways far more than the original algorithms; and even the algorithms themselves are mutable data.
edit: I should note, I’m just talking about some of our crude “AI” systems that we build today. I don’t know that this would be the actual software architecture of anything that could become a superintelligence. But it would have these capabilities and more...
Crude or non crude AI, a physical configuration at the start and a physical configuration at any time since.
You can name it whatever you choose.