The picture of Superintelligence as having and allowing a single values systems is a Yudowsky/Bostrom construct. They go down this road because they anticipate disaster along other roads.
Meanwhile, people invariably will want things that get in the way of other people’s wants.
With or without AGI, some goods will be scarce. Government and commerce will still have to distribute these goods among people.
For example, some people will wish to have as many children or other progeny as they can afford, and AI and medical technology will make it easier for people to feed and care for more children.
There is no way to accommodate all of the people who want as many children as possible exactly when they want them.
What values scheme successfully trades off among the prerogatives of all people who want many progeny? After a point, if they persist in thinking this, the many people who share this view eventually need to compromise through some mechanism.
The child-wanters will also be forced to trade off their goals with those who hope to preserve a pristine environment as much as possible.
There is no reconciling these people’s goals completely. Maybe we can arbitrate between them and prevent outcomes which satisfy nobody. Sometimes, we can show that one or another person’s goals are internally inconsistent.
There is no obvious way to show that the child-wanter’s view is superior to the environment-preserver’s view, either. Both will occasionally find themselves in conflict with those people who personally want to live for as long as they possibly can.
Neither AGI nor “Coherent Extrapolated Volition” solves the argument among child-wanters, and it does not solve the argument between child-wanters, environment-preservers and long-livers.
Perhaps some parties could be “re-educated” or medicated out of their initial belief and find themselves just as happy or happier in the end.
Perhaps at critical moments before people have fully formulated their values, it is OK for the group to steer their value system in one direction or another? We do that with children and adults all of the time.
I anticipate that IT and AI technology will make value-shifting people and populations more and more feasible.
When is that allowable? I think we need to work that one out pretty well before we start up an AGI which is even moderately good at persuading people to change their values.
The picture of Superintelligence as having and allowing a single values systems is a Yudowsky/Bostrom construct. They go down this road because they anticipate disaster along other roads.
Meanwhile, people invariably will want things that get in the way of other people’s wants.
With or without AGI, some goods will be scarce. Government and commerce will still have to distribute these goods among people.
For example, some people will wish to have as many children or other progeny as they can afford, and AI and medical technology will make it easier for people to feed and care for more children.
There is no way to accommodate all of the people who want as many children as possible exactly when they want them.
What values scheme successfully trades off among the prerogatives of all people who want many progeny? After a point, if they persist in thinking this, the many people who share this view eventually need to compromise through some mechanism.
The child-wanters will also be forced to trade off their goals with those who hope to preserve a pristine environment as much as possible.
There is no reconciling these people’s goals completely. Maybe we can arbitrate between them and prevent outcomes which satisfy nobody. Sometimes, we can show that one or another person’s goals are internally inconsistent.
There is no obvious way to show that the child-wanter’s view is superior to the environment-preserver’s view, either. Both will occasionally find themselves in conflict with those people who personally want to live for as long as they possibly can.
Neither AGI nor “Coherent Extrapolated Volition” solves the argument among child-wanters, and it does not solve the argument between child-wanters, environment-preservers and long-livers.
Perhaps some parties could be “re-educated” or medicated out of their initial belief and find themselves just as happy or happier in the end.
Perhaps at critical moments before people have fully formulated their values, it is OK for the group to steer their value system in one direction or another? We do that with children and adults all of the time.
I anticipate that IT and AI technology will make value-shifting people and populations more and more feasible.
When is that allowable? I think we need to work that one out pretty well before we start up an AGI which is even moderately good at persuading people to change their values.