Build a single AI and give it ultimate power, or build a stable ecosystem / balance of power between AIs?
Try to pass on specific values of ours, or try to ensure that life continues operating under parameters that produce some beings that have values something like that?
Each of these decisions suggests research questions.
1a. How can we extend our models of competition to hierarchical agents—agents that are composed of other agents? Is most of the competition at the top level, or at the lower levels? (For starters, is there some natural distribution of number of agents of different sizes / levels / timescales, like there is for cities of different sizes?) The purpose is to ask whether we can maintain useful competition within a singleton.
1b. For some set of competing hierarchical AIs, what circumstances make it more likely for one to conquer and subsume the others? Under what circumstances might a singleton AI split up into multiple AIs? The purpose is to estimate whether it’s possible to indefinitely avoid permanent collapse into a singleton.
2a. Try to find a candidate set of human values. Find how each is implemented neurally. The purpose is to see whether such things exist, what sorts of things they are, and whether they’re the sort of things that can be implemented in a logic.
2b. List the behaviors of a wide variety of animals. Find values/preferences/behaviors of interest, and for each, find the conditions that tend to lead animals to have / not have those behaviors, as I did for boredom in this comment. The purpose is to see what fraction of the space of behaviors is acceptable to us, and to discover the evolutionary conditions that lead to that fraction of that space. That will give us an idea of how tightly we can constrain future values by controlling the gross parameters of the ecosystem.
I think you’ll need to explain that because I don’t see that at all. We’ve made life a lot better for most people on this planet by creating power-sharing arrangements that limit any single person’s autocratic powers, and expanding franchise to all. Yet I see many people here advocating basically a return to autocratic rule by our AI overlords, with no vote for the humans left behind. Essentially, “let’s build a provably beneficial dictator!” This boggles my mind.
The alternative is to decentralize transhumanist technology and push as many people a possible through an augmentation pathway in lockstep, preserving our democratic power structures. This sidesteps the friendly AI problem entirely.
Essentially, “let’s build a provably beneficial dictator!” This boggles my mind.
Agreed, though I’m probably boggled for different reasons.
Eventually, the software will develop to the point where the human brain will be only a tiny portion of it. Or somebody will create an AI not attached to a human. The body we know will be left behind or marginalized. There’s a whole universe out there, the vast majority of it uninhabitable by humans.
I think the two most-important decisions are:
Build a single AI and give it ultimate power, or build a stable ecosystem / balance of power between AIs?
Try to pass on specific values of ours, or try to ensure that life continues operating under parameters that produce some beings that have values something like that?
Each of these decisions suggests research questions.
1a. How can we extend our models of competition to hierarchical agents—agents that are composed of other agents? Is most of the competition at the top level, or at the lower levels? (For starters, is there some natural distribution of number of agents of different sizes / levels / timescales, like there is for cities of different sizes?) The purpose is to ask whether we can maintain useful competition within a singleton.
1b. For some set of competing hierarchical AIs, what circumstances make it more likely for one to conquer and subsume the others? Under what circumstances might a singleton AI split up into multiple AIs? The purpose is to estimate whether it’s possible to indefinitely avoid permanent collapse into a singleton.
2a. Try to find a candidate set of human values. Find how each is implemented neurally. The purpose is to see whether such things exist, what sorts of things they are, and whether they’re the sort of things that can be implemented in a logic.
2b. List the behaviors of a wide variety of animals. Find values/preferences/behaviors of interest, and for each, find the conditions that tend to lead animals to have / not have those behaviors, as I did for boredom in this comment. The purpose is to see what fraction of the space of behaviors is acceptable to us, and to discover the evolutionary conditions that lead to that fraction of that space. That will give us an idea of how tightly we can constrain future values by controlling the gross parameters of the ecosystem.
Or 3) Don’t pass control to AIs at all. Don’t even build agent-y AIs. Augment humans instead.
This may be a good way to start, but it eventually leads to the same place.
I think you’ll need to explain that because I don’t see that at all. We’ve made life a lot better for most people on this planet by creating power-sharing arrangements that limit any single person’s autocratic powers, and expanding franchise to all. Yet I see many people here advocating basically a return to autocratic rule by our AI overlords, with no vote for the humans left behind. Essentially, “let’s build a provably beneficial dictator!” This boggles my mind.
The alternative is to decentralize transhumanist technology and push as many people a possible through an augmentation pathway in lockstep, preserving our democratic power structures. This sidesteps the friendly AI problem entirely.
Agreed, though I’m probably boggled for different reasons.
Eventually, the software will develop to the point where the human brain will be only a tiny portion of it. Or somebody will create an AI not attached to a human. The body we know will be left behind or marginalized. There’s a whole universe out there, the vast majority of it uninhabitable by humans.
“The software”? What software? The “software” is the human, in an augmented human. I’m not sure whatever distinction you’re drawing here is relevant.
Presumably ‘the software’ is the software that was not part of the original human.