I think this is an important subject and I agree with much of this post. However, I think the framing/perspective might be subtly but importantly wrong-or-confused.
To illustrate:
How much of the issue here is about the very singular nature of the One dominant project, vs centralization more generally into a small number of projects?
Seems to me that centralization of power per se is not the problem.
I think the problem is something more like
we want to give as much power as possible to “good” processes, e.g. a process that robustly pursues humanity’s CEV[1]; and we want to minimize the power held by “evil” processes
but: a large fraction of humans are evil, or become evil once prosocial pressures are removed; and we do not know how to reliably construct “good” AIs
and also: we (humans) are confused and in disagreement about what “good” even means
and even if it were clear what a “good goal” is, we have no reliable way of ensuring that an AI or a human institution is robustly pursuing such a goal.
I agree that (given the above conditions) concentrating power into the hands of a few humans or AIs would on expectation be (very) bad. (OTOH, a decentralized race is also very bad.) But concentration-vs-decentralization of power is just one relevant consideration among many.
Thus: if the quoted question has an implicit assumption like “the main variable to tweak is distribution-of-power”, then I think it is trying to carve the problem at unnatural joints, or making a false implicit assumption that might lead to ignoring multiple other important variables.
(And less centralization of power has serious dangers of its own. See e.g. Wei Dai’s comment.)
I think a more productive frame might be something like “how do we construct incentives, oversight, distribution of power, and other mechanisms, such that Ring Projects remain robustly aligned to ‘the greater good’?”
And maybe also “how do we become less confused about what ‘the greater good’ even is, in a way that is practically applicable to aligning Ring Projects?”
I think this is an important subject and I agree with much of this post. However, I think the framing/perspective might be subtly but importantly wrong-or-confused.
To illustrate:
Seems to me that centralization of power per se is not the problem.
I think the problem is something more like
we want to give as much power as possible to “good” processes, e.g. a process that robustly pursues humanity’s CEV[1]; and we want to minimize the power held by “evil” processes
but: a large fraction of humans are evil, or become evil once prosocial pressures are removed; and we do not know how to reliably construct “good” AIs
and also: we (humans) are confused and in disagreement about what “good” even means
and even if it were clear what a “good goal” is, we have no reliable way of ensuring that an AI or a human institution is robustly pursuing such a goal.
I agree that (given the above conditions) concentrating power into the hands of a few humans or AIs would on expectation be (very) bad. (OTOH, a decentralized race is also very bad.) But concentration-vs-decentralization of power is just one relevant consideration among many.
Thus: if the quoted question has an implicit assumption like “the main variable to tweak is distribution-of-power”, then I think it is trying to carve the problem at unnatural joints, or making a false implicit assumption that might lead to ignoring multiple other important variables.
(And less centralization of power has serious dangers of its own. See e.g. Wei Dai’s comment.)
I think a more productive frame might be something like “how do we construct incentives, oversight, distribution of power, and other mechanisms, such that Ring Projects remain robustly aligned to ‘the greater good’?”
And maybe also “how do we become less confused about what ‘the greater good’ even is, in a way that is practically applicable to aligning Ring Projects?”
If such a thing is even possible.