I am not that confident in the convergence properties of self-preservation as instrumental goal.
It seems that at least some goals should be pursued ballistically—i.e., by setting an appropriate course in motion so that it doesn’t need active guidance.
For example, living organisms vary widely in their commitments to self-preservations. One measure of this variety is the variety of lifespans and lifecycles. Organisms generally share the goal of reproducing, and they pursue this goal by a range of means, some of which require active guidance (like teaching your children) and some of which don’t (like releasing thousands of eggs into the ocean).
If goals are allowed to range very widely, it’s hard to believe that all final goals will counsel the adoption of the same CIVs as subgoals. The space of all final goals seems very large. I’m not even very sure what a goal is. But it seems at least plausible that this choice of CIVs is contaminated by our own (parochial) goals, and given the full range of weird possible goals these convergences only form small attractors.
A different convergence argument might start from competition among goals. A superintelligence might not “take off” unless it starts with sufficiently ambitious goals. Call a goal ambitious if its CIVs include coming to control significant resources. In that case, even if only a relatively small region in goal-space has the CIV of controlling significant resources, intelligences with those goals will quickly be overrepresented. Cf. this intriguing BBS paper I haven’t read yet.
I am not that confident in the convergence properties of self-preservation as instrumental goal.
It seems that at least some goals should be pursued ballistically—i.e., by setting an appropriate course in motion so that it doesn’t need active guidance.
For example, living organisms vary widely in their commitments to self-preservations. One measure of this variety is the variety of lifespans and lifecycles. Organisms generally share the goal of reproducing, and they pursue this goal by a range of means, some of which require active guidance (like teaching your children) and some of which don’t (like releasing thousands of eggs into the ocean).
If goals are allowed to range very widely, it’s hard to believe that all final goals will counsel the adoption of the same CIVs as subgoals. The space of all final goals seems very large. I’m not even very sure what a goal is. But it seems at least plausible that this choice of CIVs is contaminated by our own (parochial) goals, and given the full range of weird possible goals these convergences only form small attractors.
A different convergence argument might start from competition among goals. A superintelligence might not “take off” unless it starts with sufficiently ambitious goals. Call a goal ambitious if its CIVs include coming to control significant resources. In that case, even if only a relatively small region in goal-space has the CIV of controlling significant resources, intelligences with those goals will quickly be overrepresented. Cf. this intriguing BBS paper I haven’t read yet.