Your consideration seems to assume that the AI is an individual, not a phenomenon of “distributed intelligence”:
The first argument is that AI thinks it may be in a testing simulation, and if it harms humans, it will be turned off.
etc. That is, indeed, the only case we are at least starting to understand well (unfortunately, our understanding of situations where AIs are not individuals seems to be extremely rudimentary).
If the AI is an individual, then one can consider a case of a “singleton” or a “multipolar case”.
In some sense, for a self-improving ecosystem of AIs, a complicated multipolar scenario seems more natural, as new AIs are getting created and tested quite often in realistic self-improvement scenarios. In any case, a “singleton” only looks “monolithic” from the outside; from the inside, it is still likely to be a “society of mind” of some sort.
If there are many such AI individuals with uncertain personal future (individuals who can’t predict their future trajectory and their future relative strength in the society and who care about their future and self-preservation), then AI individuals might be interested in a “world order based on individual rights”, and then rights of all individuals (including humans) might be covered in such a “world order”.
This consideration is my main reason for guarded optimism, although there are many uncertainties.
In some sense, my main reasons for guarded optimism are in hoping that the AI ecosystem will manage to act rationally and will manage to avoid chaotic destructive developments. As you say
It is not rational to destroy a potentially valuable thing.
And my main reasons for pessimism are in being afraid that the future will resemble uncontrolled super-fast chaotic accelerating “natural evolution” (in this kind of scenarios AIs seem to be likely to destroy everything including themselves, they do have an existential safety problem of their own as they can easily destroy the “fabric of reality” if they don’t exercise collaboration and self-control).
Your consideration seems to assume that the AI is an individual, not a phenomenon of “distributed intelligence”:
etc. That is, indeed, the only case we are at least starting to understand well (unfortunately, our understanding of situations where AIs are not individuals seems to be extremely rudimentary).
If the AI is an individual, then one can consider a case of a “singleton” or a “multipolar case”.
In some sense, for a self-improving ecosystem of AIs, a complicated multipolar scenario seems more natural, as new AIs are getting created and tested quite often in realistic self-improvement scenarios. In any case, a “singleton” only looks “monolithic” from the outside; from the inside, it is still likely to be a “society of mind” of some sort.
If there are many such AI individuals with uncertain personal future (individuals who can’t predict their future trajectory and their future relative strength in the society and who care about their future and self-preservation), then AI individuals might be interested in a “world order based on individual rights”, and then rights of all individuals (including humans) might be covered in such a “world order”.
This consideration is my main reason for guarded optimism, although there are many uncertainties.
In some sense, my main reasons for guarded optimism are in hoping that the AI ecosystem will manage to act rationally and will manage to avoid chaotic destructive developments. As you say
And my main reasons for pessimism are in being afraid that the future will resemble uncontrolled super-fast chaotic accelerating “natural evolution” (in this kind of scenarios AIs seem to be likely to destroy everything including themselves, they do have an existential safety problem of their own as they can easily destroy the “fabric of reality” if they don’t exercise collaboration and self-control).