I’ve heard much about the problems of misaligned superhuman AI killing us all but the long view seems to imply that even a “well aligned” AI will prioritise inhuman instrumental goals.
I’m not quite understanding yet. Are you saying that an immortal AGI will prioritize preparing to fight an alien AGI, to the point that it won’t get anything else done? Or what?
Immortal expanding AGI is a part of classic alignment thinking, and we do assume it would either go to war or negotiate with an alien AGI if it encounters one, depending on the overlap in their alignment/goals.
Yes. It will prioritise the future over the present. The utility of all humans being destroyed by an alien AI in the future is 0. The utility of populating the future light cone is very, very large and most of that utility is in the far future. Therefore the AI should sacrifice almost everything in the near term light cone to prevent the 0 outcome. If it could digitise all humans or possibly just have a gene bank then it can still fill most of the future light cone with happy humans once all possible threats have red-shifted out of reach. Living humans are small but non-zero risk to the master plan and hence should be dispensed with.
I’m not clear on what you’re calling the “problem of superhuman AI”?
I’ve heard much about the problems of misaligned superhuman AI killing us all but the long view seems to imply that even a “well aligned” AI will prioritise inhuman instrumental goals.
I’m not quite understanding yet. Are you saying that an immortal AGI will prioritize preparing to fight an alien AGI, to the point that it won’t get anything else done? Or what?
Immortal expanding AGI is a part of classic alignment thinking, and we do assume it would either go to war or negotiate with an alien AGI if it encounters one, depending on the overlap in their alignment/goals.
Yes. It will prioritise the future over the present.
The utility of all humans being destroyed by an alien AI in the future is 0.
The utility of populating the future light cone is very, very large and most of that utility is in the far future.
Therefore the AI should sacrifice almost everything in the near term light cone to prevent the 0 outcome. If it could digitise all humans or possibly just have a gene bank then it can still fill most of the future light cone with happy humans once all possible threats have red-shifted out of reach. Living humans are small but non-zero risk to the master plan and hence should be dispensed with.