Should/will the AI assume it’s environment is universally distributed?
Maybe the AI could also ask the hypothetical human upload for the right universal distribution to use? Such an AI probably wouldn’t work out of the box, but there could be some sort of bootstrapping scheme...
Maybe the AI could also ask the hypothetical human upload for the right universal distribution to use?
Certainly, but it’s not even clear what that distribution should be over, and whether considering probability distributions at all is the right thing to do. The initial AGI needs some operating principles, but these principles should allow fundamental correction, as the research on the inside proceeds (for which they have to be good enough initially).
Maybe the AI could also ask the hypothetical human upload for the right universal distribution to use? Such an AI probably wouldn’t work out of the box, but there could be some sort of bootstrapping scheme...
Certainly, but it’s not even clear what that distribution should be over, and whether considering probability distributions at all is the right thing to do. The initial AGI needs some operating principles, but these principles should allow fundamental correction, as the research on the inside proceeds (for which they have to be good enough initially).