Maybe there are better ways to expand than through spacetime, better ways to make yourself into this sort of maximizing agent, and we are just completely unaware of them because we are comparatively dull next to the sort of AGI that has a brain the size of a planet? Some way to beat out entropy, perhaps. That’d make it inconsistent to see any sort of sky with UFAI or FAI in it.
I can somewhat imagine what these sorts of ways would be, but I have no idea if those things are likely or even feasible, since I am not a world-devouring AGI and can only do wild mass speculation at what’s beyond our current understanding of physics.
A simpler explanation could be that AGIs use stealth in pursuing their goals: the ability to camouflage oneself has always been of evolutionary import, and AGIs may find it useful to create a sky which looks like “nothing to see here” to other AGIs. (As they will likely be unfriendly toward each other) Camoflage, if good enough, would allow one to hide from predators (bigger AGI) and sneak up on prey (smaller AGI) Since we would likely be orders of magnitude worse at detecting an AGI’s camoflage, we see a sky that looks like there is nothing wrong. This doesn’t explain why we haven’t been devoured, of course, which is the weakness of the argument.
Or maybe something like acausal trade limits the expansion of AGI. If AGIs realize that fighting over resources is likely to hinder their goals more than help them in the long wrong, they might limit their expansion on the theory that there are other AGIs out there. If I think I am 1 out of a population of a billion, and I don’t want to be a target for a billion enemies at once, I might decide that taking over the entire galaxy/universe/beyond isn’t worth it. In fact, if these sorts of stand-offs become more common as the scale becomes grander, it might be motivation not to pursue such scales. The problem with this being that you would expect earlier AGIs to be more likely to just take advantage before future ones can really get to the point of being near-equals and defect on this particular dilemma. (A billion planet-eating AGI are probably not a match for one galaxy-eating AGI. So if you see a way to become the galaxy-eater before enough planet-eaters can come to the party, you go for it.)
I don’t find any of these satisfying, as one seems to require a sub-set of possibilities for unknown physics and the others seem to lean pretty heavily on the anthropic principle to explain why we, personally, are not dead yet. I see possibilities here, but none of them jump out at me as being exceptionally likely.
Maybe there are better ways to expand than through spacetime, better ways to make yourself into this sort of maximizing agent, and we are just completely unaware of them because we are comparatively dull next to the sort of AGI that has a brain the size of a planet? Some way to beat out entropy, perhaps. That’d make it inconsistent to see any sort of sky with UFAI or FAI in it.
I can somewhat imagine what these sorts of ways would be, but I have no idea if those things are likely or even feasible, since I am not a world-devouring AGI and can only do wild mass speculation at what’s beyond our current understanding of physics.
A simpler explanation could be that AGIs use stealth in pursuing their goals: the ability to camouflage oneself has always been of evolutionary import, and AGIs may find it useful to create a sky which looks like “nothing to see here” to other AGIs. (As they will likely be unfriendly toward each other) Camoflage, if good enough, would allow one to hide from predators (bigger AGI) and sneak up on prey (smaller AGI) Since we would likely be orders of magnitude worse at detecting an AGI’s camoflage, we see a sky that looks like there is nothing wrong. This doesn’t explain why we haven’t been devoured, of course, which is the weakness of the argument.
Or maybe something like acausal trade limits the expansion of AGI. If AGIs realize that fighting over resources is likely to hinder their goals more than help them in the long wrong, they might limit their expansion on the theory that there are other AGIs out there. If I think I am 1 out of a population of a billion, and I don’t want to be a target for a billion enemies at once, I might decide that taking over the entire galaxy/universe/beyond isn’t worth it. In fact, if these sorts of stand-offs become more common as the scale becomes grander, it might be motivation not to pursue such scales. The problem with this being that you would expect earlier AGIs to be more likely to just take advantage before future ones can really get to the point of being near-equals and defect on this particular dilemma. (A billion planet-eating AGI are probably not a match for one galaxy-eating AGI. So if you see a way to become the galaxy-eater before enough planet-eaters can come to the party, you go for it.)
I don’t find any of these satisfying, as one seems to require a sub-set of possibilities for unknown physics and the others seem to lean pretty heavily on the anthropic principle to explain why we, personally, are not dead yet. I see possibilities here, but none of them jump out at me as being exceptionally likely.