The relevant notion of intelligence for a singularity is optimization power, and it’s not obvious that we aren’t already witnessing the expansion of such an intelligence. You may have already had these thoughts, but you didn’t mention them, and I think they’re important to evaluating the strength of evidence we have against UFAI explosions:
What do agents with extreme optimization power look like? One way for them to look is a rapidly-expanding-space-and-resource-consuming process which at some point during our existence engulfs our region of space destroys us, which we haven’t seen happen yet. Another way for them to look is such an optimization process that has already engulfed our region of space. And if that were the case, we would necessarily be a byproduct of or noise term in that process, and the laws of physics actually meet that description.
(I had this thought when I first asked myself to look for naturally-occurring examples of extremely powerful optimization processes, and the laws of physics were the best examples I could think of. E.g., an agent wishing to minimize the L^2 norm of the differential of energy with respect to time would produce an environment that conserves energy.)
It’s difficult and conjunction-fallacious to come with an agent whose utility function would exactly be to have an environment that follows our particular laws of physics with our particular initial conditions, so one should not update heavily in favor of this idea based on observing the laws of physics to be true. What I’m saying is that we only have seen evidence against UFAIs that could have expanded from regions of nearby space and interrupted human progress, not UFAIs that could have expanded before and perhaps caused the existence of the observable universe.
This is similar to the Burning the Cosmic Commons paper by Robin Hanson, which considers whether the astronomical environment we observe might be the leftovers of a migrating extraterrestrial ecosystem that left a long time ago.
The relevant notion of intelligence for a singularity is optimization power, and it’s not obvious that we aren’t already witnessing the expansion of such an intelligence. You may have already had these thoughts, but you didn’t mention them, and I think they’re important to evaluating the strength of evidence we have against UFAI explosions:
What do agents with extreme optimization power look like? One way for them to look is a rapidly-expanding-space-and-resource-consuming process which at some point during our existence engulfs our region of space destroys us, which we haven’t seen happen yet. Another way for them to look is such an optimization process that has already engulfed our region of space. And if that were the case, we would necessarily be a byproduct of or noise term in that process, and the laws of physics actually meet that description.
(I had this thought when I first asked myself to look for naturally-occurring examples of extremely powerful optimization processes, and the laws of physics were the best examples I could think of. E.g., an agent wishing to minimize the L^2 norm of the differential of energy with respect to time would produce an environment that conserves energy.)
It’s difficult and conjunction-fallacious to come with an agent whose utility function would exactly be to have an environment that follows our particular laws of physics with our particular initial conditions, so one should not update heavily in favor of this idea based on observing the laws of physics to be true. What I’m saying is that we only have seen evidence against UFAIs that could have expanded from regions of nearby space and interrupted human progress, not UFAIs that could have expanded before and perhaps caused the existence of the observable universe.
This is similar to the Burning the Cosmic Commons paper by Robin Hanson, which considers whether the astronomical environment we observe might be the leftovers of a migrating extraterrestrial ecosystem that left a long time ago.