It would recreate humanity. I imagine that if the UFAI does enough damage that the FAI can’t remember what humanity was, it would be enough damage to lobotomize any AI and defeat it easily.
I guess that depends on the level of capability that an AGI had at that point. It supposes the level of knowledge had was enough for recreation, which may be much higher than what’s required for mere protection. It’s hard to perceive as that level of capability is way beyond our own. I’ll certainly give it some thought.
It also depends on how you define “human”. I’d hope the FAI is willing to upload us instead of wasting vast amounts of resources just so we’re instantiated in a physical universe instead of a virtual one.
It’s worth noting that the AI only has to be advanced enough to store the information. Once it’s beaten the UFAI, it has plenty of time to build up the resources and intelligence necessary to rebuild humanity.
I personally imagine that AGI will arrive well before its possible to store a full down-to-the-subatomic-level map of a person in a space that’s any smaller than the person. “Just store the humans and bring them back” implies such a massive storage requirement that its basically not much different from making a full copy of them anyway, so I wonder if such a massive storage device wouldn’t be equally vulnerable to attack.
I’m also keen to see us continue as a biological species, even if we also run simulated brains or people in parallel. Ideally I can see us doing both if we can establish a FAI. The best bet I can see so far is to make sure a FAI arrives first :-)
You don’t need accurate down to the subatomic level. You just need a human. The same human would be nice, since it means the FAI managed to keep all of those people from dying, but unless it’s programmed to only value currently alive people, that’s not a big deal.
Also, you make it sound like you’re saying we won’t develop that storage capability until well after we develop the AGI. It’s the AGI that will be developing technology. What we can do just before we make it is not a good indicator of what it can do.
Because neural pathways and other structures of the brain are pretty small, I think you’d need an extremely high resolution. However, I guess what you’re saying is that a breeding population would be enough to at least keep the species going, so I acknowledge that. Still, I’m hoping we can make something that does something in addition to that.
Your second point depends on how small the AGI can make reliable storage tech I guess.
In the perhaps this whole point is moot because its unlikely an intelligence explosion will take long enough for there to be time for other researchers to construct an alternative AGI.
Still, I’m hoping we can make something that does something in addition to that.
Their children will be fine. You don’t even need a breeding population. You just need to know how to make an egg, a sperm, and an artificial uterus.
In the perhaps this whole point is moot because its unlikely an intelligence explosion will take long enough for there to be time for other researchers to construct an alternative AGI.
It might encounter another AGI as it spreads, although I don’t think this point will matter much in the ensuing war (or treaty, if they decide on that).
It would recreate humanity. I imagine that if the UFAI does enough damage that the FAI can’t remember what humanity was, it would be enough damage to lobotomize any AI and defeat it easily.
I guess that depends on the level of capability that an AGI had at that point. It supposes the level of knowledge had was enough for recreation, which may be much higher than what’s required for mere protection. It’s hard to perceive as that level of capability is way beyond our own. I’ll certainly give it some thought.
It also depends on how you define “human”. I’d hope the FAI is willing to upload us instead of wasting vast amounts of resources just so we’re instantiated in a physical universe instead of a virtual one.
It’s worth noting that the AI only has to be advanced enough to store the information. Once it’s beaten the UFAI, it has plenty of time to build up the resources and intelligence necessary to rebuild humanity.
I personally imagine that AGI will arrive well before its possible to store a full down-to-the-subatomic-level map of a person in a space that’s any smaller than the person. “Just store the humans and bring them back” implies such a massive storage requirement that its basically not much different from making a full copy of them anyway, so I wonder if such a massive storage device wouldn’t be equally vulnerable to attack.
I’m also keen to see us continue as a biological species, even if we also run simulated brains or people in parallel. Ideally I can see us doing both if we can establish a FAI. The best bet I can see so far is to make sure a FAI arrives first :-)
You don’t need accurate down to the subatomic level. You just need a human. The same human would be nice, since it means the FAI managed to keep all of those people from dying, but unless it’s programmed to only value currently alive people, that’s not a big deal.
Also, you make it sound like you’re saying we won’t develop that storage capability until well after we develop the AGI. It’s the AGI that will be developing technology. What we can do just before we make it is not a good indicator of what it can do.
Because neural pathways and other structures of the brain are pretty small, I think you’d need an extremely high resolution. However, I guess what you’re saying is that a breeding population would be enough to at least keep the species going, so I acknowledge that. Still, I’m hoping we can make something that does something in addition to that.
Your second point depends on how small the AGI can make reliable storage tech I guess.
In the perhaps this whole point is moot because its unlikely an intelligence explosion will take long enough for there to be time for other researchers to construct an alternative AGI.
Their children will be fine. You don’t even need a breeding population. You just need to know how to make an egg, a sperm, and an artificial uterus.
It might encounter another AGI as it spreads, although I don’t think this point will matter much in the ensuing war (or treaty, if they decide on that).