This is a surprisingly good point against “AI moratorium” of even 5 minutes. Because if Eliezers beliefs are correct, and AGIs are inherently uncontrollable and will seek to malevolently optimize the entire universe, killing their creators for 0.0000000000000000001 percent more usable matter, where is everyone? Why do we exist at all?
Maybe Eliezer is wrong and AI systems saturate much sooner than he thinks.
Your suggestion that the AI would only get 1e-21 more usable matter by eliminating humans made me think about orders of magnitude a bit. According to the World Economic Forum humans have made (hence presumably used) around 1.1e15kg of matter. That’s around 2e-10 of the Earth’s mass of 5.9e24kg. Now you could argue that what should be counted is the mass that can eventually be used by a super optimizer, but then we’d have to go into the weeds of how long the system would be slowed down by trying to keep humanity alive, figuring out what is needed for that, etc.
Right plus think on a solar system or galaxy level sale.
Now consider that properly keeping humans alive—in a way actually competent not the scam life support humans offer now—involves separating their brain from their body and keeping it alive and in perfect help essentially forever using nanotechnology to replace all other organ functions etc. The human would experience a world via VR or remote surrogates.
This would cost like 10 kg of matter a human with plausible limit level tech. They can’t breed so it’s 80 billion times 10 kg....
This is a surprisingly good point against “AI moratorium” of even 5 minutes. Because if Eliezers beliefs are correct, and AGIs are inherently uncontrollable and will seek to malevolently optimize the entire universe, killing their creators for 0.0000000000000000001 percent more usable matter, where is everyone? Why do we exist at all?
Maybe Eliezer is wrong and AI systems saturate much sooner than he thinks.
Your suggestion that the AI would only get 1e-21 more usable matter by eliminating humans made me think about orders of magnitude a bit. According to the World Economic Forum humans have made (hence presumably used) around 1.1e15kg of matter. That’s around 2e-10 of the Earth’s mass of 5.9e24kg. Now you could argue that what should be counted is the mass that can eventually be used by a super optimizer, but then we’d have to go into the weeds of how long the system would be slowed down by trying to keep humanity alive, figuring out what is needed for that, etc.
Right plus think on a solar system or galaxy level sale.
Now consider that properly keeping humans alive—in a way actually competent not the scam life support humans offer now—involves separating their brain from their body and keeping it alive and in perfect help essentially forever using nanotechnology to replace all other organ functions etc. The human would experience a world via VR or remote surrogates.
This would cost like 10 kg of matter a human with plausible limit level tech. They can’t breed so it’s 80 billion times 10 kg....