Minor (?) correction: You’ve mentioned multiple times that our ASI will wipe out all value in the universe, but that’s very unlikely to happen. We won’t be the only (or the first) civilization to have created ASI, so eventually our ASI will run into other rogue/aligned ASIs and be forced to negotiate.
I think it’s more of a correction than a misunderstanding. It shouldn’t be assumed that “value” just means human civilization and its potential. Most people reading this post will assume “wiping out all value” to mean wiping out all that we value, not just wiping out humanity. But this is clearly not true, as most people value life and sentience in general, so a universe where all alien civs also end up dying due to our ASI is far worse than the one where there are survivors.
Sure; though what I imagine is more “Human ASI destroys all human value and spreads until it hits defended borders of alien ASI that has also destroyed all alien value...”
(Though I don’t think this is the case. The sun is still there, so I doubt alien ASI exists. The universe isn’t that young.)
I’m not sure if I’m in agreement with him, but it’s worth noting that Eliezer has stated on the podcast that he thinks that some (a good number of?) alien civilizations could develop AGI without going extinct. My understanding of his argument is that alien civilizations would be sufficiently biologically different from us to have ways around the problem that we do not possess.
From skimming this post it seems to me that this is probably also what @So8res thinks.
Right, but if you’re an alien civilization trying to be evil, you probably spread forever; if you’re trying to be nice, you also spread forever, but if you find a potentially life-bearing planet, you simulate it out (obviating the need for ancestor sims later). Or some such strategy. The point is there shouldn’t ever be a border facing nothing.
Minor (?) correction: You’ve mentioned multiple times that our ASI will wipe out all value in the universe, but that’s very unlikely to happen. We won’t be the only (or the first) civilization to have created ASI, so eventually our ASI will run into other rogue/aligned ASIs and be forced to negotiate.
Relevant EY tweets: https://twitter.com/ESYudkowsky/status/1558974831269273600
I believe this is a misunderstanding: ASI will wipe out all human value in the universe.
I think it’s more of a correction than a misunderstanding. It shouldn’t be assumed that “value” just means human civilization and its potential. Most people reading this post will assume “wiping out all value” to mean wiping out all that we value, not just wiping out humanity. But this is clearly not true, as most people value life and sentience in general, so a universe where all alien civs also end up dying due to our ASI is far worse than the one where there are survivors.
Sure; though what I imagine is more “Human ASI destroys all human value and spreads until it hits defended borders of alien ASI that has also destroyed all alien value...”
(Though I don’t think this is the case. The sun is still there, so I doubt alien ASI exists. The universe isn’t that young.)
https://grabbyaliens.com/
I’m not sure if I’m in agreement with him, but it’s worth noting that Eliezer has stated on the podcast that he thinks that some (a good number of?) alien civilizations could develop AGI without going extinct. My understanding of his argument is that alien civilizations would be sufficiently biologically different from us to have ways around the problem that we do not possess.
From skimming this post it seems to me that this is probably also what @So8res thinks.
Right, but if you’re an alien civilization trying to be evil, you probably spread forever; if you’re trying to be nice, you also spread forever, but if you find a potentially life-bearing planet, you simulate it out (obviating the need for ancestor sims later). Or some such strategy. The point is there shouldn’t ever be a border facing nothing.