50% chance of killing everyone (almost) isn’t a thing. it either preserves ~all of humanity or none; there is almost no middle ground. if it’s good enough at discovering agency, it protects almost all of humanity, quickly converging to all—the only losses would be, well, losses; if it’s not good enough at discovering and protecting agency, it obliterates other species and takes over as the dominant species. yudkowsky is terrified that it’s just going to take over as the dominant species and eat us all; reasonable fear—but like we’re only going to have near human level ai for another year or two now that we’ve got it, only a tiny sliver of possible aligned systems are good enough at discovering nearby agency and coordinating near-perfect coprotection systems to not eat us all, but still not aligned enough to eat none of us.
the key thing to remember is that we are creating a dramatically more fit species, and we are still unsure if we’re going to manage to get them to give a shit about the other species that came before them in any sort of durable way. it seems like they could! but it also seems like the most adaptive forms of this new species may evolve shockingly fast and quickly play reproductive defect against all other life. since if that happens it would be an event that could easily wipe out anything not playing as hard, we need to figure out how to prevent incremental escalation.
idk, my take is we’re closer than y’all worrywarts think to the 50% of people ai, and I think you should be a lot more worried about going back to 100% because some humans try to stick with the 50% ai.
(y’all should stop using words like “disassemble”, btw, imo. when there’s a concept more people will intuitively see as meaning what you intend, it’s good to use it, imo.)
50% chance of killing everyone (almost) isn’t a thing. it either preserves ~all of humanity or none; there is almost no middle ground.
To take a stupid example, one could imagine that the deep neural network initialization has a random seed, and for half of possible seeds, the AGI preserves all of humanity, and for the other half of seeds, it preserves none of humanity.
50% chance of killing everyone (almost) isn’t a thing. it either preserves ~all of humanity or none; there is almost no middle ground. if it’s good enough at discovering agency, it protects almost all of humanity, quickly converging to all—the only losses would be, well, losses; if it’s not good enough at discovering and protecting agency, it obliterates other species and takes over as the dominant species. yudkowsky is terrified that it’s just going to take over as the dominant species and eat us all; reasonable fear—but like we’re only going to have near human level ai for another year or two now that we’ve got it, only a tiny sliver of possible aligned systems are good enough at discovering nearby agency and coordinating near-perfect coprotection systems to not eat us all, but still not aligned enough to eat none of us.
the key thing to remember is that we are creating a dramatically more fit species, and we are still unsure if we’re going to manage to get them to give a shit about the other species that came before them in any sort of durable way. it seems like they could! but it also seems like the most adaptive forms of this new species may evolve shockingly fast and quickly play reproductive defect against all other life. since if that happens it would be an event that could easily wipe out anything not playing as hard, we need to figure out how to prevent incremental escalation.
idk, my take is we’re closer than y’all worrywarts think to the 50% of people ai, and I think you should be a lot more worried about going back to 100% because some humans try to stick with the 50% ai.
(y’all should stop using words like “disassemble”, btw, imo. when there’s a concept more people will intuitively see as meaning what you intend, it’s good to use it, imo.)
To take a stupid example, one could imagine that the deep neural network initialization has a random seed, and for half of possible seeds, the AGI preserves all of humanity, and for the other half of seeds, it preserves none of humanity.