I am not as convinced that there don’t exist pivotal acts that are importantly easier than directly burning all GPUs (after which I might or might not then burn most of the GPUs anyway). There’s no particular reason humans can’t perform dangerous cognition without AGI help and do some pivotal act on their own, our cognition is not exactly safe. But if I did have such an idea that I thought would work I wouldn’t write about it, and it most certainly wouldn’t be in the Overton window. Thus, I do not consider the failure of our public discourse to generate such an act to be especially strong evidence that no such act exists.
Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don’t care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI ‘this will not kill literally everyone’. Anybody telling you I’m asking for stricter ‘alignment’ than this has failed at reading comprehension.
We don’t care about trolley problems, because we’ve got that cold calculus of rationality on our side. It’s math. Infinite expected value in a post-singularity, post-scarcity transcendental world on one side of the equation, and on the other side, 99% of humans alive today. Plus, when we do pull that lever and let the aligned AGI vaporize most of humanity, we get to vaporize all of the people who didn’t agree with us about existential AI risk—it’s a 2-for-1 deal, and it was all For The Greater Good (tm).
If the reply is, “no, not killing them, but you know, forcibly imprisoning them and then implanting them with an AGI-designed chip that prevents the new thought-crime of thinking about unaligned AGI development”: oh okay my mistake I got my totalitarian dystopian futures confused.
I really wish that people stopped with this “Overton window” charade, or pretending that there’s a non-coercive way to stop people. Just say that the pivotal act involves killing a lot of people. [1]
We don’t care about trolley problems, because we’ve got that cold calculus of rationality on our side. It’s math. Infinite expected value in a post-singularity, post-scarcity transcendental world on one side of the equation, and on the other side, 99% of humans alive today. Plus, when we do pull that lever and let the aligned AGI vaporize most of humanity, we get to vaporize all of the people who didn’t agree with us about existential AI risk—it’s a 2-for-1 deal, and it was all For The Greater Good (tm).
If the reply is, “no, not killing them, but you know, forcibly imprisoning them and then implanting them with an AGI-designed chip that prevents the new thought-crime of thinking about unaligned AGI development”: oh okay my mistake I got my totalitarian dystopian futures confused.