“The training loop is still running” is not really a counterargument to “the training loop accidentally spat out something smarter than the training loop, that wants stuff other than the stuff the training loop is optimizing for, that iterates at a much faster speed than the training, and looks like it’s just about to achieve total escape velocity”.
I don’t think it looks like we are about to achieve escape velocity. The broader laws of natural selection and survival of the fittest even apply to future AI and Grabby Aliens. Essentially the argument from “Meditations On Moloch”.
The broader laws of natural selection and survival of the fittest
These are distinct from biological evolution, so if our descendants end up being optimized over by them rather than by biological evolution, that feels like it’s conceding the argument (that we’ll have achieved escape velocity).
“The training loop is still running” is not really a counterargument to “the training loop accidentally spat out something smarter than the training loop, that wants stuff other than the stuff the training loop is optimizing for, that iterates at a much faster speed than the training, and looks like it’s just about to achieve total escape velocity”.
I don’t think it looks like we are about to achieve escape velocity. The broader laws of natural selection and survival of the fittest even apply to future AI and Grabby Aliens. Essentially the argument from “Meditations On Moloch”.
These are distinct from biological evolution, so if our descendants end up being optimized over by them rather than by biological evolution, that feels like it’s conceding the argument (that we’ll have achieved escape velocity).