Indeed, I don’t want a formally-verified watertight Proof of Doom. I’d actually be a little surprised if we were doomed to mathematical standards.
I want a viral memetic Proof of Doom. A compelling argument that will convince everyone clever enough to be dangerous. The sort of thing that a marketing AI might come up with if it were constrained to only speak the truth. The sort of thing that might start a religion. A religion with a prohibition against creating God.
Indeed, I don’t want a formally-verified watertight Proof of Doom. I’d actually be a little surprised if we were doomed to mathematical standards.
I want a viral memetic Proof of Doom. A compelling argument that will convince everyone clever enough to be dangerous. The sort of thing that a marketing AI might come up with if it were constrained to only speak the truth. The sort of thing that might start a religion. A religion with a prohibition against creating God.