I was a bit confused about why you didn’t mention the canonical example of a pvitotal act: “melt all GPUs in the world and then shut down”. One reason I like this example is that it it involves a task-limited AI, which should be easier to build than an open ended agent which implements something like CEV. Another reason I like that example is that it is quite concrete, and clear about how it would shift the strategic situation.
Melting all the GPUs and then shutting down doesn’t actually count, I think (and I don’t think was intended to be the original example). Then people would just build more GPUs. It’s an important part of the problem that the system continues to melt all GPUs (at least until some better situation is achieved), and that the part where the world is like “hey, holy hell, I was using those GPUs” and tries to stop the system, is somehow resolved (either by having world governments bought into the solution, or having the system be very resistant to being stopped).
(Notably, you do eventually need to be able to stop the system somehow when you do know how to build aligned AIs so you don’t lose all most of the value of the future)
I was a bit confused about why you didn’t mention the canonical example of a pvitotal act: “melt all GPUs in the world and then shut down”. One reason I like this example is that it it involves a task-limited AI, which should be easier to build than an open ended agent which implements something like CEV. Another reason I like that example is that it is quite concrete, and clear about how it would shift the strategic situation.
Melting all the GPUs and then shutting down doesn’t actually count, I think (and I don’t think was intended to be the original example). Then people would just build more GPUs. It’s an important part of the problem that the system continues to melt all GPUs (at least until some better situation is achieved), and that the part where the world is like “hey, holy hell, I was using those GPUs” and tries to stop the system, is somehow resolved (either by having world governments bought into the solution, or having the system be very resistant to being stopped).
(Notably, you do eventually need to be able to stop the system somehow when you do know how to build aligned AIs so you don’t lose all most of the value of the future)
Yeah, good point.