My argument is that faithful exact brain uploads are guaranteed to not help unless you had already solved AI safety anyhow. I do think we can simply solve ai extinction risk anyhow, but it requires us to not only prevent AI that does not follow orders, but also prevent AI from “just following orders” to do things that some humans value but which abuse others. if we fall too far into the latter attractor—which we are at immediate risk of doing, well before stably self-reflective AGI ever happens—we become guaranteed to shortly go extinct as corporations are increasingly just an ai and a human driver. eventually the strongest corporations are abusing larger and larger portions of humanity with one human at the helm. then one day ai can drive the entire economy...
it’s pretty much just the slower version of yudkowsky’s concerns. I think he’s wrong to think self-distillation will be this quick snap-down onto the manifold of high quality hypotheses, but other than that I think he’s on point. and because of that, I think the incremental behavior of the market is likely to pull us into a defection-only-game-theory hole as society’s capabilities melt in the face of increased heat and chaos at various scales of the world.
My argument is that faithful exact brain uploads are guaranteed to not help unless you had already solved AI safety anyhow. I do think we can simply solve ai extinction risk anyhow, but it requires us to not only prevent AI that does not follow orders, but also prevent AI from “just following orders” to do things that some humans value but which abuse others. if we fall too far into the latter attractor—which we are at immediate risk of doing, well before stably self-reflective AGI ever happens—we become guaranteed to shortly go extinct as corporations are increasingly just an ai and a human driver. eventually the strongest corporations are abusing larger and larger portions of humanity with one human at the helm. then one day ai can drive the entire economy...
it’s pretty much just the slower version of yudkowsky’s concerns. I think he’s wrong to think self-distillation will be this quick snap-down onto the manifold of high quality hypotheses, but other than that I think he’s on point. and because of that, I think the incremental behavior of the market is likely to pull us into a defection-only-game-theory hole as society’s capabilities melt in the face of increased heat and chaos at various scales of the world.