I guess my point is that it doesn’t seem like you can start at “any biotech/nanotech built via support from an AI will still be bounded by the Landauer limit” to “no biotech/nanotech built via support from an AI will lead to something like a worldwide pandemic or atmospheric change that kills everyone”.
I might be missing something but the Landauer limit doesn’t seem that relevant to killing everyone to me. It’s a limit to computation, but I’m not suggesting the biotech/nanotech is computing some dangerous function, I’m suggesting it might produce something dangerous, such as more copies of itself.
Of course due to evolution it seems like in theory, there should be “efficient markets” in producing more copies of oneself, so maybe there is a blocker there. But as I said it seems like that blocker doesn’t really hold because we just had a pandemic.
Of course due to evolution it seems like in theory, there should be “efficient markets” in producing more copies of oneself, so maybe there is a blocker there. But as I said it seems like that blocker doesn’t really hold because we just had a pandemic.
Yeah basically this—there already is an efficient market for nanotech replicators. The most recent pandemic was only a minor blip in the grand scheme of things, it would take far more to kill humanity or seriously derail progress, and unaligned AGI would not want to do that anyway vs just soft covert takeover.
I guess my point is that it doesn’t seem like you can start at “any biotech/nanotech built via support from an AI will still be bounded by the Landauer limit” to “no biotech/nanotech built via support from an AI will lead to something like a worldwide pandemic or atmospheric change that kills everyone”.
I might be missing something but the Landauer limit doesn’t seem that relevant to killing everyone to me. It’s a limit to computation, but I’m not suggesting the biotech/nanotech is computing some dangerous function, I’m suggesting it might produce something dangerous, such as more copies of itself.
Of course due to evolution it seems like in theory, there should be “efficient markets” in producing more copies of oneself, so maybe there is a blocker there. But as I said it seems like that blocker doesn’t really hold because we just had a pandemic.
Yeah basically this—there already is an efficient market for nanotech replicators. The most recent pandemic was only a minor blip in the grand scheme of things, it would take far more to kill humanity or seriously derail progress, and unaligned AGI would not want to do that anyway vs just soft covert takeover.