Am I extremely sure that an unaligned AGI would cause doom?
If that’s the case, we already agree and I have nothing to add. We might disagree in the relative likelihood but that’s ok. I do agree that is a risk and we should take the matter extremely seriously
Right then, but my original claim still stands: your main point is, in fact, that it is hard to destroy the world. Like I’ve explained, this doesn’t make any sense (hacking into nuclear codes). If we create an AI better than us at code, I don’t have any doubts that it CAN easily do it, if it WANTS. My only doubt is whether it will want it or not. Not whether it will be capable, because like I said, even a very good human hacker in the future could be capable.
At least the type of AGI that I fear is one capable of Recursive Self-Improvement, which will unavoidably attain enormous capabilities. Not some prosaic non-improving AGI that is only human-level. To doubt whether the latter would have the capability to destroy the world is kinda reasonable, to doubt it about the former is not.
Am I extremely sure that an unaligned AGI would cause doom?
If that’s the case, we already agree and I have nothing to add. We might disagree in the relative likelihood but that’s ok. I do agree that is a risk and we should take the matter extremely seriously
Right then, but my original claim still stands: your main point is, in fact, that it is hard to destroy the world. Like I’ve explained, this doesn’t make any sense (hacking into nuclear codes). If we create an AI better than us at code, I don’t have any doubts that it CAN easily do it, if it WANTS. My only doubt is whether it will want it or not. Not whether it will be capable, because like I said, even a very good human hacker in the future could be capable.
At least the type of AGI that I fear is one capable of Recursive Self-Improvement, which will unavoidably attain enormous capabilities. Not some prosaic non-improving AGI that is only human-level. To doubt whether the latter would have the capability to destroy the world is kinda reasonable, to doubt it about the former is not.