Regarding disagreement (2), I think many of Yudkowsky’s “doom stories” are more intuition pumps / minimum bounds for demonstrating properties of superintelligence.
E.g. nanotech isn’t there because he necessarily thinks it’s what an unaligned AGI would do. Instead, it’s to demonstrate how high the relative tech capabilities are of the AGI.
His point (which he stresses in different ways), is “don’t look at the surface details of the story, look instead at the implied capabilities of the system”.
Similar with “imagine it self-improving in minutes”. It may or may not happen that way specifically, but the point is “computers work on such short timescales, and recursion rates can compound quickly enough, that we should expect some parts of the process to be faster than expected, including maybe the FOOM”.
It’s not supposed to be a self-contained cinematic universe, it’s supposed to be “we have little/no reason to expect it to not be at least this weird”, according to his background assumptions (which he almost always goes into more detail on anyway).
I’m not sure what you mean by “how high the relative tech capabilities are of the AGI”.
I think the general capability of the AGI itself, not “tech” capabilities specifically, are plenty dangerous themselves.
The general danger seems more like ‘a really powerful but unaligned optimizer’ that’s ‘let loose’.
I’m not sure that ‘agent-ness’ is necessary for catastrophe; just ‘strong enough optimization’ and a lack of our own capability in predicting the consequences of running the AGI.
I do agree with this:
It’s not supposed to be a self-contained cinematic universe, it’s supposed to be “we have little/no reason to expect it to not be at least this weird”, according to his background assumptions (which he almost always goes into more detail on anyway).
Regarding disagreement (2), I think many of Yudkowsky’s “doom stories” are more intuition pumps / minimum bounds for demonstrating properties of superintelligence.
E.g. nanotech isn’t there because he necessarily thinks it’s what an unaligned AGI would do. Instead, it’s to demonstrate how high the relative tech capabilities are of the AGI.
His point (which he stresses in different ways), is “don’t look at the surface details of the story, look instead at the implied capabilities of the system”.
Similar with “imagine it self-improving in minutes”. It may or may not happen that way specifically, but the point is “computers work on such short timescales, and recursion rates can compound quickly enough, that we should expect some parts of the process to be faster than expected, including maybe the FOOM”.
It’s not supposed to be a self-contained cinematic universe, it’s supposed to be “we have little/no reason to expect it to not be at least this weird”, according to his background assumptions (which he almost always goes into more detail on anyway).
I’m not sure what you mean by “how high the relative tech capabilities are of the AGI”.
I think the general capability of the AGI itself, not “tech” capabilities specifically, are plenty dangerous themselves.
The general danger seems more like ‘a really powerful but unaligned optimizer’ that’s ‘let loose’.
I’m not sure that ‘agent-ness’ is necessary for catastrophe; just ‘strong enough optimization’ and a lack of our own capability in predicting the consequences of running the AGI.
I do agree with this: