Reading through the recent Discord discussions with Eliezer, and reading and replying to comments, has given me the following impression of a crux of the takeoff debate. It may not be the crux. But it seems like a crux nonetheless, unless I’m misreading a lot of people.
Let me try to state it clearly:
The foom theorists are saying something like, “Well, you can usually-in-hindsight say that things changed gradually, or continuously, along some measure. You can use these measures after-the-fact, but that won’t tell you about the actual gradual-ness of the development of AI itself, because you won’t know which measures are gradual in advance.”
And then this addendum is also added, “Furthermore, I expect that the quantities which will experience discontinuities from the past will be those that are qualitatively important, in a way that is hard to measure. For example, ‘ability to manufacture nanobots’ or ‘ability to hack into computers’ are qualitative powers that we can expect AIs will develop rather suddenly, rather than gradually from precursor states, in the way that, e.g. progress in image classification accuracy was gradual over time. This means you can’t easily falsify the position by just pointing to straight lines on a million graphs.”
If you agree that foom is somewhat likely, then I would greatly appreciate if you think this is your crux, or if you think I’ve missed something.
If this indeed falls into one of your cruxes, then I feel like I’m in a position to say, “I kinda know what motivates your belief but I still think it’s probably wrong” at least in a weak sense, which seems important.
I lean toward the foom side, and I think I agree with the first statement. The intuition for me is that it’s kinda like p-hacking (there are very many possible graphs, and some percentage of those will be gradual), or using a log-log plot (which makes everything look like a nice straight line, but are actually very broad predictions when properly accounting for uncertainty). Not sure if I agree with the addendum or not yet, and I’m not sure how much of a crux this is for me yet.
Reading through the recent Discord discussions with Eliezer, and reading and replying to comments, has given me the following impression of a crux of the takeoff debate. It may not be the crux. But it seems like a crux nonetheless, unless I’m misreading a lot of people.
Let me try to state it clearly:
The foom theorists are saying something like, “Well, you can usually-in-hindsight say that things changed gradually, or continuously, along some measure. You can use these measures after-the-fact, but that won’t tell you about the actual gradual-ness of the development of AI itself, because you won’t know which measures are gradual in advance.”
And then this addendum is also added, “Furthermore, I expect that the quantities which will experience discontinuities from the past will be those that are qualitatively important, in a way that is hard to measure. For example, ‘ability to manufacture nanobots’ or ‘ability to hack into computers’ are qualitative powers that we can expect AIs will develop rather suddenly, rather than gradually from precursor states, in the way that, e.g. progress in image classification accuracy was gradual over time. This means you can’t easily falsify the position by just pointing to straight lines on a million graphs.”
If you agree that foom is somewhat likely, then I would greatly appreciate if you think this is your crux, or if you think I’ve missed something.
If this indeed falls into one of your cruxes, then I feel like I’m in a position to say, “I kinda know what motivates your belief but I still think it’s probably wrong” at least in a weak sense, which seems important.
I lean toward the foom side, and I think I agree with the first statement. The intuition for me is that it’s kinda like p-hacking (there are very many possible graphs, and some percentage of those will be gradual), or using a log-log plot (which makes everything look like a nice straight line, but are actually very broad predictions when properly accounting for uncertainty). Not sure if I agree with the addendum or not yet, and I’m not sure how much of a crux this is for me yet.