Fast takeoff: there will be a sudden phase shift in capabilities, and the design of whatever system first undergoes that phase shift needs to be right on the first try.
I would have said “irreversible catastrophe”, not “fast takeoff”. Isn’t that the real problem? Iterative design presumably gets you a solution eventually, if one exists, but it’s not guaranteed to get you a solution after N iterations, where N is some number determined ex ante. In extreme fast takeoff, we need to solve alignment with N=0 iterations. In slow takeoff (in a competitive uncoordinated world), we need to succeed within N<(whatever) iterations. The latter is less bad than the former, but as long as there’s a deadline, there’s a chance we’ll miss it.
(Slow takeoff is a bit worse than that because arguably as the AIs get gradually more capable, the problem keeps changing; you’re not iterating on exactly the same problem for the whole takeoff.)
You are correct. I was trying to list the two frames which I think people most often use, not necessarily the best versions of those frames, since I wanted to emphasize that there are lots of other ways the iterative design loop fails.
I would have said “irreversible catastrophe”, not “fast takeoff”. Isn’t that the real problem? Iterative design presumably gets you a solution eventually, if one exists, but it’s not guaranteed to get you a solution after N iterations, where N is some number determined ex ante. In extreme fast takeoff, we need to solve alignment with N=0 iterations. In slow takeoff (in a competitive uncoordinated world), we need to succeed within N<(whatever) iterations. The latter is less bad than the former, but as long as there’s a deadline, there’s a chance we’ll miss it.
(Slow takeoff is a bit worse than that because arguably as the AIs get gradually more capable, the problem keeps changing; you’re not iterating on exactly the same problem for the whole takeoff.)
You are correct. I was trying to list the two frames which I think people most often use, not necessarily the best versions of those frames, since I wanted to emphasize that there are lots of other ways the iterative design loop fails.