If we do get powerful AI, it seems highly plausible that even if we stay in control we will ‘go too fast’ in deploying it relative to society’s ability to adapt, if only because of the need to grow fast and stay ahead of others, and because the market doesn’t care that society wants it to go slower.
After reading my interpretation was this: assuming we stay in control, that happens only if powerful AI is aligned. The market doesn’t care that society wants to go slower, but AI will care that society wants to go slower, so when the market tries to force AI to go faster, AI will refuse.
I reflected on whether I am being too generous, but I don’t think I am. Other readings didn’t make sense to me, and I am assuming Dario is trying to make sense, while you seem doubtful. That is, I think this is plausibly Dario’s actual prediction of how fast things will go, not a hope it won’t go faster. But importantly, that is assuming alignment. Since that assumption is already hopeful, it is natural the prediction under that assumption sounds hopeful.
Paul Crowley: It’s a strange essay, in that it asks us to imagine a world in which a single datacenter contains 1E6 Nobelists expert in every field and thinking at 100x speed, and asks what happens if “sci-fi” outcomes somehow don’t happen. Of course “sci-fi” stuff happens almost immediately.
I mean, yes, sci-fi style stuff does seem rather obviously like it would happen? If it didn’t, then that’s a rather chilling indictment of the field of sci-fi?
To re-state, sci-fi outcomes don’t happen because AI is aligned. Proof: if sci-fi outcomes happened, AI would be unaligned. I actually think this point is extremely clear in the essay. It literally states: “An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks)”.
This is a good idea and it already works, it is just that AI is wholly unnecessary. Have a look at 2018 post Protecting Applications with Automated Software Diversity.