It’s interesting to look back at this question 4 years later; I think it’s a great example of the difficulty of choosing the right question to forecast in the first place.
I think it is still pretty unlikely that the criterion I outlined is met—Q2 on my survey still seems like a bottleneck. I doubt that AGI researchers would talk about instrumental convergence in the kind of conversation I outlined. But reading the motivation for the question, it sure seems like a question that reflected the motivation well would have resolved yes by now (probably some time in 2023), given the current state of discourse and the progress in the AI governance space. (Though you could argue that the governance space is still primarily focused on misuse rather than misalignment.)
I did quite deliberately include Q2 in my planned survey—I think it’s important that the people whom governments defer to in crafting policy understand the concerns, rather than simply voicing support. But I failed to notice that it is quite plausible (indeed, the default) for there to be a relatively small number of experts that understand the concerns in enough depth to produce good advice on policy, plus a large base of “voicing support” from other experts who don’t have that same deep understanding. This means that it’s very plausible that fraction defined in the question never gets anywhere close to 0.5, but nonetheless the AI community “agrees on the risk” to a sufficient degree that governance efforts do end up in a good place.
It’s interesting to look back at this question 4 years later; I think it’s a great example of the difficulty of choosing the right question to forecast in the first place.
I think it is still pretty unlikely that the criterion I outlined is met—Q2 on my survey still seems like a bottleneck. I doubt that AGI researchers would talk about instrumental convergence in the kind of conversation I outlined. But reading the motivation for the question, it sure seems like a question that reflected the motivation well would have resolved yes by now (probably some time in 2023), given the current state of discourse and the progress in the AI governance space. (Though you could argue that the governance space is still primarily focused on misuse rather than misalignment.)
I did quite deliberately include Q2 in my planned survey—I think it’s important that the people whom governments defer to in crafting policy understand the concerns, rather than simply voicing support. But I failed to notice that it is quite plausible (indeed, the default) for there to be a relatively small number of experts that understand the concerns in enough depth to produce good advice on policy, plus a large base of “voicing support” from other experts who don’t have that same deep understanding. This means that it’s very plausible that fraction defined in the question never gets anywhere close to 0.5, but nonetheless the AI community “agrees on the risk” to a sufficient degree that governance efforts do end up in a good place.