And, lest you wonder what sort of single correlated already-known-to-me variable could make my whole argument and confidence come crashing down around me, it’s whether humanity’s going to rapidly become much more competent about AGI than it appears to be about everything else.
I conclude from this that we should push on making humanity more competent at everything that affects AGI outcomes, including policy, development, deployment, and coordination. In other times I’d think that’s pretty much impossible, but on my model of how AI goes our ability to increase our competence at reasoning, evidence, argumentation, and planning is sufficiently correlated with getting closer to AGI that it’s only very hard.
I imagine you think that this is basically impossible, i.e. not worth intervening on. Does that seem right?
If so, I’d guess your reasons are something like this:
Any system that can make a big difference in these domains is extremely dangerous because it would need to be better than us at planning, and danger is a function of competent plans. Can’t find a reference but it was discussed in one of the 2021 MIRI conversations.
The coordination problem is too hard. Even if some actors have better epistemics it won’t be enough. Eliezer states this position in AGI ruin:
weaksauce Overton-abiding stuff about ‘improving public epistemology by setting GPT-4 loose on Twitter to provide scientifically literate arguments about everything’ will be cool but will not actually prevent Facebook AI Research from destroying the world six months later, or some eager open-source collaborative from destroying the world a year later if you manage to stop FAIR specifically.
Does that sound right? Are there other important reasons?
I expect the most critical reason has to do with takeoff speed; how long do we have between when AI is powerful enough to dramatically improve our institutional competence and when it poses an existential risk?
If the answer is less than e.g. 3 years (hard to imagine large institutional changes happening faster than that, even with AI help), then improving humanity’s competence is just not a tractable path to safety.
I conclude from this that we should push on making humanity more competent at everything that affects AGI outcomes, including policy, development, deployment, and coordination.
That’s the logic behind creating LessWrong, I believe?
If that was the reason for LessWrong’s existence, it was a laughable failure. This is an insular community full of nerds who talk about weird shit nobody’s ever heard of.
I conclude from this that we should push on making humanity more competent at everything that affects AGI outcomes, including policy, development, deployment, and coordination. In other times I’d think that’s pretty much impossible, but on my model of how AI goes our ability to increase our competence at reasoning, evidence, argumentation, and planning is sufficiently correlated with getting closer to AGI that it’s only very hard.
I imagine you think that this is basically impossible, i.e. not worth intervening on. Does that seem right?
If so, I’d guess your reasons are something like this:
Any system that can make a big difference in these domains is extremely dangerous because it would need to be better than us at planning, and danger is a function of competent plans. Can’t find a reference but it was discussed in one of the 2021 MIRI conversations.
The coordination problem is too hard. Even if some actors have better epistemics it won’t be enough. Eliezer states this position in AGI ruin:
Does that sound right? Are there other important reasons?
I expect the most critical reason has to do with takeoff speed; how long do we have between when AI is powerful enough to dramatically improve our institutional competence and when it poses an existential risk?
If the answer is less than e.g. 3 years (hard to imagine large institutional changes happening faster than that, even with AI help), then improving humanity’s competence is just not a tractable path to safety.
That’s the logic behind creating LessWrong, I believe?
If that was the reason for LessWrong’s existence, it was a laughable failure. This is an insular community full of nerds who talk about weird shit nobody’s ever heard of.