To do better, you need to refine your causal model of the doom.
Basically, smart AI researchers with stars in their eyes organized into teams and into communities that communicate constantly are the process that will cause the doom.
Refine that sentence into many paragraphs, then you can start to tell which interventions decrease doom a lot and which ones decrease just a little.
E.g., convincing most of the young people with the most talent for AI research that AI research is evil similar to how most of the most talented programmers starting their careers in the 1990s were convinced that working for Microsoft is evil? That would postpone our doom a lot—years probably.
Slowing down the rate of improvement of GPUs? That helps less, but still buys us significant time in expectation, as far as I can tell. Still, there is a decent chance that AI researchers can create an AI able to kill us with just the GPU designs currently available on the market, so it is not as potent an intervention as denying the doom-causing process the necessary talent to create new AI designs and new AI insights.
You can try to refine your causal model of how young people with talent for AI research decide what career to follow.
Buying humanity time by slowing down AI research is not sufficient: some “pivotal act” will have to happen that removes the danger of AI research permanently. The creation of an aligned super-intelligent AI is the prospect that gets the most ink around here, but there might be other paths that lead to a successful exit of the crisis period. You might try to refine your model of what those paths might look like.
To do better, you need a more detailed (closer to gears-level) understanding of how doom does vs. does not happen, along many different possible paths, and then you can think clearly about how new info affect all the paths.
Prior to that, though, I’d like a better sense of how I should even react to different estimates of P(doom).
For any piece of information (expert statements, benchmarks, surveys, etc.) most good faith people agree on wether it makes doom more or less likely.
But we don’t agree on how much they should move our estimates, and there’s no good way of discussing that.
How to do better?
To do better, you need to refine your causal model of the doom.
Basically, smart AI researchers with stars in their eyes organized into teams and into communities that communicate constantly are the process that will cause the doom.
Refine that sentence into many paragraphs, then you can start to tell which interventions decrease doom a lot and which ones decrease just a little.
E.g., convincing most of the young people with the most talent for AI research that AI research is evil similar to how most of the most talented programmers starting their careers in the 1990s were convinced that working for Microsoft is evil? That would postpone our doom a lot—years probably.
Slowing down the rate of improvement of GPUs? That helps less, but still buys us significant time in expectation, as far as I can tell. Still, there is a decent chance that AI researchers can create an AI able to kill us with just the GPU designs currently available on the market, so it is not as potent an intervention as denying the doom-causing process the necessary talent to create new AI designs and new AI insights.
You can try to refine your causal model of how young people with talent for AI research decide what career to follow.
Buying humanity time by slowing down AI research is not sufficient: some “pivotal act” will have to happen that removes the danger of AI research permanently. The creation of an aligned super-intelligent AI is the prospect that gets the most ink around here, but there might be other paths that lead to a successful exit of the crisis period. You might try to refine your model of what those paths might look like.
To do better, you need a more detailed (closer to gears-level) understanding of how doom does vs. does not happen, along many different possible paths, and then you can think clearly about how new info affect all the paths.
Prior to that, though, I’d like a better sense of how I should even react to different estimates of P(doom).