My understanding is that humanity is like a person stuck in a car whose brakes have failed, plummeting down a steep road towards a cliff. Timelines are about figuring out whether we have 1 minute to solve our dilemma or 10 days. The difference is very relevant indeed to what actions we strategically attempt in our limited time. Here’s my post about that: https://www.lesswrong.com/posts/wgcFStYwacRB8y3Yp/timelines-are-relevant-to-alignment-research-timelines-2-of
P(doom), given the situation, is primarily about two questions: will our strategic action succeed? Will we survive the crash?
I agree that, given we have accurate timelines and thus make correct strategic choices about which actions to attempt, the probability of our actions suceeding isn’t relevant. We must do our best. The probability that we will die if the car goes over the edge of the cliff is relevant only insofar as it motivates us to take action. We only need agreement that it is large enough to be worth worrying about, precision is irrelevant.
Currently, lots of people who could potentially be taking helpful action are instead actively making the problem worse by working on capabilities. Talking about our reasoning for our personal estimates of p(doom) is useful if and only if it helps sway some potential safety researchers into working on safety, or some capabilities researchers into stopping work on capabilities.
Setting aside how important timelines are for strategy, the fact that P(doom) combines several questions together is a good point. Another way to decompose P(doom) is:
How likely are we to survive if we do nothing about the risk? Or perhaps: How likely are we to survive if we do alignment research at the current pace?
How much can we really reduce the risk with sustained effort? How immutable is the overall risk?
Though people probably mean different things by P(doom) and seems worthwhile to disentangle them.
Talking about our reasoning for our personal estimates of p(doom) is useful if and only if it helps sway some potential safety researchers into working on safety …
Good point, P(doom) also serves a promotional role, in that it illustrates the size of the problem to others and potentially gets more people to work on alignment.
My understanding is that humanity is like a person stuck in a car whose brakes have failed, plummeting down a steep road towards a cliff. Timelines are about figuring out whether we have 1 minute to solve our dilemma or 10 days. The difference is very relevant indeed to what actions we strategically attempt in our limited time. Here’s my post about that: https://www.lesswrong.com/posts/wgcFStYwacRB8y3Yp/timelines-are-relevant-to-alignment-research-timelines-2-of P(doom), given the situation, is primarily about two questions: will our strategic action succeed? Will we survive the crash? I agree that, given we have accurate timelines and thus make correct strategic choices about which actions to attempt, the probability of our actions suceeding isn’t relevant. We must do our best. The probability that we will die if the car goes over the edge of the cliff is relevant only insofar as it motivates us to take action. We only need agreement that it is large enough to be worth worrying about, precision is irrelevant. Currently, lots of people who could potentially be taking helpful action are instead actively making the problem worse by working on capabilities. Talking about our reasoning for our personal estimates of p(doom) is useful if and only if it helps sway some potential safety researchers into working on safety, or some capabilities researchers into stopping work on capabilities.
Setting aside how important timelines are for strategy, the fact that P(doom) combines several questions together is a good point. Another way to decompose P(doom) is:
How likely are we to survive if we do nothing about the risk? Or perhaps: How likely are we to survive if we do alignment research at the current pace?
How much can we really reduce the risk with sustained effort? How immutable is the overall risk?
Though people probably mean different things by P(doom) and seems worthwhile to disentangle them.
Good point, P(doom) also serves a promotional role, in that it illustrates the size of the problem to others and potentially gets more people to work on alignment.