I think the strongest version of this idea of adding p to normality is “new evidence/knowledge that contradicts previous beliefs does not invalidate previous observations.” Therefore, when one’s actions are contingent on things happening that have already been observed to happen, things add up to normality because it is already known that those things happen—regardless of any new information.But this strict version of ‘adding up to normality’ does not apply in situations where one’s actions are contingent on unobservables. In cases where new evidence/knowledge may cause someone to dramatically revise the implications of previous observations, things don’t add up to normality. Whether this is the case or not for you as an individual depends on your gears-level understanding of your observations.
So in retrospect, the main thing I’d recommend is to promise yourself to keep steering the plane mostly as normal while you think about lift
I somewhat disagree with this. I think, in these kinds of situations, the recommendation should be more along the lines of “promise yourself to make the best risk/reward trade-off you can given your state of uncertainty.” If you’re flying in a plane that has a good track record of flying, definitely don’t touch anything because its more risky to break something that has evidence of working than it is rewarding to fix things that might not actually work. But if you’re flying in the world’s first plane and realize you don’t understand lift, land it as soon as possible.
Some Reasons Things Add Up to Normality
If you think the thing you don’t understand might be a Chesterton’s Fence, there’s a good chance it will add up to normality
If you think the thing you don’t understand can be predicted robustly by inductive reasoning and you only care about being able to accurately predict the thing itself, there’s a good chance it will add up to normality
Some Examples where Things Don’t Add Up
Example #1 (Moral Revisionism)
You’re an eco-rights activist who has tirelessly worked to make the world a better place by protecting wildlife because you believe animals have the right to live good lives on this planet too. Things are going just fine until your friend claims that R-selection implies most animals live short horrible lives and you realize you have no idea whether animals actually live good lives in the wild. Should you immediately panic in fear that you’re making things worse?
Yes. Whether or not the claim in question is accurate, your general assumption that protecting wildlife implies improved animal welfare was not well-founded enough to address significant moral risk. You should really stop doing wildlife stuff until you get this figured out or you could actually cause bad things to happen.
Example #2 (Prediction Revisionism)
You’ve built an AGI and, with all your newfound free-time and wealth, you have a lengthy chat with a mathematician. Things are going along just fine until they point out to you that your understanding of the safety measures used to ensure alignment are wrong, and that the AGI shouldn’t be aligned from the safety measures you thought were responsible.Should you immediately panic in fear that the AGI will destroy us all?
Yes. The previous observations are not sufficient to make reliable predictions. But note that a random bystander who is uninvolved with AGI development would be justified in not panicking—their gears-level understanding hinges on believing that the people who created the AGI are competent enough to address safety, not on believing that the specific details designed to make the AGI safe actually work.
I agree that carefully landing the plane is better than maintaining the course if catastrophic outcomes suddenly seem more plausible than before.
Obviously it applies if you’re the lead on a new technological project and suddenly realize a plausible catastrophic risk from it.
I don’t think it applies very strongly in your example about animal welfare, unless the protagonist has unusually high leverage on a big decision about to be made. The cost of continuing to stay in the old job for a few weeks while thinking things over (especially if leaving and then coming back would be infeasible) is plausibly worth the value of information thus gained.
I agree that carefully landing the plane is better than maintaining the course if catastrophic outcomes suddenly seem more plausible than before.
Yeah, but my point is not about catastrophic risk—it’s about the risk/reward trade-off in general. You can have risk>reward in scenarios that aren’t catastrophic. Catastrophic risk is just a good general example of where things don’t add up to normality (catastrophic risks by nature correspond to not-normal scenarios and also coincide with high risk). Don’t promise yourself to steer the plane mostly as normal, promise yourself to pursue the path that reduces risk over all outcomes you’re uncertain about.
I don’t think it applies very strongly in your example about animal welfare, unless the protagonist has unusually high leverage on a big decision about to be made. The cost of continuing to stay in the old job for a few weeks while thinking things over (especially if leaving and then coming back would be infeasible) is plausibly worth the value of information thus gained.
Good point, it really depends on the details of the example but this is just because of the different risk-reward trade-offs, not because you ought to always treat things as adding up to normality. I’ll counter that while you shouldn’t leave the job (high risk, hard to reverse), you should see if you could use your PTO as soon as possible so you can figure things out without potentially causing further negative impact. It all depends on the risk-reward trade-off:
If stopping activism corresponds to something like leaving a job, which is hard to reverse, doing so involves taking on a lot of risk if you’re uncertain and waiting for a bit can reduce that risk.
If stopping activism corresponds to something like shifting your organizations priorities, and your organization’s path can be reversed, then stopping work (after satisfying all existing contracts of course) is pretty low risk and you should stop
If stopping activism corresponds to donating large amounts of money (in earning-to-give contexts), your strategy can easily be reversed and you should stop now.
This is true even if you only have “small” amounts of impact.
Caveat:
People engage in policies for many reasons at once. So if you think the goal of your policy is X, but it’s actually X, Y and Z, then dramatic actions justified on uncertainty about X alone will probably be harmful due to Y and Z effects even if its the appropriate decision with respect to X. Because it’s easy to notice when why a thing might go wrong (like X) and hard to notice why they’re going right (like Y and Z), adding-up-to-normality serves as a way to generally protect Y and Z.
Don’t know if you saw, but I updated the post yesterday because of your (and khafra’s) points.
Also, your caveat is a good reframe of the main mechanism behind the post.
I do still disagree with you somewhat, because I think that people going through a crisis of faith are prone to flailing around and taking naive actions that they would have reconsidered after a week or month of actually thinking through the implications of their new belief. Trying to maximize utility while making a major update is safe for ideal Bayesian reasoners, but it fails badly for actual humans.
In the absence of an external crisis, taking relatively safe actions (and few irreversible actions) is correct in the short term, and the status quo is going to be reasonably safe for most people if you’ve been living it for years. If you can back off from newly-suspected-wrong activities for the time being without doing so irreversibly, then yes that’s better.
I do still disagree with you somewhat, because I think that people going through a crisis of faith are prone to flailing around and taking naive actions that they would have reconsidered after a week or month of actually thinking through the implications of their new belief. Trying to maximize utility while making a major update is safe for ideal Bayesian reasoners, but it fails badly for actual humans.
Ah, yeah I agree with this observation—and it could be good to just assume things add up to normality as a general defense against people rapidly taking naive actions. Scarcity bias is a thing after all and if you get into a mindset where now is the time to act, it’s really hard to prevent yourself from acting irrationally.
I agree with this, but a counterpoint is that it’s very hard for people to change longstanding habits and behaviors at all, and sometimes a major internal update is a good moment to make significant behavior changes because that’s the only time most people can manage major behavioral changes at all.
I think the strongest version of this idea of adding p to normality is “new evidence/knowledge that contradicts previous beliefs does not invalidate previous observations.” Therefore, when one’s actions are contingent on things happening that have already been observed to happen, things add up to normality because it is already known that those things happen—regardless of any new information.But this strict version of ‘adding up to normality’ does not apply in situations where one’s actions are contingent on unobservables. In cases where new evidence/knowledge may cause someone to dramatically revise the implications of previous observations, things don’t add up to normality. Whether this is the case or not for you as an individual depends on your gears-level understanding of your observations.
I somewhat disagree with this. I think, in these kinds of situations, the recommendation should be more along the lines of “promise yourself to make the best risk/reward trade-off you can given your state of uncertainty.” If you’re flying in a plane that has a good track record of flying, definitely don’t touch anything because its more risky to break something that has evidence of working than it is rewarding to fix things that might not actually work. But if you’re flying in the world’s first plane and realize you don’t understand lift, land it as soon as possible.
Some Reasons Things Add Up to Normality
If you think the thing you don’t understand might be a Chesterton’s Fence, there’s a good chance it will add up to normality
If you think the thing you don’t understand can be predicted robustly by inductive reasoning and you only care about being able to accurately predict the thing itself, there’s a good chance it will add up to normality
Some Examples where Things Don’t Add Up
Example #1 (Moral Revisionism)
You’re an eco-rights activist who has tirelessly worked to make the world a better place by protecting wildlife because you believe animals have the right to live good lives on this planet too. Things are going just fine until your friend claims that R-selection implies most animals live short horrible lives and you realize you have no idea whether animals actually live good lives in the wild. Should you immediately panic in fear that you’re making things worse?
Yes. Whether or not the claim in question is accurate, your general assumption that protecting wildlife implies improved animal welfare was not well-founded enough to address significant moral risk. You should really stop doing wildlife stuff until you get this figured out or you could actually cause bad things to happen.
Example #2 (Prediction Revisionism)
You’ve built an AGI and, with all your newfound free-time and wealth, you have a lengthy chat with a mathematician. Things are going along just fine until they point out to you that your understanding of the safety measures used to ensure alignment are wrong, and that the AGI shouldn’t be aligned from the safety measures you thought were responsible.Should you immediately panic in fear that the AGI will destroy us all?
Yes. The previous observations are not sufficient to make reliable predictions. But note that a random bystander who is uninvolved with AGI development would be justified in not panicking—their gears-level understanding hinges on believing that the people who created the AGI are competent enough to address safety, not on believing that the specific details designed to make the AGI safe actually work.
I agree that carefully landing the plane is better than maintaining the course if catastrophic outcomes suddenly seem more plausible than before.
Obviously it applies if you’re the lead on a new technological project and suddenly realize a plausible catastrophic risk from it.
I don’t think it applies very strongly in your example about animal welfare, unless the protagonist has unusually high leverage on a big decision about to be made. The cost of continuing to stay in the old job for a few weeks while thinking things over (especially if leaving and then coming back would be infeasible) is plausibly worth the value of information thus gained.
Yeah, but my point is not about catastrophic risk—it’s about the risk/reward trade-off in general. You can have risk>reward in scenarios that aren’t catastrophic. Catastrophic risk is just a good general example of where things don’t add up to normality (catastrophic risks by nature correspond to not-normal scenarios and also coincide with high risk). Don’t promise yourself to steer the plane mostly as normal, promise yourself to pursue the path that reduces risk over all outcomes you’re uncertain about.
Good point, it really depends on the details of the example but this is just because of the different risk-reward trade-offs, not because you ought to always treat things as adding up to normality. I’ll counter that while you shouldn’t leave the job (high risk, hard to reverse), you should see if you could use your PTO as soon as possible so you can figure things out without potentially causing further negative impact. It all depends on the risk-reward trade-off:
If stopping activism corresponds to something like leaving a job, which is hard to reverse, doing so involves taking on a lot of risk if you’re uncertain and waiting for a bit can reduce that risk.
If stopping activism corresponds to something like shifting your organizations priorities, and your organization’s path can be reversed, then stopping work (after satisfying all existing contracts of course) is pretty low risk and you should stop
If stopping activism corresponds to donating large amounts of money (in earning-to-give contexts), your strategy can easily be reversed and you should stop now.
This is true even if you only have “small” amounts of impact.
Caveat:
People engage in policies for many reasons at once. So if you think the goal of your policy is X, but it’s actually X, Y and Z, then dramatic actions justified on uncertainty about X alone will probably be harmful due to Y and Z effects even if its the appropriate decision with respect to X. Because it’s easy to notice when why a thing might go wrong (like X) and hard to notice why they’re going right (like Y and Z), adding-up-to-normality serves as a way to generally protect Y and Z.
Don’t know if you saw, but I updated the post yesterday because of your (and khafra’s) points.
Also, your caveat is a good reframe of the main mechanism behind the post.
I do still disagree with you somewhat, because I think that people going through a crisis of faith are prone to flailing around and taking naive actions that they would have reconsidered after a week or month of actually thinking through the implications of their new belief. Trying to maximize utility while making a major update is safe for ideal Bayesian reasoners, but it fails badly for actual humans.
In the absence of an external crisis, taking relatively safe actions (and few irreversible actions) is correct in the short term, and the status quo is going to be reasonably safe for most people if you’ve been living it for years. If you can back off from newly-suspected-wrong activities for the time being without doing so irreversibly, then yes that’s better.
Ah, yeah I agree with this observation—and it could be good to just assume things add up to normality as a general defense against people rapidly taking naive actions. Scarcity bias is a thing after all and if you get into a mindset where now is the time to act, it’s really hard to prevent yourself from acting irrationally.
Huzzah, convergence! I appreciate the points you’ve made.
I agree with this, but a counterpoint is that it’s very hard for people to change longstanding habits and behaviors at all, and sometimes a major internal update is a good moment to make significant behavior changes because that’s the only time most people can manage major behavioral changes at all.