This post frustrated me for a while, because it seems right but not helpful. Saying to myself, “I should be confused by fiction” doesn’t influence my present decision.
First concertize. Let’s say I have a high level world model. A few of them perhaps, to reduce the chance that one bad example results in a bad principle.
“My shower produces hot water in the morning.”
“I have fresh milk to last the next two days.”
“The roads are no longer slippery.”
What do these models exclude? “The water will be cold”, “the milk will be spoiled”, “I’ll see someone sliding at an intersection” are easy ones. Then there are weirder ones like “I don’t even own a shower”, “Someone drank all my milk in the middle of the night”, and “the roads are closed off due to an earthquake”.
I could say, “My model as stated technically disallows all these things, so if I see any, I should have a huge update”, but that’s unrealistic. The use of “easy” and “weird” implicitly show that I’m already thinking about hypotheses not as strictly allowing and disallowing, but as resulting in greater and lesser probabilistic gains/hits to my confidence.
Even if I do give up entirely on “I have fresh milk”, I usually replace it with something that is consistent with the old reasoning (not just the old observations). Perhaps I reason “The milk should have been fresh but spoiled because of a temporary power outage last night”. That’s actually a bad example because it’s not something I’d jump to if I didn’t have other observation indicating a power outage. Let’s try again. “The milk should have been fresh, but oh dang, it wasn’t.” Yes, that looks like something I’d think. What about the others? My first explanations would probably be “The roads are a little slippery some places” and “The water heater is acting up”.
So what did we just see in this totally fictional but mildly plausible-sounding anecdote? Sometimes a failed hypothesis—becomes-> failed hypothesis + some noise. Other times it’s like the water heater explanation which look pretty different. Let’s think about the first type. Is this small model-distance update heuristic justified? The new model clearly gives more probability mass to our actual observations, but that’s the representative heuristic, totally insufficient to judge whether the theory is acceptable. For that we look to Bayes:
P(H|E) = P(H) * P(E|H) / P(E)
P(E) will be the same for all hypotheses we consider, so just ignore that. P(E|H) is pretty high, since we added noise to make sure the hypothesis would predict evidence. What about P(H)? How do I practically compare the prior for different hypotheses? How do I know when adding noise to my model is good enough vs. when I need to search for new hypotheses?
Let’s think of six different methods to guess whether our new hypothesis will be good enough.
Outside view. Think of five times you’ve been confused in the past four months when dealing with spoiled foods you purchased. If you can’t, you’re well calibrated enough in the spoiled food domain and the hypothesis is fine.
Solomonoff induction says that complexity penalizes the prior probability of hypotheses. Let’s try some dirty things that look like that.
Count the words in the hypothesis.
Counts the nouns and verbs.
Count the number of conjunctions and subtract the number of disjunctions.
Count syllables or time how long it takes you to say it.
Do this a lot in your daily life so you know how big your theories generally are. Guess what your average has been. Compare to milk explanation.
Consequential salience. Think of four things your theory predicts and four things that your theory disallows. If any of those eight things makes you squick, count a point. Two squicks or more means your theory is weird and you need to look for a better hypothesis. If you spend long enough trying to think of a consequence that you notice the time, your hypothesis isn’t paying rent in expectation and you need to look for a better hypotheses.
Remember your day looking for two pieces of confirming evidence. Remember your day looking for three pieces of disconfirming evidence. Arbitrarily decide whether the hypothesis continues to jive with new evidence. If not, new hypothesis formation.
Imagine a wizard told you your new hypothesis before you tasted the spoiled milk. Imagine his clothes. Is the hypothesis sensible enough that you can trust him? Would you let him borrow a cup of sugar?
You’re wrong. Your hypothesis is simply wrong. Say it to yourself. Say that the milk is still fine. Imagine whether you could go about your day believing this. Can you drink the milk? If not, your hypothesis changed by a large amount and it’s sensible to look for alternatives rather than sticking with your old reasoning by mental inertia.
Now the critical stage!
You don’t have time to remember the last four months. Don’t even think about hypothesis priors unless you’ve already spent more than a minute trying to decide something. Milk is not a big deal, save your cognitive energy for the higher order bits of your life. Also, four months is kind of food-spoiling specific. Time frames would have to be adapted for different problems.
That is not Solomonoff induction in any way. We don’t even have a language for formally expressing high level concepts like “spoiled milk” unless you look at brain architecture to figure out how they classify reality. Also “compare” is not concrete enough.
Emotional salience fails us badly in abstract situations. Thinking of disconfirming evidence is painful; our brains won’t easily present squicky things.
Arbitrarily decide is not an actionable procedure.
This one actually seems kind of okay, unless you’re just as likely to give sugar to senseless wizards.
I’m not sure small updates have small changes in consequence value, but doing more thinking when costs are high generally doesn’t seem horrible. Maybe we should add in something to keep us from thinking longer just to procrastinate though.
Conclusions!
Priors over explanations are -hard-. Sometimes we naturally make new hypotheses, sometimes we just add some noise. Maybe take the outside view of yourself if you have time! Maybe take the outside view of the hypothesis by having a wizard tell it to you if not. Your strength as a rationalist? Not drinking spoiled milk, not wasting time thinking about spoiled milk, noticing squicks, successfully doubting yourself when you feel a squick, believing some things because they work really well even if they sound crazy when a wizard says them.
This post frustrated me for a while, because it seems right but not helpful. Saying to myself, “I should be confused by fiction” doesn’t influence my present decision.
First concertize. Let’s say I have a high level world model. A few of them perhaps, to reduce the chance that one bad example results in a bad principle.
“My shower produces hot water in the morning.” “I have fresh milk to last the next two days.” “The roads are no longer slippery.”
What do these models exclude? “The water will be cold”, “the milk will be spoiled”, “I’ll see someone sliding at an intersection” are easy ones. Then there are weirder ones like “I don’t even own a shower”, “Someone drank all my milk in the middle of the night”, and “the roads are closed off due to an earthquake”.
I could say, “My model as stated technically disallows all these things, so if I see any, I should have a huge update”, but that’s unrealistic. The use of “easy” and “weird” implicitly show that I’m already thinking about hypotheses not as strictly allowing and disallowing, but as resulting in greater and lesser probabilistic gains/hits to my confidence.
Even if I do give up entirely on “I have fresh milk”, I usually replace it with something that is consistent with the old reasoning (not just the old observations). Perhaps I reason “The milk should have been fresh but spoiled because of a temporary power outage last night”. That’s actually a bad example because it’s not something I’d jump to if I didn’t have other observation indicating a power outage. Let’s try again. “The milk should have been fresh, but oh dang, it wasn’t.” Yes, that looks like something I’d think. What about the others? My first explanations would probably be “The roads are a little slippery some places” and “The water heater is acting up”.
So what did we just see in this totally fictional but mildly plausible-sounding anecdote? Sometimes a failed hypothesis—becomes-> failed hypothesis + some noise. Other times it’s like the water heater explanation which look pretty different. Let’s think about the first type. Is this small model-distance update heuristic justified? The new model clearly gives more probability mass to our actual observations, but that’s the representative heuristic, totally insufficient to judge whether the theory is acceptable. For that we look to Bayes:
P(H|E) = P(H) * P(E|H) / P(E)
P(E) will be the same for all hypotheses we consider, so just ignore that. P(E|H) is pretty high, since we added noise to make sure the hypothesis would predict evidence. What about P(H)? How do I practically compare the prior for different hypotheses? How do I know when adding noise to my model is good enough vs. when I need to search for new hypotheses?
Let’s think of six different methods to guess whether our new hypothesis will be good enough.
Outside view. Think of five times you’ve been confused in the past four months when dealing with spoiled foods you purchased. If you can’t, you’re well calibrated enough in the spoiled food domain and the hypothesis is fine.
Solomonoff induction says that complexity penalizes the prior probability of hypotheses. Let’s try some dirty things that look like that. Count the words in the hypothesis. Counts the nouns and verbs. Count the number of conjunctions and subtract the number of disjunctions. Count syllables or time how long it takes you to say it. Do this a lot in your daily life so you know how big your theories generally are. Guess what your average has been. Compare to milk explanation.
Consequential salience. Think of four things your theory predicts and four things that your theory disallows. If any of those eight things makes you squick, count a point. Two squicks or more means your theory is weird and you need to look for a better hypothesis. If you spend long enough trying to think of a consequence that you notice the time, your hypothesis isn’t paying rent in expectation and you need to look for a better hypotheses.
Remember your day looking for two pieces of confirming evidence. Remember your day looking for three pieces of disconfirming evidence. Arbitrarily decide whether the hypothesis continues to jive with new evidence. If not, new hypothesis formation.
Imagine a wizard told you your new hypothesis before you tasted the spoiled milk. Imagine his clothes. Is the hypothesis sensible enough that you can trust him? Would you let him borrow a cup of sugar?
You’re wrong. Your hypothesis is simply wrong. Say it to yourself. Say that the milk is still fine. Imagine whether you could go about your day believing this. Can you drink the milk? If not, your hypothesis changed by a large amount and it’s sensible to look for alternatives rather than sticking with your old reasoning by mental inertia.
Now the critical stage!
You don’t have time to remember the last four months. Don’t even think about hypothesis priors unless you’ve already spent more than a minute trying to decide something. Milk is not a big deal, save your cognitive energy for the higher order bits of your life. Also, four months is kind of food-spoiling specific. Time frames would have to be adapted for different problems.
That is not Solomonoff induction in any way. We don’t even have a language for formally expressing high level concepts like “spoiled milk” unless you look at brain architecture to figure out how they classify reality. Also “compare” is not concrete enough.
Emotional salience fails us badly in abstract situations. Thinking of disconfirming evidence is painful; our brains won’t easily present squicky things.
Arbitrarily decide is not an actionable procedure.
This one actually seems kind of okay, unless you’re just as likely to give sugar to senseless wizards.
I’m not sure small updates have small changes in consequence value, but doing more thinking when costs are high generally doesn’t seem horrible. Maybe we should add in something to keep us from thinking longer just to procrastinate though.
Conclusions! Priors over explanations are -hard-. Sometimes we naturally make new hypotheses, sometimes we just add some noise. Maybe take the outside view of yourself if you have time! Maybe take the outside view of the hypothesis by having a wizard tell it to you if not. Your strength as a rationalist? Not drinking spoiled milk, not wasting time thinking about spoiled milk, noticing squicks, successfully doubting yourself when you feel a squick, believing some things because they work really well even if they sound crazy when a wizard says them.