My main point I think, is that this is a more general problem. Some configurations of observations can seem extremely unlikely, yet the sum over all of these configurations might be fairly probable. Like if an airplane has an engine failure above your home town, and is about to crash unto it. The probability of it crashing right near your house is small (if you live in a big town), but it has to crash near someone’s house. And that person that had the airplane crash right on his front lawn would go and say “What do you know, what are the odds? So unlikely!”.
So while the above example is simple to explain, what happens if someone say’s one day as a joke “I hope an airplane won’t crash on my house” and on that day an airplane does? That’s on its own seems rare enough that it shouldn’t be happening every day, or even every year, or maybe even it never happened in history (assuming reasonable stuff like ‘not in a time of war’ etc …). But that may happen to someone at some point, and we won’t go up and say “that’s insane, that couldn’t possible be true”, because we understand in some level that the probability of observing something with a low probability is very different from the probability of observing specifically that low probability event. And so, maybe that wouldn’t happen with an airplane but with a lightning strike, someone saying “I hope lightning won’t strike me today” and get struck, or a meteor, or any other of huge number of other situations. So how do we when something doesn’t fit the model? Where should I say “I should be confused by this, this phenomenon is ought not to be possible.”?
BTW—I am sure that the one in a million events happen all the time is in Methods of Rationality, but it may have some earlier references.
the probability of observing something with a low probability is very different from the probability of observing specifically that low probability event
Right. For example, suppose you have a biased coin that comes up Heads 80% of the time, and you flip it 100 times. The single most likely sequence of flips is “all Heads.” (Consider that you should bet heads on any particular flip.) But it would be incredibly shocking to actually observe 100 Headses in a row (probability 0.8¹⁰⁰ ≈ 2.037 · 10⁻¹⁰). Other sequences have less probability per individual sequence, but there are vastly more of them: there’s only one way to get “all Heads”, but there are 100 possible ways to get “99 Headses and 1 Tails” (the Tails could be the 1st flip, or the 2nd, or …), 4,950 ways to get “98 Headses and 2 Tailses”, and so on. It turns out that you’re almost certain to observe a sequence with about 20 Tailses—you can think of this as being where the “number of ways this reference class of outcomes could be realized” factor balances out the “improbability of an individual outcome” factor. For more of the theory here, see Chapter 4 of Information Theory, Inference, and Learning Algorithms.
But how can I apply this sort of logic to the problems I’ve described above? It still seems to me like I need in theory to sum over all of the probabilities in some set A that contains all these improbable events but I just don’t understand how to even properly define A, as its boundaries seem fuzzy and various thing “kinda fit” or “doesn’t quite fit, but maybe?” instead of plain true and false.
My main point I think, is that this is a more general problem. Some configurations of observations can seem extremely unlikely, yet the sum over all of these configurations might be fairly probable. Like if an airplane has an engine failure above your home town, and is about to crash unto it. The probability of it crashing right near your house is small (if you live in a big town), but it has to crash near someone’s house. And that person that had the airplane crash right on his front lawn would go and say “What do you know, what are the odds? So unlikely!”.
So while the above example is simple to explain, what happens if someone say’s one day as a joke “I hope an airplane won’t crash on my house” and on that day an airplane does? That’s on its own seems rare enough that it shouldn’t be happening every day, or even every year, or maybe even it never happened in history (assuming reasonable stuff like ‘not in a time of war’ etc …). But that may happen to someone at some point, and we won’t go up and say “that’s insane, that couldn’t possible be true”, because we understand in some level that the probability of observing something with a low probability is very different from the probability of observing specifically that low probability event. And so, maybe that wouldn’t happen with an airplane but with a lightning strike, someone saying “I hope lightning won’t strike me today” and get struck, or a meteor, or any other of huge number of other situations. So how do we when something doesn’t fit the model? Where should I say “I should be confused by this, this phenomenon is ought not to be possible.”?
BTW—I am sure that the one in a million events happen all the time is in Methods of Rationality, but it may have some earlier references.
Right. For example, suppose you have a biased coin that comes up Heads 80% of the time, and you flip it 100 times. The single most likely sequence of flips is “all Heads.” (Consider that you should bet heads on any particular flip.) But it would be incredibly shocking to actually observe 100 Headses in a row (probability 0.8¹⁰⁰ ≈ 2.037 · 10⁻¹⁰). Other sequences have less probability per individual sequence, but there are vastly more of them: there’s only one way to get “all Heads”, but there are 100 possible ways to get “99 Headses and 1 Tails” (the Tails could be the 1st flip, or the 2nd, or …), 4,950 ways to get “98 Headses and 2 Tailses”, and so on. It turns out that you’re almost certain to observe a sequence with about 20 Tailses—you can think of this as being where the “number of ways this reference class of outcomes could be realized” factor balances out the “improbability of an individual outcome” factor. For more of the theory here, see Chapter 4 of Information Theory, Inference, and Learning Algorithms.
But how can I apply this sort of logic to the problems I’ve described above? It still seems to me like I need in theory to sum over all of the probabilities in some set A that contains all these improbable events but I just don’t understand how to even properly define A, as its boundaries seem fuzzy and various thing “kinda fit” or “doesn’t quite fit, but maybe?” instead of plain true and false.