JenniferRM comments on Mistakes people make when thinking about units

JenniferRM 25 Jun 2024 18:26 UTC
8 points
0
Log odds, measured in something like “bits of evidence” or “decibels of evidence”, is the natural thing to think of yourself as “counting”. A probability of 100% would be like having infinite positive evidence for a claim and a probability of 0% is like having infinite negative evidence for a claim. Arbital has some math and Eliezer has a good old essay on this.
A good general heuristic (or widely applicable hack) to “fix your numbers to even be valid numbers” when trying to get probabilities for things based on counts (like a fast and dirty spreadsheet analysis), and never having this spit out 0% or 100% due to naive division on small numbers (like seeing 3 out of 3 of something and claiming it means the probability of that thing is probably 100%), is to use “pseudo-counting” where every category that is analytically possible is treated as having been “observed once in our imaginations”. This way, if you can fail or succeed, and you’ve seen 3 of either, and seen nothing else, you can use pseudocounts to guesstimate that whatever happened every time so far is (3+1)/(3+2) == 80% likely in the future, and whatever you’ve never seen is (0+1)/(3+2) == 20% likely.
- cubefox 26 Jun 2024 23:01 UTC
  2 points
  0
  Parent
  
  Log odds, measured in something like “bits of evidence” or “decibels of evidence”, is the natural thing to think of yourself as “counting”. A probability of 100% would be like having infinite positive evidence for a claim and a probability of 0% is like having infinite negative evidence for a claim. Arbital has some math and Eliezer has a good old essay on this.
  
  Odds (and log odds) solve some problems but they unfortunately create others.
  
  For addition and multiplication they at least seem to make things worse. We know that we can add probabilities if they are “mutually exclusive” to get the probability of their disjunction, and we know we can multiply them if they are “independent” to get the probability of their conjunction. But when can we add two odds, or multiply two odds? (Or log odds) And what would be the interpretation of the result?
  
  On the other hand, unlike for probabilities, the multiplication with constants does indeed seem unproblematic for odds. (Or the addition of constants for logs.) E.g. “doubling” some odds makes always sense due to them being unbounded from above, while doubling probabilities is not always possible. And when it is, it is questionable whether it has any sensible interpretation.
  
  But the way Arbital and Eliezer handle it doesn’t actually make use of this fact. They instead treat the likelihood ratio (or its logarithm) as evidence strength. But, as I said, the likelihood ratio is actually a ratio of probabilities, not of odds, in which case the interpretation as evidence strength is shaky. The likelihood ratio assumes that doubling a small probability of the evidence constitutes the same evidence strength as doubling a relatively large one, which seems not right.
  
  As a formal example, assume the hypothesis $H$ doubles the probability of evidence $E_{1}$ compared to $\neg H$ . That is, we have the likelihood ratio $\frac{P (E_{1} | H)}{P (E_{1} | \neg H)} = 2$ . Since ${log}_{2} (2) = 1$ , $E_{1}$ is interpreted to constitute 1 bit of evidence in favor of $H$ .
  
  Then assume we also have some evidence $E_{2}$ that is doubled by $H$ compared to $\neg H$ . So $E_{2}$ is interpreted to also be 1 bit of evidence in favor of $H$ .
  
  Does this mean both cases involve equal evidence strength? Arguably no. For example, the probability of $E_{1}$ may be quite small while the probability of $E_{2}$ may be quite large. This would mean $H$ hardly decreases the probability of $\neg E_{1}$ compared to $\neg H$ , while $H$ strongly decreases the probability of $\neg E_{2}$ compared to $\neg H$ . So $\frac{P (\neg E_{1} | H)}{P (\neg E_{1} | \neg H)} ≫ \frac{P (\neg E_{2} | H)}{P (\neg E_{2} | \neg H)}$ .
  
  So according to the likelihood ratio theory, $E_{1}$ would be moderate (1 bit) evidence for $H$ , and $E_{2}$ would be equally moderate evidence for $H$ , but $\neg E_{1}$ would be very weak evidence against $H$ while $\neg E_{2}$ would be very strong evidence against $H$ .
  
  That seems implausible. Arguably $E_{2}$ is here much stronger evidence for $H$ than $E_{1}$ .
  
  Here is a more concrete example:
  
  $H$ = The patient actually suffers from Diseasitis.
  
  $E_{1}$ = The patient suffers from Diseasitis according to test 1.
  
  $E_{2}$ = The patient suffers from Diseasitis according to test 2.
  
  $P (E_{1} | H) = 0.02$
  
  $P (E_{1} | \neg H) = 0.01$
  
  $P (E_{2} | H) = 0.98$
  
  $P (E_{2} | \neg H) = 0.49$
  
  Log likelihood ratio $E_{1}$ : ${log}_{2} (\frac{P (E_{1} | H)}{P (E_{1} | \neg H)}) = 1 bit$
  
  Log likelihood ratio $E_{2}$ : ${log}_{2} (\frac{P (E_{2} | H)}{P (E_{2} | \neg H)}) = 1 bit$
  
  So this says both tests represent equally strong evidence.
  
  What if we instead take the ratio of conditional odds, instead of the ratio of conditional probabilities (as in the likelihood ratio)?
  
  $O = \frac{P}{1 - P}$
  
  Log odds ratio $E_{1}$ : ${log}_{2} (\frac{O (E_{1} | H)}{O (E_{1} | \neg H)}) = 1.0146 bits$
  
  Log odds ratio $E_{2}$ : ${log}_{2} (\frac{O (E_{2} | H)}{O (E_{2} | \neg H)}) = 5.6724 bits$
  
  So the odds ratios are actually pretty different. Unlike the likelihood ratio, the odds ratio agrees with my argument that $E_{2}$ is significantly stronger evidence than $E_{1}$ .