cubefox comments on Mistakes people make when thinking about units

cubefox 26 Jun 2024 23:01 UTC
2 points
0

Log odds, measured in something like “bits of evidence” or “decibels of evidence”, is the natural thing to think of yourself as “counting”. A probability of 100% would be like having infinite positive evidence for a claim and a probability of 0% is like having infinite negative evidence for a claim. Arbital has some math and Eliezer has a good old essay on this.

Odds (and log odds) solve some problems but they unfortunately create others.

For addition and multiplication they at least seem to make things worse. We know that we can add probabilities if they are “mutually exclusive” to get the probability of their disjunction, and we know we can multiply them if they are “independent” to get the probability of their conjunction. But when can we add two odds, or multiply two odds? (Or log odds) And what would be the interpretation of the result?

On the other hand, unlike for probabilities, the multiplication with constants does indeed seem unproblematic for odds. (Or the addition of constants for logs.) E.g. “doubling” some odds makes always sense due to them being unbounded from above, while doubling probabilities is not always possible. And when it is, it is questionable whether it has any sensible interpretation.

But the way Arbital and Eliezer handle it doesn’t actually make use of this fact. They instead treat the likelihood ratio (or its logarithm) as evidence strength. But, as I said, the likelihood ratio is actually a ratio of probabilities, not of odds, in which case the interpretation as evidence strength is shaky. The likelihood ratio assumes that doubling a small probability of the evidence constitutes the same evidence strength as doubling a relatively large one, which seems not right.

As a formal example, assume the hypothesis $H$ doubles the probability of evidence $E_{1}$ compared to $\neg H$ . That is, we have the likelihood ratio $\frac{P (E_{1} | H)}{P (E_{1} | \neg H)} = 2$ . Since ${log}_{2} (2) = 1$ , $E_{1}$ is interpreted to constitute 1 bit of evidence in favor of $H$ .

Then assume we also have some evidence $E_{2}$ that is doubled by $H$ compared to $\neg H$ . So $E_{2}$ is interpreted to also be 1 bit of evidence in favor of $H$ .

Does this mean both cases involve equal evidence strength? Arguably no. For example, the probability of $E_{1}$ may be quite small while the probability of $E_{2}$ may be quite large. This would mean $H$ hardly decreases the probability of $\neg E_{1}$ compared to $\neg H$ , while $H$ strongly decreases the probability of $\neg E_{2}$ compared to $\neg H$ . So $\frac{P (\neg E_{1} | H)}{P (\neg E_{1} | \neg H)} ≫ \frac{P (\neg E_{2} | H)}{P (\neg E_{2} | \neg H)}$ .

So according to the likelihood ratio theory, $E_{1}$ would be moderate (1 bit) evidence for $H$ , and $E_{2}$ would be equally moderate evidence for $H$ , but $\neg E_{1}$ would be very weak evidence against $H$ while $\neg E_{2}$ would be very strong evidence against $H$ .

That seems implausible. Arguably $E_{2}$ is here much stronger evidence for $H$ than $E_{1}$ .

Here is a more concrete example:

$H$ = The patient actually suffers from Diseasitis.

$E_{1}$ = The patient suffers from Diseasitis according to test 1.

$E_{2}$ = The patient suffers from Diseasitis according to test 2.

$P (E_{1} | H) = 0.02$

$P (E_{1} | \neg H) = 0.01$

$P (E_{2} | H) = 0.98$

$P (E_{2} | \neg H) = 0.49$

Log likelihood ratio $E_{1}$ : ${log}_{2} (\frac{P (E_{1} | H)}{P (E_{1} | \neg H)}) = 1 bit$

Log likelihood ratio $E_{2}$ : ${log}_{2} (\frac{P (E_{2} | H)}{P (E_{2} | \neg H)}) = 1 bit$

So this says both tests represent equally strong evidence.

What if we instead take the ratio of conditional odds, instead of the ratio of conditional probabilities (as in the likelihood ratio)?

$O = \frac{P}{1 - P}$

Log odds ratio $E_{1}$ : ${log}_{2} (\frac{O (E_{1} | H)}{O (E_{1} | \neg H)}) = 1.0146 bits$

Log odds ratio $E_{2}$ : ${log}_{2} (\frac{O (E_{2} | H)}{O (E_{2} | \neg H)}) = 5.6724 bits$

So the odds ratios are actually pretty different. Unlike the likelihood ratio, the odds ratio agrees with my argument that $E_{2}$ is significantly stronger evidence than $E_{1}$ .