What precisely does “There is a 70% chance of rain tomorrow” mean?
If you offered me the choice of two “lottery tickets”, one of which paid $30 if it rained tomorrow and one of which paid $70 if it didn’t, I wouldn’t care which one I took.
That surely can’t be the right general answer, because the relationship between your attitude to getting $30 and to getting $70 will depend on your wealth now. (And also on your tolerance for risk, but you might be willing to argue that risk aversion is always irrational except in so far as it derives from diminishing marginal utility.)
You could switch from dollars to utilons, but then I think you have a different problem: we don’t have direct access to our utility functions, and I think the best techniques for figuring them out depend on probability, which is going to lead to trouble if probabilities are defined in terms of utilities.
My question was about the probability of rain, not about what you would be willing to bet on. Besides, who’s that “me”, a perfect rational Homo Economicus or a real person? Offering bets to an idealized concept seems like an iffy idea :-)
“Probability of appreciable rainfall” * “fraction of specified area which will receive it” is 0.7.
Or, I guess more properly it should be an integral over possible rainfall patterns. But “70% of London will definitely see lots of rain, and 30% will see none” and “we have 70% credence that all of London will see lots of rain, and 30% credence that no rain will fall in London” would both be reported as 70% chance of rain in London.
You just replaced the word probability with credence/chance without explaining what’s meant with it on a more basic level.
The people where you complain that they don’t know what probability means also won’t know what credence means.
I think then you evaded the question Lumifer asked. The original post stated I'd like to see people have a clue what a probability actually is. Then Lumifer asked what it actually is. Explaining weather forcasts is besides the main point.
Yes, I wasn’t answering the question as intended. But both kithpendragon and Lumifer were talking about the weather forecast, and it does seem at least vaguely relevant that even if you know exactly what probability is, that’s not sufficient to understand “70% chance of rain”.
One possible answer, related to the concept of calibration, is this: it means that it rained in 70% of the cases when you predicted 70% chance of rain.
You’ve defined your calibration with respect to predicting rain. But I am not interested in calibration, I’m interested in the weather tomorrow. Does the probability of rain tomorrow even exist?
Naively, I would expect it to mean that if you take sufficiently many predictions (i.e. there’s one made every day), and you group them by predicted chance (70%, 80%, etc. at e.g. 10% granularity), then in each bin, the proportion of correct predictions should match the bin’s assigned chance (e.g. between 75% and 85% for the 80% bin). And so given enough predictions, your expected probability for a single prediction coming true should approach the predicted chance. With more predictions, you can make smaller bins (to within 1%, etc).
So, you’re taking the frequentist approach, the probability is the fraction of the times the event happened as n goes to infinity? But tomorrow is unique. It will never repeat again—n is always equal to 1.
And, as mentioned in another reply, calibration and probability are different things.
But tomorrow is unique. It will never repeat again—n is always equal to 1.
The prediction is not unique. I group predictions (with some binning of similar-enough predictions), not days. Then if I’ve seen enough past predictions to be justified that they’re well calibrated, I can use the predicted probability as my subjective probability (or a factor of it).
The trouble with this approach is that it breaks down when we want to describe uncertain events that are unique. The question of who will win the 2016 presidential election is one that we still want to be able to describe with probabilities, even though it doesn’t make great sense to aggregate probabilities across different presidential elections.
In order to explain what a single probability means, instead of what calibration means, you need to describe it as a measure of uncertainty. The three main ‘correctness’ questions then are 1) how well it corresponds to the actual future, 2) how well it corresponds to known clues at the time, and 3) how precisely I’m reporting it.
That’s correct: my approach doesn’t generalize to unique/rare events. The ‘naive’ or frequentist approach seems to work for weather predictions, and creates a simple intuition that’s easier IMO to explain to laymen than more general approaches.
Perhaps I could have better phrased the complaint; I wasn’t attempting to dive into the philosophical. The point was that the meteorologist is not “wrong” if it rains on a 30% chance or if the high temperature is off by a couple of degrees. Meteorologists deal with a lot of uncertainty (that they don’t always communicate to us effectively). People need to understand that a 30% chance of rain only means that it likely won’t rain (roughly 2:1 against). Still wouldn’t hurt to take an umbrella.
As for the philosophical, I’d have to claim that a Probability is a quantitative expression of predictive uncertainty that exists within an informational system such as the human brain or, yes, weather prediction models. Come to think of it, that might actually be helpful for people to understand the weather report. I just don’t trust my coworkers to be able to parse most of those words.
The point was that the meteorologist is not “wrong” if it rains on a 30% chance
Well, is the forecast falsifiable, then? Can it be wrong? How would you know?
Probability is a quantitative expression of predictive uncertainty that exists within an informational system such as the human brain or, yes, weather prediction models.
So the probability exists purely in the map, but not in the territory? I am not sure quantum mechanics would agree.
Is the forecast falsifiable, then? Can it be wrong? How would you know?
Same way you know if other probabilistic prediction systems are “wrong”: keep track of accurate and inaccurate predictions, weighted by confidence levels, and develop model of the system’s reliability. Unreliable systems are probably “wrong” in some way. Individual predictions that express extreme confidence in an outcome that is not observed are “wrong”. But I cannot recall having reason to accuse any meteorologists of either error. (Full disclosure: I don’t care enough to make detailed records.)
I would also point out that the audience adds another level down the predictive rabbit hole. Weather forecasts usually predict for a large area. I’ve observed that weather can be significantly different between Hershey and Harrisburg in Pennsylvania. The two are less than a half-hour apart, and usually have identical forecast conditions. This further confounds the issue by adding the question of who is included in that 30% chance of rain. You could interpret it to mean a high degree of confidence that 30% of the forecast area will see rain. I have not seen an interview with a meteorologist that addressed that particular wrinkle.
So the probability exists purely in the map, but not in the territory? I am not sure quantum mechanics would agree.
Can’t speak on quantum mechanics with much authority, but my suspicion is that there’s something going on that we haven’t yet learned to predict (or maybe don’t have direct access to) on a quantum level. I seem to remember that quantum physics predicts more than [3 space + 1 time] dimensions. Since I don’t appear to have access to these “extra” dimensions, it seems intuitive that I would be as ineffective at predicting events within them as Flatlanders would be at predicting a game of pool as seen from a single slice perpendicular to the table. They might be able to state a likelihood that (for example) the red circle would appear between times T1 and T2 and between points P1 and P2, but without a view of the plane parallel to the table and intersecting with the balls they would really only be making an educated guess. The uncertainty exists in my mind (as limited by my view), not in the game. I suspect something similar is likely true of Physics, though I’m aware that there are plenty of other theories competing with that one. The fact of multiple competing theories is, in itself, evidence that we are missing some important piece of information.
Same way you know if other probabilistic prediction systems are “wrong”
I asked about a single forecast, not about a prediction system (for which, of course, it’s possible to come up with various metrics of accuracy, etc.). Can the forecast of 70% chance of rain tomorrow be wrong, without the quotes? How could you tell without access to the underlying forecasting system?
but my suspicion is that there’s something going on that we haven’t yet learned to predict
So your position is that reality is entirely deterministic, there is no “probability” at all in the territory?
Unfortunately my weather forcast doesn’t tell me it’s between 10 and 15 degrees with 80% probability but the weather forcast for tomorrow is 12 degrees. As such it makes more sense to say it was wrong.
Certainly it is easier to say it was wrong. Meteorologists actually do see the error bars &c., then they dumb it down so most people can grasp what they’re saying. I understand there is ongoing discussion as to what kind of balance is appropriate between being precise and being understandable. Unfortunately, status quo bias seems to be dictating the outcome of that discussion, and much of the information in meteorological models is never provided to the general public as a result.
I think most people would be perfectly able to understand: The temperature is going to be between 10 and 15 degrees instead of the temperature is going to be 12 degrees.
Then the metrologist can use whatever probability he considers to be appropriate.
Unfortunately, status quo bias seems to be dictating the outcome of that discussion
Yes, and the status quo is wrong. It’s makes sense to say it’s wrong. People in charge really do screw up by staying with the status quo. Making excuses for it doesn’t help.
That’s especially true today where I get my weather information from Google or from Windows. In both cases it would be easy to provide a UX interface that allows me to see proper statistics about the weather.
Google knows a lot about me. It could even guess that I want proper statistics.
The status quo is certainly wrong when it comes to the presentation of weather related data. The report is badly oversimplified due to several effects including the (over)estimated gap in understanding of statistics between meteorologists and the general public.
A 30% chance of precipitation is not, however, “wrong” if it does in fact rain. It merely expresses a fairly high degree of uncertainty in the claim “it will/won’t rain today”. The claim that such a report means the meteorologist was wrong (or somehow lying) is the subject of my complaint, not the format of the report itself (which I agree is abysmally deficient).
I’d like to see people have a clue what a probability actually is. I’m tired of hearing how the weather forecast was “wrong”.
Heh. It isn’t that simple.
What precisely does “There is a 70% chance of rain tomorrow” mean?
If you offered me the choice of two “lottery tickets”, one of which paid $30 if it rained tomorrow and one of which paid $70 if it didn’t, I wouldn’t care which one I took.
That surely can’t be the right general answer, because the relationship between your attitude to getting $30 and to getting $70 will depend on your wealth now. (And also on your tolerance for risk, but you might be willing to argue that risk aversion is always irrational except in so far as it derives from diminishing marginal utility.)
You could switch from dollars to utilons, but then I think you have a different problem: we don’t have direct access to our utility functions, and I think the best techniques for figuring them out depend on probability, which is going to lead to trouble if probabilities are defined in terms of utilities.
My question was about the probability of rain, not about what you would be willing to bet on. Besides, who’s that “me”, a perfect rational Homo Economicus or a real person? Offering bets to an idealized concept seems like an iffy idea :-)
“Probability of appreciable rainfall” * “fraction of specified area which will receive it” is 0.7.
Or, I guess more properly it should be an integral over possible rainfall patterns. But “70% of London will definitely see lots of rain, and 30% will see none” and “we have 70% credence that all of London will see lots of rain, and 30% credence that no rain will fall in London” would both be reported as 70% chance of rain in London.
https://en.wikipedia.org/wiki/Probability_of_precipitation
And what does that mean?
You just replaced the word probability with credence/chance without explaining what’s meant with it on a more basic level. The people where you complain that they don’t know what probability means also won’t know what credence means.
I was talking about weather forecasts, not trying to explain probability.
I think then you evaded the question Lumifer asked. The original post stated
I'd like to see people have a clue what a probability actually is.Then Lumifer asked what it actually is. Explaining weather forcasts is besides the main point.Yes, I wasn’t answering the question as intended. But both kithpendragon and Lumifer were talking about the weather forecast, and it does seem at least vaguely relevant that even if you know exactly what probability is, that’s not sufficient to understand “70% chance of rain”.
Okay, I might have been to harsh.
One possible answer, related to the concept of calibration, is this: it means that it rained in 70% of the cases when you predicted 70% chance of rain.
You’ve defined your calibration with respect to predicting rain. But I am not interested in calibration, I’m interested in the weather tomorrow. Does the probability of rain tomorrow even exist?
Naively, I would expect it to mean that if you take sufficiently many predictions (i.e. there’s one made every day), and you group them by predicted chance (70%, 80%, etc. at e.g. 10% granularity), then in each bin, the proportion of correct predictions should match the bin’s assigned chance (e.g. between 75% and 85% for the 80% bin). And so given enough predictions, your expected probability for a single prediction coming true should approach the predicted chance. With more predictions, you can make smaller bins (to within 1%, etc).
So, you’re taking the frequentist approach, the probability is the fraction of the times the event happened as n goes to infinity? But tomorrow is unique. It will never repeat again—n is always equal to 1.
And, as mentioned in another reply, calibration and probability are different things.
The prediction is not unique. I group predictions (with some binning of similar-enough predictions), not days. Then if I’ve seen enough past predictions to be justified that they’re well calibrated, I can use the predicted probability as my subjective probability (or a factor of it).
The trouble with this approach is that it breaks down when we want to describe uncertain events that are unique. The question of who will win the 2016 presidential election is one that we still want to be able to describe with probabilities, even though it doesn’t make great sense to aggregate probabilities across different presidential elections.
In order to explain what a single probability means, instead of what calibration means, you need to describe it as a measure of uncertainty. The three main ‘correctness’ questions then are 1) how well it corresponds to the actual future, 2) how well it corresponds to known clues at the time, and 3) how precisely I’m reporting it.
That’s correct: my approach doesn’t generalize to unique/rare events. The ‘naive’ or frequentist approach seems to work for weather predictions, and creates a simple intuition that’s easier IMO to explain to laymen than more general approaches.
What do you mean?
What Vaniver said: my approach breaks down for unique events. Edited for clarity.
It means that the proportion of meteorological models that predict rain to those that don’t is 7:3. Take an umbrella. ;)
Yeah, that’s an old joke, except it’s told about meteorologists and not models.
But the question of “what a probability actually is” stands. You are not going to argue that it’s a ratio of model outcomes, are you?
Perhaps I could have better phrased the complaint; I wasn’t attempting to dive into the philosophical. The point was that the meteorologist is not “wrong” if it rains on a 30% chance or if the high temperature is off by a couple of degrees. Meteorologists deal with a lot of uncertainty (that they don’t always communicate to us effectively). People need to understand that a 30% chance of rain only means that it likely won’t rain (roughly 2:1 against). Still wouldn’t hurt to take an umbrella.
As for the philosophical, I’d have to claim that a Probability is a quantitative expression of predictive uncertainty that exists within an informational system such as the human brain or, yes, weather prediction models. Come to think of it, that might actually be helpful for people to understand the weather report. I just don’t trust my coworkers to be able to parse most of those words.
Well, is the forecast falsifiable, then? Can it be wrong? How would you know?
So the probability exists purely in the map, but not in the territory? I am not sure quantum mechanics would agree.
Same way you know if other probabilistic prediction systems are “wrong”: keep track of accurate and inaccurate predictions, weighted by confidence levels, and develop model of the system’s reliability. Unreliable systems are probably “wrong” in some way. Individual predictions that express extreme confidence in an outcome that is not observed are “wrong”. But I cannot recall having reason to accuse any meteorologists of either error. (Full disclosure: I don’t care enough to make detailed records.)
I would also point out that the audience adds another level down the predictive rabbit hole. Weather forecasts usually predict for a large area. I’ve observed that weather can be significantly different between Hershey and Harrisburg in Pennsylvania. The two are less than a half-hour apart, and usually have identical forecast conditions. This further confounds the issue by adding the question of who is included in that 30% chance of rain. You could interpret it to mean a high degree of confidence that 30% of the forecast area will see rain. I have not seen an interview with a meteorologist that addressed that particular wrinkle.
Can’t speak on quantum mechanics with much authority, but my suspicion is that there’s something going on that we haven’t yet learned to predict (or maybe don’t have direct access to) on a quantum level. I seem to remember that quantum physics predicts more than [3 space + 1 time] dimensions. Since I don’t appear to have access to these “extra” dimensions, it seems intuitive that I would be as ineffective at predicting events within them as Flatlanders would be at predicting a game of pool as seen from a single slice perpendicular to the table. They might be able to state a likelihood that (for example) the red circle would appear between times T1 and T2 and between points P1 and P2, but without a view of the plane parallel to the table and intersecting with the balls they would really only be making an educated guess. The uncertainty exists in my mind (as limited by my view), not in the game. I suspect something similar is likely true of Physics, though I’m aware that there are plenty of other theories competing with that one. The fact of multiple competing theories is, in itself, evidence that we are missing some important piece of information.
I expect time will tell.
I asked about a single forecast, not about a prediction system (for which, of course, it’s possible to come up with various metrics of accuracy, etc.). Can the forecast of 70% chance of rain tomorrow be wrong, without the quotes? How could you tell without access to the underlying forecasting system?
So your position is that reality is entirely deterministic, there is no “probability” at all in the territory?
I feel that is most likely, yes.
Unfortunately my weather forcast doesn’t tell me it’s between 10 and 15 degrees with 80% probability but the weather forcast for tomorrow is 12 degrees. As such it makes more sense to say it was wrong.
Certainly it is easier to say it was wrong. Meteorologists actually do see the error bars &c., then they dumb it down so most people can grasp what they’re saying. I understand there is ongoing discussion as to what kind of balance is appropriate between being precise and being understandable. Unfortunately, status quo bias seems to be dictating the outcome of that discussion, and much of the information in meteorological models is never provided to the general public as a result.
I think most people would be perfectly able to understand: The temperature is going to be between 10 and 15 degrees instead of the temperature is going to be 12 degrees.
Then the metrologist can use whatever probability he considers to be appropriate.
Yes, and the status quo is wrong. It’s makes sense to say it’s wrong. People in charge really do screw up by staying with the status quo. Making excuses for it doesn’t help.
That’s especially true today where I get my weather information from Google or from Windows. In both cases it would be easy to provide a UX interface that allows me to see proper statistics about the weather.
Google knows a lot about me. It could even guess that I want proper statistics.
The status quo is certainly wrong when it comes to the presentation of weather related data. The report is badly oversimplified due to several effects including the (over)estimated gap in understanding of statistics between meteorologists and the general public.
A 30% chance of precipitation is not, however, “wrong” if it does in fact rain. It merely expresses a fairly high degree of uncertainty in the claim “it will/won’t rain today”. The claim that such a report means the meteorologist was wrong (or somehow lying) is the subject of my complaint, not the format of the report itself (which I agree is abysmally deficient).
Do you think
I was just dumbing things downis generally a valid excuse when people state that you are making wrong statements?I think lying does include an attempt at deception which I agree isn’t there on the part of meteorologists.