For instance, say you have a two deals, both with a 98% chance of making $1 Million. One of them has a 2% chance of making nothing, the other has a 2% chance of losing $50 Million. A 95% confidence interval would treat these two identically.
First, these two deals have quite different expected values. Second, I’m not sure that for a binary outcome the concept of a “95% confidence interval” makes any sense to start with.
Propagation of error is to propagation of uncertainty what expected value is to mean; it’s somewhat of a specific focus of that concept.
Huh? Mean is very specific concept, it doesn’t need any more “focus”. Similarly, in stats “expected value” is a well-defined concept. These two terms are connected, but it’s not accurate to say that one is “a specific focus” of another.
While I am reluctant to propose a new term like expected error
Um, do you really believe you are proposing a newterm? You think that up until now no statisticians pondered the question of how close their forecasts are going to be to actual realizations?
I was expecting comments like this, which is one reason the post was mostly a defense.
First, these two deals have quite different expected values. Second, I’m not sure that for a binary outcome the concept of a “95% confidence interval” makes any sense to start with.
I’m not sure if we’re disagreeing here. I agree confidence intervals aren’t great in these cases. In this specific case the EV would be different, but it would be trivial to come up with a case where that doesn’t occur. That said, I would imagine that even if the EV were different, it would be beneficial for the chosen parameter of variation would be better than confidence intervals for these cases.
Huh? Mean is very specific concept, it doesn’t need any more “focus”. Similarly, in stats “expected value” is a well-defined concept. These two terms are connected, but it’s not accurate to say that one is “a specific focus” of another.
I think you’re arguing over my use of the word ‘focus.’ Maybe you define a bit differently. Expected value implies the mean, but is only used in some cases. Or, in every case there is an expected value, mean is used for it, but in many cases of means being used, they are in the context of something other than expected values.
Um, do you really believe you are proposing a new term? You think that up until now no statisticians pondered the question of how close their forecasts are going to be to actual realizations?
Do you think I wasn’t previously knowledgable about those concepts? I was considering writing differences between what I focussed on and them in this article, but assumed that few people would bring that up.
Specifically, ‘forecasting accuracy’ metrics, from what I’ve read, are defined very specifically as being after the fact, not before the fact.
The forecast error (also known as a residual) is the difference between the actual value and the forecast value for the corresponding period.
These metrics are carried out after the final answer is known, typically when the prediction was made as a single point. The similarity is one reason why I suggested that the mean absolute error be used for expected error.
Statisticians (and many other professionals) have obviously pondered many of these same questions and have figured out the main mathematics, as I pointed out above. However, I get the impression that there may be a gap in this specific area. I consider the idea of expected error to be very much in the vein of (applied information economics)[https://en.wikipedia.org/wiki/Applied_information_economics], which I do think has been relatively overlooked for whatever reason.
I wouldn’t say it has been overlooked that much. lukeprog prominently reviewed How to Measure Anything: Finding the Value of Intangibles in Business, and in any case, it’s just a specific methodology for applying decision theory—advice on expert elicitation, calibration, statistical modeling, Value of Information calculations, and sequential decision making.
it would be beneficial for the chosen parameter of variation would be better than confidence intervals for these cases.
I don’t understand.
in every case there is an expected value, mean is used for it
This is a really weird way of putting it. In the context of models, typically you do not have a mean, but you do have an expected value. Or, expected value is what the sample mean converges to as the sample size goes to infinity (with some exceptions).
Specifically, ‘forecasting accuracy’ metrics, from what I’ve read, are defined very specifically as being after the fact, not before the fact.
You need to read more. I recommend Hastie, et al The Elements of Statistical Learning. There is, certainly, such a thing as post factum accuracy metrics, but any predictive model should tell you what it thinks its error would be. I grant you that it’s a subject which can complicated quickly, but even at the basic level when you fit a standard OLS regression you get what’s know as a standard error. It is, subject to some assumptions, your before-the-fact forecast accuracy metric.
One thing to keep in mind is that, just because something already exists somewhere on earth, doesn’t make it useless on LW. The thing that – in theory – makes this site valuable in my experience, is that you have a guarantee of content being high quality if it is being received well. Sure I could study for years and read all content of the sequences from various fields, but I can’t read them all in one place without anything wasting my time in between.
So I don’t think “this has already been figured out in book XX” implies that it isn’t worth reading. Because I won’t go out to read book XX, but I might read this post.
First, these two deals have quite different expected values. Second, I’m not sure that for a binary outcome the concept of a “95% confidence interval” makes any sense to start with.
Huh? Mean is very specific concept, it doesn’t need any more “focus”. Similarly, in stats “expected value” is a well-defined concept. These two terms are connected, but it’s not accurate to say that one is “a specific focus” of another.
Um, do you really believe you are proposing a new term? You think that up until now no statisticians pondered the question of how close their forecasts are going to be to actual realizations?
I was expecting comments like this, which is one reason the post was mostly a defense.
I’m not sure if we’re disagreeing here. I agree confidence intervals aren’t great in these cases. In this specific case the EV would be different, but it would be trivial to come up with a case where that doesn’t occur. That said, I would imagine that even if the EV were different, it would be beneficial for the chosen parameter of variation would be better than confidence intervals for these cases.
I think you’re arguing over my use of the word ‘focus.’ Maybe you define a bit differently. Expected value implies the mean, but is only used in some cases. Or, in every case there is an expected value, mean is used for it, but in many cases of means being used, they are in the context of something other than expected values.
Do you think I wasn’t previously knowledgable about those concepts? I was considering writing differences between what I focussed on and them in this article, but assumed that few people would bring that up.
Specifically, ‘forecasting accuracy’ metrics, from what I’ve read, are defined very specifically as being after the fact, not before the fact.
The forecast error (also known as a residual) is the difference between the actual value and the forecast value for the corresponding period.
These metrics are carried out after the final answer is known, typically when the prediction was made as a single point. The similarity is one reason why I suggested that the mean absolute error be used for expected error.
Statisticians (and many other professionals) have obviously pondered many of these same questions and have figured out the main mathematics, as I pointed out above. However, I get the impression that there may be a gap in this specific area. I consider the idea of expected error to be very much in the vein of (applied information economics)[https://en.wikipedia.org/wiki/Applied_information_economics], which I do think has been relatively overlooked for whatever reason.
I wouldn’t say it has been overlooked that much. lukeprog prominently reviewed How to Measure Anything: Finding the Value of Intangibles in Business, and in any case, it’s just a specific methodology for applying decision theory—advice on expert elicitation, calibration, statistical modeling, Value of Information calculations, and sequential decision making.
I don’t understand.
This is a really weird way of putting it. In the context of models, typically you do not have a mean, but you do have an expected value. Or, expected value is what the sample mean converges to as the sample size goes to infinity (with some exceptions).
You need to read more. I recommend Hastie, et al The Elements of Statistical Learning. There is, certainly, such a thing as post factum accuracy metrics, but any predictive model should tell you what it thinks its error would be. I grant you that it’s a subject which can complicated quickly, but even at the basic level when you fit a standard OLS regression you get what’s know as a standard error. It is, subject to some assumptions, your before-the-fact forecast accuracy metric.
I think we all need to read more, thanks for the book recommendation.
One thing to keep in mind is that, just because something already exists somewhere on earth, doesn’t make it useless on LW. The thing that – in theory – makes this site valuable in my experience, is that you have a guarantee of content being high quality if it is being received well. Sure I could study for years and read all content of the sequences from various fields, but I can’t read them all in one place without anything wasting my time in between.
So I don’t think “this has already been figured out in book XX” implies that it isn’t worth reading. Because I won’t go out to read book XX, but I might read this post.