Thank you! That was clarifying especially the explanation of epistemic uncertainty for y.
1. I’ve been thinking about epistemic uncertainty more in terms of ‘possible alternative qualities present’, where
you don’t know the probability of a certain quality being present for x (e.g. what’s the chance of the die having an extended three-sided base?).
or might not even be aware of some of the possible qualities that x might have (e.g. you don’t know a triangular prism die can exist).
2. Your take on epistemic uncertainty for that figure seems to be
you know of x’s possible quality dimensions (e.g. relative lengths and angles of sides at corners).
but given a set configuration of x (e.g. triangular prism with equilateral triangle sides = 1, rectangular lengths = 2 ), you don’t know yet the probabilities of outcomes for y (what’s the probability of landing face up for base1, base2, rect1, rect2, rect3?).
Both seem to fit the definition of epistemic uncertainty. Do correct me here!
Edit: Rough difference in focus: 1. Recognition and Representation vs. 2. Sampling and Prediction
Good point, my example with the figure is lacking in regards to 1 simply because we are assuming that x is known completely and that the observed y are true instances of what we want to measure. And from this I realize that I am confused about when some uncertainties should be called aleagoric or epistemic.
When I think I can correctly point out epistemic uncertainty:
If the y that are observed are not the ones that we actually want then I’d call this uncertainty epistemic. This could be if we are using tired undergrads to count the number of pips of each rolled die and they miscount for some fraction of the dice.
If you haven’t seen similar x before then you have epistemic uncertainty because you have uncertainty about which model or model parameters to use when estimating y. (This is the one I wrote about previously and the one shown in the figure)
My confusion from 1:
If the conditions of the experiment changes. Our undergrads start to pull dice from another bag with an entirely different distribution p(y|x), then we have insufficient knowledge to estimate y and I would call this epistemic uncertainty.
If x is lacking in some information to do good estimates of y. x is the color of the die and when we have thrown enough dice from our experimental distribution we get a good estimate of p(y|x) and our uncertainty doesn’t increase with more rolls, which makes me think that it is aleatoric uncertainty. But on the other hand x is not sufficient to spot when we have a new type of die (see previous point) and if we knew more about the dice we could do better estimates which makes me think that it is epistemic uncertainty.
You bring up a good point in 1 and I agree that this feels like it should be epistemic uncertainty, but at some point the boundary between inherent uncertainty in the process and uncertainty from knowing too little about the process becomes vague to me and I can’t really tell when a process is aleatoric or epistemic.
I also noticed I was confused. Feels like we’re at least disentangling cases and making better distinctions here. BTW, just realised that a problem with my triangular prism example is that theoretically no willrectangular side can face up parallel to the floor at the same time, just two at 60º angles).
But on the other hand x is not sufficient to spot when we have a new type of die (see previous point) and if we knew more about the dice we could do better estimates which makes me think that it is epistemic uncertainty.
This is interesting. This seems to ask the question ‘Is a change in the quality of x like colour actually causal to outcomes y?’ Difficulty here is that you can never fully be certain empirically, just get closer to [change in roll probability] for [limit number of rolls → infinity] = 0.
Aleatoric uncertainty is from inherent stochasticity and does not reduce with more data.
Epistemic uncertainty is from lack of knowledge and/or data and can be further reduced by improving the model with more knowledge and/or data.
However, I found some useful tidbits
Uncertainties are characterized as epistemic, if the modeler sees a possibility to reduce them by gathering more data or by refining models. Uncertainties are categorized as aleatory if the modeler does not foresee the possibility of reducing them. [Aleatory or epistemic? Does it matter?]
Which sources of uncertainty, variables, or probabilities are labelled epistemic and which are labelled aleatory depends upon the mission of the study. [...] One cannot make the distinction between aleatory and epistemic uncertainties purely through physical properties or the experts’ judgments. The same quantity in one study may be treated as having aleatory uncertainty while in another study the uncertainty maybe treated as epistemic. [Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management]
[E]pistemic uncertainty means not being certain what the relevant probability distribution is, and aleatoric uncertainty means not being certain what a random sample drawn from a probability distribution will be. [Uncertainty quantification]
With this my updated view is that our confusion is probably because there is a free parameter in where to draw the line between aleatoric and epistemic uncertainty.
This seems reasonable as more information can always lead to better estimates (at least down to considering wavefunctions I suppose) but in most cases having this kind of information and using it is infeasible and thus having the distinction between aleatoric and epistemic depend on the problem at hand seems reasonable.
This seems to ask the question ‘Is a change in the quality of x like colour actually causal to outcomes y?’
Yes, I think you are right. Usually when modeling you can learn correlations that are useful for predictions but if the correlations are spurious they might disappear when the distributions changes. As such to know if p(y|x) changes from only observing x, then we would probably need that all causal relationships to y are captured in x?
Thank you! That was clarifying especially the explanation of epistemic uncertainty for y.
1. I’ve been thinking about epistemic uncertainty more in terms of ‘possible alternative qualities present’, where
you don’t know the probability of a certain quality being present for x (e.g. what’s the chance of the die having an extended three-sided base?).
or might not even be aware of some of the possible qualities that x might have (e.g. you don’t know a triangular prism die can exist).
2. Your take on epistemic uncertainty for that figure seems to be
you know of x’s possible quality dimensions (e.g. relative lengths and angles of sides at corners).
but given a set configuration of x (e.g. triangular prism with equilateral triangle sides = 1, rectangular lengths = 2 ), you don’t know yet the probabilities of outcomes for y (what’s the probability of landing face up for base1, base2, rect1, rect2, rect3?).
Both seem to fit the definition of epistemic uncertainty. Do correct me here!
Edit: Rough difference in focus:
1. Recognition and Representation
vs.
2. Sampling and Prediction
Good point, my example with the figure is lacking in regards to 1 simply because we are assuming that x is known completely and that the observed y are true instances of what we want to measure. And from this I realize that I am confused about when some uncertainties should be called aleagoric or epistemic.
When I think I can correctly point out epistemic uncertainty:
If the y that are observed are not the ones that we actually want then I’d call this uncertainty epistemic. This could be if we are using tired undergrads to count the number of pips of each rolled die and they miscount for some fraction of the dice.
If you haven’t seen similar x before then you have epistemic uncertainty because you have uncertainty about which model or model parameters to use when estimating y. (This is the one I wrote about previously and the one shown in the figure)
My confusion from 1:
If the conditions of the experiment changes. Our undergrads start to pull dice from another bag with an entirely different distribution p(y|x), then we have insufficient knowledge to estimate y and I would call this epistemic uncertainty.
If x is lacking in some information to do good estimates of y. x is the color of the die and when we have thrown enough dice from our experimental distribution we get a good estimate of p(y|x) and our uncertainty doesn’t increase with more rolls, which makes me think that it is aleatoric uncertainty. But on the other hand x is not sufficient to spot when we have a new type of die (see previous point) and if we knew more about the dice we could do better estimates which makes me think that it is epistemic uncertainty.
You bring up a good point in 1 and I agree that this feels like it should be epistemic uncertainty, but at some point the boundary between inherent uncertainty in the process and uncertainty from knowing too little about the process becomes vague to me and I can’t really tell when a process is aleatoric or epistemic.
I also noticed I was confused. Feels like we’re at least disentangling cases and making better distinctions here.
BTW, just realised that a problem with my triangular prism example is that theoretically no will rectangular side can face up parallel to the floor at the same time, just two at 60º angles).
This is interesting. This seems to ask the question ‘Is a change in the quality of x like colour actually causal to outcomes y?’ Difficulty here is that you can never fully be certain empirically, just get closer to [change in roll probability] for [limit number of rolls → infinity] = 0.
To disentangle the confusion I took a look around about a few different definitions of the concepts. The definitions were mostly the same kind of vague statement of the type:
Aleatoric uncertainty is from inherent stochasticity and does not reduce with more data.
Epistemic uncertainty is from lack of knowledge and/or data and can be further reduced by improving the model with more knowledge and/or data.
However, I found some useful tidbits
With this my updated view is that our confusion is probably because there is a free parameter in where to draw the line between aleatoric and epistemic uncertainty.
This seems reasonable as more information can always lead to better estimates (at least down to considering wavefunctions I suppose) but in most cases having this kind of information and using it is infeasible and thus having the distinction between aleatoric and epistemic depend on the problem at hand seems reasonable.
This is clarifying, thank you!
Good catch
Yes, I think you are right. Usually when modeling you can learn correlations that are useful for predictions but if the correlations are spurious they might disappear when the distributions changes. As such to know if p(y|x) changes from only observing x, then we would probably need that all causal relationships to y are captured in x?