A final thought that came to mind, regarding the following passage:
It seems possible for person X to predict a fair number of a more epistemically competent person Y’s beliefs—even before person X is as epistemically competent as Y. And in that case, doing so is evidence that person X is moving in the right direction.
I think that that’s is a good and interesting point.
But I imagine there would also be many cases in which X develops an intuitive ability to predict Y’s beliefs quite well in a given set of domains, but in which that ability doesn’t transferring to new domains. It’s possible that this would be because X’s “black box” simulation of Y’s beliefs is more epistemically competent than Y in this new domain. But it seems more likely that Y is somewhat similarly epistemically competent in this new domain as in the old domain, but has to draw on different reasoning processes, knowledge, theories, intuitions, etc., and X’s intuitions aren’t calibrated for how Y is now thinking.
I think we could usefully think of this issue as a question of robustness to distributional shift.
I think the same issue could probably also occur even if X has a more explicit process for predicting Y’s beliefs. E.g., even if X believes they understand what sort of sources of information Y considers and how Y evaluates it and X tries to replicate that (rather than just trying to more intuitively guess what Y will say), the process X uses may not be robust to distributional shift.
But I’d guess that more explicit, less “black box” approaches for predicting what Y will say will tend to either be more robust to distributional shift or more able to fail gracefully, such as recognising that uncertainty is now much higher and there’s a need to think more carefully.
(None of this means I disagree with the quoted passage; I’m just sharing some additional thoughts that came to mind when I read it, which seem relevant and maybe useful.)
This sounds roughly right to me. I think concretely this wouldn’t catch people off guard very often. We have a lot of experience trying to model the thoughts of other people, in large part because we need to do this to communicate with them. I’d feel pretty comfortable basically saying, “I bet I could predict what Stuart will think in areas of Anthropology, but I really don’t know his opinions of British politics”.
If forecasters are calibrated, then on average they shouldn’t be overconfident. It’s expected there will be pockets where they are, but I think the damage caused here isn’t particularly high.
But it seems like you’re just saying the issue I’m gesturing at shouldn’t cause mis-calibration or overconfidence, rather than that it won’t reduce the resolution/accuracy or the practical usefulness of a system based on X predicting what Y will think?
That sounds right. However, I think that being properly calibrated is a really big deal, and a major benefit compared to other approaches.
On the part:
But I’d guess that more explicit, less “black box” approaches for predicting what Y will say will tend to either be more robust to distributional shift or more able to fail gracefully, such as recognising that uncertainty is now much higher and there’s a need to think more carefully.
If there are good additional approaches that are less black-box, I see them ideally being additions to this rough framework. There are methods to encourage discussion and information sharing, including with the Judge / the person’s beliefs who is being predicted.
A final thought that came to mind, regarding the following passage:
I think that that’s is a good and interesting point.
But I imagine there would also be many cases in which X develops an intuitive ability to predict Y’s beliefs quite well in a given set of domains, but in which that ability doesn’t transferring to new domains. It’s possible that this would be because X’s “black box” simulation of Y’s beliefs is more epistemically competent than Y in this new domain. But it seems more likely that Y is somewhat similarly epistemically competent in this new domain as in the old domain, but has to draw on different reasoning processes, knowledge, theories, intuitions, etc., and X’s intuitions aren’t calibrated for how Y is now thinking.
I think we could usefully think of this issue as a question of robustness to distributional shift.
I think the same issue could probably also occur even if X has a more explicit process for predicting Y’s beliefs. E.g., even if X believes they understand what sort of sources of information Y considers and how Y evaluates it and X tries to replicate that (rather than just trying to more intuitively guess what Y will say), the process X uses may not be robust to distributional shift.
But I’d guess that more explicit, less “black box” approaches for predicting what Y will say will tend to either be more robust to distributional shift or more able to fail gracefully, such as recognising that uncertainty is now much higher and there’s a need to think more carefully.
(None of this means I disagree with the quoted passage; I’m just sharing some additional thoughts that came to mind when I read it, which seem relevant and maybe useful.)
This sounds roughly right to me. I think concretely this wouldn’t catch people off guard very often. We have a lot of experience trying to model the thoughts of other people, in large part because we need to do this to communicate with them. I’d feel pretty comfortable basically saying, “I bet I could predict what Stuart will think in areas of Anthropology, but I really don’t know his opinions of British politics”.
If forecasters are calibrated, then on average they shouldn’t be overconfident. It’s expected there will be pockets where they are, but I think the damage caused here isn’t particularly high.
That makes sense to me.
But it seems like you’re just saying the issue I’m gesturing at shouldn’t cause mis-calibration or overconfidence, rather than that it won’t reduce the resolution/accuracy or the practical usefulness of a system based on X predicting what Y will think?
That sounds right. However, I think that being properly calibrated is a really big deal, and a major benefit compared to other approaches.
On the part:
If there are good additional approaches that are less black-box, I see them ideally being additions to this rough framework. There are methods to encourage discussion and information sharing, including with the Judge / the person’s beliefs who is being predicted.