I feel like I’ve long underappreciated the importance of introspectability in information & prediction systems.
Say you have a system that produces interesting probabilities pn for various statements. The value that an agent gets from them is not directly correlating to the accuracy of these probabilities, but rather to the expected utility gain they get after using information of these probabilities in corresponding Bayesian-approximating updates. Perhaps more directly, something related to the difference between one’s prior and posterior after updated on pn.
Assuming that prediction systems produce varying levels of quality results, agents will need to know more about these predictions to really optimally update accordingly.
A very simple example would be something like a bunch of coin flips. Say there were 5 coins flipped, I see 3 of them, and I want to estimate the number that were heads. A predictor tells me that their prediction has a mean probability of 40% heads. This is useful, but what would be much more useful is a list of which specific coins the predictor saw and what their values were. Then I could get a much more confident answer; possibly a perfect answer.
Financial markets are very black-box like. Many large changes in company prices never really get explained publicly. My impression is that no one really understands the reasons for many significant market moves.
This seems really suboptimal and I’m sure no one wanted this property to be the case.[1]
Similarly, when trying to model the future of our own prediction capacities, I really don’t think they should be like financial markets in this specific way.
[1] I realize that participants in the market try to keep things hidden, but I mean the specific point that few people think that “Stock Market being a black box” = “A good thing for society.”
In some sense, markets have a particular built-in interpretability: for any trade, someone made that trade, and so there is at least one person who can explain it. And any larger market move is just a combination of such smaller trades.
This is different from things like the huge recommender algorithms running YouTube, where it is not the case that for each recommendation, there is someone who understands that recommendation.
However, the above argument fails in more nuanced cases:
Just because for every trade there’s someone who can explain it, doesn’t mean that there is a particular single person who can explain all trades
Some trades might be made by black-box algorithms
There can be weird “beauty contest” dynamics where two people do something only because the other person did it
Good point, though I think the “more nuanced cases” are very common cases.
The 2010 flash crash seems relevant; it seems like it was caused by chaotic feedback loops with algorithmic components, that as a whole, are very difficult to understand. While that example was particularly algorithmic-induced, other examples also could come from very complex combinations of trades between many players, and when one agent attempts to debug what happened, most of the traders won’t even be available or willing to explain their parts.
The 2007-2008 crisis may have been simpler, but even that has 14 listed causes on Wikipedia and still seems hotly debated.
In comparison, YouTube I think algorithms may be even simpler, though they are still quite messy.
I feel like I’ve long underappreciated the importance of introspectability in information & prediction systems.
Say you have a system that produces interesting probabilities pn for various statements. The value that an agent gets from them is not directly correlating to the accuracy of these probabilities, but rather to the expected utility gain they get after using information of these probabilities in corresponding Bayesian-approximating updates. Perhaps more directly, something related to the difference between one’s prior and posterior after updated on pn.
Assuming that prediction systems produce varying levels of quality results, agents will need to know more about these predictions to really optimally update accordingly.
A very simple example would be something like a bunch of coin flips. Say there were 5 coins flipped, I see 3 of them, and I want to estimate the number that were heads. A predictor tells me that their prediction has a mean probability of 40% heads. This is useful, but what would be much more useful is a list of which specific coins the predictor saw and what their values were. Then I could get a much more confident answer; possibly a perfect answer.
Financial markets are very black-box like. Many large changes in company prices never really get explained publicly. My impression is that no one really understands the reasons for many significant market moves.
This seems really suboptimal and I’m sure no one wanted this property to be the case.[1]
Similarly, when trying to model the future of our own prediction capacities, I really don’t think they should be like financial markets in this specific way.
[1] I realize that participants in the market try to keep things hidden, but I mean the specific point that few people think that “Stock Market being a black box” = “A good thing for society.”
In some sense, markets have a particular built-in interpretability: for any trade, someone made that trade, and so there is at least one person who can explain it. And any larger market move is just a combination of such smaller trades.
This is different from things like the huge recommender algorithms running YouTube, where it is not the case that for each recommendation, there is someone who understands that recommendation.
However, the above argument fails in more nuanced cases:
Just because for every trade there’s someone who can explain it, doesn’t mean that there is a particular single person who can explain all trades
Some trades might be made by black-box algorithms
There can be weird “beauty contest” dynamics where two people do something only because the other person did it
Good point, though I think the “more nuanced cases” are very common cases.
The 2010 flash crash seems relevant; it seems like it was caused by chaotic feedback loops with algorithmic components, that as a whole, are very difficult to understand. While that example was particularly algorithmic-induced, other examples also could come from very complex combinations of trades between many players, and when one agent attempts to debug what happened, most of the traders won’t even be available or willing to explain their parts.
The 2007-2008 crisis may have been simpler, but even that has 14 listed causes on Wikipedia and still seems hotly debated.
In comparison, YouTube I think algorithms may be even simpler, though they are still quite messy.