The basic idea of a trapped prior is purely epistemic. It can happen (in theory) even in someone who doesn’t feel emotions at all. If you gather sufficient evidence that there are no polar bears near you, and your algorithm for combining prior with new experience is just a little off, then you can end up rejecting all apparent evidence of polar bears as fake, and trapping your anti-polar-bear prior. This happens without any emotional component.
I either don’t follow your more general / “purely epistemic” point, or disagree. If a person’s algorithm is doing correct Bayesian epistemology, a low prior of polar bears won’t obscure the accumulating likelihood ratios in favor of polar bears; a given observation will just be classified as “an update in favor of polar bears maybe being a thing, though they’re still very very unlikely even after this datapoint”; priors don’t mess with the direction of the update.
I guess you’re trying to describe some other situation than correct Bayesian updating, when you talk about a person/alien/AI’s algorithm being “just a little off”. But I can’t figure out what kind of “a little off” you are imagining, that would yield this.
My (very non-confident) guess at what is going on with self-reinforcing prejudices/phobias/etc. in humans, is that it involves actively insulating some part of the mind from the data (as you suggest), and that this is not the sort of phenomenon that would happen with an algorithm that didn’t go out of its way to do something like compartmentalization.
If there’s a mechanism you’re proposing that doesn’t require compartmentalization, might you clarify it?
Here’s my current understanding of what Scott meant by “just a little off”.
I think exact Bayesian inference via Solomonoff induction doesn’t run into the trapped prior problem. Unfortunately, bounded agents like us can’t do exact Bayesian inference via Solomonoff induction, since we can only consider a finite set of hypotheses at any given point. I think we try to compensate for this by recognizing that this list of hypotheses is incomplete, and appending it with new hypotheses whenever it seems like our current hypotheses are doing a sufficiently terrible job of explaining the input data.
One side effect is that if the true hypothesis (eg “polar bears are real”) is not among our currently considered hypotheses, but our currently considered hypotheses are doing a sufficiently non-terrible job of explaining the input data (eg if the hypothesis “polar bears aren’t real, but there’s a lot of bad evidence suggesting that they are” is included, and the data is noisy enough that this hypothesis is reasonable), we just never even end up considering the true hypothesis. There wouldn’t be accumulating likelihood ratios in favor of polar bears, because actual polar bears were never considered in the first place.
I think something similar is happening with phobias. For example, for someone with a phobia of dogs, I think the (subconscious, non-declarative) hypothesis “dogs are safe” doesn’t actually get considered until the subject is well into exposure therapy, after which they’ve accumulated enough evidence that’s sufficiently inconsistent with their prior hypotheses of dogs being scary and dangerous that they start considering alternative hypotheses.
In some sense this algorithm is “going out of its way to do something like compartmentalization”, in that it’s actively trying to fit all input data into its current hypotheses (/ “compartments”) until this method no longer works.
I don’t think that’s what’s happening in me or other people. Or at least, I don’t think it’s a full description. One reason I don’t, is that after I’ve e.g. been camping for a long time, with a lot of room for quiet, it becomes easier than it has been to notice that I don’t have to see things the way I’ve been seeing them. My priors become “less stuck”, if you like. I don’t see why that would be, on your (zhukeepa’s) model.
Introspectively, I think it’s more like, that sometimes facing an unknown hypothesis (or rather, a hypothesis that’ll send the rest of my map into unknownness) is too scary to manage to see as a possibility at all.
and your algorithm for combining prior with new experience is just a little off
And while he doesn’t give a more detailed explanation I think it is clear that humans are not Bayesian but our brains use some kind of hack or approximation and that it’s plausible that trapped priors can happen.
I either don’t follow your more general / “purely epistemic” point, or disagree. If a person’s algorithm is doing correct Bayesian epistemology, a low prior of polar bears won’t obscure the accumulating likelihood ratios in favor of polar bears; a given observation will just be classified as “an update in favor of polar bears maybe being a thing, though they’re still very very unlikely even after this datapoint”; priors don’t mess with the direction of the update.
I guess you’re trying to describe some other situation than correct Bayesian updating, when you talk about a person/alien/AI’s algorithm being “just a little off”. But I can’t figure out what kind of “a little off” you are imagining, that would yield this.
My (very non-confident) guess at what is going on with self-reinforcing prejudices/phobias/etc. in humans, is that it involves actively insulating some part of the mind from the data (as you suggest), and that this is not the sort of phenomenon that would happen with an algorithm that didn’t go out of its way to do something like compartmentalization.
If there’s a mechanism you’re proposing that doesn’t require compartmentalization, might you clarify it?
Here’s my current understanding of what Scott meant by “just a little off”.
I think exact Bayesian inference via Solomonoff induction doesn’t run into the trapped prior problem. Unfortunately, bounded agents like us can’t do exact Bayesian inference via Solomonoff induction, since we can only consider a finite set of hypotheses at any given point. I think we try to compensate for this by recognizing that this list of hypotheses is incomplete, and appending it with new hypotheses whenever it seems like our current hypotheses are doing a sufficiently terrible job of explaining the input data.
One side effect is that if the true hypothesis (eg “polar bears are real”) is not among our currently considered hypotheses, but our currently considered hypotheses are doing a sufficiently non-terrible job of explaining the input data (eg if the hypothesis “polar bears aren’t real, but there’s a lot of bad evidence suggesting that they are” is included, and the data is noisy enough that this hypothesis is reasonable), we just never even end up considering the true hypothesis. There wouldn’t be accumulating likelihood ratios in favor of polar bears, because actual polar bears were never considered in the first place.
I think something similar is happening with phobias. For example, for someone with a phobia of dogs, I think the (subconscious, non-declarative) hypothesis “dogs are safe” doesn’t actually get considered until the subject is well into exposure therapy, after which they’ve accumulated enough evidence that’s sufficiently inconsistent with their prior hypotheses of dogs being scary and dangerous that they start considering alternative hypotheses.
In some sense this algorithm is “going out of its way to do something like compartmentalization”, in that it’s actively trying to fit all input data into its current hypotheses (/ “compartments”) until this method no longer works.
I agree an algorithm could do as you describe.
I don’t think that’s what’s happening in me or other people. Or at least, I don’t think it’s a full description. One reason I don’t, is that after I’ve e.g. been camping for a long time, with a lot of room for quiet, it becomes easier than it has been to notice that I don’t have to see things the way I’ve been seeing them. My priors become “less stuck”, if you like. I don’t see why that would be, on your (zhukeepa’s) model.
Introspectively, I think it’s more like, that sometimes facing an unknown hypothesis (or rather, a hypothesis that’ll send the rest of my map into unknownness) is too scary to manage to see as a possibility at all.
Scott says
And while he doesn’t give a more detailed explanation I think it is clear that humans are not Bayesian but our brains use some kind of hack or approximation and that it’s plausible that trapped priors can happen.