It’s Easy to Overestimate The Degree to which Agents Minimize Prediction Error
I often enjoy variety—in food, television, etc—and observe other humans doing so. Naively, it seems like humans sometimes prefer predictability and sometimes prefer variety.
However: any learning agent, almost no matter its values, will tend to look like it is seeking predictability once it has learned its environment well. It is taking actions it has taken before, and steering toward the environmental states similar to what it always steers for. So, one could understandably reach the conclusion that it is reliability itself which the agent likes.
In other words: if I seem to eat the same foods quite often (despite claiming to like variety), you might conclude that I like familiarity when it’s actually just that I like what I like. I’ve found a set of foods which I particularly enjoy (which I can rotate between for the sake of variety). That doesn’t mean it is familiarity itself which I enjoy.
I’m not denying that mere familiarity has some positive valence for humans; I’m just saying that for arbitrary agents, it seems easy to over-estimate the importance of familiarity in their values, so we should be a bit suspicious about it for humans too. And I’m saying that it seems like humans enjoy surprises sometimes, and there’s evolutionary/machine-learning reasoning to explain why this might be the case.
I’ve replied about surprise, its benefits, and its mechanism a coupletimes now. My theory is that surprise is by itself bad but can be made good by having control systems that expect surprise and send a good signal when surprise is seen. Depending on how this gets weighted, this creates a net positive mixed emotion where surprise is experienced as something good and serves many useful purposes.
I think this mostly dissolves the other points you bring up that I read as contingent on thinking the theory doesn’t predict humans would find variety and surprise good in some circumstances, but if not please let me know what the remaining concerns are in light of this explanation (or possibly object to my explanation of why we expect surprise to sometimes be net good).
I think this mostly dissolves the other points you bring up that I read as contingent on thinking the theory doesn’t predict humans would find variety and surprise good in some circumstances, but if not please let me know what the remaining concerns are in light of this explanation (or possibly object to my explanation of why we expect surprise to sometimes be net good).
Yeah, I noted that I and other humans often seem to enjoy surprise, but I also had a different point I was trying to make—the claim that it makes sense that you’d observe competent agents doing many things which can be explained by minimizing prediction error, no matter what their goals.
But, it isn’t important for you to respond further to this point if you don’t feel it accounts for your observations.
I’ve replied about surprise, its benefits, and its mechanism a couple times now. My theory is that surprise is by itself bad but can be made good by having control systems that expect surprise and send a good signal when surprise is seen. Depending on how this gets weighted, this creates a net positive mixed emotion where surprise is experienced as something good and serves many useful purposes.
I think this mostly dissolves the other points you bring up that I read as contingent on thinking the theory doesn’t predict humans would find variety and surprise good in some circumstances, but if not please let me know what the remaining concerns are in light of this explanation (or possibly object to my explanation of why we expect surprise to sometimes be net good).
Yeah, I noted that I and other humans often seem to enjoy surprise, but I also had a different point I was trying to make—the claim that it makes sense that you’d observe competent agents doing many things which can be explained by minimizing prediction error, no matter what their goals.
But, it isn’t important for you to respond further to this point if you don’t feel it accounts for your observations.