The problem is that the standard justifications of Bayesian probability are in a framework where the facts that you are uncertain about are not in any way affected by whether or not you believe them!
I want to point out that this is not an esoteric abstract problem but a concrete issue that actual humans face all the time. There’s a large class of propositions whose truth value is heavily affected by how much you believe (and by “believe” I mean “alieve”) them—e.g. propositions about yourself like “I am confident” or even “I am attractive”—and I think the LW zeitgeist doesn’t really engage with this. Your beliefs about yourself express themselves in muscle tension which has real effects on your body, and from there leak out in your body language to affect how other people treat you; you are almost always in the state Harry describes in HPMoR of having your cognition constrained by the direct effects of believing things on the world as opposed to just by the effects of actions you take on the basis of your beliefs.
There’s an amusing tie-in here to one of the standard ways to break the prediction market game we used to play at CFAR workshops. At the beginning we claim “the best strategy is to always write down your true probability at any time,” but the argument that’s supposed to establish this has a hidden assumption that the act of doing so doesn’t affect the situation the prediction market is about, and it’s easy to write down prediction markets violating this assumption, e.g. “the last bet on this prediction market will be under 50%.”
I think the LW zeitgeist doesn’t really engage with this.
Really? I feel quite the opposite, unless you’re saying we could do still more. I think LW is actually one of the few communities that take this sort of non-dualism/naturalism in arriving at a probabilistic judgement (and all its meta levels) seriously. We’ve been repeatedly exposed to the fact that Newcomblike problemsareeverywhere since a long time ago, and then relatively recently, with Simler’s wonderful post on crony beliefs (and now, his even more delightful book with Hanson, of course).
ETA: I’m missing quite a few posts that were even older (Wei Dai’s? Drescher’s? yvain had something too IIRC), it’d be nice if someone else who does remember posted them here.
I think your links are a good indication of the way that LW has engaged with a relatively narrow aspect of this, and with a somewhat biased manner. “Crony beliefs” is a good example—starting right from the title, it sets up a dichotomy of “merit beliefs” versus “crony beliefs”, with the not-particularly-subtle-connotation of “merit beliefs are this great thing that models reality and in an ideal world we’d only have merit beliefs, but in the real world, we also have to deal with the fact that it’s useful to have crony beliefs for the purpose of manipulating others and securing social alliances”.
Which… yes, that is one aspect of this. But the more general point of the original post is that there are a wide variety of beliefs which are underdetermined by external reality. It’s not that you intentionally have fake beliefs which out of alignment with the world, it’s that some beliefs are to some extent self-fulfilling, and their truth value just is whatever you decide to believe in. If your deep-level alief is that “I am confident”, then you will be confident; if your deep-level alief is that “I am unconfident”, then you will be that.
Another way of putting it: what is the truth value of the belief “I will go to the beach this evening”? Well, if I go to the beach this evening, then it is true; if I don’t go to the beach this evening, it’s false. Its truth is determined by the actions of the agent, rather than the environment.
The predictive processing thing could be said to take this even further: it hypothesizes that all action is caused by these kinds of self-fulfilling beliefs; on some level our brain believes that we’ll take an action, and then it ends up fulfilling that prediction:
About a third of Surfing Uncertainty is on the motor system, it mostly didn’t seem that interesting to me, and I don’t have time to do it justice here (I might make another post on one especially interesting point). But this has been kind of ignored so far. If the brain is mostly just in the business of making predictions, what exactly is the motor system doing?
Based on a bunch of really excellent experiments that I don’t have time to describe here, Clark concludes: it’s predicting action, which causes the action to happen.
This part is almost funny. Remember, the brain really hates prediction error and does its best to minimize it. With failed predictions about eg vision, there’s not much you can do except change your models and try to predict better next time. But with predictions about proprioceptive sense data (ie your sense of where your joints are), there’s an easy way to resolve prediction error: just move your joints so they match the prediction. So (and I’m asserting this, but see Chapters 4 and 5 of the book to hear the scientific case for this position) if you want to lift your arm, your brain just predicts really really strongly that your arm has been lifted, and then lets the lower levels’ drive to minimize prediction error do the rest.
Under this model, the “prediction” of a movement isn’t just the idle thought that a movement might occur, it’s the actual motor program. This gets unpacked at all the various layers – joint sense, proprioception, the exact tension level of various muscles – and finally ends up in a particular fluid movement
Now, I’ve mostly been talking about cases where the truth of a belief is determined purely by our choices. But as the OP suggests, there are often complex interplays between the agent and the environment. For instance, if you believe that “I will be admitted to Example University if I study hard enough to get in”, then that belief may become self-fulfilling in that it causes you to study hard enough to get in. But at the same time, you may simply not be good enough, so the truth value of this belief is determined both by whether you believe in it, and by whether you actually are good enough.
With regard to the thing about confidence; people usually aren’t just confident in general, they are confident about something in particular. I’m much more confident in my ability to write on a keyboard, than I am in my ability to do brain surgery. You could say that my confidence in my ability to do X, is the probability that I assign to doing X successfully.
And it’s often important that I’m not overconfident. Yes, if I’m really confident in my ability to do something, then other people will give me more respect. But the reason why they do that, is that confidence is actually a bit of a costly signal. So far I’ve said that an agent’s decisions determine the truth-values of many beliefs, but it’s also the other way around: the agent’s beliefs determine the agent’s actions. If I believe myself to be really good at brain surgery even when I’m not, I may be able to talk myself into a situation where I’m allowed to do brain surgery, but the result will be a dead patient. And it’s not going to take many dead patients before people realize I’m a fraud and put me in prison. But if I’m completely deluded and firmly believe myself to be a master brain surgeon, that belief will cause me to continue carrying out brain surgeries, even when it would be better from a self-interested perspective to stop doing that.
So there’s a complicated thing where beliefs have several effects: they determine your predictions about the world and they determine your future actions and they determine the subconscious signals that you send to others. You have an interest in being overconfident for the sake of persuading others, and for the sake of getting yourself to do things, but also in being just-appropriately-confident for the sake of being able to predict the consequences of your own future actions better.
An important framing here is “your beliefs determine your actions, so how do you get the beliefs which cause the best actions”. There have beensomeposts offering tools for belief-modification which had the goal of causing change, but this mostly hasn’t been stated explicitly, and even some of the posts which have offered tools for this (e.g. Nate’s “Dark Arts of Rationality”) have still talked about it being a “Dark Art” thing which is kinda dirty to engage in. Which I think is dangerous, because getting an epistemically correct map is only half of what you need for success, with the “have beliefs which cause you to take the actions that you need to succeed” being the other half that’s just as important to get right. (Except, as noted, they are not two independent things but intertwined with each other in complicated ways.)
There’s a thing MIRI people talk about, about the distinction between “cartesian” and “naturalized” agents: a cartesian agent is something like AIXI that has a “cartesian boundary” separating itself from the environment, so it can try to have accurate beliefs about the environment, then try to take the best actions on the environment given those beliefs. But a naturalized agent, which is what we actually are and what any AI we build actually is, is part of the environment; there is no cartesian boundary. Among other things this means that the environment is too big to fully model, and it’s much less clear what it even means for the agent to contemplate taking different actions. Scott Garrabrant has said that he does not understand what naturalized agency means; among other things this means we don’t have a toy model that deserves to be called “naturalized AIXI.”
There’s a way in which I think the LW zeitgeist treats humans as cartesian agents, and I think fully internalizing that you’re a naturalized agent looks very different, although my concepts and words around this are still relatively nebulous.
I’m confused by this. Sure, your body has involuntary mechanisms that truthfully signal your beliefs to others. But the only reason these mechanisms could exist is to help your genes! Yours specifically! That means you shouldn’t try to override them when your interests coincide with those of your genes. In particular, you shouldn’t force yourself to believe that you’re attractive. Am I missing something?
In particular, you shouldn’t force yourself to believe that you’re attractive.
And I never said this.
But there’s a thing that can happen when someone else gaslights you into believing that you’re unattractive, which makes it true, and you might be interested in undoing that damage, for example.
It seems pretty easy for such mechanisms to be adapted for maximizing reproduction in some ancestral excitement but maladapted for maximizing your preferences in the modern environment.
I think I agree that your point is generally under-considered, especially by the sort of people who compulsively tear down Chesterton’s fences.
What rossry said, but also, why do you expect to be “winning” all arms races here? Genes in other people may have led to development of meme-hacks that you don’t know are actually giving someone else an edge in a zero sum game.
In particular, they might call you fat or stupid or incompetent and you might end up believing it.
I want to point out that this is not an esoteric abstract problem but a concrete issue that actual humans face all the time. There’s a large class of propositions whose truth value is heavily affected by how much you believe (and by “believe” I mean “alieve”) them—e.g. propositions about yourself like “I am confident” or even “I am attractive”—and I think the LW zeitgeist doesn’t really engage with this. Your beliefs about yourself express themselves in muscle tension which has real effects on your body, and from there leak out in your body language to affect how other people treat you; you are almost always in the state Harry describes in HPMoR of having your cognition constrained by the direct effects of believing things on the world as opposed to just by the effects of actions you take on the basis of your beliefs.
There’s an amusing tie-in here to one of the standard ways to break the prediction market game we used to play at CFAR workshops. At the beginning we claim “the best strategy is to always write down your true probability at any time,” but the argument that’s supposed to establish this has a hidden assumption that the act of doing so doesn’t affect the situation the prediction market is about, and it’s easy to write down prediction markets violating this assumption, e.g. “the last bet on this prediction market will be under 50%.”
Really? I feel quite the opposite, unless you’re saying we could do still more. I think LW is actually one of the few communities that take this sort of non-dualism/naturalism in arriving at a probabilistic judgement (and all its meta levels) seriously. We’ve been repeatedly exposed to the fact that Newcomblike problems are everywhere since a long time ago, and then relatively recently, with Simler’s wonderful post on crony beliefs (and now, his even more delightful book with Hanson, of course).
ETA: I’m missing quite a few posts that were even older (Wei Dai’s? Drescher’s? yvain had something too IIRC), it’d be nice if someone else who does remember posted them here.
I think your links are a good indication of the way that LW has engaged with a relatively narrow aspect of this, and with a somewhat biased manner. “Crony beliefs” is a good example—starting right from the title, it sets up a dichotomy of “merit beliefs” versus “crony beliefs”, with the not-particularly-subtle-connotation of “merit beliefs are this great thing that models reality and in an ideal world we’d only have merit beliefs, but in the real world, we also have to deal with the fact that it’s useful to have crony beliefs for the purpose of manipulating others and securing social alliances”.
Which… yes, that is one aspect of this. But the more general point of the original post is that there are a wide variety of beliefs which are underdetermined by external reality. It’s not that you intentionally have fake beliefs which out of alignment with the world, it’s that some beliefs are to some extent self-fulfilling, and their truth value just is whatever you decide to believe in. If your deep-level alief is that “I am confident”, then you will be confident; if your deep-level alief is that “I am unconfident”, then you will be that.
Another way of putting it: what is the truth value of the belief “I will go to the beach this evening”? Well, if I go to the beach this evening, then it is true; if I don’t go to the beach this evening, it’s false. Its truth is determined by the actions of the agent, rather than the environment.
The predictive processing thing could be said to take this even further: it hypothesizes that all action is caused by these kinds of self-fulfilling beliefs; on some level our brain believes that we’ll take an action, and then it ends up fulfilling that prediction:
Now, I’ve mostly been talking about cases where the truth of a belief is determined purely by our choices. But as the OP suggests, there are often complex interplays between the agent and the environment. For instance, if you believe that “I will be admitted to Example University if I study hard enough to get in”, then that belief may become self-fulfilling in that it causes you to study hard enough to get in. But at the same time, you may simply not be good enough, so the truth value of this belief is determined both by whether you believe in it, and by whether you actually are good enough.
With regard to the thing about confidence; people usually aren’t just confident in general, they are confident about something in particular. I’m much more confident in my ability to write on a keyboard, than I am in my ability to do brain surgery. You could say that my confidence in my ability to do X, is the probability that I assign to doing X successfully.
And it’s often important that I’m not overconfident. Yes, if I’m really confident in my ability to do something, then other people will give me more respect. But the reason why they do that, is that confidence is actually a bit of a costly signal. So far I’ve said that an agent’s decisions determine the truth-values of many beliefs, but it’s also the other way around: the agent’s beliefs determine the agent’s actions. If I believe myself to be really good at brain surgery even when I’m not, I may be able to talk myself into a situation where I’m allowed to do brain surgery, but the result will be a dead patient. And it’s not going to take many dead patients before people realize I’m a fraud and put me in prison. But if I’m completely deluded and firmly believe myself to be a master brain surgeon, that belief will cause me to continue carrying out brain surgeries, even when it would be better from a self-interested perspective to stop doing that.
So there’s a complicated thing where beliefs have several effects: they determine your predictions about the world and they determine your future actions and they determine the subconscious signals that you send to others. You have an interest in being overconfident for the sake of persuading others, and for the sake of getting yourself to do things, but also in being just-appropriately-confident for the sake of being able to predict the consequences of your own future actions better.
An important framing here is “your beliefs determine your actions, so how do you get the beliefs which cause the best actions”. There have been some posts offering tools for belief-modification which had the goal of causing change, but this mostly hasn’t been stated explicitly, and even some of the posts which have offered tools for this (e.g. Nate’s “Dark Arts of Rationality”) have still talked about it being a “Dark Art” thing which is kinda dirty to engage in. Which I think is dangerous, because getting an epistemically correct map is only half of what you need for success, with the “have beliefs which cause you to take the actions that you need to succeed” being the other half that’s just as important to get right. (Except, as noted, they are not two independent things but intertwined with each other in complicated ways.)
Yes, this.
There’s a thing MIRI people talk about, about the distinction between “cartesian” and “naturalized” agents: a cartesian agent is something like AIXI that has a “cartesian boundary” separating itself from the environment, so it can try to have accurate beliefs about the environment, then try to take the best actions on the environment given those beliefs. But a naturalized agent, which is what we actually are and what any AI we build actually is, is part of the environment; there is no cartesian boundary. Among other things this means that the environment is too big to fully model, and it’s much less clear what it even means for the agent to contemplate taking different actions. Scott Garrabrant has said that he does not understand what naturalized agency means; among other things this means we don’t have a toy model that deserves to be called “naturalized AIXI.”
There’s a way in which I think the LW zeitgeist treats humans as cartesian agents, and I think fully internalizing that you’re a naturalized agent looks very different, although my concepts and words around this are still relatively nebulous.
I’m confused by this. Sure, your body has involuntary mechanisms that truthfully signal your beliefs to others. But the only reason these mechanisms could exist is to help your genes! Yours specifically! That means you shouldn’t try to override them when your interests coincide with those of your genes. In particular, you shouldn’t force yourself to believe that you’re attractive. Am I missing something?
And I never said this.
But there’s a thing that can happen when someone else gaslights you into believing that you’re unattractive, which makes it true, and you might be interested in undoing that damage, for example.
It seems pretty easy for such mechanisms to be adapted for maximizing reproduction in some ancestral excitement but maladapted for maximizing your preferences in the modern environment.
I think I agree that your point is generally under-considered, especially by the sort of people who compulsively tear down Chesterton’s fences.
What rossry said, but also, why do you expect to be “winning” all arms races here? Genes in other people may have led to development of meme-hacks that you don’t know are actually giving someone else an edge in a zero sum game.
In particular, they might call you fat or stupid or incompetent and you might end up believing it.