There is a third use of Bayesianism, the way that sophisticated economists and political scientists use it: as a useful fiction for modeling agents who try to make good decisions in light of their beliefs and preferences. I’d guess that this is useful for AI, too. These will be really complicated systems and we don’t know much about their details yet, but it will plausibly be reasonable to model them as “trying to make good decisions in light of their beliefs and preferences”.
Perhaps a fourth use is that we might actively want to try to make our systems more like Bayesian reasoners, at least in some cases.
My post was intended to critique these positions too. In particular, the responses I’d give are that:
There are many ways to model agents as “trying to make good decisions in light of their beliefs and preferences”. I expect bayesian ideas to be useful for very simple models, where you can define a set of states to have priors and preferences over. For more complex and interesting models, I think most of the work is done by considering the cognition the agents are doing, and I don’t think bayesianism gives you particular insight into that for the same reasons I don’t think it gives you particular insight into human cognition.
In response to “The Bayesian framework plausibly allows us to see failure modes that are common to many boundedly rational agents”: in general I believe that looking at things from a wide range of perspectives allows you to identify more failure modes—for example, thinking of an agent as a chaotic system might inspire you to investigate adversarial examples. Nevertheless, apart from this sort of inspiration, I think that the bayesian framework is probably harmful when applied to complex systems because it pushes people into using misleading concepts like “boundedly rational” (compare your claim with the claim that a model in which all animals are infinitely large helps us identify properties that are common to “boundedly sized” animals).
“We might actively want to try to make our systems more like Bayesian reasoners”: I expect this not to be a particularly useful approach, insofar as bayesian reasoners don’t do “reasoning”. If we have no good reason to think that explicit utility functions are something that is feasible in practical AGI, except that it’s what ideal bayesian reasoners do, then I want to discourage people from spending their time on that instead of something else.
I don’t think bayesianism gives you particular insight into that for the same reasons I don’t think it gives you particular insight into human cognition
In the areas I focus on, at least, I wouldn’t know where to start if I couldn’t model agents using Bayesian tools. Game-theoretic concepts like social dilemma, equilibrium selection, costly signaling, and so on seem indispensable, and you can’t state these crisply without a formal model of preferences and beliefs. You might disagree that these are useful concepts, but at this point I feel like the argument has to take place at the level of individual applications of Bayesian modeling, rather than a wholesale judgement about Bayesianism.
misleading concepts like “boundedly rational” (compare your claim with the claim that a model in which all animals are infinitely large helps us identify properties that are common to “boundedly sized” animals)
I’m not saying that the idealized model helps us identify properties common to more realistic agents just because it’s idealized. I agree that many idealized models may be useless for their intended purpose. I’m saying that, as it happens, whenever I think of various agentlike systems it strikes me as useful to model those systems in a Bayesian way when reasoning about some of their aspects—even though the details of their architectures may differ a lot.
I didn’t quite understand why you said “boundedly rational” is a misleading concept, I’d be interested to see you elaborate.
if we have no good reason to think that explicit utility functions are something that is feasible in practical AGI
I’m not saying that we should try to design agents who are literally doing expected utility calculations over some giant space of models all the time. My suggestion was that it might be good—for the purpose of attempting to guarantee safe behavior—to design agents which in limited circumstances make decisions by explicitly distilling their preferences and beliefs into utilities and probabilities. It’s not obvious to me that this is intractable. Anyway, I don’t think this point is central to the disagreement.
Game-theoretic concepts like social dilemma, equilibrium selection, costly signaling, and so on seem indispensable
I agree with this. I think I disagree that “stating them crisply” is indispensable.
I wouldn’t know where to start if I couldn’t model agents using Bayesian tools.
To be a little contrarian, I want to note that this phrasing has a certain parallel with the streetlight effect: you wouldn’t know how to look for your keys if you didn’t have the light from the streetlamp. In particular, this is also what someone would say if we currently had no good methods for modelling agents, but bayesian tools were the ones which seemed good.
Anyway, I’d be interested in having a higher-bandwidth conversation with you about this topic. I’ll get in touch :)
My post was intended to critique these positions too. In particular, the responses I’d give are that:
There are many ways to model agents as “trying to make good decisions in light of their beliefs and preferences”. I expect bayesian ideas to be useful for very simple models, where you can define a set of states to have priors and preferences over. For more complex and interesting models, I think most of the work is done by considering the cognition the agents are doing, and I don’t think bayesianism gives you particular insight into that for the same reasons I don’t think it gives you particular insight into human cognition.
In response to “The Bayesian framework plausibly allows us to see failure modes that are common to many boundedly rational agents”: in general I believe that looking at things from a wide range of perspectives allows you to identify more failure modes—for example, thinking of an agent as a chaotic system might inspire you to investigate adversarial examples. Nevertheless, apart from this sort of inspiration, I think that the bayesian framework is probably harmful when applied to complex systems because it pushes people into using misleading concepts like “boundedly rational” (compare your claim with the claim that a model in which all animals are infinitely large helps us identify properties that are common to “boundedly sized” animals).
“We might actively want to try to make our systems more like Bayesian reasoners”: I expect this not to be a particularly useful approach, insofar as bayesian reasoners don’t do “reasoning”. If we have no good reason to think that explicit utility functions are something that is feasible in practical AGI, except that it’s what ideal bayesian reasoners do, then I want to discourage people from spending their time on that instead of something else.
In the areas I focus on, at least, I wouldn’t know where to start if I couldn’t model agents using Bayesian tools. Game-theoretic concepts like social dilemma, equilibrium selection, costly signaling, and so on seem indispensable, and you can’t state these crisply without a formal model of preferences and beliefs. You might disagree that these are useful concepts, but at this point I feel like the argument has to take place at the level of individual applications of Bayesian modeling, rather than a wholesale judgement about Bayesianism.
I’m not saying that the idealized model helps us identify properties common to more realistic agents just because it’s idealized. I agree that many idealized models may be useless for their intended purpose. I’m saying that, as it happens, whenever I think of various agentlike systems it strikes me as useful to model those systems in a Bayesian way when reasoning about some of their aspects—even though the details of their architectures may differ a lot.
I didn’t quite understand why you said “boundedly rational” is a misleading concept, I’d be interested to see you elaborate.
I’m not saying that we should try to design agents who are literally doing expected utility calculations over some giant space of models all the time. My suggestion was that it might be good—for the purpose of attempting to guarantee safe behavior—to design agents which in limited circumstances make decisions by explicitly distilling their preferences and beliefs into utilities and probabilities. It’s not obvious to me that this is intractable. Anyway, I don’t think this point is central to the disagreement.
I agree with this. I think I disagree that “stating them crisply” is indispensable.
To be a little contrarian, I want to note that this phrasing has a certain parallel with the streetlight effect: you wouldn’t know how to look for your keys if you didn’t have the light from the streetlamp. In particular, this is also what someone would say if we currently had no good methods for modelling agents, but bayesian tools were the ones which seemed good.
Anyway, I’d be interested in having a higher-bandwidth conversation with you about this topic. I’ll get in touch :)