Daniel Herrmann

• Daniel Herrmann 31 Dec 2024 17:49 UTC

  4 points

  1

  on: Is “VNM-agent” one of sev­eral op­tions, for what minds can grow up into?

  Thanks for the post. I think there are at least two ways that we could try to look for different rational-mind-patterns, in the way this post suggests. The first is to keep the underlying mathematical framework of options the same (in VNM, the set of gambles, outcomes, etc.), and looks at different patterns of preference/​behaviour/​valuing/​etc. The second is to change the background mathematical framework in order to have some more realistic, or at the very least different idealizing assumptions. Within the different framework we can then explore also different preference structures/​behaviour norms, etc. Here I will focus more on the second approach.

  In particular, I want to point folks towards a decision theory framework that I think has a lot of virtues (no doubt many readers on LW will already be familiar with it). The Jeffrey-Bolker framework provides a concrete example of the kind of alternative “mathematically describable mind-patterns” that the post and clarifying comment talks about. Like the VNM framework, it proves that rational preferences can be represented as expected utility maximization (although things are subtle, as we conditional on acts as opposed to treating them like exogenous probability distributions/​functions/​random objects, so the mathematics is a bit different). But it does so with very different assumptions about agency, and the background conceptual space in which preference and agency operate.

  I have a write-up here of some key differences between Jeffrey-Bolker and Savage (which is ~kind of~ like a subjective probability version of VNM) that I find exciting for an embedded agency point of view. Here are two quick examples. First, VNM requires a very rich domain of preference – typically preference is defined over all probability distributions over conseqeunces. Savage similarly requires that an agent have preferences defined over all functions from states to consequences. This forces agents to rank logically impossible or causally incoherent scenarios. Jeffrey-Bolker instead only requires preferences over propositions closed under logical operations, allowing agents to only evaluate scenarios they consider possible. Second, Savage style approaches require act-state independence—agents can’t think their actions influence the world’s state. Jeffrey-Bolker drops this, letting agents model themselves as part of the world they’re reasoning about. Both differences stem from Jeffrey-Bolker’s core conceptual/​formal innovation: treating acts, states, and consequences as the same type of object in a unified algebra, rather than fundamentally different things.

  Considering the Jeffrey-Bolker framework is valuable in two ways: first, as an alternative ‘stable attractor’ for minds that avoids VNM’s peculiarities, and second, as a framework within which we can precisely express different decision theories like EDT and CDT. This highlights how progress on alternative models of agency requires both exploring different background frameworks for modeling rational choice AND exploring different decision rules within those frameworks. Rather than assuming minds must converge to VNM-style agency as they get smarter, we should actively investigate what shape the background decision context takes for real agents.
  
  For more detail in general and on my take in particular, I recommend:
  
  My writeup with Gerard about Jeffrey-Bolker (linked above as well).
  The first chapter of my thesis, which gives my approach to embedded agency and my take of why Jeffrey-Bolker in particular is a very attractive decision theory.
  This great writeup of different decision theory frameworks by Fishburn that gives a sense of how many different alternatives to VNM there are. Ends with a brief description of Jeffrey-Bolker, but more more detail about earlier decision frameworks.
  
  

• Daniel Herrmann 13 Sep 2022 9:22 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: Hoagy’s comment on: Bridg­ing Ex­pected Utility Max­i­miza­tion and Optimization

  Thank you for this. Yes, the problem is that (in some cases) we think it can sometimes be difficult to specify what the probability distribution would be without the agent. One strategy would be to define some kind of counterfactual distribution that would obtain if there were no agent, but then we need to have some principled way to get this counterfactual (which might be possible). I think this is easier in situations in which the presence of an agent/​optimizer is only one possibility, in which case we have a defined probability distribution, conditional on there not being an agent. Perhaps that is all that matters (I am somewhat partial to this), but then I don’t think of this as giving us a definition of an optimizing system (since, conditional on their being an optimizing system, there would cease to be an optimizing system—for a similar idea, see Vingean Agency).

  I like your suggestions for connecting (1) and (3).
  
  And thanks for the correction!

• Daniel Herrmann 13 Sep 2022 9:11 UTC

  LW: 1 AF: 1

  1

  AF

  in reply to: Rohin Shah’s comment on: Bridg­ing Ex­pected Utility Max­i­miza­tion and Optimization

  Thanks for this. We agree it’s natural to think that a stronger optimizer means less information from seeing the end state, but the question shows up again here. The general tension is that one version of thinking of optimization is something like, the optimizer has a high probability of hitting a narrow target. But the narrowness notion is often what is doing the work in making this seem intuitive, and under seemingly relevant notions of narrowness (how likely is this set of outcomes to be realized), then the set of outcomes we wanted to say is narrow is, in fact, not narrow at all. The lesson we take is that a lot of the ways we want to measure the underlying space rely on choices we make in describing the (size of the) space. If the choices reflect our uncertainty, then we get the puzzle we describe. I don’t see how moving to thinking in terms of entropy would address this. Given that we are working in continuous spaces, I think one way to see that we often makes choices like this, even with entropy, is to look at continuous generalizations of entropy. When we move to the continuous case, things become more subtle. Differential entropy (the most natural generalization) lacks some of the important properties that makes entropy a useful measure of uncertainty (it can be negative, and it is not invariant under continuous coordinate transformations). You can move to relative entropy to try to fix these problems, but this depends on a choice of an underlying measure m. What we see in both these cases is that the generalizations of entropy—both differential and relative—rely on some choice of a way to describe the underlying space (for differential, it is the choice of coordinate system, and for relative, the underlying measure m).