how do we reconcile the ideas that 1) most imaginable expected utility maximizers would drive humanity to extinction, and 2) we humans are still alive, even though every single physical system in existence is mathematically equivalent to some class of flawless expected utility maximizers?
First of all, if everything is mathematically equivalent to an EU maximizer, then saying that something is an EU maximizer no longer represents meaningful knowledge, since it no longer distinguishes between fiction and reality. As Eliezer once beautifully put it:
Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge. The strength of a model is not what it can explain, but what it can’t, for only prohibitions constrain anticipation. If you don’t notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.
So if you are capable of taking any self-consistent physical description of a system, whether it actually conforms to reality or not, and saying that this can be modeled as an EU maximizer set-up, then the fact that something is an EU maximizer no longer constrains expectations by prohibiting certain world-states. Since we do actually have certain expectations related to EU maximizers, this suggests that the discussion you are starting here is primarily semantic and not substantive (as you are using the concept of EU maximizers in a manner that John Wentworth has described as “confused”).
With this in mind, the resolution to the apparent tension between the two statements you included in the section I quoted at the top of this comment is that it[1] implicitly relies on the use of a counting argument without properly considering what compact manifold it arises out of. Put differently, we already know that humans are still alive, which screens off other considerations that are not as narrowly tailored to this fact. When you condition on any piece of knowledge, you explicitly change the probabilities of all other events in a manner that makes them compatible with what you are conditioning. It could not have been any other way. So the space of “imaginable EU maximizers”, after you condition, does not get a uniform subjective probability distribution or anything close to it, but rather one that puts mass only (or, at least, primarily) on those hypotheses that ensure humans would survive until now (since we know that we, in fact, have).
First of all, if everything is mathematically equivalent to an EU maximizer, then saying that something is an EU maximizer no longer represents meaningful knowledge, since it no longer distinguishes between fiction and reality.
I’m confused about your claim. For example, I can model (nearly?) everything with quantum mechanics, so then does calling something a quantum mechanical system not confer meaningful knowledge?
There are physical models which are not based on Quantum Mechanics, and are in fact incompatible with it. For example, to a physicist in the 19th century, a world that functioned on the basis of (very slight modifications of) Newtonian Mechanics and Classical E&M would have seemed very plausible.
The fact that reality turned out not to be this way does not imply the physical theory was internally inconsistent, but rather that it was incompatible with empirical observations that eventually led to the creation of the QM theory. So the point is that you cannot actually model nearly everything in conceptspace with QM, it’s just that reality turns out to be well-approximated by it, while (realistic) fiction like Newtonian Mechanics is not (for example, at the atomic & subatomic level).
This is what makes calling something a QM system an example of meaningful knowledge: it approximates reality better than it does something that is not real, exactly part of Your Strength as a Rationalist. By contrast, whatever story I give you, true or not, can be viewed as flowing from the Texas Sharpshooter Utility Function in exactly the same way that you said reality does:
All you have to do is define a utility function which, at time T, takes in all the relevant context within and around a given physical system, and assigns the highest expected utility to whatever actions that system actually takes to produce its state at time T+1.
So the fact that you “know” something is an EU maximizer, under OP’s definition of that term (which, as I mentioned above, is confused and confusing), does not constrain your expectations in any meaningful way because it does not rule out future world-states (because both true and false predictions are equally compatible with the EU process, as described).
By contrast, knowing something follows QM principles does constrain expectations significantly, as we can design self-consistent models and imagined future world-states which do not follow it (as I mentioned above). For example, the quantization of energy levels, the photoelectric effect, quantum tunneling, quantum entanglement, the anomalous magnetic moment of the electron, specific predictions about the spectra of atoms and molecules, etc., are all predictions given directly by QM; as such, the theory invalidates world-states in which we design proper experiments that do not find all of these.
But there is no articulable future world-state which is ruled out by OP’s conception of EU maximization.
For example, I can model (nearly?) everything with quantum mechanics, so then does calling something a quantum mechanical system not confer meaningful knowledge?
Actually no. Quantum mechanics is pretty well established, and we may suppose that it describes everything (at least, in low gravitational fields). Given that, pointing at a thing and saying “quantum mechanics!” adds no new information. That is not a model. An actual model would allow making predictions about the thing, or at least calculating (not merely fitting to) known properties. There aren’t all that many systems we can do that for. The successes of quantum mechanics, which are many, are found in the systems simple enough that we can.
Quantum mechanics is pretty well established, and we may suppose that it describes everything (at least, in low gravitational fields). Given that, pointing at a thing and saying “quantum mechanics!” adds no new information.
Are you making this argument?
P1: Quantum mechanics is well established.
P2: Quantum mechanics describes everything in low gravitational fields.
C1: So, calling a thing a “quantum system” doesn’t convey any information.
I wouldn’t state P1 and P2 as dogmatically as that, but rounding the uncertainty off to zero, yes. If everything is known to be described by quantum mechanics, pointing at something and saying “this is described by quantum mechanics” adds no new information.
First of all, if everything is mathematically equivalent to an EU maximizer, then saying that something is an EU maximizer no longer represents meaningful knowledge, since it no longer distinguishes between fiction and reality. As Eliezer once beautifully put it:
So if you are capable of taking any self-consistent physical description of a system, whether it actually conforms to reality or not, and saying that this can be modeled as an EU maximizer set-up, then the fact that something is an EU maximizer no longer constrains expectations by prohibiting certain world-states. Since we do actually have certain expectations related to EU maximizers, this suggests that the discussion you are starting here is primarily semantic and not substantive (as you are using the concept of EU maximizers in a manner that John Wentworth has described as “confused”).
With this in mind, the resolution to the apparent tension between the two statements you included in the section I quoted at the top of this comment is that it[1] implicitly relies on the use of a counting argument without properly considering what compact manifold it arises out of. Put differently, we already know that humans are still alive, which screens off other considerations that are not as narrowly tailored to this fact. When you condition on any piece of knowledge, you explicitly change the probabilities of all other events in a manner that makes them compatible with what you are conditioning. It could not have been any other way. So the space of “imaginable EU maximizers”, after you condition, does not get a uniform subjective probability distribution or anything close to it, but rather one that puts mass only (or, at least, primarily) on those hypotheses that ensure humans would survive until now (since we know that we, in fact, have).
By which I am referring to “the feeling that the statements are contradictory,” not “your post.”
I’m confused about your claim. For example, I can model (nearly?) everything with quantum mechanics, so then does calling something a quantum mechanical system not confer meaningful knowledge?
There are physical models which are not based on Quantum Mechanics, and are in fact incompatible with it. For example, to a physicist in the 19th century, a world that functioned on the basis of (very slight modifications of) Newtonian Mechanics and Classical E&M would have seemed very plausible.
The fact that reality turned out not to be this way does not imply the physical theory was internally inconsistent, but rather that it was incompatible with empirical observations that eventually led to the creation of the QM theory. So the point is that you cannot actually model nearly everything in conceptspace with QM, it’s just that reality turns out to be well-approximated by it, while (realistic) fiction like Newtonian Mechanics is not (for example, at the atomic & subatomic level).
This is what makes calling something a QM system an example of meaningful knowledge: it approximates reality better than it does something that is not real, exactly part of Your Strength as a Rationalist. By contrast, whatever story I give you, true or not, can be viewed as flowing from the Texas Sharpshooter Utility Function in exactly the same way that you said reality does:
So the fact that you “know” something is an EU maximizer, under OP’s definition of that term (which, as I mentioned above, is confused and confusing), does not constrain your expectations in any meaningful way because it does not rule out future world-states (because both true and false predictions are equally compatible with the EU process, as described).
By contrast, knowing something follows QM principles does constrain expectations significantly, as we can design self-consistent models and imagined future world-states which do not follow it (as I mentioned above). For example, the quantization of energy levels, the photoelectric effect, quantum tunneling, quantum entanglement, the anomalous magnetic moment of the electron, specific predictions about the spectra of atoms and molecules, etc., are all predictions given directly by QM; as such, the theory invalidates world-states in which we design proper experiments that do not find all of these.
But there is no articulable future world-state which is ruled out by OP’s conception of EU maximization.
Actually no. Quantum mechanics is pretty well established, and we may suppose that it describes everything (at least, in low gravitational fields). Given that, pointing at a thing and saying “quantum mechanics!” adds no new information. That is not a model. An actual model would allow making predictions about the thing, or at least calculating (not merely fitting to) known properties. There aren’t all that many systems we can do that for. The successes of quantum mechanics, which are many, are found in the systems simple enough that we can.
Are you making this argument?
P1: Quantum mechanics is well established.
P2: Quantum mechanics describes everything in low gravitational fields.
C1: So, calling a thing a “quantum system” doesn’t convey any information.
I wouldn’t state P1 and P2 as dogmatically as that, but rounding the uncertainty off to zero, yes. If everything is known to be described by quantum mechanics, pointing at something and saying “this is described by quantum mechanics” adds no new information.