I agree with this; the ‘e.g.’ was meant to point toward the most similar theories that have names, not pin down exactly what Eliezer is doing here. I though that it would be better to refer to the class of similar theories here since there is enough uncertainty that we don’t really have details.
endoself
Karma: 1,979
Yeah, this whole line of reasoning fails if you can get to 3^^^3 utilons without creating ~3^^^3 sentients to distribute them among.
Overall I’m having a really surprising amount of difficulty thinking up an example where you have a lot of causal importance but no anthropic counter-evidence.
I’m not sure what you mean. If you use an anthropic theory like what Eliezer is using here (e.g. SSA, UDASSA) then an amount of causal importance that is large compared to the rest of your reference class implies few similar members of the reference class, which is anthropic counter-evidence, so of course it would be impossible to think of an example. Even if nonsentients can contribute to utility, if I can create 3^^^3 utilons using nonsentients, than some other people probably can to, so I don’t have a lot of causal importance compared to them.
Anyway, does “anthropic” even really have anything to do with qualia? The way people talk about it it clearly does, but I’m not sure it even shows up in the definition—a non-sentient optimizer could totally make anthropic updates.
This is the contrapositive of the grandparent. I was saying that if we assume that the reference class is sentients, then nonsentients need to reason using different rules i.e. a different reference class. You are saying that if nonsentients should reason using the same rules, then the reference class cannot comprise only sentients. I actually agree with the latter much more strongly, and I only brought up the former because it seemed similar to the argument you were trying to remember.
There are really two separate questions here, that of how to reason anthropically and that of how magic reality-fluid is distributed. Confusing these is common, since the same sort of considerations affect both of them and since they are both badly understood, though I would say that due to UDT/ADT, we now understand the former much better, while acknowledging the possibility of unknown unknowns. (Our current state of knowledge where we confuse these actually feels a lot like people who have never learnt to separate the descriptive and the normative.)
The way Eliezer presented things in the post, it is not entirely clear which of the two he meant to be responsible for the leverage penalty. It seems like he meant for it to be an epistemic consideration due to anthropic reasoning, but this seems obviously wrong given UDT. In the Tegmark IV model that he describes, the leverage penalty is caused by reality-fluid, but it seems like he only intended that as an analogy. It seems a lot more probable to me though, and it is possible that Eliezer would express uncertainty as to whether the leverage penalty is actually caused by reality-fluid, so that it is a bit more than an analogy. There is also a third mathematically equivalent possibility where the leverage penalty is about values, and we just care less about individual people when there are more of them, but Eliezer obviously does not hold that view.
Maybe I was unclear. I don’t dismiss Y=TL4 as wrong, I ignore it as untestable and therefore useless for justifying anything interesting, like how an AI ought to deal with tiny probabilities of enormous utilities.
He’s not saying that the leverage penalty might be correct because we might live in a certain type of Tegmark IV, he’s saying that the fact that the leverage penalty would be correct if we did live in Tegmark IV + some other assumptions shows (a) that it is a consistent decision procedure and¹ (b) it is the sort of decision procedure that emerges reasonably naturally and is thus a more reasonable hypothesis than if we didn’t know it comes up natuarally like that.
It is possible that it is hard to communicate here since Eliezer is making analogies to model theory, and I would assume that you are not familiar with model theory.
¹ The word ‘and’ isn’t really correct here. It’s very likely that EY means one of (a) and (b), and possibly both.
Pascal’s mugging is less of a problem if your utility function is bounded, and it completely goes away if the bound is reasonably low, since then there just isn’t any amount of utility that would outweight the improbability of the mugger being truthful.
I’m referring to an infinity of possible outcomes, not an infinity of possible choices. This problem still applies if the agent must pick from a finite list of actions.
Specifically, I’m referring to the problem discussed in this paper, which is mostly the same problem as Pascal’s mugging.
If Pascal’s mugger was a force of nature—a new theory of physics, maybe—then the case for keeping to expected utility maximisation may be quite strong.
There’s still the failure of convergence. If the theory that made you think that it would be a good idea to accept Pascal’s mugging tells you to sum an infinite series, and that infinite series diverges, then the theory is wrong.
You still get a probability function without Savage’s P6 and P7, you just don’t get a utility function with codomain the reals, and you don’t get expectations over infinite outcome spaces. If we add real-valued probabilities, for example by assuming Savage’s P6′, you even get finite expectations, assuming I haven’t made an error.
I can find one discussion where, when the question of bounded utility functions came up, Eliezer responded, “[To avert a certain problem] the bound would also have to be substantially less than 3^^^^3.”—but this indicates a misunderstanding of the idea of utility, because utility functions can be arbitrarily (positively) rescaled or recentered. Individual utility “numbers” are not meaningful; only ratios of utility differences.
I think he was assuming a natural scale. After all, you can just pick some everyday-sized utility difference to use as your unit, and measure everytihng on that scale. It wouldn’t really matter what utility difference you pick as long as it is a natural size, since multiplying by 3^^^3 is easily enough for the argument to go through.
Quantum mechanics actually has lead to some study of negative probabilities, though I’m not familiar with the details. I agree that they don’t come up in the standard sort of QM and that they don’t seem helpful here.
IIRC putting all possible observers in the same reference class leads to bizarre conclusions...? I can’t immediately re-derive why that would be.
The only reason that I have ever thought of is that our reference class should intuitively consist of only sentient beings, but that nonsentient beings should still be able to reason. Is this what you were thinking of? Whether it applies in a given context may depend on what exactly you mean by a reference class in that context.
If what he says is true, then there will be 3^^^3 years of life in the universe. Then, assuming this anthropic framework is correct, it’s very unlikely to find yourself at the beginning rather than at any other point in time, so this provides 3^^^3-sized evidence against this scenario.
I don’t know of any set of axioms that imply that you should take expected utilities when considering infinite sets of possible outcomes that do not also imply that the utility function is bounded. If we think that our utility functions are unbounded and we want to use the Solomonoff prior, why are we still taking expectations?
(I suppose because we don’t know how else to aggregate the utilities over possible worlds. Last week, I tried to see how far I could get if I weakened a few of the usual assumptions. I couldn’t really get anywhere interesting because my axioms weren’t strong enough to tell you how to decide in many cases, even when the generalized probabilities and generalized utilities are known.)
Intuitively? Yes, of course I do. I don’t trust that intuition too strongly, but this thought experiment does make me update a lot about the value of a lot of ideas about Pascal’s mugging.
That’s probably true in many cases, but the “mugger” scenario is really designed to test our limits. If 3^^^3 doesn’t work, then probably 3^^^^3 will.
The probability that humans will eventually be capable of creating x utility given that the mugger is capable of creating x utility probably converges to some constant as x goes to infinity. (Of course, this still isn’t a solution as expected utility still doesn’t convege.)
Actually, there is no order of summation in which the sum will converge, since the terms get arbitrary large. The theorem you are thinking of applies to conditionally convergent series, not all divergent series.
I don’t think I’ve been clear. I’m saying that the categories of abstract and concrete objects are themselves generated by experience and are intended to reflect natural categories, and that it’s not useful to think about what abstraction is without thinking about particular abstract objects and what makes us consider them abstract.
I don’t think we really can. The categories of concrete and abstract objects are supposed to carve reality at its joins: I see a chair, I prove a theorem. You can’t really do this sort of analysis without reference to the chairs and the theorems, and if you do make those references, you must have already settled the question of whether a chair is concrete, and a fortiori whether concrete objects exist. The alternative, studying concepts that were originally intended to carve reality at its joins without intending to do so yourself, has historically been unproductive, except to some extent in math.
In other words, I would be comfortable saying that my office chair and the number 3 both plexist (Platonic-exist), whereas my office chair mexists (materially exists) whereas 3 does not.
I agree that this is useful, but it is essential to recognize that these words are just wrapping up our confusion, and that there are other questions that are still left unanswered when we have answered yours. It can sometimes help to determine which things plexist and which mexist, but we still don’t really know what we mean when we say these, and having words for them can sometimes cause us to forget that. (I suppose I should refer to phlogiston here.) I think that Tegmark-platonism is probably a step towards resolving that confusion, but I doubt that any current metaphysical theory that has completed the job; I certainly don’t know of any that doesn’t leave me confused.
If I understand this correctly, I disagree. Modern philosophical platonism means different things by ‘abstract’ than Tegmark’s platonism. In philosophical platonism, I accept your definition that something is abstract if it is causally inert and non-spatiotemporal. For Tegmark, this doesn’t really make sense though, since the universe is causal in the same sense that a mathematical model of a dynamical system is causal, and it is spatiotemporal in the same sense that the mathematical concept of Minkowski spacetime is spatiotemporal, since the universe is just (approximately) a dynamical system on (approximately) Minkowski spacetime. The usual definition of an abstract object implies that physical, spatiotemporal objects are not abstract, which contradicts the MUH. I don’t think we really have a precise definition of abstract object that makes sense in Tegmark’s platonism, since something like ‘mathematical structure’ is obviously imprecise.
From If Many-Worlds had Come First:
Obviously this is a parody and Eliezer is making an argument for many worlds. However, this isn’t that far from how the thought experiment is presented in introductory books and even popularizations. Why, then, don’t more people realize that many worlds is correct? Why aren’t tons of bright middle-school children who read science fiction and popular science spontaneously rediscovering many worlds?