if it’s possible to have “pivotal” decisions that affect 3^^^3 people, then it’s also possible to have 3^^^3 people in “normal” situations all making their separate (correlated) decisions, eating 3^^^3 sandwiches, and so the stakes of everything are similarly mathematically big.
Agreed.
This seems to put you in a strange position though: you are not only saying that high-value outcomes are unlikely, but that you have no preferences about them. That is, they aren’t merely impossible-in-reality, they are impossible-in-thought-experiments.
Perhaps I’m being dense, but I don’t follow this point. If I deny that my epistemic probabilities are closed under countable weighted sums, and assert that the hypothesis “you can actually play a St. Petersburg game for n steps” is less likely than it is easy-to-describe (as n gets large), in what sense does that render me unable to consider St. Petersburg games in thought experiments?
How would you learn that there may or may not be a 10^100 future people with our choices as the fulcrum? Why would the same process not generalize? (And if it may happen in the future but not now, is that 0 probability?)
The same process generalizes.
My point was not “it’s especially hard to learn that there are 3^^^3 people with our choices as the fulcrum”. Rather, consider the person who says “but shouldn’t our choices be dominated by our current best guesses about what makes the universe seem most enormous, more or less regardless of how implausibly bad those best guesses seem?”. More concretely, perhaps they say “but shouldn’t we do whatever seems most plausibly likely to satisfy the simulator-gods, because if there are simulator gods and we do please them then we could get mathematically large amounts of utility, and this argument is bad but it’s not 1 in 3^^^3 bad, so.” One of my answers to this is “don’t worry about the 3^^^3 happy people until you believe yourself upstream of 3^^^3 happy people in the analogous fashion to how we currently think we’re upstream of 10^50 happy people”.
And for the record, I agree that “maximize option value, figure out what’s going on, stay sane” is another fine response. (As is “I think you have made an error in assessing your insane plan as having higher EV than business-as-usual”, which is perhaps one argument-step upstream of that.)
I don’t feel too confused about how to act in real life; I do feel somewhat confused about how to formally justify that sort of reasoning.
My personal answer is that infinite universes don’t seem infinitely more important than finite universes, and that 2x bigger universes generally don’t seem 2x as important. (I tentatively feel that amongst reasonably-large universes, value is almost independent of size—while thinking that within any given universe 2x more flourishing is much closer to 2x as valuable.)
That sounds like you’re asserting that the amount of possible flourishing limits to some maximum value (as, eg, the universe gets large enough to implement all possible reasonably-distinct combinations of flourishing civilizations)?
I’m sympathetic to this view. I’m not fully sold, of course. (Example confusion between me and that view: I have conflicting intuitions about whether running an extra identical copy of the same simulated happy people is ~useless or ~twice as good, and as such I’m uncertain about whether tiling copies of all combinations of flourishing civilizations is better in a way that doesn’t decay.)
While we’re listing guesses, a few of my other guesses include:
Naturalism resolves the issue somehow. Like, perhaps the fact that you need to be embedded somewhere inside the world with a long St. Petersburg game drives its probability lower than the length of the sentence “a long St. Petersburg game” in a relevant way, and this phenomenon generalizes, or something. (Presumably this would have to come hand-in-hand with some sort of finitist philosophy, that denies that epistemic probabilities are closed under countable combination, due to your argument above.)
There is a maximum utility, namely “however good the best arrangement of the entire mathematical multiverse could be”, and even if it does wind up being the case that the amount of flourishing you can get per-instantiation fails to converge as space increases, or even if it does turn out that instantiating all the flourishing n times is n times as good, there’s still some maximal number of instantiations that the multiverse is capable of supporting or something, and the maximum utility remains well-defined.
The whole utility-function story is just borked. Like, we already know the concept is philosophically fraught. There’s plausibly a utility number, which describes how good the mathematical multiverse is, but the other multiverses we intuitively want to evaluate are counterfactual, and counterfactual mathematical multiverses are dubious above and beyond the already-dubious mathematical multiverse. Maybe once we’re deconfused about this whole affair, we’ll retreat to somewhere like “utility functions are a useful abstraction on local scales” while having some global theory of a pretty different character.
Some sort of ultrafinitism wins the day, and once we figure out how to be suitably ultrafinitist, we don’t go around wanting countable combinations of epistemic probabilities or worrying too much about particularly big numbers. Like, such a resolution could have a flavor where “Nate’s utilities are unbounded” becomes the sort of thing that infinitists say about Nate, but not the sort of thing a properly operating ultrafinitist says about themselves, and things turn out to work for the ultrafinitists even if the infinitists say their utilities are unbounded or w/e.
To be clear, I haven’t thought about this stuff all that much, and it’s quite plausible to me that someone is less confused than me here. (That said, most accounts that I’ve heard, as far as I’ve managed to understand them, sound less to me like they come from a place of understanding, and more like the speaker has prematurely committed to a resolution.)
One of my answers to this is “don’t worry about the 3^^^3 happy people until you believe yourself upstream of 3^^^3 happy people in the analogous fashion to how we currently think we’re upstream of 10^50 happy people”.
My point was that this doesn’t seem consistent with anything like a leverage penalty.
And for the record, I agree that “maximize option value, figure out what’s going on, stay sane” is another fine response.
My point was that we can say lots about which actions are more or less likely to generate 3^^^3 utility even without knowing how the universe got so large. (And then this appears to have relatively clear implications for our behavior today, e.g. by influencing our best guesses about the degree of moral convergence.)
That sounds like you’re asserting that the amount of possible flourishing limits to some maximum value (as, eg, the universe gets large enough to implement all possible reasonably-distinct combinations of flourishing civilizations)?
In terms of preferences, I’m just saying that it’s not the case that for every universe, there is another possible universe so much bigger that I care only 1% as much about what happens in the smaller universe. If you look at a 10^20 universe and the 10^30 universe that are equally simple, I’m like “I care about what happens in both of those universes. It’s possible I care about the 10^30 universe 2x as much, but it might be more like 1.000001x as much or 1x as much, and it’s not plausible I care 10^10 as much.” That means I care about each individual life less if it happens in a big universe.
This isn’t why I believe the view, but one way you might be able to better sympathize is by thinking: “There is another universe that is like the 10^20 universe but copied 10^10 times. That’s not that much more complex than the 10^20 universe. And in fact total observer counts were already dominated by copies of those universes that were tiled 3^^^3 times, and the description complexity difference between 3^^^3 and 10^10 x 3^^^3 are not very large.” Of course unbounded utilities don’t admit that kind of reasoning, because they don’t admit any kind of reasoning. And indeed, the fact that the expectations diverge seem very closely related to the exact reasoning you would care most about doing in order to actually assess the relative importance of different decisions, so I don’t think the infinity thing is a weird case, it seems absolutely central and I don’t even know how to talk about what the view should be if the infinites didn’t diverge.
I’m not very intuitively drawn to views like “count the distinct experiences,” and I think that in addition to being kind of unappealing those views also have some pretty crazy consequences (at least for all the concrete versions I can think of).
I basically agree that someone who has the opposite view—that for every universe there is a bigger universe that dwarfs its importance—has a more complicated philosophical question and I don’t know the answer. That said, I think it’s plausible they are in the same position as someone who has strong brute intuitions that A>B, B>C, and C>A for some concrete outcomes A, B, C—no amount of philosophical progress will help them get out of the inconsistency. I wouldn’t commit to that pessimistic view, but I’d give it maybe 50/50---I don’t see any reason that there needs to be a satisfying resolution to this kind of paradox.
Perhaps I’m being dense, but I don’t follow this point. If I deny that my epistemic probabilities are closed under countable weighted sums, and assert that the hypothesis “you can actually play a St. Petersburg game for n steps” is less likely than it is easy-to-describe (as n gets large), in what sense does that render me unable to consider St. Petersburg games in thought experiments?
Do you have preferences over the possible outcomes of thought experiments? Does it feel intuitively like they should satisfy dominance principles? If so, it seems like it’s just as troubling that there are thought experiments. Analogously, if I had the strong intuition that A>B>C>A, and someone said “Ah but don’t worry, B could never happen in the real world!” I wouldn’t be like “Great that settles it, no longer feel confused+troubled.”
My point was that this doesn’t seem consistent with anything like a leverage penalty.
I’m not particulalry enthusiastic about “artificial leverage penalties” that manually penalize the hypothesis you can get 3^^^3 happy people by a factor of 1/3^^^3 (and so insofar as that’s what you’re saying, I agree).
From my end, the core of my objection feels more like “you have an extra implicit assumption that lotteries are closed under countable combination, and I’m not sold on that.” The part where I go “and maybe some sufficiently naturalistic prior ends up thinking long St. Petersburg games are ultimately less likely than they are simple???” feels to me more like a parenthetical, and a wild guess about how the weakpoint in your argument could resolve.
(My guess is that you mean something more narrow and specific by “leverage penalty” than I did, and that me using those words caused confusion. I’m happy to retreat to a broader term, that includes things like “big gambles just turn out not to unbalance naturalistic reasoning when you’re doing it properly (eg. b/c finding-yourself-in-the-universe correctly handles this sort of thing somehow)”, if you have one.)
(My guess is that part of the difference in framing in the above paragraphs, and in my original comment, is due to me updating in response to your comments, and retreating my position a bit. Thanks for the points that caused me to update somewhat!)
My point was that we can say lots about which actions are more or less likely to generate 3^^^3 utility even without knowing how the universe got so large.
I agree.
In terms of preferences, I’m just saying...
This seems like a fine guess to me. I don’t feel sold on it, but that could ofc be because you’ve resolved confusions that I have not. (The sort of thing that would persuade me would be you demonstrating at least as much mastery of my own confusions than I possess, and then walking me through the resolution. (Which I say for the purpose of being upfront about why I have not yet updated in favor of this view. In particular, it’s not a request. I’d be happy for more thoughts on it if they’re cheap and you find generating them to be fun, but don’t think this is terribly high-priority.))
That means I care about each individual life less if it happens in a big universe.
I indeed find this counter-intuitive. Hooray for flatly asserting things I might find counter-intuitive!
Let me know if you want me to flail in the direction of confusions that stand between me and what I understand to be your view. The super short version is something like “man, I’m not even sure whether logic or physics comes first, so I get off that train waaay before we get to the Tegmark IV logical multiverse”.
(Also, to be clear, I don’t find UDASSA particularly compelling, mainly b/c of how confused I remain in light of it. Which I note in case you were thinking that the inferential gap you need to span stretches only to UDASSA-town.)
Do you have preferences over the possible outcomes of thought experiments? Does it feel intuitively like they should satisfy dominance principles? If so, it seems like it’s just as troubling that there are thought experiments.
You’ve lost me somewhere. Maybe try backing up a step or two? Why are we talking about thought experiments?
One of my best explicit hypotheses for what you’re saying is “it’s one thing to deny closure of epistemic probabiltiies under countable weighted combination in real life, and another to deny them in thought experiments; are you not concerned that denying them in thought experiments is troubling?”, but this doesn’t seem like a very likely argument for you to be making, and so I mostly suspect I’ve lost the thread.
(I stress again that, from my perspective, the heart of my objection is your implicit assumption that lotteries are closed under countable combination. If you’re trying to object to some other thing I said about leverage penalties, my guess is that I micommunicated my position (perhaps due to a poor choice of words) or shifted my position in response to your previous comments, and that our arguments are now desynched.)
Backing up to check whether I’m just missing something obvious, and trying to sharpen my current objection:
It seems to me that your argument contains a fourth, unlisted assumption, which is that lotteries are closed under countable combination. Do you agree? Am I being daft and missing that, like, some basic weak dominance assumption implies closure of lotteries under countable combination? Assuming I’m not being daft, do you agree that your argument sure seems to leave the door open for people who buy antisymmetry, dominance, and unbounded utilities, but reject countable combination of lotteries?
From my end, the core of my objection feels more like “you have an extra implicit assumption that lotteries are closed under countable combination, and I’m not sold on that.” [...] It seems to me that your argument contains a fourth, unlisted assumption, which is that lotteries are closed under countable combination. Do you agree?
My formal argument is even worse than that: I assume you have preferences over totally arbitrary probability distributions over outcomes!
I don’t think this is unlisted though—right at the beginning I said we were proving theorems about a preference ordering < defined over the space of probability distributions over a space of outcomes Ω. I absolutely think it’s plausible to reject that starting premise (and indeed I suggest that someone with “unbounded utilities” ought to reject this premise in an even more dramatic way).
If you’re trying to object to some other thing I said about leverage penalties, my guess is that I miscommunicated my position
It seems to me that our actual situation (i.e. my actual subjective distribution over possible worlds) is divergent in the same way as the St Petersburg lottery, at least with respect to quantities like expected # of happy people. So I’m less enthusiastic about talking about ways of restricting the space of probability distributions to avoid St Petersburg lotteries. This is some of what I’m getting at in the parent, and I now see that it may not be responsive to your view. But I’ll elaborate a bit anyway.
There are universes with populations of 3↑↑↑3 that seem only 21000 times less likely than our own. It would be very surprising and confusing to learn that not only am I wrong but this epistemic state ought to have been unreachable, that anyone must assign those universes probability at most 1/3↑↑↑3. I’ve heard it argued that you should be confident that the world is giant, based on anthropic views like SIA, but I’ve never heard anyone seriously argue that you should be perfectly confident that the world isn’t giant.
If you agree with me that in fact our current epistemic state looks like a St Petersburg lottery with respect to # of people, then I hope you can sympathize with my lack of enthusiasm.
All that is to say: it may yet be that preferences are defined over a space of probability distributions small enough to evade the argument in the OP. But at that point it seems much more likely that preferences just aren’t defined over probability distributions at all—it seems odd to hold onto probability distributions as the object of preferences while restricting the space of probability distributions far enough that they appear to exclude our current situation.
You’ve lost me somewhere. Maybe try backing up a step or two? Why are we talking about thought experiments?
Suppose that you have some intuition that implies A > B > C > A.
At first you are worried that this intuition must be unreliable. But then you realize that actually B is impossible in reality, so consistency is restored.
I claim that you should be skeptical of the original intuition anyway. We have gotten some evidence that the intuition isn’t really tracking preferences in the way you might have hoped that it was—because if it were correctly tracking preferences it wouldn’t be inconsistent like that.
The fact that B can never come about in reality doesn’t really change the situation, you still would have expected consistently-correct intuitions to yield consistent answers.
(The only way I’d end up forgiving the intuition is if I thought it was correctly tracking the impossibility of B. But in this case I don’t think so. I’m pretty sure my intuition that you should be willing to take a 1% risk in order to double the size of the world isn’t tracking some deep fact that would make certain epistemic states inaccessible.)
(That all said, a mark against an intuition isn’t a reason to dismiss it outright, it’s just one mark against it.)
Ok, cool, I think I see where you’re coming from now.
I don’t think this is unlisted though …
Fair! To a large degree, I was just being daft. Thanks for the clarification.
It seems to me that our actual situation (i.e. my actual subjective distribution over possible worlds) is divergent in the same way as the St Petersburg lottery, at least with respect to quantities like expected # of happy people.
I think this is a good point, and I hadn’t had this thought quite this explicitly myself, and it shifts me a little. (Thanks!)
(I’m not terribly sold on this point myself, but I agree that it’s a crux of the matter, and I’m sympathetic.)
But at that point it seems much more likely that preferences just aren’t defined over probability distributions at all
This might be where we part ways? I’m not sure. A bunch of my guesses do kinda look like things you might describe as “preferences not being defined over probability distributions” (eg, “utility is a number, not a function”). But simultaneously, I feel solid in my ability to use probabliity distributions and utility functions in day-to-day reasoning problems after I’ve chunked the world into a small finite number of possible actions and corresponding outcomes, and I can see a bunch of reasons why this is a good way to reason, and whatever the better preference-formalism turns out to be, I expect it to act a lot like probability distributions and utility functions in the “local” situation after the reasoner has chunked the world.
Like, when someone comes to me and says “your small finite considerations in terms of actions and outcomes are super simplified, and everything goes nutso when we remove all the simplifications and take things to infinity, but don’t worry, sanity can be recovered so long as you (eg) care less about each individual life in a big universe than in a small universe”, then my response is “ok, well, maybe you removed the simplifications in the wrong way? or maybe you took limits in a bad way? or maybe utility is in fact bounded? or maybe this whole notion of big vs small universes was misguided?”
It looks to me like you’re arguing that one should either accept bounded utilities, or reject the probability/utility factorization in normal circumstances, whereas to me it looks like there’s still a whole lot of flex (ex: ‘outcomes’ like “I come back from the store with milk” and “I come back from the store empty-handed” shouldn’t have been treated the same way as ‘outcomes’ like “Tegmark 3 multiverse branch A, which looks like B” and “Conway’s game of life with initial conditions X, which looks like Y”, and something was going wrong in our generalization from the everyday to the metaphysical, and we shouldn’t have been identifying outcomes with universes and expecting preferences to be a function of probability distributions on those universes, but thinking of “returning with milk” as an outcome is still fine).
And maybe you’d say that this is just conceding your point? That when we pass from everyday reasoning about questions like “is there milk at the store, or not?” to metaphysical reasoning like “Conway’s Life, or Tegmark 3?”, we should either give up on unbounded utilities, or give up on thinking of preferences as defined on probability distributions on outcomes? I more-or-less buy that phrasing, with the caveat that I am open to the weak-point being this whole idea that metaphysical universes are outcomes and that probabilities on outcome-collections that large are reasonable objects (rather than the weakpoint being the probablity/utility factorization per se).
it seems odd to hold onto probability distributions as the object of preferences while restricting the space of probability distributions far enough that they appear to exclude our current situation
I agree that would be odd.
One response I have is similar to the above: I’m comfortable using probability distributions for stuff like “does the store have milk or not?” and less comfortable using them for stuff like “Conway’s Life or Tegmark 3?”, and wouldn’t be surprised if thinking of mathematical universes as “outcomes” was a Bad Plan and that this (or some other such philosophically fraught assumption) was the source of the madness.
Also, to say a bit more on why I’m not sold that the current situation is divergent in the St. Petersburg way wrt, eg, amount of Fun: if I imagine someone in Vegas offering me a St. Petersburg gamble, I imagine thinking through it and being like “nah, you’d run out of money too soon for this to be sufficiently high EV”. If you’re like “ok, but imagine that the world actually did look like it could run the gamble infinitely”, my gut sense is “wow, that seems real sus”. Maybe the source of the susness is that eventually it’s just not possible to get twice as much Fun. Or maybe it’s that nobody anywhere is ever in a physical position to reliably double the amount of Fun in the region that they’re able to affect. Or something.
And, I’m sympathetic to the objection “well, you surely shouldn’t assign probability less than <some vanishingly small but nonzero number> that you’re in such a situation!”. And maybe that’s true; it’s definitely on my list of guesses. But I don’t by any means feel forced into that corner. Like, maybe it turns out that the lightspeed limit in our universe is a hint about what sort of universes can be real at all (whatever the heck that turns out to mean), and an agent can’t ever face a St. Petersburgish choice in some suitably general way. Or something. I’m more trying to gesture at how wide the space of possibilities seems to me from my state of confusion, than to make specific counterproposals that I think are competitive.
(And again, I note that the reason I’m not updating (more) towards your apparently-narrower stance, is that I’m uncertain about whether you see a narrower space of possible resolutions on account of being less confused than I am, vs because you are making premature philosophical commitments.)
To be clear, I agree that you need to do something weirder than “outcomes are mathematical universes, preferences are defined on (probability distributions over) those” if you’re going to use unbounded utilities. And again, I note that “utility is bounded” is reasonably high on my list of guesses. But I’m just not all that enthusiastic about “outcomes are mathematical universes” in the first place, so \shrug.
The fact that B can never come about in reality doesn’t really change the situation, you still would have expected consistently-correct intuitions to yield consistent answers.
I think I understand what you’re saying about thought experiments, now. In my own tongue: even if you’ve convinced yourself that you can’t face a St. Petersburg gamble in real life, it still seems like St. Petersburg gambles form a perfectly lawful thought experiment, and it’s at least suspicious if your reasoning procedures would break down facing a perfectly lawful scenario (regardless of whether you happen to face it in fact).
I basically agree with this, and note that, insofar as my confusions resolve in the “unbounded utilities” direction, I expect some sort of account of metaphysical/anthropic/whatever reasoning that reveals St. Petersburg gambles (and suchlike) to be somehow ill-conceived or ill-typed. Like, in that world, what’s supposed to happen when someone is like “but imagine you’re offered a St. Petersburg bet” is roughly the same as what’s supposed to happen when someone’s like “but imagine a physically identical copy of you that lacks qualia”—you’re supposed to say “no”, and then be able to explain why.
(Or, well, you’re always supposed to say “no” to the gamble and be able to explain why, but what’s up for grabs is whether the “why” is “because utility is bounded”, or some other thing, where I at least am confused enough to still have some of my chips on “some other thing”.)
To be explicit, the way that my story continues to shift in response to what you’re saying, is an indication of continued updating & refinement of my position. Yay; thanks.
I expect it to act a lot like probability distributions and utility functions in the “local” situation after the reasoner has chunked the world.
I agree with this: (i) it feels true and would be surprising not to add up to normality, (ii) coherence theorems suggest that any preferences can be represented as probabilities+utilities in the case of finitely many outcomes.
“utility is a number, not a function”
This is my view as well, but you still need to handle the dependence on subjective uncertainty. I think the core thing at issue is whether that uncertainty is represented by a probability distribution (where utility is an expectation).
(Slightly less important: my most naive guess is that the utility number is itself represented as a sum over objects, and then we might use “utility function” to refer to the thing being summed.)
Also, to say a bit more on why I’m not sold that the current situation is divergent in the St. Petersburg way wrt, eg, amount of Fun...
I don’t mean that we face some small chance of encountering a St Petersburg lottery. I mean that when I actually think about the scale of the universe, and what I ought to believe about physics, I just immediately run into St Petersburg-style cases:
It’s unclear whether we can have an extraordinarily long-lived civilization if we reduce entropy consumption to ~0 (e.g. by having a reversible civilization). That looks like at least 5% probability, and would suggest the number of happy lives is much more than 10100 times larger than I might have thought. So does it dominate the expectation?
But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe. Maybe that happens with only 1% probability, but it corresponds to yet bigger civilization. So does that mean we should think that colonizing faster increases the value of the future by 1%, or by 100% since these possibilities are bigger and better and dominate the expectation?
But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it (but a large fraction of them are nevertheless filled with other civilizations like ours). Perhaps that’s 20%. So it actually looks more likely than the “long-lived reversible civilization” and implies more total flourishing. And on those perspectives not going extinct is more important than going faster, for the usual reasons. So does that dominate the calculus instead?
Perhaps rather than having a single set of physical constants, our universe runs every possible set. If that’s 5%, and could stack on top of any of the above while implying another factor of 10100 of scale. And if the standard model had no magic constants maybe this possibility would be 1% instead of 5%. So should I updated by a factor of 5 that we won’t discover that the standard model has fewer magic constants, because then “continuum of possible copies of our universe running in parallel” has only 1% chance instead of 5%?
Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10100 more people, then it starts to seem like even less of a further ask to assume that there were actually 101000 more people instead. So should we assume that multiple of those enhugening assumptions are true (since each one increases values by more than it decreases probability), or just take our favorite and then keep cranking up the numbers larger and larger (with each cranking being more probable than the last and hence more probable than adding a second enhugening assumption)?
Those are very naive physically statements of the possibilities, but the point is that it seems easy to imagine the possibility that populations could be vastly larger than we think “by default”, and many of those possibilities seem to have reasonable chances rather than being vanishingly unlikely. And at face value you might have thought those possibilities were actually action-relevant (e.g. the possibility of exponential returns to resources dominates the EV and means we should rush to colonize after all), but once you actually look at the whole menu, and see how the situation is just obviously paradoxical in every dimension, I think it’s pretty clear that you should cut off this line of thinking.
A bit more precisely: this situation is structurally identical to someone in a St Petersburg paradox shuffling around the outcomes and finding that they can justify arbitrary comparisons because everything has infinite EV and it’s easy to rewrite. That is, we can match each universe U with “U but with long-lived reversible civilizations,” and we find that the long-lived reversible civilizations dominate the calculus. Or we can match each universe U with “U but vast” and find the vast universes dominate the calculus. Or we can match “long-lived reversible civilizations” with “vast” and find that we can ignore long-lived reversible civilizations. It’s just like matching up the outcomes in the St Petersburg paradox in order to show that any outcome dominates itself.
The unbounded utility claim seems precisely like the claim that each of those less-likely-but-larger universes ought to dominate our concern, compared to the smaller-but-more-likely universe we expect by default. And that way of reasoning seems like it leads directly to these contradictions at the very first time you try to apply it to our situation (indeed, I think every time I’ve seen someone use this assumption in a substantive way it has immediately seemed to run into paradoxes, which are so severe that they mean they could as well have reached the opposite conclusion by superficially-equally-valid reasoning).
I totally believe you might end up with a very different way of handling big universes than “bounded utilities,” but I suspect it will also lead to the conclusion that “the plausible prospect of a big universe shouldn’t dominate our concern.” And I’d probably be fine with the result. Once you divorce unbounded utilities from the usual theory about how utilities work, and also divorce them from what currently seems like their main/only implication, I expect I won’t have anything more than a semantic objection.
More specifically and less confidently, I do think there’s a pretty good chance that whatever theory you end up with will agree roughly with the way that I handle big universes—we’ll just use our real probabilities of each of these universes rather than focusing on the big ones in virtue of their bigness, and within each universe we’ll still prefer have larger flourishing populations. I do think that conclusion is fairly uncertain, but I tentatively think it’s more likely we’ll give up on the principle “a bigger civilization is nearly-linearly better within a given universe” than on the principle “a bigger universe is much less than linearly more important.”
And from a practical perspective, I’m not sure what interim theory you use to reason about these things. I suspect it’s mostly academic for you because e.g. you think alignment is a 99% risk of death instead of a 20% risk of death and hence very few other questions about the future matter. But if you ever did find yourself having to reason about humanity’s long-term future (e.g. to assess the value of extinction risk vs faster colonization, or the extent of moral convergence), then it seems like you should use an interim theory which isn’t fanatical about the possibility of big universes—because the fanatical theories just don’t work, and spit out inconsistent results if combined with our current framework. You can also interpret my argument as strongly objecting to the use of unbounded utilities in that interim framework.
(I, in fact, lifted it off of you, a number of years ago :-p)
but you still need to handle the dependence on subjective uncertainty.
Of course. (And noting that I am, perhaps, more openly confused about how to handle the subjective uncertainty than you are, given my confusions around things like logical uncertainty and whether difficult-to-normalize arithmetical expressions meaningfully denote numbers.)
Running through your examples:
It’s unclear whether we can have an extraordinarily long-lived civilization …
I agree. Separately, I note that I doubt total Fun is linear in how much compute is available to civilization; continuity with the past & satisfactory completion of narrative arcs started in the past is worth something, from which we deduce that wiping out civilization and replacing it with another different civilization of similar flourish and with 2x as much space to flourish in, is not 2x as good as leaving the original civilization alone. But I’m basically like “yep, whether we can get reversibly-computed Fun chugging away through the high-entropy phase of the universe seems like an empiricle question with cosmically large swings in utility associated therewith.”
But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe.
This seems fairly plausible to me! For instance, my best guess is that you can get more than 2x the Fun by computing two people interacting than by computing two individuals separately. (Although my best guess is also that this effect diminishes at scale, \shrug.)
By my lights, it sure would be nice to have more clarity on this stuff before needing to decide how much to rush our expansion. (Although, like, 1st world problems.)
But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it
Sure, this is pretty plausible, but (arguendo) it shouldn’t really be factoring into our action analysis, b/c of the part where we can’t reach it. \shrug
Perhaps rather than having a single set of physical constants, our universe runs every possible set.
Sure. And again (arguendo) this doesn’t much matter to us b/c the others are beyond our sphere of influence.
Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10^100 more people, then it starts to seem like even less of a further ask to assume that there were actually 10^1000 more people instead
I think this is where I get off the train (at least insofar as I entertain unbounded-utility hypotheses). Like, our ability to reversibly compute in the high-entropy regime is bounded by our error-correction capabilities, and we really start needing to upend modern physics as I understand it to make the numbers really huge. (Like, maybe 10^1000 is fine, but it’s gonna fall off a cliff at some point.)
I have a sense that I’m missing some deeper point you’re trying to make.
I also have a sense that… how to say… like, suppose someone argued “well, you don’t have 1/∞ probability that “infinite utility” makes sense, so clearly you’ve got to take infinite utilities seriously”. My response would be something like “That seems mixed up to me. Like, on my current understanding, “infinite utility” is meaningless, it’s a confusion, and I just operate day-to-day without worrying about it. It’s not so much that my operating model assigns probability 0 to the proposition “infinite utilities are meaningful”, as that infinite utilities simply don’t fit into my operating model, they don’t make sense, they don’t typecheck. And separately, I’m not yet philosophically mature, and I can give you various meta-probabilities about what sorts of things will and won’t typecheck in my operating model tomorrow. And sure, I’m not 100% certain that we’ll never find a way to rescue the idea of infinite utilities. But that meta-uncertainty doesn’t bleed over into my operating model, and I’m not supposed to ram infinities into a place where they don’t fit just b/c I might modify the type signatures tomorrow.”
When you bandy around plausible ways that the universe could be real large, it doesn’t look obviously divergent to me. Some of the bullets you’re handling are ones that I am just happy to bite, and others involve stuff that I’m not sure I’m even going to think will typecheck, once I understand wtf is going on. Like, just as I’m not compelled by “but you have more than 0% probability that ‘infinite utility’ is meaningful” (b/c it’s mixing up the operating model and my philosophical immaturity), I’m not compelled by “but your operating model, which says that X, Y, and Z all typecheck, is badly divergent”. Yeah, sure, and maybe the resolution is that utilities are bounded, or maybe it’s that my operating model is too permissive on account of my philosophical immaturity. Philosophical immaturity can lead to an operating model that’s too permisive (cf. zombie arguments) just as easily as one that’s too strict.
Like… the nature of physical law keeps seeming to play games like “You have continua!! But you can’t do an arithmetic encoding. There’s infinite space!! But most of it is unreachable. Time goes on forever!! But most of it is high-entropy. You can do reversible computing to have Fun in a high-entropy universe!! But error accumulates.” And this could totally be a hint about how things that are real can’t help but avoid the truly large numbers (never mind the infinities), or something, I don’t know, I’m philisophically immature. But from my state of philosophical immaturity, it looks like this could totally still resolve in a “you were thinking about it wrong; the worst enhugening assumptions fail somehow to typecheck” sort of way.
Trying to figure out the point that you’re making that I’m missing, it sounds like you’re trying to say something like “Everyday reasoning at merely-cosmic scales already diverges, even without too much weird stuff. We already need to bound our utilities, when we shift from looking at the milk in the supermarket to looking at the stars in the sky (nevermind the rest of the mathematical multiverse, if there is such a thing).” Is that about right?
If so, I indeed do not yet buy it. Perhaps spell it out in more detail, for someone who’s suspicious of any appeals to large swaths of terrain that we can’t affect (eg, variants of this universe w/ sufficiently different cosmological constants, at least in the regions where the locals aren’t thinking about us-in-particular); someone who buys reversible computing but is going to get suspicious when you try to drive the error rate to shockingly low lows?
To be clear, insofar as modern cosmic-scale reasoning diverges (without bringing in considerations that I consider suspicious and that I suspect I might later think belong in the ‘probably not meaningful (in the relevant way)’ bin), I do start to feel the vice grips on me, and I expect I’d give bounded utilities another look if I got there.
A side note: IB physicalisms solves at least a large chunk of naturalism/counterfactuals/anthropics but is almost orthogonal to this entire issue (i.e. physicalist loss functions should still be bounded for the same reason cartesian loss functions should be bounded), so I’m pretty skeptical there’s anything in that direction. The only part which is somewhat relevant is: IB physicalists have loss functions that depend on which computations are running so two exact copies of the same thing definitely count as the same and not twice as much (except potentially in some indirect way, such as being involved together in a single more complex computation).
I am definitely entertaining the hypothesis that the solution to naturalism/anthropics is in no way related to unbounded utilities. (From my perspective, IB physicalism looks like a guess that shows how this could be so, rather than something I know to be a solution, ofc. (And as I said to Paul, the observation that would update me in favor of it would be demonstrated mastery of, and unravelling of, my own related confusions.))
Those & others. I flailed towards a bunch of others in my thread w/ Paul. Throwing out some taglines:
“does logic or physics come first???”
“does it even make sense to think of outcomes as being mathematical universes???”
“should I even be willing to admit that the expression “3^^^3″ denotes a number before taking time proportional to at least log(3^^^3) to normalize it?”
“is the thing I care about more like which-computations-physics-instantiates, or more like the-results-of-various-computations??? is there even a difference?”
“how does the fact that larger quantum amplitudes correspond to more magical happening-ness relate to the question of how much more I should care about a simulation running on a computer with wires that are twice as thick???”
Note that these aren’t supposed to be particularly well-formed questions. (They’re more like handles for my own confusions.)
Note that I’m open to the hypothesis that you can resolve some but not others. From my own state of confusion, I’m not sure which issues are interwoven, and it’s plausible to me that you, from a state of greater clarity, can see independences that I cannot.
Note that I’m not asking for you to show me how IB physicalism chooses a consistent set of answers to some formal interpretations of my confusion-handles. That’s the sort of (non-trivial and virtuous!) feat that causes me to rate IB physicalism as a “plausible guess”.
In the specific case of IB physicalism, I’m like “maaaybe? I don’t yet see how to relate this Γ that you suggestively refer to as a ‘map from programs to results’ to a philosophical stance on computation and instantiation that I understand” and “I’m still not sold on the idea of handling non-realizability with inframeasures (on account of how I still feel confused about a bunch of things that inframeasures seem like a plausible guess for how to solve)” and etc.
Maybe at some point I’ll write more about the difference, in my accounting, between plausible guesses and solutions.
Hmm… I could definitely say stuff about, what’s the IB physicalism take on those questions. But this would be what you specifically said you’re not asking me to do. So, from my perspective addressing your confusion seems like a completely illegible task atm. Maybe the explanation you alluded to in the last paragraph would help.
I’d be happy to read it if you’re so inclined and think the prompt would help you refine your own thoughts, but yeah, my anticipation is that it would mostly be updating my (already decent) probability that IB physicalism is a reasonable guess.
A few words on the sort of thing that would update me, in hopes of making it slightly more legible sooner rather than later/never: there’s a difference between giving the correct answer to metaethics (“‘goodness’ refers to an objective (but complicated, and not objectively compelling) logical fact, which was physically shadowed by brains on account of the specifics of natural selection and the ancestral environment”), and the sort of argumentation that, like, walks someone from their confused state to the right answer (eg, Eliezer’s metaethics sequence). Like, the confused person is still in a state of “it seems to me that either morality must be objectively compelling, or nothing truly matters”, and telling them your favorite theory isn’t really engaging with their intuitions. Demonstrating that your favorite theory can give consistent answers to all their questions is something, it’s evidence that you have at least produced a plausible guess. But from their confused perspective, lots of people (including the nihilists, including the Bible-based moral realists) can confidently provide answers that seem superficially consistent.
The compelling thing, at least to me and my ilk, is the demonstration of mastery and the ability to build a path from the starting intuitions to the conclusion. In the case of a person confused about metaethics, this might correspond to the ability to deconstruct the “morality must be objectively compelling, or nothing truly matters” intuition, right in front of them, such that they can recognize all the pieces inside themselves, and with a flash of clarity see the knot they were tying themselves into. At which point you can help them untie the knot, and tug on the strings, and slowly work your way up to the answer.
(The metaethics sequence is, notably, a tad longer than the answer itself.)
(If I were to write this whole concept of solutions-vs-answers up properly, I’d attempt some dialogs that make the above more concrete and less metaphorical, but \shrug.)
In the case of IB physicalism (and IB more generally), I can see how it’s providing enough consistent answers that it counts as a plausible guess. But I don’t see how to operate it to resolve my pre-existing confusions. Like, we work with (infra)measures over ΣR×Φ, and we say some fancy words about how ΣR is our “beliefs about the computations”, but as far as I’ve been able to make out this is just a neato formalism; I don’t know how to get to that endpoint by, like, starting from my own messy intuitions about when/whether/how physical processes reflect some logical procedure. I don’t know how to, like, look inside myself, and find confusions like “does logic or physics come first?” or “do I switch which algorithm I’m instantiating when I drink alcohol?”, and disassemble them into their component parts, and gain new distinctions that show me how the apparent conflicts weren’t true conflicts and all my previous intuitions were coming at things from slightly the wrong angle, and then shift angles and have a bunch of things click into place, and realize that the seeds of the answer were inside me all along, and that the answer is clearly that the universe isn’t really just a physical arrangement of particles (or a wavefunction thereon, w/e), but one of those plus a mapping from syntax-trees to bits (here taking |R|=2). Or whatever the philosophy corresponding to “a hypothesis is a ΣR×Φ” is supposed to be. Like, I understand that it’s a neat formalism that does cool math things, and I see how it can be operated to produce consistent answers to various philosophical questions, but that’s a long shot from seeing it solve the philosophical problems at hand. Or, to say it another way, answering my confusion handles consistently is not nearly enough to get me to take a theory philosophically seriously, like, it’s not enough to convince me that the universe actually has an assignment of syntax-trees to bits in addition to the physical state, which is what it looks to me like I’d need to believe if I actually took IB physicalism seriously.
I don’t think I’m capable of writing something like the metaethics sequence about IB, that’s a job for someone else. My own way of evaluating philosophical claims is more like:
Can we a build an elegant, coherent mathematical theory around the claim?
Does the theory meet reasonable desiderata?
Does the theory play nicely with other theories we have high confidence of?
If there are compelling desiderata the theory doesn’t meet, can we show that meeting them is impossible?
For example, the way I understood objective morality is wrong was by (i) seeing that there’s a coherent theory of agents with any utility function whatsoever (ii) understanding that, in terms of the physical world, “Vanessa’s utility function” is more analogous to “coastline of Africa” than to “fundamental equations of physics”.
I agree that explaining why we have certain intuitions is a valuable source of evidence, but it’s entangled with messy details of human psychology that create a lot of noise. (Notice that I’m not saying you shouldn’t use intuition, obviously intuition is an irreplaceable core part of cognition. I’m saying that explaining intuition using models of the mind, while possible and desirable, is also made difficult by the messy complexity of human minds, which in particular introduces a lot of variables that vary between people.)
Also, I want to comment on your last tagline, just because it’s too tempting:
how does the fact that larger quantum amplitudes correspond to more magical happening-ness relate to the question of how much more I should care about a simulation running on a computer with wires that are twice as thick???
I haven’t written the proofs cleanly yet (because prioritizing other projects atm), but it seems that IB physicalism produces a rather elegant interpretation of QM. Many-worlds turns out to be false. The wavefunction is not “a thing that exists”. Instead, what exists is the outcomes of all possible measurements. The universe samples those outcomes from a distribution that is determined by two properties: (i) the marginal distribution of each measurement has to obey the Born rule (ii) the overall amount of computation done by the universe should be minimal. It follows that, outside of weird thought experiments (i.e. as long as decoherence applies), agents don’t get split into copies and quantum randomness is just ordinary randomness. (Another nice consequence is that Boltzmann brains don’t have qualia.)
Agreed.
Perhaps I’m being dense, but I don’t follow this point. If I deny that my epistemic probabilities are closed under countable weighted sums, and assert that the hypothesis “you can actually play a St. Petersburg game for n steps” is less likely than it is easy-to-describe (as n gets large), in what sense does that render me unable to consider St. Petersburg games in thought experiments?
The same process generalizes.
My point was not “it’s especially hard to learn that there are 3^^^3 people with our choices as the fulcrum”. Rather, consider the person who says “but shouldn’t our choices be dominated by our current best guesses about what makes the universe seem most enormous, more or less regardless of how implausibly bad those best guesses seem?”. More concretely, perhaps they say “but shouldn’t we do whatever seems most plausibly likely to satisfy the simulator-gods, because if there are simulator gods and we do please them then we could get mathematically large amounts of utility, and this argument is bad but it’s not 1 in 3^^^3 bad, so.” One of my answers to this is “don’t worry about the 3^^^3 happy people until you believe yourself upstream of 3^^^3 happy people in the analogous fashion to how we currently think we’re upstream of 10^50 happy people”.
And for the record, I agree that “maximize option value, figure out what’s going on, stay sane” is another fine response. (As is “I think you have made an error in assessing your insane plan as having higher EV than business-as-usual”, which is perhaps one argument-step upstream of that.)
I don’t feel too confused about how to act in real life; I do feel somewhat confused about how to formally justify that sort of reasoning.
That sounds like you’re asserting that the amount of possible flourishing limits to some maximum value (as, eg, the universe gets large enough to implement all possible reasonably-distinct combinations of flourishing civilizations)?
I’m sympathetic to this view. I’m not fully sold, of course. (Example confusion between me and that view: I have conflicting intuitions about whether running an extra identical copy of the same simulated happy people is ~useless or ~twice as good, and as such I’m uncertain about whether tiling copies of all combinations of flourishing civilizations is better in a way that doesn’t decay.)
While we’re listing guesses, a few of my other guesses include:
Naturalism resolves the issue somehow. Like, perhaps the fact that you need to be embedded somewhere inside the world with a long St. Petersburg game drives its probability lower than the length of the sentence “a long St. Petersburg game” in a relevant way, and this phenomenon generalizes, or something. (Presumably this would have to come hand-in-hand with some sort of finitist philosophy, that denies that epistemic probabilities are closed under countable combination, due to your argument above.)
There is a maximum utility, namely “however good the best arrangement of the entire mathematical multiverse could be”, and even if it does wind up being the case that the amount of flourishing you can get per-instantiation fails to converge as space increases, or even if it does turn out that instantiating all the flourishing n times is n times as good, there’s still some maximal number of instantiations that the multiverse is capable of supporting or something, and the maximum utility remains well-defined.
The whole utility-function story is just borked. Like, we already know the concept is philosophically fraught. There’s plausibly a utility number, which describes how good the mathematical multiverse is, but the other multiverses we intuitively want to evaluate are counterfactual, and counterfactual mathematical multiverses are dubious above and beyond the already-dubious mathematical multiverse. Maybe once we’re deconfused about this whole affair, we’ll retreat to somewhere like “utility functions are a useful abstraction on local scales” while having some global theory of a pretty different character.
Some sort of ultrafinitism wins the day, and once we figure out how to be suitably ultrafinitist, we don’t go around wanting countable combinations of epistemic probabilities or worrying too much about particularly big numbers. Like, such a resolution could have a flavor where “Nate’s utilities are unbounded” becomes the sort of thing that infinitists say about Nate, but not the sort of thing a properly operating ultrafinitist says about themselves, and things turn out to work for the ultrafinitists even if the infinitists say their utilities are unbounded or w/e.
To be clear, I haven’t thought about this stuff all that much, and it’s quite plausible to me that someone is less confused than me here. (That said, most accounts that I’ve heard, as far as I’ve managed to understand them, sound less to me like they come from a place of understanding, and more like the speaker has prematurely committed to a resolution.)
My point was that this doesn’t seem consistent with anything like a leverage penalty.
My point was that we can say lots about which actions are more or less likely to generate 3^^^3 utility even without knowing how the universe got so large. (And then this appears to have relatively clear implications for our behavior today, e.g. by influencing our best guesses about the degree of moral convergence.)
In terms of preferences, I’m just saying that it’s not the case that for every universe, there is another possible universe so much bigger that I care only 1% as much about what happens in the smaller universe. If you look at a 10^20 universe and the 10^30 universe that are equally simple, I’m like “I care about what happens in both of those universes. It’s possible I care about the 10^30 universe 2x as much, but it might be more like 1.000001x as much or 1x as much, and it’s not plausible I care 10^10 as much.” That means I care about each individual life less if it happens in a big universe.
This isn’t why I believe the view, but one way you might be able to better sympathize is by thinking: “There is another universe that is like the 10^20 universe but copied 10^10 times. That’s not that much more complex than the 10^20 universe. And in fact total observer counts were already dominated by copies of those universes that were tiled 3^^^3 times, and the description complexity difference between 3^^^3 and 10^10 x 3^^^3 are not very large.” Of course unbounded utilities don’t admit that kind of reasoning, because they don’t admit any kind of reasoning. And indeed, the fact that the expectations diverge seem very closely related to the exact reasoning you would care most about doing in order to actually assess the relative importance of different decisions, so I don’t think the infinity thing is a weird case, it seems absolutely central and I don’t even know how to talk about what the view should be if the infinites didn’t diverge.
I’m not very intuitively drawn to views like “count the distinct experiences,” and I think that in addition to being kind of unappealing those views also have some pretty crazy consequences (at least for all the concrete versions I can think of).
I basically agree that someone who has the opposite view—that for every universe there is a bigger universe that dwarfs its importance—has a more complicated philosophical question and I don’t know the answer. That said, I think it’s plausible they are in the same position as someone who has strong brute intuitions that A>B, B>C, and C>A for some concrete outcomes A, B, C—no amount of philosophical progress will help them get out of the inconsistency. I wouldn’t commit to that pessimistic view, but I’d give it maybe 50/50---I don’t see any reason that there needs to be a satisfying resolution to this kind of paradox.
Do you have preferences over the possible outcomes of thought experiments? Does it feel intuitively like they should satisfy dominance principles? If so, it seems like it’s just as troubling that there are thought experiments. Analogously, if I had the strong intuition that A>B>C>A, and someone said “Ah but don’t worry, B could never happen in the real world!” I wouldn’t be like “Great that settles it, no longer feel confused+troubled.”
I’m not particulalry enthusiastic about “artificial leverage penalties” that manually penalize the hypothesis you can get 3^^^3 happy people by a factor of 1/3^^^3 (and so insofar as that’s what you’re saying, I agree).
From my end, the core of my objection feels more like “you have an extra implicit assumption that lotteries are closed under countable combination, and I’m not sold on that.” The part where I go “and maybe some sufficiently naturalistic prior ends up thinking long St. Petersburg games are ultimately less likely than they are simple???” feels to me more like a parenthetical, and a wild guess about how the weakpoint in your argument could resolve.
(My guess is that you mean something more narrow and specific by “leverage penalty” than I did, and that me using those words caused confusion. I’m happy to retreat to a broader term, that includes things like “big gambles just turn out not to unbalance naturalistic reasoning when you’re doing it properly (eg. b/c finding-yourself-in-the-universe correctly handles this sort of thing somehow)”, if you have one.)
(My guess is that part of the difference in framing in the above paragraphs, and in my original comment, is due to me updating in response to your comments, and retreating my position a bit. Thanks for the points that caused me to update somewhat!)
I agree.
This seems like a fine guess to me. I don’t feel sold on it, but that could ofc be because you’ve resolved confusions that I have not. (The sort of thing that would persuade me would be you demonstrating at least as much mastery of my own confusions than I possess, and then walking me through the resolution. (Which I say for the purpose of being upfront about why I have not yet updated in favor of this view. In particular, it’s not a request. I’d be happy for more thoughts on it if they’re cheap and you find generating them to be fun, but don’t think this is terribly high-priority.))
I indeed find this counter-intuitive. Hooray for flatly asserting things I might find counter-intuitive!
Let me know if you want me to flail in the direction of confusions that stand between me and what I understand to be your view. The super short version is something like “man, I’m not even sure whether logic or physics comes first, so I get off that train waaay before we get to the Tegmark IV logical multiverse”.
(Also, to be clear, I don’t find UDASSA particularly compelling, mainly b/c of how confused I remain in light of it. Which I note in case you were thinking that the inferential gap you need to span stretches only to UDASSA-town.)
You’ve lost me somewhere. Maybe try backing up a step or two? Why are we talking about thought experiments?
One of my best explicit hypotheses for what you’re saying is “it’s one thing to deny closure of epistemic probabiltiies under countable weighted combination in real life, and another to deny them in thought experiments; are you not concerned that denying them in thought experiments is troubling?”, but this doesn’t seem like a very likely argument for you to be making, and so I mostly suspect I’ve lost the thread.
(I stress again that, from my perspective, the heart of my objection is your implicit assumption that lotteries are closed under countable combination. If you’re trying to object to some other thing I said about leverage penalties, my guess is that I micommunicated my position (perhaps due to a poor choice of words) or shifted my position in response to your previous comments, and that our arguments are now desynched.)
Backing up to check whether I’m just missing something obvious, and trying to sharpen my current objection:
It seems to me that your argument contains a fourth, unlisted assumption, which is that lotteries are closed under countable combination. Do you agree? Am I being daft and missing that, like, some basic weak dominance assumption implies closure of lotteries under countable combination? Assuming I’m not being daft, do you agree that your argument sure seems to leave the door open for people who buy antisymmetry, dominance, and unbounded utilities, but reject countable combination of lotteries?
My formal argument is even worse than that: I assume you have preferences over totally arbitrary probability distributions over outcomes!
I don’t think this is unlisted though—right at the beginning I said we were proving theorems about a preference ordering < defined over the space of probability distributions over a space of outcomes Ω. I absolutely think it’s plausible to reject that starting premise (and indeed I suggest that someone with “unbounded utilities” ought to reject this premise in an even more dramatic way).
It seems to me that our actual situation (i.e. my actual subjective distribution over possible worlds) is divergent in the same way as the St Petersburg lottery, at least with respect to quantities like expected # of happy people. So I’m less enthusiastic about talking about ways of restricting the space of probability distributions to avoid St Petersburg lotteries. This is some of what I’m getting at in the parent, and I now see that it may not be responsive to your view. But I’ll elaborate a bit anyway.
There are universes with populations of 3↑↑↑3 that seem only 21000 times less likely than our own. It would be very surprising and confusing to learn that not only am I wrong but this epistemic state ought to have been unreachable, that anyone must assign those universes probability at most 1/3↑↑↑3. I’ve heard it argued that you should be confident that the world is giant, based on anthropic views like SIA, but I’ve never heard anyone seriously argue that you should be perfectly confident that the world isn’t giant.
If you agree with me that in fact our current epistemic state looks like a St Petersburg lottery with respect to # of people, then I hope you can sympathize with my lack of enthusiasm.
All that is to say: it may yet be that preferences are defined over a space of probability distributions small enough to evade the argument in the OP. But at that point it seems much more likely that preferences just aren’t defined over probability distributions at all—it seems odd to hold onto probability distributions as the object of preferences while restricting the space of probability distributions far enough that they appear to exclude our current situation.
Suppose that you have some intuition that implies A > B > C > A.
At first you are worried that this intuition must be unreliable. But then you realize that actually B is impossible in reality, so consistency is restored.
I claim that you should be skeptical of the original intuition anyway. We have gotten some evidence that the intuition isn’t really tracking preferences in the way you might have hoped that it was—because if it were correctly tracking preferences it wouldn’t be inconsistent like that.
The fact that B can never come about in reality doesn’t really change the situation, you still would have expected consistently-correct intuitions to yield consistent answers.
(The only way I’d end up forgiving the intuition is if I thought it was correctly tracking the impossibility of B. But in this case I don’t think so. I’m pretty sure my intuition that you should be willing to take a 1% risk in order to double the size of the world isn’t tracking some deep fact that would make certain epistemic states inaccessible.)
(That all said, a mark against an intuition isn’t a reason to dismiss it outright, it’s just one mark against it.)
Ok, cool, I think I see where you’re coming from now.
Fair! To a large degree, I was just being daft. Thanks for the clarification.
I think this is a good point, and I hadn’t had this thought quite this explicitly myself, and it shifts me a little. (Thanks!)
(I’m not terribly sold on this point myself, but I agree that it’s a crux of the matter, and I’m sympathetic.)
This might be where we part ways? I’m not sure. A bunch of my guesses do kinda look like things you might describe as “preferences not being defined over probability distributions” (eg, “utility is a number, not a function”). But simultaneously, I feel solid in my ability to use probabliity distributions and utility functions in day-to-day reasoning problems after I’ve chunked the world into a small finite number of possible actions and corresponding outcomes, and I can see a bunch of reasons why this is a good way to reason, and whatever the better preference-formalism turns out to be, I expect it to act a lot like probability distributions and utility functions in the “local” situation after the reasoner has chunked the world.
Like, when someone comes to me and says “your small finite considerations in terms of actions and outcomes are super simplified, and everything goes nutso when we remove all the simplifications and take things to infinity, but don’t worry, sanity can be recovered so long as you (eg) care less about each individual life in a big universe than in a small universe”, then my response is “ok, well, maybe you removed the simplifications in the wrong way? or maybe you took limits in a bad way? or maybe utility is in fact bounded? or maybe this whole notion of big vs small universes was misguided?”
It looks to me like you’re arguing that one should either accept bounded utilities, or reject the probability/utility factorization in normal circumstances, whereas to me it looks like there’s still a whole lot of flex (ex: ‘outcomes’ like “I come back from the store with milk” and “I come back from the store empty-handed” shouldn’t have been treated the same way as ‘outcomes’ like “Tegmark 3 multiverse branch A, which looks like B” and “Conway’s game of life with initial conditions X, which looks like Y”, and something was going wrong in our generalization from the everyday to the metaphysical, and we shouldn’t have been identifying outcomes with universes and expecting preferences to be a function of probability distributions on those universes, but thinking of “returning with milk” as an outcome is still fine).
And maybe you’d say that this is just conceding your point? That when we pass from everyday reasoning about questions like “is there milk at the store, or not?” to metaphysical reasoning like “Conway’s Life, or Tegmark 3?”, we should either give up on unbounded utilities, or give up on thinking of preferences as defined on probability distributions on outcomes? I more-or-less buy that phrasing, with the caveat that I am open to the weak-point being this whole idea that metaphysical universes are outcomes and that probabilities on outcome-collections that large are reasonable objects (rather than the weakpoint being the probablity/utility factorization per se).
I agree that would be odd.
One response I have is similar to the above: I’m comfortable using probability distributions for stuff like “does the store have milk or not?” and less comfortable using them for stuff like “Conway’s Life or Tegmark 3?”, and wouldn’t be surprised if thinking of mathematical universes as “outcomes” was a Bad Plan and that this (or some other such philosophically fraught assumption) was the source of the madness.
Also, to say a bit more on why I’m not sold that the current situation is divergent in the St. Petersburg way wrt, eg, amount of Fun: if I imagine someone in Vegas offering me a St. Petersburg gamble, I imagine thinking through it and being like “nah, you’d run out of money too soon for this to be sufficiently high EV”. If you’re like “ok, but imagine that the world actually did look like it could run the gamble infinitely”, my gut sense is “wow, that seems real sus”. Maybe the source of the susness is that eventually it’s just not possible to get twice as much Fun. Or maybe it’s that nobody anywhere is ever in a physical position to reliably double the amount of Fun in the region that they’re able to affect. Or something.
And, I’m sympathetic to the objection “well, you surely shouldn’t assign probability less than <some vanishingly small but nonzero number> that you’re in such a situation!”. And maybe that’s true; it’s definitely on my list of guesses. But I don’t by any means feel forced into that corner. Like, maybe it turns out that the lightspeed limit in our universe is a hint about what sort of universes can be real at all (whatever the heck that turns out to mean), and an agent can’t ever face a St. Petersburgish choice in some suitably general way. Or something. I’m more trying to gesture at how wide the space of possibilities seems to me from my state of confusion, than to make specific counterproposals that I think are competitive.
(And again, I note that the reason I’m not updating (more) towards your apparently-narrower stance, is that I’m uncertain about whether you see a narrower space of possible resolutions on account of being less confused than I am, vs because you are making premature philosophical commitments.)
To be clear, I agree that you need to do something weirder than “outcomes are mathematical universes, preferences are defined on (probability distributions over) those” if you’re going to use unbounded utilities. And again, I note that “utility is bounded” is reasonably high on my list of guesses. But I’m just not all that enthusiastic about “outcomes are mathematical universes” in the first place, so \shrug.
I think I understand what you’re saying about thought experiments, now. In my own tongue: even if you’ve convinced yourself that you can’t face a St. Petersburg gamble in real life, it still seems like St. Petersburg gambles form a perfectly lawful thought experiment, and it’s at least suspicious if your reasoning procedures would break down facing a perfectly lawful scenario (regardless of whether you happen to face it in fact).
I basically agree with this, and note that, insofar as my confusions resolve in the “unbounded utilities” direction, I expect some sort of account of metaphysical/anthropic/whatever reasoning that reveals St. Petersburg gambles (and suchlike) to be somehow ill-conceived or ill-typed. Like, in that world, what’s supposed to happen when someone is like “but imagine you’re offered a St. Petersburg bet” is roughly the same as what’s supposed to happen when someone’s like “but imagine a physically identical copy of you that lacks qualia”—you’re supposed to say “no”, and then be able to explain why.
(Or, well, you’re always supposed to say “no” to the gamble and be able to explain why, but what’s up for grabs is whether the “why” is “because utility is bounded”, or some other thing, where I at least am confused enough to still have some of my chips on “some other thing”.)
To be explicit, the way that my story continues to shift in response to what you’re saying, is an indication of continued updating & refinement of my position. Yay; thanks.
I agree with this: (i) it feels true and would be surprising not to add up to normality, (ii) coherence theorems suggest that any preferences can be represented as probabilities+utilities in the case of finitely many outcomes.
This is my view as well, but you still need to handle the dependence on subjective uncertainty. I think the core thing at issue is whether that uncertainty is represented by a probability distribution (where utility is an expectation).
(Slightly less important: my most naive guess is that the utility number is itself represented as a sum over objects, and then we might use “utility function” to refer to the thing being summed.)
I don’t mean that we face some small chance of encountering a St Petersburg lottery. I mean that when I actually think about the scale of the universe, and what I ought to believe about physics, I just immediately run into St Petersburg-style cases:
It’s unclear whether we can have an extraordinarily long-lived civilization if we reduce entropy consumption to ~0 (e.g. by having a reversible civilization). That looks like at least 5% probability, and would suggest the number of happy lives is much more than 10100 times larger than I might have thought. So does it dominate the expectation?
But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe. Maybe that happens with only 1% probability, but it corresponds to yet bigger civilization. So does that mean we should think that colonizing faster increases the value of the future by 1%, or by 100% since these possibilities are bigger and better and dominate the expectation?
But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it (but a large fraction of them are nevertheless filled with other civilizations like ours). Perhaps that’s 20%. So it actually looks more likely than the “long-lived reversible civilization” and implies more total flourishing. And on those perspectives not going extinct is more important than going faster, for the usual reasons. So does that dominate the calculus instead?
Perhaps rather than having a single set of physical constants, our universe runs every possible set. If that’s 5%, and could stack on top of any of the above while implying another factor of 10100 of scale. And if the standard model had no magic constants maybe this possibility would be 1% instead of 5%. So should I updated by a factor of 5 that we won’t discover that the standard model has fewer magic constants, because then “continuum of possible copies of our universe running in parallel” has only 1% chance instead of 5%?
Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10100 more people, then it starts to seem like even less of a further ask to assume that there were actually 101000 more people instead. So should we assume that multiple of those enhugening assumptions are true (since each one increases values by more than it decreases probability), or just take our favorite and then keep cranking up the numbers larger and larger (with each cranking being more probable than the last and hence more probable than adding a second enhugening assumption)?
Those are very naive physically statements of the possibilities, but the point is that it seems easy to imagine the possibility that populations could be vastly larger than we think “by default”, and many of those possibilities seem to have reasonable chances rather than being vanishingly unlikely. And at face value you might have thought those possibilities were actually action-relevant (e.g. the possibility of exponential returns to resources dominates the EV and means we should rush to colonize after all), but once you actually look at the whole menu, and see how the situation is just obviously paradoxical in every dimension, I think it’s pretty clear that you should cut off this line of thinking.
A bit more precisely: this situation is structurally identical to someone in a St Petersburg paradox shuffling around the outcomes and finding that they can justify arbitrary comparisons because everything has infinite EV and it’s easy to rewrite. That is, we can match each universe U with “U but with long-lived reversible civilizations,” and we find that the long-lived reversible civilizations dominate the calculus. Or we can match each universe U with “U but vast” and find the vast universes dominate the calculus. Or we can match “long-lived reversible civilizations” with “vast” and find that we can ignore long-lived reversible civilizations. It’s just like matching up the outcomes in the St Petersburg paradox in order to show that any outcome dominates itself.
The unbounded utility claim seems precisely like the claim that each of those less-likely-but-larger universes ought to dominate our concern, compared to the smaller-but-more-likely universe we expect by default. And that way of reasoning seems like it leads directly to these contradictions at the very first time you try to apply it to our situation (indeed, I think every time I’ve seen someone use this assumption in a substantive way it has immediately seemed to run into paradoxes, which are so severe that they mean they could as well have reached the opposite conclusion by superficially-equally-valid reasoning).
I totally believe you might end up with a very different way of handling big universes than “bounded utilities,” but I suspect it will also lead to the conclusion that “the plausible prospect of a big universe shouldn’t dominate our concern.” And I’d probably be fine with the result. Once you divorce unbounded utilities from the usual theory about how utilities work, and also divorce them from what currently seems like their main/only implication, I expect I won’t have anything more than a semantic objection.
More specifically and less confidently, I do think there’s a pretty good chance that whatever theory you end up with will agree roughly with the way that I handle big universes—we’ll just use our real probabilities of each of these universes rather than focusing on the big ones in virtue of their bigness, and within each universe we’ll still prefer have larger flourishing populations. I do think that conclusion is fairly uncertain, but I tentatively think it’s more likely we’ll give up on the principle “a bigger civilization is nearly-linearly better within a given universe” than on the principle “a bigger universe is much less than linearly more important.”
And from a practical perspective, I’m not sure what interim theory you use to reason about these things. I suspect it’s mostly academic for you because e.g. you think alignment is a 99% risk of death instead of a 20% risk of death and hence very few other questions about the future matter. But if you ever did find yourself having to reason about humanity’s long-term future (e.g. to assess the value of extinction risk vs faster colonization, or the extent of moral convergence), then it seems like you should use an interim theory which isn’t fanatical about the possibility of big universes—because the fanatical theories just don’t work, and spit out inconsistent results if combined with our current framework. You can also interpret my argument as strongly objecting to the use of unbounded utilities in that interim framework.
(I, in fact, lifted it off of you, a number of years ago :-p)
Of course. (And noting that I am, perhaps, more openly confused about how to handle the subjective uncertainty than you are, given my confusions around things like logical uncertainty and whether difficult-to-normalize arithmetical expressions meaningfully denote numbers.)
Running through your examples:
I agree. Separately, I note that I doubt total Fun is linear in how much compute is available to civilization; continuity with the past & satisfactory completion of narrative arcs started in the past is worth something, from which we deduce that wiping out civilization and replacing it with another different civilization of similar flourish and with 2x as much space to flourish in, is not 2x as good as leaving the original civilization alone. But I’m basically like “yep, whether we can get reversibly-computed Fun chugging away through the high-entropy phase of the universe seems like an empiricle question with cosmically large swings in utility associated therewith.”
This seems fairly plausible to me! For instance, my best guess is that you can get more than 2x the Fun by computing two people interacting than by computing two individuals separately. (Although my best guess is also that this effect diminishes at scale, \shrug.)
By my lights, it sure would be nice to have more clarity on this stuff before needing to decide how much to rush our expansion. (Although, like, 1st world problems.)
Sure, this is pretty plausible, but (arguendo) it shouldn’t really be factoring into our action analysis, b/c of the part where we can’t reach it. \shrug
Sure. And again (arguendo) this doesn’t much matter to us b/c the others are beyond our sphere of influence.
I think this is where I get off the train (at least insofar as I entertain unbounded-utility hypotheses). Like, our ability to reversibly compute in the high-entropy regime is bounded by our error-correction capabilities, and we really start needing to upend modern physics as I understand it to make the numbers really huge. (Like, maybe 10^1000 is fine, but it’s gonna fall off a cliff at some point.)
I have a sense that I’m missing some deeper point you’re trying to make.
I also have a sense that… how to say… like, suppose someone argued “well, you don’t have 1/∞ probability that “infinite utility” makes sense, so clearly you’ve got to take infinite utilities seriously”. My response would be something like “That seems mixed up to me. Like, on my current understanding, “infinite utility” is meaningless, it’s a confusion, and I just operate day-to-day without worrying about it. It’s not so much that my operating model assigns probability 0 to the proposition “infinite utilities are meaningful”, as that infinite utilities simply don’t fit into my operating model, they don’t make sense, they don’t typecheck. And separately, I’m not yet philosophically mature, and I can give you various meta-probabilities about what sorts of things will and won’t typecheck in my operating model tomorrow. And sure, I’m not 100% certain that we’ll never find a way to rescue the idea of infinite utilities. But that meta-uncertainty doesn’t bleed over into my operating model, and I’m not supposed to ram infinities into a place where they don’t fit just b/c I might modify the type signatures tomorrow.”
When you bandy around plausible ways that the universe could be real large, it doesn’t look obviously divergent to me. Some of the bullets you’re handling are ones that I am just happy to bite, and others involve stuff that I’m not sure I’m even going to think will typecheck, once I understand wtf is going on. Like, just as I’m not compelled by “but you have more than 0% probability that ‘infinite utility’ is meaningful” (b/c it’s mixing up the operating model and my philosophical immaturity), I’m not compelled by “but your operating model, which says that X, Y, and Z all typecheck, is badly divergent”. Yeah, sure, and maybe the resolution is that utilities are bounded, or maybe it’s that my operating model is too permissive on account of my philosophical immaturity. Philosophical immaturity can lead to an operating model that’s too permisive (cf. zombie arguments) just as easily as one that’s too strict.
Like… the nature of physical law keeps seeming to play games like “You have continua!! But you can’t do an arithmetic encoding. There’s infinite space!! But most of it is unreachable. Time goes on forever!! But most of it is high-entropy. You can do reversible computing to have Fun in a high-entropy universe!! But error accumulates.” And this could totally be a hint about how things that are real can’t help but avoid the truly large numbers (never mind the infinities), or something, I don’t know, I’m philisophically immature. But from my state of philosophical immaturity, it looks like this could totally still resolve in a “you were thinking about it wrong; the worst enhugening assumptions fail somehow to typecheck” sort of way.
Trying to figure out the point that you’re making that I’m missing, it sounds like you’re trying to say something like “Everyday reasoning at merely-cosmic scales already diverges, even without too much weird stuff. We already need to bound our utilities, when we shift from looking at the milk in the supermarket to looking at the stars in the sky (nevermind the rest of the mathematical multiverse, if there is such a thing).” Is that about right?
If so, I indeed do not yet buy it. Perhaps spell it out in more detail, for someone who’s suspicious of any appeals to large swaths of terrain that we can’t affect (eg, variants of this universe w/ sufficiently different cosmological constants, at least in the regions where the locals aren’t thinking about us-in-particular); someone who buys reversible computing but is going to get suspicious when you try to drive the error rate to shockingly low lows?
To be clear, insofar as modern cosmic-scale reasoning diverges (without bringing in considerations that I consider suspicious and that I suspect I might later think belong in the ‘probably not meaningful (in the relevant way)’ bin), I do start to feel the vice grips on me, and I expect I’d give bounded utilities another look if I got there.
A side note: IB physicalisms solves at least a large chunk of naturalism/counterfactuals/anthropics but is almost orthogonal to this entire issue (i.e. physicalist loss functions should still be bounded for the same reason cartesian loss functions should be bounded), so I’m pretty skeptical there’s anything in that direction. The only part which is somewhat relevant is: IB physicalists have loss functions that depend on which computations are running so two exact copies of the same thing definitely count as the same and not twice as much (except potentially in some indirect way, such as being involved together in a single more complex computation).
I am definitely entertaining the hypothesis that the solution to naturalism/anthropics is in no way related to unbounded utilities. (From my perspective, IB physicalism looks like a guess that shows how this could be so, rather than something I know to be a solution, ofc. (And as I said to Paul, the observation that would update me in favor of it would be demonstrated mastery of, and unravelling of, my own related confusions.))
In the parenthetical remark, are you talking about confusions related to Pascal-mugging-type thought experiments, or other confusions?
Those & others. I flailed towards a bunch of others in my thread w/ Paul. Throwing out some taglines:
“does logic or physics come first???”
“does it even make sense to think of outcomes as being mathematical universes???”
“should I even be willing to admit that the expression “3^^^3″ denotes a number before taking time proportional to at least log(3^^^3) to normalize it?”
“is the thing I care about more like which-computations-physics-instantiates, or more like the-results-of-various-computations??? is there even a difference?”
“how does the fact that larger quantum amplitudes correspond to more magical happening-ness relate to the question of how much more I should care about a simulation running on a computer with wires that are twice as thick???”
Note that these aren’t supposed to be particularly well-formed questions. (They’re more like handles for my own confusions.)
Note that I’m open to the hypothesis that you can resolve some but not others. From my own state of confusion, I’m not sure which issues are interwoven, and it’s plausible to me that you, from a state of greater clarity, can see independences that I cannot.
Note that I’m not asking for you to show me how IB physicalism chooses a consistent set of answers to some formal interpretations of my confusion-handles. That’s the sort of (non-trivial and virtuous!) feat that causes me to rate IB physicalism as a “plausible guess”.
In the specific case of IB physicalism, I’m like “maaaybe? I don’t yet see how to relate this Γ that you suggestively refer to as a ‘map from programs to results’ to a philosophical stance on computation and instantiation that I understand” and “I’m still not sold on the idea of handling non-realizability with inframeasures (on account of how I still feel confused about a bunch of things that inframeasures seem like a plausible guess for how to solve)” and etc.
Maybe at some point I’ll write more about the difference, in my accounting, between plausible guesses and solutions.
Hmm… I could definitely say stuff about, what’s the IB physicalism take on those questions. But this would be what you specifically said you’re not asking me to do. So, from my perspective addressing your confusion seems like a completely illegible task atm. Maybe the explanation you alluded to in the last paragraph would help.
I’d be happy to read it if you’re so inclined and think the prompt would help you refine your own thoughts, but yeah, my anticipation is that it would mostly be updating my (already decent) probability that IB physicalism is a reasonable guess.
A few words on the sort of thing that would update me, in hopes of making it slightly more legible sooner rather than later/never: there’s a difference between giving the correct answer to metaethics (“‘goodness’ refers to an objective (but complicated, and not objectively compelling) logical fact, which was physically shadowed by brains on account of the specifics of natural selection and the ancestral environment”), and the sort of argumentation that, like, walks someone from their confused state to the right answer (eg, Eliezer’s metaethics sequence). Like, the confused person is still in a state of “it seems to me that either morality must be objectively compelling, or nothing truly matters”, and telling them your favorite theory isn’t really engaging with their intuitions. Demonstrating that your favorite theory can give consistent answers to all their questions is something, it’s evidence that you have at least produced a plausible guess. But from their confused perspective, lots of people (including the nihilists, including the Bible-based moral realists) can confidently provide answers that seem superficially consistent.
The compelling thing, at least to me and my ilk, is the demonstration of mastery and the ability to build a path from the starting intuitions to the conclusion. In the case of a person confused about metaethics, this might correspond to the ability to deconstruct the “morality must be objectively compelling, or nothing truly matters” intuition, right in front of them, such that they can recognize all the pieces inside themselves, and with a flash of clarity see the knot they were tying themselves into. At which point you can help them untie the knot, and tug on the strings, and slowly work your way up to the answer.
(The metaethics sequence is, notably, a tad longer than the answer itself.)
(If I were to write this whole concept of solutions-vs-answers up properly, I’d attempt some dialogs that make the above more concrete and less metaphorical, but \shrug.)
In the case of IB physicalism (and IB more generally), I can see how it’s providing enough consistent answers that it counts as a plausible guess. But I don’t see how to operate it to resolve my pre-existing confusions. Like, we work with (infra)measures over ΣR×Φ, and we say some fancy words about how ΣR is our “beliefs about the computations”, but as far as I’ve been able to make out this is just a neato formalism; I don’t know how to get to that endpoint by, like, starting from my own messy intuitions about when/whether/how physical processes reflect some logical procedure. I don’t know how to, like, look inside myself, and find confusions like “does logic or physics come first?” or “do I switch which algorithm I’m instantiating when I drink alcohol?”, and disassemble them into their component parts, and gain new distinctions that show me how the apparent conflicts weren’t true conflicts and all my previous intuitions were coming at things from slightly the wrong angle, and then shift angles and have a bunch of things click into place, and realize that the seeds of the answer were inside me all along, and that the answer is clearly that the universe isn’t really just a physical arrangement of particles (or a wavefunction thereon, w/e), but one of those plus a mapping from syntax-trees to bits (here taking |R|=2). Or whatever the philosophy corresponding to “a hypothesis is a ΣR×Φ” is supposed to be. Like, I understand that it’s a neat formalism that does cool math things, and I see how it can be operated to produce consistent answers to various philosophical questions, but that’s a long shot from seeing it solve the philosophical problems at hand. Or, to say it another way, answering my confusion handles consistently is not nearly enough to get me to take a theory philosophically seriously, like, it’s not enough to convince me that the universe actually has an assignment of syntax-trees to bits in addition to the physical state, which is what it looks to me like I’d need to believe if I actually took IB physicalism seriously.
I don’t think I’m capable of writing something like the metaethics sequence about IB, that’s a job for someone else. My own way of evaluating philosophical claims is more like:
Can we a build an elegant, coherent mathematical theory around the claim?
Does the theory meet reasonable desiderata?
Does the theory play nicely with other theories we have high confidence of?
If there are compelling desiderata the theory doesn’t meet, can we show that meeting them is impossible?
For example, the way I understood objective morality is wrong was by (i) seeing that there’s a coherent theory of agents with any utility function whatsoever (ii) understanding that, in terms of the physical world, “Vanessa’s utility function” is more analogous to “coastline of Africa” than to “fundamental equations of physics”.
I agree that explaining why we have certain intuitions is a valuable source of evidence, but it’s entangled with messy details of human psychology that create a lot of noise. (Notice that I’m not saying you shouldn’t use intuition, obviously intuition is an irreplaceable core part of cognition. I’m saying that explaining intuition using models of the mind, while possible and desirable, is also made difficult by the messy complexity of human minds, which in particular introduces a lot of variables that vary between people.)
Also, I want to comment on your last tagline, just because it’s too tempting:
I haven’t written the proofs cleanly yet (because prioritizing other projects atm), but it seems that IB physicalism produces a rather elegant interpretation of QM. Many-worlds turns out to be false. The wavefunction is not “a thing that exists”. Instead, what exists is the outcomes of all possible measurements. The universe samples those outcomes from a distribution that is determined by two properties: (i) the marginal distribution of each measurement has to obey the Born rule (ii) the overall amount of computation done by the universe should be minimal. It follows that, outside of weird thought experiments (i.e. as long as decoherence applies), agents don’t get split into copies and quantum randomness is just ordinary randomness. (Another nice consequence is that Boltzmann brains don’t have qualia.)
What’s ordinary randomness?