I think your intuition is relying a little too much on the absurdity heuristic (e.g., “It boggles my mind...”) and flat out assertion (e.g., “But the set itself doesn’t have any independent existence.”). Metaphysical intuition is really misleading. I think most people underestimate that, especially because the absurdity heuristic is strong and therefore it’s easy to reach a reductio ad absurdum that is nonetheless true. I’ll give an example.
Once upon a time I didn’t think copies ‘counted’ in a multiverse, either morally or for purposes of anthropic reasoning. 200 Jacks had the same weight as 1 Mary. The opposite was absurd, you see: You’re claiming that 3 copies of the exact same computation are worth more than 2 computations of 2 different people, leading separate and diverse lives? Absurd! My moral and metaphysical intuition balks at such an idea! I came up with, like, 3 reductio ad absurdums to prove my point. Eliezer, Wei Dai, Steven Kaas, Nick Bostrom, what did they know? And there was some pride, too, because they way I was thinking about it meant I could easily deal with indexical uncertainty, and the others seemed clueless. … Well, turns out those reductios weren’t absurd: I just hadn’t learned to think like reality. I had to update, because that’s where the decision theory led, and it’s hard to argue with mathematics. And it came to my attention that thinking doubled computations had the same measure had a lot of problems as well. Since then, I’ve been a lot more careful about asserting my intuition when it disagrees with people who seem to have thought about it a lot more than I have.
In the case of the Mathematical Universe Hypothesis or permutations thereof (Eliezer seems to think the mysterious ‘reality fluid’ or ‘measure’ has a lot to do with directed acyclic graphs, for instance), there’s a lot of mental firepower aimed against you. Why do you believe what you believe? If it turns out the reason is metaphysical intuition, be on guard. Acknowledge your intuition, but don’t believe everything you think.
I think your intuition is relying a little too much on the absurdity heuristic (e.g., “It boggles my mind...”) and flat out assertion (e.g., “But the set itself doesn’t have any independent existence.”). Metaphysical intuition is really misleading. I think most people underestimate that, especially because the absurdity heuristic is strong and therefore it’s easy to reach a reductio ad absurdum that is nonetheless true. I’ll give an example.
Once upon a time I didn’t think copies ‘counted’ in a multiverse, either morally or for purposes of anthropic reasoning. 200 Jacks had the same weight as 1 Mary. The opposite was absurd, you see: You’re claiming that 3 copies of the exact same computation are worth more than 2 computations of 2 different people, leading separate and diverse lives? Absurd! My moral and metaphysical intuition balks at such an idea! I came up with, like, 3 reductio ad absurdums to prove my point. Eliezer, Wei Dai, Steven Kaas, Nick Bostrom, what did they know? And there was some pride, too, because they way I was thinking about it meant I could easily deal with indexical uncertainty, and the others seemed clueless. … Well, turns out those reductios weren’t absurd: I just hadn’t learned to think like reality. I had to update, because that’s where the decision theory led, and it’s hard to argue with mathematics. And it came to my attention that thinking doubled computations had the same measure had a lot of problems as well. Since then, I’ve been a lot more careful about asserting my intuition when it disagrees with people who seem to have thought about it a lot more than I have.
In the case of the Mathematical Universe Hypothesis or permutations thereof (Eliezer seems to think the mysterious ‘reality fluid’ or ‘measure’ has a lot to do with directed acyclic graphs, for instance), there’s a lot of mental firepower aimed against you. Why do you believe what you believe? If it turns out the reason is metaphysical intuition, be on guard. Acknowledge your intuition, but don’t believe everything you think.