(Too bad this post wasn’t made by Nick Beckstead, because then he would be able to receive notice when someone posts here. I guess I’ll send a PM alerting him to this comment.)
I’d like to suggest that Nick do a sequence of posts on the main novel arguments in his thesis, both to draw more attention to them, and to focus discussion. Right now it’s hard to get much of a public discussion going because if I read one section of his thesis and post a comment on that, most other people will either have read that section a long time ago and have forgotten much of it, or will read it in the future and therefore can’t respond.
That aside, I do have an object-level comment. Nick states (in section 6.3.1) that Period Independence is incompatible with bounded utility function, but I think that’s wrong. Consider a total utilitarian who exponentially discounts each person-stage according to their distance from some chosen space-time event. Then the utility function is both bounded (assuming the undiscounted utility for each person-stage is bounded) and satisfies Period Independence. Another idea for a bounded utility function satisfying Period Independence, which I previously suggested on LW and was originally motivated by multiverse-related considerations, is to discount or bound the utility assigned to each person-stage by their algorithmic probability.
That aside, I do have an object-level comment. Nick states (in section 6.3.1) that Period Independence is incompatible with bounded utility function, but I think that’s wrong. Consider a total utilitarian who exponentially discounts each person-stage according to their distance from some chosen space-time event. Then the utility function is both bounded (assuming the undiscounted utility for each person-stage is bounded) and satisfies Period Independence.
I agree with this. I think I was implicitly assuming some additional premises, particularly Temporal Impartiality. I believe that bounded utility + Temporal Impartiality is inconsistent with bounded utility. (Even saying this implicitly assumes other stuff, like transitive rankings, etc., though I agree that Temporal Impartiality is much more substantive.)
Another idea for a bounded utility function satisfying Period Independence, which I previously suggested on LW and was originally motivated by multiverse-related considerations, is to discount or bound the utility assigned to each person-stage by their algorithmic probability.
I am having a hard time parsing this. Could you explain where the following argument breaks down?
Let A(n,X) be a world in which there are n periods of quality X.
The value of what happens during a period is a function of what happens during that period, and not a function of what happens in other periods.
If the above premise is true, then there exists a positive period quality X such that, for any n, A(n,X) is a possible world.
Assuming Period Independence and Temporal Impartiality, as n approaches infinity, the value of A(n,X) approaches infinity.
Therefore, Period Independence and Temporal Impartiality imply an unbounded utility function.
The first premise here is something I articulate in Section 3.2, but may not be totally clear given the informal statement of Period Independence that I run with.
Let me note that one thing about your proposal confuses me, and could potentially be related to why I don’t see which step of the above argument you deny. I primarily think of probability as a property of possible worlds, rather than individuals. Perhaps you are thinking of probability as a property of centered possible worlds? Is your proposal that the goodness of a world A with is of the form:
g(A) = well-being of person 1 prior centered world probability of person 1 in world A + well-being of person 2 prior centered world probability of person 2 in A + …
? If it is, this is a proposal I have not thought about and would be interested in hearing more about its merits and why it is bounded.
Could you explain where the following argument breaks down?
My proposal violates Temporal Impartiality.
I primarily think of probability as a property of possible worlds, rather than individuals. Perhaps you are thinking of probability as a property of centered possible worlds?
Yes, sort of. When I said “algorithmic probability” I was referring to the technical concept divorced from standard connotations of “probability”, but my idea is also somewhat related to the idea of probability as a property of centered possible worlds.
I guess there’s a bit of an inferential gap between us that makes it hard for me to quickly explain the idea to you. From my perspective, it would be much easier if you were already familiar with Algorithmic Information Theory and my UDT ideas, but I’m not sure if you want to read up on all that. Do you see Paul Christiano often? If so, he can probably explain it to you in person fairly quickly. Or, since you’re at FHI, Stuart Armstrong might also know enough about my ideas to explain them to you.
OK, I”ll ask Paul or Stewart next time I see them.
Does your proposal also violate #1 because the simplicity of an observer-situated-in-a-world is a holistic property of the the observer-situated-in-a-world rather than a local one?
Does your proposal also violate #1 because the simplicity of an observer-situated-in-a-world is a holistic property of the the observer-situated-in-a-world rather than a local one?
Yes (assuming by #1 you mean Period Independence), but it’s not clear to what extent. For example there are at least two kinds of programs that can output a human brain. A) simulate a world and output the object at some space-time location. B) simulate a world and scan for an object matching some criteria, then output such an object. If a time period gets repeated exactly, people’s algorithmic probability from A gets doubled, but algorithmic probability from B doesn’t. I’m not sure at this point whether A dominates B or vice versa.
Also, it’s not clear to me that strict Period Independence is a good thing. It seems reasonable to not value a time period as much if you knew it was an exact repetition of a previous time period. I wrote a post that’s related to this.
Also, it’s not clear to me that strict Period Independence is a good thing. It seems reasonable to not value a time period as much if you knew it was an exact repetition of a previous time period. I wrote a post that’s related to this.
I agree that Period Independence may break in the kind of case you describe, though I’m not sure. I don’t think that the kind of case you are describing here is a strong consideration against using Period Independence in cases that don’t involve exact repetition. I think your main example in the post is excellent.
I don’t think that the kind of case you are describing here is a strong consideration against using Period Independence in cases that don’t involve exact repetition.
What if we assume Period Independence except for exact repetitions, where the value of extra repetitions eventually go to zero? Perhaps this could be a way to be “timid” while making the downsides of “timidity” seem not so bad or even reasonable? For example in section 6.3.2, such a person would only choose deal 1 over deal 2 if the years of happy lives offered in deal 1 are such that he would already have repeated all possible happy time periods so many times that he values more repetitions very little.
BTW what do you think about my suggestion to do a sequence of blog posts based on your thesis? Or maybe you can at least do one post as a trial run? Also as an unrelated comment, the font in your thesis seems to be such that it’s pretty uncomfortable to read in Adobe Acrobat, unless I zoom in to make the text much larger than I usually have to. Not sure if it’s something you can easily fix. If not, I can try to help if you email me the source of the PDF.
What if we assume Period Independence except for exact repetitions, where the value of extra repetitions eventually go to zero? Perhaps this could be a way to be “timid” while making the downsides of “timidity” seem not so bad or even reasonable? For example in section 6.3.2, such a person would only choose deal 1 over deal 2 if the years of happy lives offered in deal 1 are such that he would already have repeated all possible happy time periods so many times that he values more repetitions very little.
I think it would be interesting if you could show that the space of possible periods-of-lives is structured in such a way that, when combined with a reasonable rule for discounting repetitions, yields a bounded utility function. I don’t have fully developed views on the repetition issue and can imagine that the view has some weird consequences, but if you could do this I would count it as a significant mark in favor of the perspective.
BTW what do you think about my suggestion to do a sequence of blog posts based on your thesis?
I think this would have some value but isn’t at the top of my list right now.
Also as an unrelated comment, the font in your thesis seems to be such that it’s pretty uncomfortable to read in Adobe Acrobat, unless I zoom in to make the text much larger than I usually have to. Not sure if it’s something you can easily fix. If not, I can try to help if you email me the source of the PDF.
I think I’ll keep with the current format for citation consistency for now. But I have added a larger font version here.
(Too bad this post wasn’t made by Nick Beckstead, because then he would be able to receive notice when someone posts here. I guess I’ll send a PM alerting him to this comment.)
I’d like to suggest that Nick do a sequence of posts on the main novel arguments in his thesis, both to draw more attention to them, and to focus discussion. Right now it’s hard to get much of a public discussion going because if I read one section of his thesis and post a comment on that, most other people will either have read that section a long time ago and have forgotten much of it, or will read it in the future and therefore can’t respond.
That aside, I do have an object-level comment. Nick states (in section 6.3.1) that Period Independence is incompatible with bounded utility function, but I think that’s wrong. Consider a total utilitarian who exponentially discounts each person-stage according to their distance from some chosen space-time event. Then the utility function is both bounded (assuming the undiscounted utility for each person-stage is bounded) and satisfies Period Independence. Another idea for a bounded utility function satisfying Period Independence, which I previously suggested on LW and was originally motivated by multiverse-related considerations, is to discount or bound the utility assigned to each person-stage by their algorithmic probability.
I agree with this. I think I was implicitly assuming some additional premises, particularly Temporal Impartiality. I believe that bounded utility + Temporal Impartiality is inconsistent with bounded utility. (Even saying this implicitly assumes other stuff, like transitive rankings, etc., though I agree that Temporal Impartiality is much more substantive.)
I am having a hard time parsing this. Could you explain where the following argument breaks down?
Let A(n,X) be a world in which there are n periods of quality X.
The value of what happens during a period is a function of what happens during that period, and not a function of what happens in other periods.
If the above premise is true, then there exists a positive period quality X such that, for any n, A(n,X) is a possible world.
Assuming Period Independence and Temporal Impartiality, as n approaches infinity, the value of A(n,X) approaches infinity.
Therefore, Period Independence and Temporal Impartiality imply an unbounded utility function.
The first premise here is something I articulate in Section 3.2, but may not be totally clear given the informal statement of Period Independence that I run with.
Let me note that one thing about your proposal confuses me, and could potentially be related to why I don’t see which step of the above argument you deny. I primarily think of probability as a property of possible worlds, rather than individuals. Perhaps you are thinking of probability as a property of centered possible worlds? Is your proposal that the goodness of a world A with is of the form:
g(A) = well-being of person 1 prior centered world probability of person 1 in world A + well-being of person 2 prior centered world probability of person 2 in A + …
? If it is, this is a proposal I have not thought about and would be interested in hearing more about its merits and why it is bounded.
My proposal violates Temporal Impartiality.
Yes, sort of. When I said “algorithmic probability” I was referring to the technical concept divorced from standard connotations of “probability”, but my idea is also somewhat related to the idea of probability as a property of centered possible worlds.
I guess there’s a bit of an inferential gap between us that makes it hard for me to quickly explain the idea to you. From my perspective, it would be much easier if you were already familiar with Algorithmic Information Theory and my UDT ideas, but I’m not sure if you want to read up on all that. Do you see Paul Christiano often? If so, he can probably explain it to you in person fairly quickly. Or, since you’re at FHI, Stuart Armstrong might also know enough about my ideas to explain them to you.
OK, I”ll ask Paul or Stewart next time I see them.
Does your proposal also violate #1 because the simplicity of an observer-situated-in-a-world is a holistic property of the the observer-situated-in-a-world rather than a local one?
Yes (assuming by #1 you mean Period Independence), but it’s not clear to what extent. For example there are at least two kinds of programs that can output a human brain. A) simulate a world and output the object at some space-time location. B) simulate a world and scan for an object matching some criteria, then output such an object. If a time period gets repeated exactly, people’s algorithmic probability from A gets doubled, but algorithmic probability from B doesn’t. I’m not sure at this point whether A dominates B or vice versa.
Also, it’s not clear to me that strict Period Independence is a good thing. It seems reasonable to not value a time period as much if you knew it was an exact repetition of a previous time period. I wrote a post that’s related to this.
I agree that Period Independence may break in the kind of case you describe, though I’m not sure. I don’t think that the kind of case you are describing here is a strong consideration against using Period Independence in cases that don’t involve exact repetition. I think your main example in the post is excellent.
What if we assume Period Independence except for exact repetitions, where the value of extra repetitions eventually go to zero? Perhaps this could be a way to be “timid” while making the downsides of “timidity” seem not so bad or even reasonable? For example in section 6.3.2, such a person would only choose deal 1 over deal 2 if the years of happy lives offered in deal 1 are such that he would already have repeated all possible happy time periods so many times that he values more repetitions very little.
BTW what do you think about my suggestion to do a sequence of blog posts based on your thesis? Or maybe you can at least do one post as a trial run? Also as an unrelated comment, the font in your thesis seems to be such that it’s pretty uncomfortable to read in Adobe Acrobat, unless I zoom in to make the text much larger than I usually have to. Not sure if it’s something you can easily fix. If not, I can try to help if you email me the source of the PDF.
I think it would be interesting if you could show that the space of possible periods-of-lives is structured in such a way that, when combined with a reasonable rule for discounting repetitions, yields a bounded utility function. I don’t have fully developed views on the repetition issue and can imagine that the view has some weird consequences, but if you could do this I would count it as a significant mark in favor of the perspective.
I think this would have some value but isn’t at the top of my list right now.
I think I’ll keep with the current format for citation consistency for now. But I have added a larger font version here.