Wow, thank you this is a really well made point. I see now how accounting for future lives seems like double counting their desires with our own desires to have their desires fulfilled.
You already put a lot of effort into a response so don’t feel obliged to respond but some things in my mind that I need to work out about this:
Can’t this argument do a bit too much “I’m not factoring in the utility of X (future peoples happiness) but I am instead factoring in my desires to make X a reality” in a way that you could apply it to any utility function parameter. For instance, “my utility calculation doesn’t factor in actually (preventing malaria), but I am instead factoring in my desires to (prevent malaria).” Maybe while the sentiment is true, it seems like it can apply to everything.
Another unrelated thought I am working with. If the goal is to have grandpa on his deathbed happy thinking that the future will be okay, wouldn’t this goal be satisfied by lying to grandpa? In other words if we have an all powerful aligned AGI and tell it give it the utility function you outlined, where we maximize our desires to have the future happy. Wouldn’t it just find a way to make us think the future would be okay by doing things we think would work? As opposed to actually assigning utility to future people actually being happy, which the AGI would then actually improve the future.
You helped me see the issues with assigning any utility to future people. You changed my opinion that that isn’t great. I guess I am struggling to accept your alternative as I think it may have a lot of the same issues if not a few more.
Can’t this argument do a bit too much “I’m not factoring in the utility of X (future peoples happiness) but I am instead factoring in my desires to make X a reality” in a way that you could apply it to any utility function parameter. For instance, “my utility calculation doesn’t factor in actually (preventing malaria), but I am instead factoring in my desires to (prevent malaria).” Maybe while the sentiment is true, it seems like it can apply to everything.
I think this operates on two different level.
The level I am discussing is: “I consider all sentient beings currently alive as moral subject; I consider any potential but not-yet-existing sentient beings not subjects, but objects of the existing beings’ values”.
The one you’re proposing is one far removed, sort of the “solipsist” level: “I consider only myself as the sole existing moral subject, as I can only be sure of my own inner experiences; however I feel empathy and compassions for these possibly-P-zombie creatures that move around me, thus I do good because I enjoy the warm fuzzies that follow”.
Ultimately I do think in some sense all our morality is rooted in that sort of personal moral intuition. If no one felt any kind of empathy or concern for others we probably wouldn’t have morality at all! But yeah, I do decide out of sheer symmetry that it’s not likely that somehow I am the only special human being who experiences the world, and thus I should treat all others as I would myself.
Meanwhile with future humans the asymmetry is real. Time translation doesn’t work the same way as space translation does. Also, even if I wanted to follow a logical extension step of “well, most humans care about the future, therefore I may as well take a shortcut and consider future humans themselves as moral subjects, to account in a more direct way for the same effect”, then I should do so by weighing approximately how actual living humans care about the future, which in overwhelming majority means “our kids and grandkids”, and not the longtermists’ future hypothetical em clusters.
If the goal is to have grandpa on his deathbed happy thinking that the future will be okay, wouldn’t this goal be satisfied by lying to grandpa? In other words if we have an all powerful aligned AGI and tell it give it the utility function you outlined, where we maximize our desires to have the future happy. Wouldn’t it just find a way to make us think the future would be okay by doing things we think would work? As opposed to actually assigning utility to future people actually being happy, which the AGI would then actually improve the future.
Well, I suppose yes, you could in theory end up with an AGI whose goal is simply letting us all die believing that everyone will be happy after us. The fiction would be horribly complex; we’d need to be all sterilised but actually deliver babies that are the AGI’s construct (lest more humans come into the equation who need to be deceived!). Then when the last true human dies, the AGI goes “haha, got you suckers” and starts disassembling everything; or turns itself off, having completed its purpose. Not sure how to dodge that (or even whether I can say it’s strictly speaking bad, though I guess it is insofar as people don’t like being lied to), but I think it’d be a lot simpler for the AGI to just actually make the future good.
There is a meta question here whether morality is based on personal intuition or calculations. My own inclination is that utility calculations would only make a difference “in the margin” but the high level decision are made by our moral intuition.
That is, we can do calculations to decide if we fund Charity A or Charity B in similar areas, but I doubt that for most people major moral decisions actually (or should) boil down to calculating utility functions.
But of course to each their own, and if someone finds math useful to make such decisions then whom am I to tell them not to do it.
Yeah, I think calculations can be a tool but ultimately when deciding a framework we’re trying to synthesise our intuitions into a simple set of axioms from which everything proceeds. But the intuitions remain the origin of it all. You could design some game theoretical framework for what guarantees a society to run best without appealing to any moral intuition, but that would probably look quite alien and cold. Morality is one of our terminal values, we just try to make sense of it.
Wow, thank you this is a really well made point. I see now how accounting for future lives seems like double counting their desires with our own desires to have their desires fulfilled.
You already put a lot of effort into a response so don’t feel obliged to respond but some things in my mind that I need to work out about this:
Can’t this argument do a bit too much “I’m not factoring in the utility of X (future peoples happiness) but I am instead factoring in my desires to make X a reality” in a way that you could apply it to any utility function parameter. For instance, “my utility calculation doesn’t factor in actually (preventing malaria), but I am instead factoring in my desires to (prevent malaria).” Maybe while the sentiment is true, it seems like it can apply to everything.
Another unrelated thought I am working with. If the goal is to have grandpa on his deathbed happy thinking that the future will be okay, wouldn’t this goal be satisfied by lying to grandpa? In other words if we have an all powerful aligned AGI and tell it give it the utility function you outlined, where we maximize our desires to have the future happy. Wouldn’t it just find a way to make us think the future would be okay by doing things we think would work? As opposed to actually assigning utility to future people actually being happy, which the AGI would then actually improve the future.
You helped me see the issues with assigning any utility to future people. You changed my opinion that that isn’t great. I guess I am struggling to accept your alternative as I think it may have a lot of the same issues if not a few more.
I think this operates on two different level.
The level I am discussing is: “I consider all sentient beings currently alive as moral subject; I consider any potential but not-yet-existing sentient beings not subjects, but objects of the existing beings’ values”.
The one you’re proposing is one far removed, sort of the “solipsist” level: “I consider only myself as the sole existing moral subject, as I can only be sure of my own inner experiences; however I feel empathy and compassions for these possibly-P-zombie creatures that move around me, thus I do good because I enjoy the warm fuzzies that follow”.
Ultimately I do think in some sense all our morality is rooted in that sort of personal moral intuition. If no one felt any kind of empathy or concern for others we probably wouldn’t have morality at all! But yeah, I do decide out of sheer symmetry that it’s not likely that somehow I am the only special human being who experiences the world, and thus I should treat all others as I would myself.
Meanwhile with future humans the asymmetry is real. Time translation doesn’t work the same way as space translation does. Also, even if I wanted to follow a logical extension step of “well, most humans care about the future, therefore I may as well take a shortcut and consider future humans themselves as moral subjects, to account in a more direct way for the same effect”, then I should do so by weighing approximately how actual living humans care about the future, which in overwhelming majority means “our kids and grandkids”, and not the longtermists’ future hypothetical em clusters.
Well, I suppose yes, you could in theory end up with an AGI whose goal is simply letting us all die believing that everyone will be happy after us. The fiction would be horribly complex; we’d need to be all sterilised but actually deliver babies that are the AGI’s construct (lest more humans come into the equation who need to be deceived!). Then when the last true human dies, the AGI goes “haha, got you suckers” and starts disassembling everything; or turns itself off, having completed its purpose. Not sure how to dodge that (or even whether I can say it’s strictly speaking bad, though I guess it is insofar as people don’t like being lied to), but I think it’d be a lot simpler for the AGI to just actually make the future good.
There is a meta question here whether morality is based on personal intuition or calculations. My own inclination is that utility calculations would only make a difference “in the margin” but the high level decision are made by our moral intuition.
That is, we can do calculations to decide if we fund Charity A or Charity B in similar areas, but I doubt that for most people major moral decisions actually (or should) boil down to calculating utility functions.
But of course to each their own, and if someone finds math useful to make such decisions then whom am I to tell them not to do it.
Yeah, I think calculations can be a tool but ultimately when deciding a framework we’re trying to synthesise our intuitions into a simple set of axioms from which everything proceeds. But the intuitions remain the origin of it all. You could design some game theoretical framework for what guarantees a society to run best without appealing to any moral intuition, but that would probably look quite alien and cold. Morality is one of our terminal values, we just try to make sense of it.