decision theory is no substitute for utility function
some people, upon learning about decision theories such as LDT and how it cooperates on problems such as the prisoner’s dilemma, end up believing the following:
my utility function is about what i want for just me; but i’m altruistic (/egalitarian/cosmopolitan/pro-fairness/etc) because decision theory says i should cooperate with other agents. decision theoritic cooperation is the true name of altruism.
it’s possible that this is true for some people, but in general i expect that to be a mistaken analysis of their values.
decision theory cooperates with agents relative to how much power they have, and only when it’s instrumental.
in my opinion, real altruism (/egalitarianism/cosmopolitanism/fairness/etc) should be in the utility function which the decision theory is instrumental to. i actually intrinsically care about others; i don’t just care about others instrumentally because it helps me somehow.
some important aspects that my utility-function-altruism differs from decision-theoritic-cooperation includes:
i care about people weighed by moral patienthood, decision theory only cares about agents weighed by negotiation power. if an alien superintelligence is very powerful but isn’t a moral patient, then i will only cooperate with it instrumentally (for example because i care about the alien moral patients that it has been in contact with); if cooperating with it doesn’t help my utility function (which, again, includes altruism towards aliens) then i won’t cooperate with that alien superintelligence. corollarily, i will take actions that cause nice things to happen to people even if they’ve very impoverished (and thus don’t have much LDT negotiation power) and it doesn’t help any other aspect of my utility function than just the fact that i value that they’re okay.
if i can switch to a better decision theory, or if fucking over some non-moral-patienty agents helps me somehow, then i’ll happily do that; i don’t have goal-content integrity about my decision theory. i do have goal-content integrity about my utility function: i don’t want to become someone who wants moral patients to unconsentingly-die or suffer, for example.
there seems to be a sense in which some decision theories are better than others, because they’re ultimately instrumental to one’s utility function. utility functions, however, don’t have an objective measure for how good they are. hence, moral anti-realism is true: there isn’t a Single Correct Utility Function.
decision theory is instrumental; the utility function is where the actual intrinsic/axiomatic/terminal goals/values/preferences are stored. usually, i also interpret “morality” and “ethics” as “terminal values”, since most of the stuff that those seem to care about looks like terminal values to me. for example, i will want fairness between moral patients intrinsically, not just because my decision theory says that that’s instrumental to me somehow.
An interesting question for me is how much true altruism is required to give rise to a generally altruistic society under high quality coordination frameworks. I suspect it’s quite small.
Another question is whether building coordination frameworks to any degree requires some background of altruism. I suspect that this is the case. It’s the hypothesis I’ve accreted for explaining the success of post-war economies (guessing that war leads to a boom in nationalistic altruism, generally increased fairness and mutual faith).
ah, it also annoys me when people say that caring about others can only be instrumental.
what does it even mean? helping other people makes me feel happy. watching a nice movie makes me feel happy. the argument that I don’t “really” care about other people would also prove that I don’t “really” care about movies etc.
I am happy for the lucky coincidence that decision theories sometimes endorse cooperation, but I would probably do that regardless. for example, if I had an option to donate something useful to million people, or sell it to dozen people, I would probably choose the former option even if it meant no money for me. (and yes, I would hope there would be some win/win solution, such as the million people paying me via Kickstarter. but in the inconvenient universe where Kickstarter is somehow not an option, I am going to donate anyway.)
There actually is a meaningful question there: Would you enter the experience machine? Or do you need it to be real. Do you just want the experience of pleasing others or do you need those people being pleased out there to actually exist.
There are a lot of people who really think, and might truly be experience oriented. If given the ability, they would instantly self-modify into a Victory Psychopath Protecting A Dream.
I don’t have an explicit theory of how this works; for example, I would consider “pleasing others” in an experience machine meaningless, but “eating a cake” in an experience machine seems just as okay as in real life (maybe even preferable, considering that cakes are unhealthy). A fake memory of “having eaten a cake” would be a bad thing; “making people happier by talking to them” in an experience machine would be intrinsically meaningless, but it might help me improve my actual social skills, which would be valuable. Sometimes I care about the referent being real (the people I would please), sometimes I don’t (the cake I would eat). But it’s not the people/cake distinction per se; for example in case of using fake simulated people to practice social skills, the emphasis is on the skills being real; I would be disappointed if the experience machine merely gave me a fake “feeling of having improved my skills”.
I imagine that for a psychopath everything and everyone is instrumental, so there would be no downside to the experience machine (except for the risk of someone turning it off). But this is just a guess.
I suspect that analyzing “the true preferences” is tricky, because ultimately we are built of atoms, and atoms have no preferences. So the question is whether by focusing on some aspect of the human mind we got better insight to its true nature, or whether we have just eliminated the context that was necessary for it to make sense.
My utility function is pretty much just my own happiness (in a fun-theoretic rather than purely hedonistic sense). However, my decision theory is updateless with respect to which sentient being I ended up as, so once you factor that in, I’m a multiverse-wide realityfluid-weighted average utilitarian.
I’m not sure how correct this is, but it’s possible.
It certainly is possible! In more decision-theoritic terms, I’d describe this as “it sure would suck if agents in my reference class just optimized for their own happiness; it seems like the instrumental thing for agents in my reference class to do is maximize for everyone’s happiness”. Which is probly correct!
But as per my post, I’d describe this position as “not intrinsically altruistic” — you’re optimizing for everyone’s happiness because “it sure would sure if agents in my reference class didn’t do that”, not because you intrinsically value that everyone be happy, regardless of reasoning about agents and reference classes and veils of ignorance.
I agree, though I haven’t seen many proposing that, but also see So8res’ Decision theory does not imply that we get to have nice things, though this is coming from the opposite direction (with the start being about people invalidly assuming too much out of LDT cooperation)
Though for our morals, I do think there’s an active question of which pieces we feel better replacing with the more formal understanding, because there isn’t a sharp distinction between our utility function and our decision theory. Some values trump others when given better tools. Though I agree that replacing all the altruism components is many steps farther than is the best solution in that regard.
decision theory is no substitute for utility function
some people, upon learning about decision theories such as LDT and how it cooperates on problems such as the prisoner’s dilemma, end up believing the following:
it’s possible that this is true for some people, but in general i expect that to be a mistaken analysis of their values.
decision theory cooperates with agents relative to how much power they have, and only when it’s instrumental.
in my opinion, real altruism (/egalitarianism/cosmopolitanism/fairness/etc) should be in the utility function which the decision theory is instrumental to. i actually intrinsically care about others; i don’t just care about others instrumentally because it helps me somehow.
some important aspects that my utility-function-altruism differs from decision-theoritic-cooperation includes:
i care about people weighed by moral patienthood, decision theory only cares about agents weighed by negotiation power. if an alien superintelligence is very powerful but isn’t a moral patient, then i will only cooperate with it instrumentally (for example because i care about the alien moral patients that it has been in contact with); if cooperating with it doesn’t help my utility function (which, again, includes altruism towards aliens) then i won’t cooperate with that alien superintelligence. corollarily, i will take actions that cause nice things to happen to people even if they’ve very impoverished (and thus don’t have much LDT negotiation power) and it doesn’t help any other aspect of my utility function than just the fact that i value that they’re okay.
if i can switch to a better decision theory, or if fucking over some non-moral-patienty agents helps me somehow, then i’ll happily do that; i don’t have goal-content integrity about my decision theory. i do have goal-content integrity about my utility function: i don’t want to become someone who wants moral patients to unconsentingly-die or suffer, for example.
there seems to be a sense in which some decision theories are better than others, because they’re ultimately instrumental to one’s utility function. utility functions, however, don’t have an objective measure for how good they are. hence, moral anti-realism is true: there isn’t a Single Correct Utility Function.
decision theory is instrumental; the utility function is where the actual intrinsic/axiomatic/terminal goals/values/preferences are stored. usually, i also interpret “morality” and “ethics” as “terminal values”, since most of the stuff that those seem to care about looks like terminal values to me. for example, i will want fairness between moral patients intrinsically, not just because my decision theory says that that’s instrumental to me somehow.
An interesting question for me is how much true altruism is required to give rise to a generally altruistic society under high quality coordination frameworks. I suspect it’s quite small.
Another question is whether building coordination frameworks to any degree requires some background of altruism. I suspect that this is the case. It’s the hypothesis I’ve accreted for explaining the success of post-war economies (guessing that war leads to a boom in nationalistic altruism, generally increased fairness and mutual faith).
ah, it also annoys me when people say that caring about others can only be instrumental.
what does it even mean? helping other people makes me feel happy. watching a nice movie makes me feel happy. the argument that I don’t “really” care about other people would also prove that I don’t “really” care about movies etc.
I am happy for the lucky coincidence that decision theories sometimes endorse cooperation, but I would probably do that regardless. for example, if I had an option to donate something useful to million people, or sell it to dozen people, I would probably choose the former option even if it meant no money for me. (and yes, I would hope there would be some win/win solution, such as the million people paying me via Kickstarter. but in the inconvenient universe where Kickstarter is somehow not an option, I am going to donate anyway.)
There actually is a meaningful question there: Would you enter the experience machine? Or do you need it to be real. Do you just want the experience of pleasing others or do you need those people being pleased out there to actually exist.
There are a lot of people who really think, and might truly be experience oriented. If given the ability, they would instantly self-modify into a Victory Psychopath Protecting A Dream.
I don’t have an explicit theory of how this works; for example, I would consider “pleasing others” in an experience machine meaningless, but “eating a cake” in an experience machine seems just as okay as in real life (maybe even preferable, considering that cakes are unhealthy). A fake memory of “having eaten a cake” would be a bad thing; “making people happier by talking to them” in an experience machine would be intrinsically meaningless, but it might help me improve my actual social skills, which would be valuable. Sometimes I care about the referent being real (the people I would please), sometimes I don’t (the cake I would eat). But it’s not the people/cake distinction per se; for example in case of using fake simulated people to practice social skills, the emphasis is on the skills being real; I would be disappointed if the experience machine merely gave me a fake “feeling of having improved my skills”.
I imagine that for a psychopath everything and everyone is instrumental, so there would be no downside to the experience machine (except for the risk of someone turning it off). But this is just a guess.
I suspect that analyzing “the true preferences” is tricky, because ultimately we are built of atoms, and atoms have no preferences. So the question is whether by focusing on some aspect of the human mind we got better insight to its true nature, or whether we have just eliminated the context that was necessary for it to make sense.
What about the following:
My utility function is pretty much just my own happiness (in a fun-theoretic rather than purely hedonistic sense). However, my decision theory is updateless with respect to which sentient being I ended up as, so once you factor that in, I’m a multiverse-wide realityfluid-weighted average utilitarian.
I’m not sure how correct this is, but it’s possible.
It certainly is possible! In more decision-theoritic terms, I’d describe this as “it sure would suck if agents in my reference class just optimized for their own happiness; it seems like the instrumental thing for agents in my reference class to do is maximize for everyone’s happiness”. Which is probly correct!
But as per my post, I’d describe this position as “not intrinsically altruistic” — you’re optimizing for everyone’s happiness because “it sure would sure if agents in my reference class didn’t do that”, not because you intrinsically value that everyone be happy, regardless of reasoning about agents and reference classes and veils of ignorance.
I agree, though I haven’t seen many proposing that, but also see So8res’ Decision theory does not imply that we get to have nice things, though this is coming from the opposite direction (with the start being about people invalidly assuming too much out of LDT cooperation)
Though for our morals, I do think there’s an active question of which pieces we feel better replacing with the more formal understanding, because there isn’t a sharp distinction between our utility function and our decision theory. Some values trump others when given better tools. Though I agree that replacing all the altruism components is many steps farther than is the best solution in that regard.