What if the utilities don’t exactly counterbalance?
The idea is that it’d be great to have a formalism where they do by construction.
Also, when there’s no third party, it’s not distinct enough from Pascal’s Wager as to demand extra terminology that focusses on the third party, such as “Pascal’s Mugging”. If it is just agent doing contemplations by itself, that’s the agent making a wager on it’s hypotheses, not getting mugged by someone.
I’ll just go ahead and use “Pascal Scam” to describe a situation where an in-distinguished agent promises unusually huge pay off, and the mark erroneously gives in due to some combination of bad priors and bad utility evaluation. The common errors seem to be 1: omit the consequence of keeping the money for a more distinguished agent, 2: assign too high prior, 3: and, when picking between approaches, ignore the huge cost of acting in a manner which encourages disinformation. All those errors act in favour of the scammer (and some are optional), while non-erroneous processing would assign huge negative utility to paying up even given high priors.
The idea is that it’d be great to have a formalism where they do by construction.
There is no real way of doing that without changing your probability function or your utility function. However you can’t change those. The real problem is with the expected utility function and I don’t see any way of fixing it, though perhaps I missed something.
Also, when there’s no third party, it’s not distinct enough from Pascal’s Wager as to demand extra terminology that focusses on the third party, such as “Pascal’s Mugging”. If it is just agent doing contemplations by itself, that’s the agent making a wager on it’s hypotheses, not getting mugged by someone.
Any agent subject to Pascal’s Mugging would fall pray to this problem first, and it would be far worse. While the mugger is giving his scenario, the agent could imagine an even more unlikely scenario. Say one where the mugger actually gives him 3^^^^^^3 units of utility if he does some arbitrary task, instead of 3^^^3. This possibility immediately gets so much utility that it far outweighs anything the mugger has to say after that. Then the agent may imagine an even more unlikely scenario where it gets 3^^^^^^^^^^3 units of utility, and so on.
I don’t really know what an agent would do if the expected utility of any action approached infinity. Perhaps it would generally work out as some things would approach infinity faster than others. I admit I didn’t consider that. But I don’t know if that would necessarily be the case. Even if it is it seems “wrong” for expected utilities of everything to be infinite and only tiny probabilities to matter for anything. And if so then it would work out for the pascal’s mugging scenario too I think.
There is no real way of doing that without changing your probability function or your utility function. However you can’t change those.
Last time I checked, priors were fairly subjective even here. We don’t know what is the best way to assign priors. Things like “Solomonoff induction” depend to arbitrary choice of machine.
Any agent subject to Pascal’s Mugging would fall pray to this problem first, and it would be far worse.
Nope, people who end up 419-scammed or waste a lot of money investing into someone like Randel L Mills or Andrea Rossi live through their life ok until they read a harmful string in a harmful set of circumstances (bunch of other believers around for example).
Last time I checked, priors were fairly subjective even here. We don’t know what is the best way to assign priors. Things like “Solomonoff induction” depend to arbitrary choice of machine.
Priors are indeed up for grabs, but a set of priors about the universe ought be consistent with itself, no? A set of priors based only on complexity may indeed not be the best set of priors—that’s what all the discussions about “leverage penalties” and the like are about, enhancing Solomonoff induction with something extra. But what you seem to suggest is a set of priors about the universe that are designed for the express purposes of making human utility calculations balance out? Wouldn’t such a set of priors require the anthroporphization of the universe, and effectively mean sacrificing all sense of epistemic rationality?
The best “priors” about the universe are 1 for what that universe right around you is, and 0 for everything else. Other priors are a compromise, an engineering decision.
What I am thinking is that
there is a considerably better way to assign priors which we do not know of yet—the way which will assign equal probabilities to each side of a die if it has no reason to prefer one over the other—the way that does correspond to symmetries in the evidence.
We don’t know that there will still be same problem when we have a non-stupid way to assign priors (especially as the non-stupid way ought to be considerably more symmetric). And it may be that some value systems are intrinsically incoherent. Suppose you wanted to maximize blerg without knowing what blerg even really is. That wouldn’t be possible, you can’t maximize something without having a measure of it. But I still can tell you i’d give you 3^^^^3 blergs for a dollar, without either of us knowing what blerg is supposed to be or whenever 3^^^^3 blergs even make sense (if blerg is an unique good book of up to 1000 page length, it doesn’t because duplicates aren’t blerg).
Last time I checked, priors were fairly subjective even here. We don’t know what is the best way to assign priors. Things like “Solomonoff induction” depend to arbitrary choice of machine.
True, but the goal of a probability function is to represent the actual probability of an event happening as closely as possible. The map should correspond to the territory. If your map is good, you shouldn’t change it unless you observe actual changes in the territory.
Nope, people who end up 419-scammed or waste a lot of money investing into someone like Randel L Mills or Andrea Rossi live through their life ok until they read a harmful string in a harmful set of circumstances (bunch of other believers around for example).
I don’t know if those things have such extremes in low probability vs high utility to be called pascal’s mugging. But even so, the human brain doesn’t operate on anything like Solomonoff induction, Bayesian probability theory, or expected utility maximization.
The actual probability is either 0 or 1 (either happens or doesn’t happen). Values in-between quantify ignorance and partial knowledge (e.g. when you have no reason to prefer one side of the die to the other), or, at times, are chosen very arbitrarily (what is the probability that a physics theory is “correct”).
I don’t know if those things have such extremes in low probability vs high utility to be called pascal’s mugging.
New names for same things are kind of annoying, to be honest, especially ill chosen… if it happens by your own contemplation, I’d call it Pascal’s Wager. Mugging implies someone making threats, scam is more general and can involve promises of reward. Either way the key is the high payoff proposition wrecking some havoc, either through it’s prior probability being too high, other propositions having been omitted, or the like.
But even so, the human brain doesn’t operate on anything like Solomonoff induction, Bayesian probability theory, or expected utility maximization.
The actual probability is either 0 or 1 (either happens or doesn’t happen).
Yes but the goal is to assign whatever outcome that will actually happen with the highest probability as possible, using whatever information we have. The fact that some outcomes result in ridiculously huge utility gains does not imply anything about how likely they are to happen, so there is no reason that should be taken into account (unless it actually does, in which case it should.)
New names for same things are kind of annoying, to be honest, especially ill chosen… if it happens by your own contemplation, I’d call it Pascal’s Wager. Mugging implies someone making threats, scam is more general and can involve promises of reward. Either way the key is the high payoff proposition wrecking some havoc, either through it’s prior probability being too high, other propositions having been omitted, or the like.
Pascal’s mugging was an absurd scenario with absurd rewards that approach infinity. What you are talking about is just normal everyday scams. Most scams do not promise such huge rewards or have such low probabilities (if you didn’t know any better it is feasible that someone could have an awesome invention or need your help with transaction fees.)
And the problem with scams is that people overestimate their probability. If they were to consider how many emails in the world are actually from Nigerian Princes vs scammers, or how many people promise awesome inventions without any proof they will actually work, they would reconsider. In pascal’s mugging, you fall for it even after having considered the probability of it happening in detail.
Your probability estimation could be absolutely correct. Maybe 1 out of a trillion times a person meets someone claiming to be a matrix lord, they are actually telling the truth. And they still end up getting scammed, so that the 1 in a trillionth counter-factual of themselves gets infinite reward.
But even so, the human brain doesn’t operate on anything like Solomonoff induction, Bayesian probability theory, or expected utility maximization.
People are still agents, though.
They are agents, but they aren’t subject to this specific problem because we don’t really use expected utility maximization. At best maybe some kind of poor approximation of it. But it is a problem for building AIs or any kind of computer system that makes decisions based on probabilities.
Maybe 1 out of a trillion times a person meets someone claiming to be a matrix lord, they are actually telling the truth
I think you’re considering a different problem than Pascal’s Mugging, if you’re taking it as a given that the probabilities are indeed 1 in a trillion (or for that matter 1 in 10). The original problem doesn’t make such an assumption.
What you have in mind, the case of definitely known probabilities, seems to me more like The LifeSpan dilemma where e.g. “an unbounded utility on lifespan implies willingness to trade an 80% probability of living some large number of years for a 1/(3^^^3) probability of living some sufficiently longer lifespan”
The wiki page on it seems to suggest that this is the problem.
If an agent’s utilities over outcomes can potentially grow much faster than the probability of those outcomes diminishes, then it will be dominated by tiny probabilities of hugely important outcomes; speculations about low-probability-high-stakes scenarios will come to dominate his moral decision making… The agent would always have to take those kinds of actions with far-fetched results, that have low but non-negligible probabilities but extremely high returns.
This is seen as an unreasonable result. Intuitively, one is not inclined to acquiesce to the mugger’s demands—or even pay all that much attention one way or another—but what kind of prior does this imply?
Also this
Peter de Blanc has proven[1] that if an agent assigns a finite probability to all computable hypotheses and assigns unboundedly large finite utilities over certain environment inputs, then the expected utility of any outcome is undefined.
which is pretty concerning.
I’m curious what you think the problem with Pascal’s Mugging is though. That you can’t easily estimate the probability of such a situation? Well that is true of anything and isn’t really unique to Pascal’s Mugging. But we can still approximate probabilities. A necessary evil to live in a probabilistic world without the ability to do perfect Bayesian updates on all available information, or unbiased priors.
I’m curious what you think the problem with Pascal’s Mugging is though.
Bad math being internally bad, that’s the problem. Nothing to do with any worlds, real or imaginary, just a case of internally bad math—utilities are undefined, it is undefined if you pay up or not, the actions chosen are undefined. Akin to maximizing blerg without any definition of what blerg even is—maximizing “expected utility” without having defined it.
Speed prior works, for example (it breaks some assumptions of Blanc. Namely, the probability is not bounded from below by any computable function of length of the hypothesis).
utilities are undefined, it is undefined if you pay up or not, the actions chosen are undefined. Akin to maximizing blerg without any definition of what blerg even is—maximizing “expected utility” without having defined it.
Call it undefined if you like, but I’d still prefer 3^^^3 people not suffer. It would be pretty weird to argue that human lives decay in utility based on how many there are. If you found out that the universe was bigger than you thought, that there really were far more humans in the universe somehow, would you just stop caring about human life?
It would also be pretty hard to argue that at least some small amount of money isn’t worth giving in order to save a human life, or that giving a small amount of money isn’t worth a small probability of saving enough lives to make up for the small probability.
It would be pretty weird to argue that human lives decay in utility based on how many there are.
Well, suppose there’s mind uploads, and one mind upload is very worried about himself so he runs himself multiply redundant with 5 exact copies. Should this upload be a minor utility monster?
3^^^3 is far more than there are possible people.
If you found out that the universe was bigger than you thought, that there really were far more humans in the universe somehow, would you just stop caring about human life?
Bounded doesn’t mean it just hits a cap and stays there. Also, if you scale all utilities that you can effect down it changes nothing about actions (another confusion—mapping the utility to how much one cares).
And yes there are definitely cases where money are worth small probability of saving lives, and everyone agrees on such—e.g. if we find out that an asteroid has certain chance to hit Earth, we’d give money to space agencies, even when chance is rather minute (we’d not give money to cold fusion crackpots though). There’s nothing fundamentally wrong with spending a bit to avert a small probability of something terrible happening. The problem arises when the probability is overestimated, when the consequences are poorly evaluated, etc. It is actively harmful for example to encourage boys to cry wolf needlessly. I’m thinking people sort of feel innately that if they are giving money away—losing—some giant fairness fairy is going to make the result more likely good than bad for everyone. World doesn’t work like this; all those naive folks who jump on opportunity to give money to someone promising to save the world, no matter how ignorant, uneducated, or crackpotty that person is in the fields where correctness can be checked at all, they are increasing risk, not decreasing.
It would be pretty weird to argue that human lives decay in utility based on how many there are.
Maybe not as weird as all that. Given a forced choice between killing A and B where I know nothing about them, I flip a coin, but add the knowledge that A is a duplicate of C and B is not a duplicate of anyone, and I choose A quite easily. I conclude from this that I value unique human lives quite a lot more than I value non-unique human lives. As others have pointed out, the number of unique human lives is finite, and the number of lives I consider worth living necessarily even lower, so the more people there are living lives worth living, the less unique any individual is, and therefore the less I value any individual life. (Insofar as my values are consistent, anyway. Which of course they aren’t, but this whole “lets pretend” game of utility calculation that we enjoy playing depends on treating them as though they were.)
There is no evidence for the actual existence of neatly walled-of and unupdateable utility functions or probability functions, any more than there is for a luz’.
Utility and probability functions are not perfect or neatly walled off. But that doesn’t mean you should change them to fix a problem with your expected utility function. The goal of a probability function is to represent the actual probability of an event happening as closely as possible. And the goal of a utility function is to represent what you states you would prefer the universe to be in. This also shouldn’t change unless you’ve actually changed your preferences.
And the goal of a utility function is to represent what you states you would prefer the universe to be in. This also shouldn’t change unless you’ve actually changed your preferences.
There’s plenty of evidence of people changing their preferences over significant periods of time: it would be weird not to.
Of course people can change their preferences. But if your preferences are not consistent you will likely end up in situations that are less preferable than if you had the same preferences the entire time. It also makes you a potential money pump.
And I am well aware that the theory of stable utility functions is standardly patched up with a further theory of terminal values, for which there is also no direct evidence.
What? Terminal values are not a patch for utility functions. It’s basically another word that means the same thing, what state you would prefer the world to end up in. And how can there be evidence for a decision theory?
Terminal values are not a patch for utility functions.
Well, I’ve certainly seen discussions here in which the observed inconsistency among our professed values is treated as a non-problem on the grounds that those are mere instrumental values, and our terminal values are presumed to be more consistent than that.
Insofar as stable utility functions depend on consistent values, it’s not unreasonable to describe such discussions as positing consistent terminal values in order to support a belief in stable utility functions.
I don’t know what you mean. All I’m saying is that you shouldn’t change your preferences because of a problem with your expected utility function. Your preferences are just what you want. Utility functions are just a mathematical way of expressing that.
I don’t see why our preferences can’t be expressed by a utility function even as they are. The only reason it wouldn’t work out is if there were circular preferences, and I don’t think most peoples preferences would work out to be truly circular if they were to think about the specific occurrence and decide what they really preferred.
Though mapping out which outcomes are more preferred than others is not enough to assign them an actual utility, you’d somehow have to guess how much more preferable one outcome is to another quantitatively.But even then I think most people could if they thought about it enough. The problem is that our utility functions are complex and we don’t really know what they are, not that they don’t exist.
I don’t see why our preferences can’t be expressed by a utility function even as they are. The only reason it wouldn’t work out is if there were circular preferences, and I don’t think most peoples preferences would work out to be truly circular if they were to think about the specific occurrence and decide what they really preferred.
Or they might violate the independence axiom, but in any case what do you mean by ” think about the specific occurrence and decide what they really preferred”, since the result of such thinking is likely to depend on the exact order they thought about things in.
The idea is that it’d be great to have a formalism where they do by construction.
Also, when there’s no third party, it’s not distinct enough from Pascal’s Wager as to demand extra terminology that focusses on the third party, such as “Pascal’s Mugging”. If it is just agent doing contemplations by itself, that’s the agent making a wager on it’s hypotheses, not getting mugged by someone.
I’ll just go ahead and use “Pascal Scam” to describe a situation where an in-distinguished agent promises unusually huge pay off, and the mark erroneously gives in due to some combination of bad priors and bad utility evaluation. The common errors seem to be 1: omit the consequence of keeping the money for a more distinguished agent, 2: assign too high prior, 3: and, when picking between approaches, ignore the huge cost of acting in a manner which encourages disinformation. All those errors act in favour of the scammer (and some are optional), while non-erroneous processing would assign huge negative utility to paying up even given high priors.
There is no real way of doing that without changing your probability function or your utility function. However you can’t change those. The real problem is with the expected utility function and I don’t see any way of fixing it, though perhaps I missed something.
Any agent subject to Pascal’s Mugging would fall pray to this problem first, and it would be far worse. While the mugger is giving his scenario, the agent could imagine an even more unlikely scenario. Say one where the mugger actually gives him 3^^^^^^3 units of utility if he does some arbitrary task, instead of 3^^^3. This possibility immediately gets so much utility that it far outweighs anything the mugger has to say after that. Then the agent may imagine an even more unlikely scenario where it gets 3^^^^^^^^^^3 units of utility, and so on.
I don’t really know what an agent would do if the expected utility of any action approached infinity. Perhaps it would generally work out as some things would approach infinity faster than others. I admit I didn’t consider that. But I don’t know if that would necessarily be the case. Even if it is it seems “wrong” for expected utilities of everything to be infinite and only tiny probabilities to matter for anything. And if so then it would work out for the pascal’s mugging scenario too I think.
Last time I checked, priors were fairly subjective even here. We don’t know what is the best way to assign priors. Things like “Solomonoff induction” depend to arbitrary choice of machine.
Nope, people who end up 419-scammed or waste a lot of money investing into someone like Randel L Mills or Andrea Rossi live through their life ok until they read a harmful string in a harmful set of circumstances (bunch of other believers around for example).
Priors are indeed up for grabs, but a set of priors about the universe ought be consistent with itself, no? A set of priors based only on complexity may indeed not be the best set of priors—that’s what all the discussions about “leverage penalties” and the like are about, enhancing Solomonoff induction with something extra. But what you seem to suggest is a set of priors about the universe that are designed for the express purposes of making human utility calculations balance out? Wouldn’t such a set of priors require the anthroporphization of the universe, and effectively mean sacrificing all sense of epistemic rationality?
The best “priors” about the universe are 1 for what that universe right around you is, and 0 for everything else. Other priors are a compromise, an engineering decision.
What I am thinking is that
there is a considerably better way to assign priors which we do not know of yet—the way which will assign equal probabilities to each side of a die if it has no reason to prefer one over the other—the way that does correspond to symmetries in the evidence.
We don’t know that there will still be same problem when we have a non-stupid way to assign priors (especially as the non-stupid way ought to be considerably more symmetric). And it may be that some value systems are intrinsically incoherent. Suppose you wanted to maximize blerg without knowing what blerg even really is. That wouldn’t be possible, you can’t maximize something without having a measure of it. But I still can tell you i’d give you 3^^^^3 blergs for a dollar, without either of us knowing what blerg is supposed to be or whenever 3^^^^3 blergs even make sense (if blerg is an unique good book of up to 1000 page length, it doesn’t because duplicates aren’t blerg).
True, but the goal of a probability function is to represent the actual probability of an event happening as closely as possible. The map should correspond to the territory. If your map is good, you shouldn’t change it unless you observe actual changes in the territory.
I don’t know if those things have such extremes in low probability vs high utility to be called pascal’s mugging. But even so, the human brain doesn’t operate on anything like Solomonoff induction, Bayesian probability theory, or expected utility maximization.
The actual probability is either 0 or 1 (either happens or doesn’t happen). Values in-between quantify ignorance and partial knowledge (e.g. when you have no reason to prefer one side of the die to the other), or, at times, are chosen very arbitrarily (what is the probability that a physics theory is “correct”).
New names for same things are kind of annoying, to be honest, especially ill chosen… if it happens by your own contemplation, I’d call it Pascal’s Wager. Mugging implies someone making threats, scam is more general and can involve promises of reward. Either way the key is the high payoff proposition wrecking some havoc, either through it’s prior probability being too high, other propositions having been omitted, or the like.
People are still agents, though.
Yes but the goal is to assign whatever outcome that will actually happen with the highest probability as possible, using whatever information we have. The fact that some outcomes result in ridiculously huge utility gains does not imply anything about how likely they are to happen, so there is no reason that should be taken into account (unless it actually does, in which case it should.)
Pascal’s mugging was an absurd scenario with absurd rewards that approach infinity. What you are talking about is just normal everyday scams. Most scams do not promise such huge rewards or have such low probabilities (if you didn’t know any better it is feasible that someone could have an awesome invention or need your help with transaction fees.)
And the problem with scams is that people overestimate their probability. If they were to consider how many emails in the world are actually from Nigerian Princes vs scammers, or how many people promise awesome inventions without any proof they will actually work, they would reconsider. In pascal’s mugging, you fall for it even after having considered the probability of it happening in detail.
Your probability estimation could be absolutely correct. Maybe 1 out of a trillion times a person meets someone claiming to be a matrix lord, they are actually telling the truth. And they still end up getting scammed, so that the 1 in a trillionth counter-factual of themselves gets infinite reward.
They are agents, but they aren’t subject to this specific problem because we don’t really use expected utility maximization. At best maybe some kind of poor approximation of it. But it is a problem for building AIs or any kind of computer system that makes decisions based on probabilities.
I think you’re considering a different problem than Pascal’s Mugging, if you’re taking it as a given that the probabilities are indeed 1 in a trillion (or for that matter 1 in 10). The original problem doesn’t make such an assumption.
What you have in mind, the case of definitely known probabilities, seems to me more like The LifeSpan dilemma where e.g. “an unbounded utility on lifespan implies willingness to trade an 80% probability of living some large number of years for a 1/(3^^^3) probability of living some sufficiently longer lifespan”
The wiki page on it seems to suggest that this is the problem.
Also this
which is pretty concerning.
I’m curious what you think the problem with Pascal’s Mugging is though. That you can’t easily estimate the probability of such a situation? Well that is true of anything and isn’t really unique to Pascal’s Mugging. But we can still approximate probabilities. A necessary evil to live in a probabilistic world without the ability to do perfect Bayesian updates on all available information, or unbiased priors.
I abhor using unnecessary novel jargon.
Bad math being internally bad, that’s the problem. Nothing to do with any worlds, real or imaginary, just a case of internally bad math—utilities are undefined, it is undefined if you pay up or not, the actions chosen are undefined. Akin to maximizing blerg without any definition of what blerg even is—maximizing “expected utility” without having defined it.
Speed prior works, for example (it breaks some assumptions of Blanc. Namely, the probability is not bounded from below by any computable function of length of the hypothesis).
Call it undefined if you like, but I’d still prefer 3^^^3 people not suffer. It would be pretty weird to argue that human lives decay in utility based on how many there are. If you found out that the universe was bigger than you thought, that there really were far more humans in the universe somehow, would you just stop caring about human life?
It would also be pretty hard to argue that at least some small amount of money isn’t worth giving in order to save a human life, or that giving a small amount of money isn’t worth a small probability of saving enough lives to make up for the small probability.
Well, suppose there’s mind uploads, and one mind upload is very worried about himself so he runs himself multiply redundant with 5 exact copies. Should this upload be a minor utility monster?
3^^^3 is far more than there are possible people.
Bounded doesn’t mean it just hits a cap and stays there. Also, if you scale all utilities that you can effect down it changes nothing about actions (another confusion—mapping the utility to how much one cares).
And yes there are definitely cases where money are worth small probability of saving lives, and everyone agrees on such—e.g. if we find out that an asteroid has certain chance to hit Earth, we’d give money to space agencies, even when chance is rather minute (we’d not give money to cold fusion crackpots though). There’s nothing fundamentally wrong with spending a bit to avert a small probability of something terrible happening. The problem arises when the probability is overestimated, when the consequences are poorly evaluated, etc. It is actively harmful for example to encourage boys to cry wolf needlessly. I’m thinking people sort of feel innately that if they are giving money away—losing—some giant fairness fairy is going to make the result more likely good than bad for everyone. World doesn’t work like this; all those naive folks who jump on opportunity to give money to someone promising to save the world, no matter how ignorant, uneducated, or crackpotty that person is in the fields where correctness can be checked at all, they are increasing risk, not decreasing.
Maybe not as weird as all that. Given a forced choice between killing A and B where I know nothing about them, I flip a coin, but add the knowledge that A is a duplicate of C and B is not a duplicate of anyone, and I choose A quite easily. I conclude from this that I value unique human lives quite a lot more than I value non-unique human lives. As others have pointed out, the number of unique human lives is finite, and the number of lives I consider worth living necessarily even lower, so the more people there are living lives worth living, the less unique any individual is, and therefore the less I value any individual life. (Insofar as my values are consistent, anyway. Which of course they aren’t, but this whole “lets pretend” game of utility calculation that we enjoy playing depends on treating them as though they were.)
There is no evidence for the actual existence of neatly walled-of and unupdateable utility functions or probability functions, any more than there is for a luz’.
Utility and probability functions are not perfect or neatly walled off. But that doesn’t mean you should change them to fix a problem with your expected utility function. The goal of a probability function is to represent the actual probability of an event happening as closely as possible. And the goal of a utility function is to represent what you states you would prefer the universe to be in. This also shouldn’t change unless you’ve actually changed your preferences.
There’s plenty of evidence of people changing their preferences over significant periods of time: it would be weird not to. And I am well aware that the theory of stable utility functions is standardly patched up with a further theory of terminal values, for which there is also no direct evidence.
Of course people can change their preferences. But if your preferences are not consistent you will likely end up in situations that are less preferable than if you had the same preferences the entire time. It also makes you a potential money pump.
What? Terminal values are not a patch for utility functions. It’s basically another word that means the same thing, what state you would prefer the world to end up in. And how can there be evidence for a decision theory?
Well, I’ve certainly seen discussions here in which the observed inconsistency among our professed values is treated as a non-problem on the grounds that those are mere instrumental values, and our terminal values are presumed to be more consistent than that.
Insofar as stable utility functions depend on consistent values, it’s not unreasonable to describe such discussions as positing consistent terminal values in order to support a belief in stable utility functions.
Well, how is this different from changing our preferences to utility functions to fix problems with our naive preferences?
I don’t know what you mean. All I’m saying is that you shouldn’t change your preferences because of a problem with your expected utility function. Your preferences are just what you want. Utility functions are just a mathematical way of expressing that.
Human preferences don’t naturally satisfy the VNM axioms, thus by expressing them as a utility function you’ve already changed them.
I don’t see why our preferences can’t be expressed by a utility function even as they are. The only reason it wouldn’t work out is if there were circular preferences, and I don’t think most peoples preferences would work out to be truly circular if they were to think about the specific occurrence and decide what they really preferred.
Though mapping out which outcomes are more preferred than others is not enough to assign them an actual utility, you’d somehow have to guess how much more preferable one outcome is to another quantitatively.But even then I think most people could if they thought about it enough. The problem is that our utility functions are complex and we don’t really know what they are, not that they don’t exist.
Or they might violate the independence axiom, but in any case what do you mean by ” think about the specific occurrence and decide what they really preferred”, since the result of such thinking is likely to depend on the exact order they thought about things in.