Why should I behave as if my values satisfy the VNM axioms?
Rationality here is typically defined as either epistemic (make sure your mental models match reality well) or instrumental (make sure the steps you take actually lead to your goals).
Defining rationality as “you MUST have a single utility function which MUST follow VNM” doesn’t strike me as a good idea.
Because the VNM axioms seem so intuitively obvious that violating them strongly feels like making an error. Of course I cannot prove them without introducing another set of axioms which can be questioned in turn etc. You always need to start with some assumptions.
I would reject the completeness axiom. I often face choices where I don’t know which option I prefer, but where I would not agree that I am indifferent. And I’m okay with this fact.
I also reject the transitivity axiom—intransitive preference is an observed fact for real humans in a wide variety of settings. And you might say this is irrational, but my preference are what they are.
Can you give an example of situations A, B, C for which your preferences are A > B, B > C, C > A? What would you do if you need to choose between A, B, C?
Sure. I’ll go to the grocery store and have three kinds of tomato sauce and I’ll look at A and B, and pick B, then B and C, pick C, and C and A, and pick A. And I’ll stare at them indecisively until my preferences shift. It’s sort of ridiculous—it can take something like a minute to decide. This is NOT the same as feeling indifferent, in which case I would just pick one and go.
I have similar experiences when choosing between entertainment options, transport, etc. My impression is that this is an experience that many people have.
It seems to me that what you’re describing are not preferences but spur of the moment decisions. A preference should be thought of as in CEV: the thing you would prefer if you thought about it long enough, knew enough, were more the person you want to be etc. The mere fact you somehow decide between the sauces in the end suggests you’re not describing a preference. Also I doubt that you have terminal values related to tomato sauce. More likely, your terminal values involve something like “experiencing pleasure” and your problem here is epistemic rather than “moral”: you’re not sure which sauce would give you more pleasure.
You are using preference to mean something other than I thought you were.
I’m not convinced that the CEV definition of preference is useful. No actual human ever has infinite time or information; we are always making decisions while we are limited computationally and informationally. You can’t just define away those limits. And I’m not at all convinced that our preferences would converge even given infinite time. That’s an assumption, not a theorem.
When buying pasta sauce, I have multiple incommensurable values: money, health, and taste. And in general, when you have multiple criteria, there’s no non-paradoxical way to do rankings. (This is basically Arrow’s theorem). And I suspect that’s the cause for my lack of preference ordering.
No actual human ever has infinite time or information
Of course. But rationality means your decisions should be as close as possible to the decisions you would make if you had infinite time and information.
When buying pasta sauce, I have multiple incommensurable values: money, health, and taste.
Money is not a terminal value for most people. I suspect you want money because of the things it can buy you, not as a value in itself. I think health is also instrumental. We value health because illness is unpleasant, might lead to death and generally interferes with taking actions to optimize our values. The unpleasant sensations of illness might well be commensurable with the pleasant sensations of taste. For example you would probably pass up a gourmet meal if eating it implies getting cancer.
But rationality means your decisions should be as close as possible to the decisions you would make if you had infinite time and information.
However you can not know what decisions you would make if you had infinite time and information. You can make guesses based on your ideas of convergence, but that’s about it.
I think it should be possible to compute probabilities and expectation values of absolutely anything. However to put it on a sound mathematical basis we need a theory of logical uncertainty.
I think so because conceptually a Bayesian expectation value is your “best effort” to estimate something. Since you can always do your “best effort” you can always compute the expectation value. Of course, for this to fully make sense we must take computing resource limits into account. So we need a theory of probability with limited computing resources aka a theory of logical uncertainty.
conceptually a Bayesian expectation value is your “best effort” to estimate something.
Not quite. Conceptually a Bayesian expectation is your attempt to rationally quantify your beliefs which may or may not involve best efforts. That requires these beliefs to exist. I don’t see why it isn’t possible to have no beliefs with regard to some topic.
Since you can always do your “best effort” you can always compute the expectation value.
That’s not very meaningful. You can always output some number, but so what? If you have no information you have no information and your number is going to be bogus.
If you don’t believe that the process of thought asymptotically converges to some point called “truth” (at least approximately), what does it mean to have a correct answer to any question?
Meta-remark: Whoever is downvoting all of my comments in this thread, do you really think I’m not arguing in good faith? Or are you downvoting just because you disagree? If it’s the latter, do you think it’s good practice or you just haven’t given it thought?
If you don’t believe that the process of thought asymptotically converges to some point called “truth” (at least approximately), what does it mean to have a correct answer to any question?
There is that thing called reality. Reality determines what constitutes a correct answer to a question (for that subset of questions which actually have “correct” answers).
I see no reason to believe that the process of thought converges at all, never mind asymptotically to ‘some point called “truth”’.
Interaction with reality only gives you raw sensory experiences. It doesn’t allow you to deduce anything. When you compute 12 x 12 and it turns out to be 144, you believe 144 is the one correct answer. Therefore you implicitly assume that tomorrow you won’t somehow realize the answer is 356.
Not quite. I can test whether a rock is hard by kicking it.
But this byte-sized back-and-forth doesn’t look particularly useful. I don’t understand where you are coming from—to me it seems that you consider your thought processes primary and reality secondary. Truth, say you, is whatever the thought processes converge to, regardless of reality. That doesn’t make sense to me.
When you kick the rock, all you get is a sensory experience (a quale, if you like). You interpret this experience as a sensation arising from your foot. You assume this sensation is the result of your leg undergoing something called “collision” with something called “rock”. You deduce that the rock probably has a property called “hard”. All of those are deductions you do using your model of reality. This model is generated from memories of previous experiences by a process of thought based on something like Occam’s razor.
Since the only access to truth we might have is through our own thought, if the latter doesn’t converge to truth (at least approximately) then truth is completely inaccessible.
if the latter doesn’t converge to truth (at least approximately) then truth is completely inaccessible.
Why not? Granted that we have access to reality only through mental constructs and so any approximations to “the truth” are our own thoughts, but I don’t see any problems with stating that sometimes these mental constructs adequately reflect reality (=truth) and sometimes they don’t. I don’t see where this whole idea of asymptotic convergence is coming from. There is no guarantee that more thinking will get you closer to the truth, but on the other hand sometimes the truth is right there, easily accessible.
Before we even get to VNM axioms I would like to point out that humans do not operate in a VNM setting where a single consequentialist entity is faced with a sequence of lotteries and is able to express his preferences as one-dimensional rankings.
Haven’t there been a lot of discussion about the applicability of VNM to human ethical systems? It looks like a well-trodden ground to me.
Before we even get to VNM axioms I would like to point out that humans do not operate in a VNM setting where a single consequentialist entity is faced with a sequence of lotteries and is able to express his preferences as one-dimensional rankings.
He doesn’t express the entire ranking, but he does still have to choose the best option.
What would be a good criterion? You cannot pull yourself up by your bootstraps. You need to start from something.
Before we even get to VNM axioms I would like to point out that humans do not operate in a VNM setting where a single consequentialist entity is faced with a sequence of lotteries and is able to express his preferences as one-dimensional rankings.
How would you want to operate? You mentioned instrumental rationality. I don’t know how to define instrumental rationality without the VNM setting (or something similar).
Instrumental rationality: achieving your values. Not necessarily “your values” in the sense of being selfish values or unshared values: “your values” means anything you care about. The art of choosing actions that steer the future toward outcomes ranked higher in your preferences. On LW we sometimes refer to this as “winning”.
Well, you originally said ” violating them strongly feels like making an error. ” I said that “feels like” is a weak point. You asked for an alternative. I suggested mismatch with reality. As in “violating X leads to results which do not agree with what we know of reality”.
This is a nice motto, but how do you make a mathematical model out of it?
We were talking about how would a human qualify as a “rational agent”. I see no need to make mathematical models here.
...”violating X leads to results which are do not agree with what we know of reality”.
This only makes sense in epistemic context, not in instrumental one. How can a way of making decisions “not agree with what we know of reality”? Note that I’m making a normative statement (what one should do), not a descriptive statement (“people usually behave in such-and-such way”).
We were talking about how would a human qualify as a “rational agent”. I see no need to make mathematical models here.
There is always a need to make mathematical models since before you have a mathematical model your understanding is imprecise. For example, a mathematical model allows you to prove than under certain assumptions diversifying donations is irrational.
There is nothing intrinsic to the action of “praying for a miracle” which “disagrees with reality”. It’s only when we view this action in the context of a decision theory which says e.g. “choose the action which leads to maximal expected utility under the Solomonoff prior” can we say the action is “irrational” because, in fact, it does not lead to maximal expected utility. But in order to make this argument you need to assume a decision theory.
There is nothing intrinsic to the action of “praying for a miracle” which “disagrees with reality”.
Given the definition of a miracle, I think there is, but anyway—I’m willing to go out on a limb, take the shortcut, and pronounce praying for a miracle to fail instrumental rationality. Without first constructing a rigorous mathematical model of the expected utility under the Solomonoff prior. YMMV, of course.
That seems to be a rather narrow and not very useful definition of a “rational agent” as applied to humans.
I think it is the correct definition in the sense that you should behave like one.
Why should I behave as if my values satisfy the VNM axioms?
Rationality here is typically defined as either epistemic (make sure your mental models match reality well) or instrumental (make sure the steps you take actually lead to your goals).
Defining rationality as “you MUST have a single utility function which MUST follow VNM” doesn’t strike me as a good idea.
Because the VNM axioms seem so intuitively obvious that violating them strongly feels like making an error. Of course I cannot prove them without introducing another set of axioms which can be questioned in turn etc. You always need to start with some assumptions.
Which VNM axiom would you reject?
I would reject the completeness axiom. I often face choices where I don’t know which option I prefer, but where I would not agree that I am indifferent. And I’m okay with this fact.
I also reject the transitivity axiom—intransitive preference is an observed fact for real humans in a wide variety of settings. And you might say this is irrational, but my preference are what they are.
Can you give an example of situations A, B, C for which your preferences are A > B, B > C, C > A? What would you do if you need to choose between A, B, C?
Sure. I’ll go to the grocery store and have three kinds of tomato sauce and I’ll look at A and B, and pick B, then B and C, pick C, and C and A, and pick A. And I’ll stare at them indecisively until my preferences shift. It’s sort of ridiculous—it can take something like a minute to decide. This is NOT the same as feeling indifferent, in which case I would just pick one and go.
I have similar experiences when choosing between entertainment options, transport, etc. My impression is that this is an experience that many people have.
If you google “intransitive preference” you get a bunch of references—this one has cites to the original experiements: http://www.stanford.edu/class/symbsys170/Preference.pdf
It seems to me that what you’re describing are not preferences but spur of the moment decisions. A preference should be thought of as in CEV: the thing you would prefer if you thought about it long enough, knew enough, were more the person you want to be etc. The mere fact you somehow decide between the sauces in the end suggests you’re not describing a preference. Also I doubt that you have terminal values related to tomato sauce. More likely, your terminal values involve something like “experiencing pleasure” and your problem here is epistemic rather than “moral”: you’re not sure which sauce would give you more pleasure.
You are using preference to mean something other than I thought you were.
I’m not convinced that the CEV definition of preference is useful. No actual human ever has infinite time or information; we are always making decisions while we are limited computationally and informationally. You can’t just define away those limits. And I’m not at all convinced that our preferences would converge even given infinite time. That’s an assumption, not a theorem.
When buying pasta sauce, I have multiple incommensurable values: money, health, and taste. And in general, when you have multiple criteria, there’s no non-paradoxical way to do rankings. (This is basically Arrow’s theorem). And I suspect that’s the cause for my lack of preference ordering.
Of course. But rationality means your decisions should be as close as possible to the decisions you would make if you had infinite time and information.
Money is not a terminal value for most people. I suspect you want money because of the things it can buy you, not as a value in itself. I think health is also instrumental. We value health because illness is unpleasant, might lead to death and generally interferes with taking actions to optimize our values. The unpleasant sensations of illness might well be commensurable with the pleasant sensations of taste. For example you would probably pass up a gourmet meal if eating it implies getting cancer.
However you can not know what decisions you would make if you had infinite time and information. You can make guesses based on your ideas of convergence, but that’s about it.
A Bayesian never “knows” anything. She can only compute probabilities and expectation values.
Can she compute probabilities and expectation values with respect to decisions she would make if she had infinite time and information?
I think it should be possible to compute probabilities and expectation values of absolutely anything. However to put it on a sound mathematical basis we need a theory of logical uncertainty.
On the basis of what do you think so? And what entity will be doing the computing?
I think so because conceptually a Bayesian expectation value is your “best effort” to estimate something. Since you can always do your “best effort” you can always compute the expectation value. Of course, for this to fully make sense we must take computing resource limits into account. So we need a theory of probability with limited computing resources aka a theory of logical uncertainty.
Not quite. Conceptually a Bayesian expectation is your attempt to rationally quantify your beliefs which may or may not involve best efforts. That requires these beliefs to exist. I don’t see why it isn’t possible to have no beliefs with regard to some topic.
That’s not very meaningful. You can always output some number, but so what? If you have no information you have no information and your number is going to be bogus.
If you don’t believe that the process of thought asymptotically converges to some point called “truth” (at least approximately), what does it mean to have a correct answer to any question?
Meta-remark: Whoever is downvoting all of my comments in this thread, do you really think I’m not arguing in good faith? Or are you downvoting just because you disagree? If it’s the latter, do you think it’s good practice or you just haven’t given it thought?
There is that thing called reality. Reality determines what constitutes a correct answer to a question (for that subset of questions which actually have “correct” answers).
I see no reason to believe that the process of thought converges at all, never mind asymptotically to ‘some point called “truth”’.
How do you know anything about reality if not through your own thought process?
Through interaction with reality. Are you arguing from a brain-in-the-vat position?
Interaction with reality only gives you raw sensory experiences. It doesn’t allow you to deduce anything. When you compute 12 x 12 and it turns out to be 144, you believe 144 is the one correct answer. Therefore you implicitly assume that tomorrow you won’t somehow realize the answer is 356.
And what does that have to do with knowing anything about reality? Your thought process is not a criterion of whether anything is true.
But it is the only criterion you are able to apply.
Not quite. I can test whether a rock is hard by kicking it.
But this byte-sized back-and-forth doesn’t look particularly useful. I don’t understand where you are coming from—to me it seems that you consider your thought processes primary and reality secondary. Truth, say you, is whatever the thought processes converge to, regardless of reality. That doesn’t make sense to me.
When you kick the rock, all you get is a sensory experience (a quale, if you like). You interpret this experience as a sensation arising from your foot. You assume this sensation is the result of your leg undergoing something called “collision” with something called “rock”. You deduce that the rock probably has a property called “hard”. All of those are deductions you do using your model of reality. This model is generated from memories of previous experiences by a process of thought based on something like Occam’s razor.
OK, and how do we get from that to ‘the process of thought asymptotically converges to some point called “truth”’?
Since the only access to truth we might have is through our own thought, if the latter doesn’t converge to truth (at least approximately) then truth is completely inaccessible.
Why not? Granted that we have access to reality only through mental constructs and so any approximations to “the truth” are our own thoughts, but I don’t see any problems with stating that sometimes these mental constructs adequately reflect reality (=truth) and sometimes they don’t. I don’t see where this whole idea of asymptotic convergence is coming from. There is no guarantee that more thinking will get you closer to the truth, but on the other hand sometimes the truth is right there, easily accessible.
I apologize but this discussion seems to be going nowhere.
Agreed.
So you care more about following the VNM axioms, then which utility function you are maximizing? That behavior is itself not VNM rational.
If you don’t follow the VNM axioms you are not maximizing any utility function.
So why do you care about maximizing any utility function?
What would constitute a valid answer to that question, from your point of view?
I can’t think of one. You’re the one arguing for what appears to be an inconsistent position.
What is the inconsistency?
Saying one should maximize a utility function, but not caring which utility function is maximized.
Who said I don’t care which utility function is maximized?
“feels like” is a notoriously bad criterion :-)
Before we even get to VNM axioms I would like to point out that humans do not operate in a VNM setting where a single consequentialist entity is faced with a sequence of lotteries and is able to express his preferences as one-dimensional rankings.
Haven’t there been a lot of discussion about the applicability of VNM to human ethical systems? It looks like a well-trodden ground to me.
He doesn’t express the entire ranking, but he does still have to choose the best option.
What would be a good criterion? You cannot pull yourself up by your bootstraps. You need to start from something.
How would you want to operate? You mentioned instrumental rationality. I don’t know how to define instrumental rationality without the VNM setting (or something similar).
Mismatch with reality.
Well, the locally canonical definition is this:
I see nothing about VNM there.
I’m not following
This is a nice motto, but how do you make a mathematical model out of it?
Well, you originally said ” violating them strongly feels like making an error. ” I said that “feels like” is a weak point. You asked for an alternative. I suggested mismatch with reality. As in “violating X leads to results which do not agree with what we know of reality”.
We were talking about how would a human qualify as a “rational agent”. I see no need to make mathematical models here.
This only makes sense in epistemic context, not in instrumental one. How can a way of making decisions “not agree with what we know of reality”? Note that I’m making a normative statement (what one should do), not a descriptive statement (“people usually behave in such-and-such way”).
There is always a need to make mathematical models since before you have a mathematical model your understanding is imprecise. For example, a mathematical model allows you to prove than under certain assumptions diversifying donations is irrational.
Ever heard of someone praying for a miracle?
Bollocks! I guess next you’ll be telling me I can not properly understand anything which is not expressed in numbers… :-P
There is nothing intrinsic to the action of “praying for a miracle” which “disagrees with reality”. It’s only when we view this action in the context of a decision theory which says e.g. “choose the action which leads to maximal expected utility under the Solomonoff prior” can we say the action is “irrational” because, in fact, it does not lead to maximal expected utility. But in order to make this argument you need to assume a decision theory.
Given the definition of a miracle, I think there is, but anyway—I’m willing to go out on a limb, take the shortcut, and pronounce praying for a miracle to fail instrumental rationality. Without first constructing a rigorous mathematical model of the expected utility under the Solomonoff prior. YMMV, of course.