In the interest of giving you better positive feedback: awesome article! I had previously suspected that you might not have a good justification for your claims and force your conclusions based on weak data. This suspicion is dead for now. Please continue with your long and in-depth posts.
What I particularly liked is that this article appears much less handwavey than usual. Typically, you demonstrate one particular experiment or line of evidence and then just go, “X and Y did dozens of related studies[way][too][many][references]”, but I’m lazy and even though I actually download all these papers, it will probably take me weeks or months before I read them all and until then, your posts seem much weaker than they really are. Seeing multiple different approaches at once is much better, especially of the form “basic model” → “problems with model” → “proposed alternative” → “lots of evidence that fits predictions (especially evidence for actual moving parts)”.
Also, I like your use of summaries here (and the repetition). One problem of past posts (e.g. 123) was that your summaries and the actual points made in the article didn’t quite match. The conclusions often seemed much stronger than whatever argument you actually made, especially because these posts are more isolated. I was unsure where you were going, so I feared the worst. (“He’s throwing away all subjective experience!” instead of the sane “It isn’t easy to talk accurately about subjective experience.”) This time, the summaries are limited to actual previous points. I prefer to keep lessons (especially somewhat tentative ones) apart from the evidence for them. Whenever someone combines them, I always suspect some kind of bottom line argument, probably unfairly so in your case. Also, this time you acknowledge remaining black boxes. You were aware and open about them in the past, but that often became apparent only in some deeply buried comment.
Finally, this post has convinced me to move Glimcher way up on my reading list.
The idea that mere choice set size would matter didn’t really click for me before (and why). I knew that having many similar choices is harmful, so I made sure items in my daily todo system are very diverse. I totally ignored the size and so I have 25 different items to choose from right now. When I felt paralyzed, I thought I just needed better information. I added columns for how much time I had spent on each item and so on, so I could easily tell which items were over-saturated and which were more urgent. I was still paralyzed and really puzzled, “All the information you need for deciding is right there! Look, it’s even color-coded so you only have to pick one of these three red ones. Just pick one, any one! Don’t just sit there! Gah!” I’ll try automating this and hide as much information as possible so I actually can make decisions.
Luke, there’s a serious and common misconception in your explanation of the independence axiom (serious enough that I don’t consider this nitpicking). If you could, please fix it as soon as you can to prevent the spread of this unfortunate misunderstanding. I wrote a post to try and dispell misconceptions such as this one, because utility theory is used in a lot of toy decision theory problems, versions of which might actually be encountered by utility-seeking AIs:
For example, the independence axiom of expected utility theory says that if you prefer one apple to one orange, you must also prefer one apple plus a tiny bit more apple over one orange plus that same tiny bit of apple. If a subject prefers A to B, then the subject can’t also prefer B+C to A+C. But Allais (1953) found that subjects do violate this basic assumption under some conditions.
This is not what the independence axiom says. What it says is that, for example, if you prefer an apple over an orange, then you must prefer the gamble [72% chance you get an apple, otherwise you get a cat] over the gamble [72% chance you get an orange, otherwise you get a cat]. The axiom is about mixing probabilistic outcomes, not mixing amounts of various commodities.
This distinction is important, because for example, if you’d rather have 1 apple than 1 orange, but you’d rather have 1 orange
and 0.2 apples than 1.2 apples, you’re not violating the independence axiom, nor instantiating the Allais paradox. You simply don’t like having too much apple, which is fine as far as EU is concerned: apple can have negative marginal utility after a certain point. Such explanations are an essential feature, not a shortcomming, of utility theory.
The Allais paradox is a legitimate failure of utility theory in describing human behavior, though, so you’re of course right that expected utility theory is very useless as a predictive tool. I doubt any powerful AGI would commit the Allais paradox, though.
Otherwise, thanks for the incredibly informative post!
When I read about the Allais paradox (in lukeprog’s post, after he fixed your objection), my first thought was that this violation would occur when the cat was actually something very like an orange, such as a grapefruit. For example, suppose that the cat actually is an orange. So you prefer an apple to an orange, but you prefer an orange to a gamble which is 70% apple and 30% orange. And the neoclassical utility theorist would explain this by saying that you prefer certainty to uncertainty, so adding a term for certainty to the utility function. And then, if the choice is really between 70% apple and 30% grapefruit versus 70% orange and 30% grapefruit, the latter is still more certain than the former (although not completely certain), so might well be preferred.
This sounds like I’m trying to come up with a way to save utility theory, but actually that’s not how it went. My immediate intuitive reaction to reading lukeprog’s paraphrase of your example was ‹I’ll bet that this happens when the cat is similar to the orange.›, without any conscious reasoning behind it, and it was only after thinking about this hypothesis that I realised that it suggested a way to save utility theory. So I’m quite curious: Does the Allais paradox appear only when the cat is similar to an orange, or does it also appear when the cat is (as the terms ‘apple’, ‘orange’, and ‘cat’ imply) really quite different?
Thank you for this. I had seen this several times and didn’t understand how they derived this from the independence axiom. I think even Foundations of Neuroecon states the axiom this way.
In the interest of giving you better positive feedback: awesome article! I had previously suspected that you might not have a good justification for your claims and force your conclusions based on weak data. This suspicion is dead for now. Please continue with your long and in-depth posts.
What I particularly liked is that this article appears much less handwavey than usual. Typically, you demonstrate one particular experiment or line of evidence and then just go, “X and Y did dozens of related studies[way][too][many][references]”, but I’m lazy and even though I actually download all these papers, it will probably take me weeks or months before I read them all and until then, your posts seem much weaker than they really are. Seeing multiple different approaches at once is much better, especially of the form “basic model” → “problems with model” → “proposed alternative” → “lots of evidence that fits predictions (especially evidence for actual moving parts)”.
Also, I like your use of summaries here (and the repetition). One problem of past posts (e.g. 1 2 3) was that your summaries and the actual points made in the article didn’t quite match. The conclusions often seemed much stronger than whatever argument you actually made, especially because these posts are more isolated. I was unsure where you were going, so I feared the worst. (“He’s throwing away all subjective experience!” instead of the sane “It isn’t easy to talk accurately about subjective experience.”) This time, the summaries are limited to actual previous points. I prefer to keep lessons (especially somewhat tentative ones) apart from the evidence for them. Whenever someone combines them, I always suspect some kind of bottom line argument, probably unfairly so in your case. Also, this time you acknowledge remaining black boxes. You were aware and open about them in the past, but that often became apparent only in some deeply buried comment.
Finally, this post has convinced me to move Glimcher way up on my reading list.
The idea that mere choice set size would matter didn’t really click for me before (and why). I knew that having many similar choices is harmful, so I made sure items in my daily todo system are very diverse. I totally ignored the size and so I have 25 different items to choose from right now. When I felt paralyzed, I thought I just needed better information. I added columns for how much time I had spent on each item and so on, so I could easily tell which items were over-saturated and which were more urgent. I was still paralyzed and really puzzled, “All the information you need for deciding is right there! Look, it’s even color-coded so you only have to pick one of these three red ones. Just pick one, any one! Don’t just sit there! Gah!” I’ll try automating this and hide as much information as possible so I actually can make decisions.
Thanks for going out of your way to give me such actionable feedback!
Luke, there’s a serious and common misconception in your explanation of the independence axiom (serious enough that I don’t consider this nitpicking). If you could, please fix it as soon as you can to prevent the spread of this unfortunate misunderstanding. I wrote a post to try and dispell misconceptions such as this one, because utility theory is used in a lot of toy decision theory problems, versions of which might actually be encountered by utility-seeking AIs:
This is not what the independence axiom says. What it says is that, for example, if you prefer an apple over an orange, then you must prefer the gamble [72% chance you get an apple, otherwise you get a cat] over the gamble [72% chance you get an orange, otherwise you get a cat]. The axiom is about mixing probabilistic outcomes, not mixing amounts of various commodities.
This distinction is important, because for example, if you’d rather have 1 apple than 1 orange, but you’d rather have 1 orange and 0.2 apples than 1.2 apples, you’re not violating the independence axiom, nor instantiating the Allais paradox. You simply don’t like having too much apple, which is fine as far as EU is concerned: apple can have negative marginal utility after a certain point. Such explanations are an essential feature, not a shortcomming, of utility theory.
The Allais paradox is a legitimate failure of utility theory in describing human behavior, though, so you’re of course right that expected utility theory is very useless as a predictive tool. I doubt any powerful AGI would commit the Allais paradox, though.
Otherwise, thanks for the incredibly informative post!
Fixed, thanks!
I also updated the PDF.
When I read about the Allais paradox (in lukeprog’s post, after he fixed your objection), my first thought was that this violation would occur when the cat was actually something very like an orange, such as a grapefruit. For example, suppose that the cat actually is an orange. So you prefer an apple to an orange, but you prefer an orange to a gamble which is 70% apple and 30% orange. And the neoclassical utility theorist would explain this by saying that you prefer certainty to uncertainty, so adding a term for certainty to the utility function. And then, if the choice is really between 70% apple and 30% grapefruit versus 70% orange and 30% grapefruit, the latter is still more certain than the former (although not completely certain), so might well be preferred.
This sounds like I’m trying to come up with a way to save utility theory, but actually that’s not how it went. My immediate intuitive reaction to reading lukeprog’s paraphrase of your example was ‹I’ll bet that this happens when the cat is similar to the orange.›, without any conscious reasoning behind it, and it was only after thinking about this hypothesis that I realised that it suggested a way to save utility theory. So I’m quite curious: Does the Allais paradox appear only when the cat is similar to an orange, or does it also appear when the cat is (as the terms ‘apple’, ‘orange’, and ‘cat’ imply) really quite different?
Thank you for this. I had seen this several times and didn’t understand how they derived this from the independence axiom. I think even Foundations of Neuroecon states the axiom this way.