ETA 1/12: This review is critical and at times harsh, not because I want to harshly criticize the post or the author, but because I did not consider harshness of criticism when writing. I still think the post is positive-net-value, and might even vote it up in the review. I especially want to emphasize that I do not think it is in any way useful to blame or punish the author for the things I complain about below; this is intended as a “pointing out a problematic habit which a lot of people have and society often encourages” criticism, not a “bad thing must be punished” criticism.
When this post first came out, I said something felt off about it. The same thing still feels off about it, but I no longer endorse my original explanation of what-felt-off. So here’s another attempt.
First, what this post does well. There’s a core model which says something like “people with the power to structure incentives tend get the appearance of what they ask for, which often means bad behavior is hidden”. It’s a useful and insightful model, and the post presents it with lots of examples, producing a well-written and engaging explanation. The things which the post does well more than outweigh the problems below; it’s a great post.
On to the problem. Let’s use the slave labor example, because that’s the first spot where the problem comes up:
No company goes “I’m going to go out and enslave people today” (especially not publicly), but not paying people is sometimes cheaper than paying them, so financial pressure will push towards slavery. Public pressure pushes in the opposite direction, so companies try not to visibly use slave labor. But they can’t control what their subcontractors do, and especially not what their subcontractors’ subcontractors’ subcontractors do, and sometimes this results in workers being unpaid and physically blocked from leaving.
… so far, so good. This is generally solid analysis of an interesting phenomenon.
But then we get to the next sentence:
Who’s at fault for the subcontractor(^3)’s slave labor?
… and this where I want to say NO. My instinct says DO NOT EVER ASK THAT QUESTION, it is a WRONG QUESTION, you will be instantly mindkilled every time you ask “who should be blamed for X?”.
… on reflection, I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective. Asking “who should we blame?” is always engaging in a status fight. Status fights are generally mindkillers, and should be kept strictly separate from modelling and epistemics.
Now, this does not mean that we shouldn’t model status fights. Rather, it means that we should strive to avoid engaging in status fights when modelling them. Concretely: rather than ask “who should we blame?”, ask “what incentives do we create by blaming <actor>?”. This puts the question in an analytical frame, rather than a “we’re having a status fight right now” frame.
Some sections of the post do discuss things in that sort of analytical frame, and those sections are where I see most of the value. Unfortunately, they’re mixed in with parts which don’t use an analytical frame. For instance, in the “Dating an Artist” example, we see “It wasn’t my fault he felt that pressure...” - a sentence which is engaging in a status fight. But then two sentences later, there’s a great analysis of the incentives created by precommitting to housing provision. If the sentences about how “Time in my spare bedroom was a gift to him I had no obligation to keep giving” were stripped out, and just the analysis were left behind, then the post would be dramatically better.
Now, I’m sure somebody’s going to come along and object that we can never fully separate the status fights from the status-fight-models in reality; people will always sneak their status fights into epistemic discussions. My answer is that this is the fallacy of gray:
The Sophisticate: “The world isn’t black and white. No one does pure good or pure bad. It’s all gray. Therefore, no one is better than anyone else.”
The Zetet: “Knowing only gray, you conclude that all grays are the same shade. You mock the simplicity of the two-color view, yet you replace it with a one-color view . . .”
—Marc Stiegler, David’s Sling
Yes, the status fights will sometimes sneak into epistemic discussions. Ideally, we want epistemic discussion norms which are robust to the presence of status fights. “Don’t engage in explicit status fights, use an analytic frame instead” is one such norm—it is not sufficient on its own, but it seems to take a large step in the right direction, and in practice it is the norm which seems most important for keeping epistemic discussion sane.
I can still see where there’s room to quibble about using this norm all the time, but at the very least it is a norm which would improve overall epistemics if applied to posts like this.
… on reflection, I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective. Asking “who should we blame?” is always engaging in a status fight. Status fights are generally mindkillers, and should be kept strictly separate from modelling and epistemics.
Now, this does not mean that we shouldn’t model status fights. Rather, it means that we should strive to avoid engaging in status fights when modelling them. Concretely: rather than ask “who should we blame?”, ask “what incentives do we create by blaming <actor>?”. This puts the question in an analytical frame, rather than a “we’re having a status fight right now” frame.
This was a pretty important couple of points. I’m not sure I agree with them as worded, but point towards something that I think is close to a pareto improvement, at least for LessWrong and maybe for the whole world.
I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective
The key problem is… sometimes you actually just do need to have status fights, and you still want to have as-good-epistemics-as-possible given that you’re in a status fight. So a binary distinction of “trying to have good epistemics” vs “not” isn’t the right frame.
I think this might actually be a pretty good distinction for LessWrong’s frontpage – “status fight or no?” is close to the question that our Frontpage ‘politics’ distinction is aiming at. I do think it is probably reasonable that if you’re trying to write a frontpage page, you follow the “what incentives do we create by blaming?” rule, and if you want to more directly talk about “no actually we should blame Bob for X” then you write a personal blogpost.
The key problem is… sometimes you actually just do need to have status fights, and you still want to have as-good-epistemics-as-possible given that you’re in a status fight. So a binary distinction of “trying to have good epistemics” vs “not” isn’t the right frame.
Part of my model here is that moral/status judgements (like “we should blame X for Y”) like to sneak into epistemic models and masquerade as weight-bearing components of predictions. The “virtue theory of metabolism”, which Yudkowsky jokes about a few times in the sequences, is an excellent example of this sort of thing, though I think it happens much more often and usually much more subtly than that.
My answer to that problem on a personal level is to rip out the weeds wherever I notice them, and build a dome around the garden to keep the spores out. In other words: keep morality/status fights strictly out of epistemics in my own head. In principle, there is zero reason why status-laden value judgements should ever be directly involved in predictive matters. (Even when we’re trying to model our own value judgements, the analysis/engagement distinction still applies.)
Epistemics will still be involved in status fights, but the goal is to make that a one-way street as much as possible. Epistemics should influence status, not the other way around.
In practice it’s never that precise even when it works, largely because value connotations in everyday language can compactly convey epistemically-useful information - e.g. the weeds analogy above. But it’s still useful to regularly check that the value connotations can be taboo’d without the whole model ceasing to make sense, and it’s useful to perform that sort of check automatically when value judgements play a large role.
John and I had a fantastic offline discussion and I’m currently revising this in light of that. We’re also working on a postmortem on the whole thing that I expect to be very informative. I keep mission creeping on my edits and response and it’s going to take a while so I’m writing the bare minimum comment to register that this is happening.
That explanation what is suspicious about the post drives two intuitions on me how that gets at something and how it is too black and white.
The danger is having a logic of “We had a bad harvest this year. We need to burn more witches so that our harvests are better”. Having a guilty party makes it easy to stop being curious about the mechanics and fuels a very flawed theory of remedy.
But then if there is a car accident and somebody tries to find whos insurance company should be paying the repair bills going in that situation and saying “You are committing a grievious error if you try to find a blame party” seems wrong. There it still seems there are more and less productive ways about it. A court probably would not think who we should bill but rather who is on the hook for the bill. Likewise a airplane crash investigation is very interested in the causes and is likely to be basis for future preventative action. The kind of question of “a plane crashed and we have no clue why” screams to have a high quality, correct answer. It also seems typical that in such investigations multiple contribuitng hypotheses are examined closely.
In serial show The Boys one of the characters spins a plane crash for politcal grief over airspace control. That fictional situation seems like an example of how to do it wrong where it is pretty clear how to do it right.
I guess itmight just be that “How this happened?” is a way more justifiable question rather than “Whos life we should make diffcult based on this?”
I’ve been thinking a lot about this comment, and wanted to think more, but it seems useful to have something up as voting starts, so....
I think there’s A Thing JW both agree is harmful (around assigning people moral responsibility when they’re responding to incentives), and that I was trying to fight against. One thing I took from this comment is there’s a good chance I had only a partial victory against Harmful Thing, and tried to pull down the master’s house with the master’s tools. I’d be very interested in exploring that further. (I also think it’s possible JW is doing the same thing… it’s a hard trap to escape)
I don’t think giving up the question “Who should we blame?” entirely is a good idea. Possibly the benefits of the norm would outweigh the costs for LessWrong in particular, but I don’t believe such a norm would be a pareto improvement.
ETA 1/12: This review is critical and at times harsh, not because I want to harshly criticize the post or the author, but because I did not consider harshness of criticism when writing. I still think the post is positive-net-value, and might even vote it up in the review. I especially want to emphasize that I do not think it is in any way useful to blame or punish the author for the things I complain about below; this is intended as a “pointing out a problematic habit which a lot of people have and society often encourages” criticism, not a “bad thing must be punished” criticism.
When this post first came out, I said something felt off about it. The same thing still feels off about it, but I no longer endorse my original explanation of what-felt-off. So here’s another attempt.
First, what this post does well. There’s a core model which says something like “people with the power to structure incentives tend get the appearance of what they ask for, which often means bad behavior is hidden”. It’s a useful and insightful model, and the post presents it with lots of examples, producing a well-written and engaging explanation. The things which the post does well more than outweigh the problems below; it’s a great post.
On to the problem. Let’s use the slave labor example, because that’s the first spot where the problem comes up:
… so far, so good. This is generally solid analysis of an interesting phenomenon.
But then we get to the next sentence:
… and this where I want to say NO. My instinct says DO NOT EVER ASK THAT QUESTION, it is a WRONG QUESTION, you will be instantly mindkilled every time you ask “who should be blamed for X?”.
… on reflection, I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective. Asking “who should we blame?” is always engaging in a status fight. Status fights are generally mindkillers, and should be kept strictly separate from modelling and epistemics.
Now, this does not mean that we shouldn’t model status fights. Rather, it means that we should strive to avoid engaging in status fights when modelling them. Concretely: rather than ask “who should we blame?”, ask “what incentives do we create by blaming <actor>?”. This puts the question in an analytical frame, rather than a “we’re having a status fight right now” frame.
Some sections of the post do discuss things in that sort of analytical frame, and those sections are where I see most of the value. Unfortunately, they’re mixed in with parts which don’t use an analytical frame. For instance, in the “Dating an Artist” example, we see “It wasn’t my fault he felt that pressure...” - a sentence which is engaging in a status fight. But then two sentences later, there’s a great analysis of the incentives created by precommitting to housing provision. If the sentences about how “Time in my spare bedroom was a gift to him I had no obligation to keep giving” were stripped out, and just the analysis were left behind, then the post would be dramatically better.
Now, I’m sure somebody’s going to come along and object that we can never fully separate the status fights from the status-fight-models in reality; people will always sneak their status fights into epistemic discussions. My answer is that this is the fallacy of gray:
Yes, the status fights will sometimes sneak into epistemic discussions. Ideally, we want epistemic discussion norms which are robust to the presence of status fights. “Don’t engage in explicit status fights, use an analytic frame instead” is one such norm—it is not sufficient on its own, but it seems to take a large step in the right direction, and in practice it is the norm which seems most important for keeping epistemic discussion sane.
I can still see where there’s room to quibble about using this norm all the time, but at the very least it is a norm which would improve overall epistemics if applied to posts like this.
This was a pretty important couple of points. I’m not sure I agree with them as worded, but point towards something that I think is close to a pareto improvement, at least for LessWrong and maybe for the whole world.
The key problem is… sometimes you actually just do need to have status fights, and you still want to have as-good-epistemics-as-possible given that you’re in a status fight. So a binary distinction of “trying to have good epistemics” vs “not” isn’t the right frame.
I think this might actually be a pretty good distinction for LessWrong’s frontpage – “status fight or no?” is close to the question that our Frontpage ‘politics’ distinction is aiming at. I do think it is probably reasonable that if you’re trying to write a frontpage page, you follow the “what incentives do we create by blaming?” rule, and if you want to more directly talk about “no actually we should blame Bob for X” then you write a personal blogpost.
Part of my model here is that moral/status judgements (like “we should blame X for Y”) like to sneak into epistemic models and masquerade as weight-bearing components of predictions. The “virtue theory of metabolism”, which Yudkowsky jokes about a few times in the sequences, is an excellent example of this sort of thing, though I think it happens much more often and usually much more subtly than that.
My answer to that problem on a personal level is to rip out the weeds wherever I notice them, and build a dome around the garden to keep the spores out. In other words: keep morality/status fights strictly out of epistemics in my own head. In principle, there is zero reason why status-laden value judgements should ever be directly involved in predictive matters. (Even when we’re trying to model our own value judgements, the analysis/engagement distinction still applies.)
Epistemics will still be involved in status fights, but the goal is to make that a one-way street as much as possible. Epistemics should influence status, not the other way around.
In practice it’s never that precise even when it works, largely because value connotations in everyday language can compactly convey epistemically-useful information - e.g. the weeds analogy above. But it’s still useful to regularly check that the value connotations can be taboo’d without the whole model ceasing to make sense, and it’s useful to perform that sort of check automatically when value judgements play a large role.
John and I had a fantastic offline discussion and I’m currently revising this in light of that. We’re also working on a postmortem on the whole thing that I expect to be very informative. I keep mission creeping on my edits and response and it’s going to take a while so I’m writing the bare minimum comment to register that this is happening.
That explanation what is suspicious about the post drives two intuitions on me how that gets at something and how it is too black and white.
The danger is having a logic of “We had a bad harvest this year. We need to burn more witches so that our harvests are better”. Having a guilty party makes it easy to stop being curious about the mechanics and fuels a very flawed theory of remedy.
But then if there is a car accident and somebody tries to find whos insurance company should be paying the repair bills going in that situation and saying “You are committing a grievious error if you try to find a blame party” seems wrong. There it still seems there are more and less productive ways about it. A court probably would not think who we should bill but rather who is on the hook for the bill. Likewise a airplane crash investigation is very interested in the causes and is likely to be basis for future preventative action. The kind of question of “a plane crashed and we have no clue why” screams to have a high quality, correct answer. It also seems typical that in such investigations multiple contribuitng hypotheses are examined closely.
In serial show The Boys one of the characters spins a plane crash for politcal grief over airspace control. That fictional situation seems like an example of how to do it wrong where it is pretty clear how to do it right.
I guess itmight just be that “How this happened?” is a way more justifiable question rather than “Whos life we should make diffcult based on this?”
I’ve been thinking a lot about this comment, and wanted to think more, but it seems useful to have something up as voting starts, so....
I think there’s A Thing JW both agree is harmful (around assigning people moral responsibility when they’re responding to incentives), and that I was trying to fight against. One thing I took from this comment is there’s a good chance I had only a partial victory against Harmful Thing, and tried to pull down the master’s house with the master’s tools. I’d be very interested in exploring that further. (I also think it’s possible JW is doing the same thing… it’s a hard trap to escape)
I don’t think giving up the question “Who should we blame?” entirely is a good idea. Possibly the benefits of the norm would outweigh the costs for LessWrong in particular, but I don’t believe such a norm would be a pareto improvement.