Hmm. I think I meant something more like your second interpretation than your first interpretation but I think I actually meant a third thing and am not confident we aren’t still misunderstanding each other.
An intended implication, (which comes with an if-then suggestion, which was not an essential part of my original claim but I think is relevant) is:
If you value being able to think freely and have epistemologically sound thoughts, it is important to be able to think thoughts that you will neither be rewarded nor punished for… [edit: or be extremely confident than you have accounted for your biases towards reward gradients]. And the rewards are only somewhat less bad than the punishments.
A followup implication is that this is not possible to maintain humanity-wide if thought-privacy is removed (which legalizing blackmail would contribution somewhat towards). And that this isn’t just a fact about our current equilibria, it’s intrinsic to human biology.
It seems plausible (although I am quite skeptical) that a small group of humans might be able to construct an epistemically sound world that includes lack-of-intellectual-privacy, but they’d have to have correctly accounted for wide variety of subtle errors.
[edit: all of this assumes you are running on human wetware. If you remove that as a constraint other things may be possible]
further update: I do think rewards are something like 10x less problematic than punishments, because humans are risk averse and fear punishment more than they desire reward. (“10x” is a stand-in for “whatever the psychological research says on how big the difference is between human response to rewards and punishments”)
[note: this subthread is far afield from the article—LW is about publication, not private thoughts (unless there’s a section I don’t know about where only specifically invited people can see things) . And LW karma is far from the sanctions under discussion in the rest of the post.]
Have you considered things to reduce the assymetric impact of up- and down-votes? Cap karma value at −5? Use downvotes as a divisor for upvotes (say, score is upvotes / (1 + 0.25 * downvotes)) rather than simple subtraction?
We’ve thought about things in that space, although any of the ideas would be a fairly major change, and we haven’t come up with anything we feel good enough about to commit to.
(We have done some subtle things to avoid making downvotes feel worse than they need to, such as not including the explicit number of downvotes)
Do you think that thoughts are too incentivised or not incentivised enough on the margin, for the purpose of epistemically sound thinking? If they’re too incentivised, have you considered dampening LWs karma system? If they’re not incentivised enough, what makes you believe that legalising blackmail will worsen the epistemic quality of thoughts?
The LW karma obviously has its flaws, per Goodhart’s law. It is used anyway, because the alternative is having other problems, and for the moment this seems like a reasonable trade-off.
The punishment for “heresies” is actually very mild. As long as one posts respected content in general, posting a “heretical” comment every now and then does not ruin their karma. (Compare to people having their lives changed dramatically because of one tweet.) The punishment accumulates mostly for people whose only purpose here is to post “heresies”. Also, LW karma does not prevent anyone from posting “heresies” on a different website. Thus, people can keep positive LW karma even if their main topic is talking how LW is fundamentally wrong as long as they can avoid being annoying (for example by posting hundred LW-critical posts on their personal website, posting a short summary with hyperlinks on LW, and afterwards using LW mostly to debate other topics).
Blackmail typically attacks you in real life, i.e. you can’t limit the scope of impact. If losing an online account on a website X would be the worst possible outcome of one’s behavior at the website X, life would be easy. (You would only need to keep your accounts on different websites separated from each other.) It was already mentioned somewhere in this debate that blackmail often uses the difference between norms in different communities, i.e. that your local-norm-following behavior in one context can be local-norm-breaking in another context. This is quite unlike LW karma.
I’d say thoughts aren’t incentivized enough on the margin, but:
1. A major bottleneck is how fine-tuned and useful the incentives are. (i.e. I’d want to make LW karma more closely track “reward good epistemic processes” before I made the signal stronger. I think it currently tracks that well enough that I prefer it over no-karma).
2. It’s important that people can still have private thoughts separate from the LW karma system. LW is where you come when you have thoughts that seem good enough to either contribute to the commons, or to get feedback on so you can improve your thought process… after having had time to mull things over privately without worrying about what anyone will think of you.
(But, I also think, on the margin, people should be much less scared about sharing their private thoughts than they currently are. Many people seem to be scared about sharing unfinished thoughts at all, and my actual model of what is “threatening” says that there’s a much narrower domain where you need to be worried in the current environment)
3. One conscious decision we made was not not display “number of downvotes” on a post (we tried it out privately for admins for awhile). Instead we just included “total number of votes”. Explicitly knowing how much one’s post got downvoted felt much worse than having a vague sense of how good it was overall + a rough sense of how many people *may* have downvoted it. This created a stronger punishment signal than seemed actually appropriate.
(Separately, I am right now making arguments in terms that I’m fairly confident both of us value, but I also think there are reasons to want private thoughts that are more like “having a Raemon_healthy soul”, than like being able to contribute usefully to the intellectual commons.
(I noticed while writing this that the latter might be most of what a Benquo finds important for having a healthy soul, but unsure. In any case healthy souls are more complicated and I’m avoiding making claims about them for now)
Hmm. I think I meant something more like your second interpretation than your first interpretation but I think I actually meant a third thing and am not confident we aren’t still misunderstanding each other.
An intended implication, (which comes with an if-then suggestion, which was not an essential part of my original claim but I think is relevant) is:
If you value being able to think freely and have epistemologically sound thoughts, it is important to be able to think thoughts that you will neither be rewarded nor punished for… [edit: or be extremely confident than you have accounted for your biases towards reward gradients]. And the rewards are only somewhat less bad than the punishments.
A followup implication is that this is not possible to maintain humanity-wide if thought-privacy is removed (which legalizing blackmail would contribution somewhat towards). And that this isn’t just a fact about our current equilibria, it’s intrinsic to human biology.
It seems plausible (although I am quite skeptical) that a small group of humans might be able to construct an epistemically sound world that includes lack-of-intellectual-privacy, but they’d have to have correctly accounted for wide variety of subtle errors.
[edit: all of this assumes you are running on human wetware. If you remove that as a constraint other things may be possible]
further update: I do think rewards are something like 10x less problematic than punishments, because humans are risk averse and fear punishment more than they desire reward. (“10x” is a stand-in for “whatever the psychological research says on how big the difference is between human response to rewards and punishments”)
[note: this subthread is far afield from the article—LW is about publication, not private thoughts (unless there’s a section I don’t know about where only specifically invited people can see things) . And LW karma is far from the sanctions under discussion in the rest of the post.]
Have you considered things to reduce the assymetric impact of up- and down-votes? Cap karma value at −5? Use downvotes as a divisor for upvotes (say, score is upvotes / (1 + 0.25 * downvotes)) rather than simple subtraction?
We’ve thought about things in that space, although any of the ideas would be a fairly major change, and we haven’t come up with anything we feel good enough about to commit to.
(We have done some subtle things to avoid making downvotes feel worse than they need to, such as not including the explicit number of downvotes)
Do you think that thoughts are too incentivised or not incentivised enough on the margin, for the purpose of epistemically sound thinking? If they’re too incentivised, have you considered dampening LWs karma system? If they’re not incentivised enough, what makes you believe that legalising blackmail will worsen the epistemic quality of thoughts?
The LW karma obviously has its flaws, per Goodhart’s law. It is used anyway, because the alternative is having other problems, and for the moment this seems like a reasonable trade-off.
The punishment for “heresies” is actually very mild. As long as one posts respected content in general, posting a “heretical” comment every now and then does not ruin their karma. (Compare to people having their lives changed dramatically because of one tweet.) The punishment accumulates mostly for people whose only purpose here is to post “heresies”. Also, LW karma does not prevent anyone from posting “heresies” on a different website. Thus, people can keep positive LW karma even if their main topic is talking how LW is fundamentally wrong as long as they can avoid being annoying (for example by posting hundred LW-critical posts on their personal website, posting a short summary with hyperlinks on LW, and afterwards using LW mostly to debate other topics).
Blackmail typically attacks you in real life, i.e. you can’t limit the scope of impact. If losing an online account on a website X would be the worst possible outcome of one’s behavior at the website X, life would be easy. (You would only need to keep your accounts on different websites separated from each other.) It was already mentioned somewhere in this debate that blackmail often uses the difference between norms in different communities, i.e. that your local-norm-following behavior in one context can be local-norm-breaking in another context. This is quite unlike LW karma.
I’d say thoughts aren’t incentivized enough on the margin, but:
1. A major bottleneck is how fine-tuned and useful the incentives are. (i.e. I’d want to make LW karma more closely track “reward good epistemic processes” before I made the signal stronger. I think it currently tracks that well enough that I prefer it over no-karma).
2. It’s important that people can still have private thoughts separate from the LW karma system. LW is where you come when you have thoughts that seem good enough to either contribute to the commons, or to get feedback on so you can improve your thought process… after having had time to mull things over privately without worrying about what anyone will think of you.
(But, I also think, on the margin, people should be much less scared about sharing their private thoughts than they currently are. Many people seem to be scared about sharing unfinished thoughts at all, and my actual model of what is “threatening” says that there’s a much narrower domain where you need to be worried in the current environment)
3. One conscious decision we made was not not display “number of downvotes” on a post (we tried it out privately for admins for awhile). Instead we just included “total number of votes”. Explicitly knowing how much one’s post got downvoted felt much worse than having a vague sense of how good it was overall + a rough sense of how many people *may* have downvoted it. This created a stronger punishment signal than seemed actually appropriate.
(Separately, I am right now making arguments in terms that I’m fairly confident both of us value, but I also think there are reasons to want private thoughts that are more like “having a Raemon_healthy soul”, than like being able to contribute usefully to the intellectual commons.
(I noticed while writing this that the latter might be most of what a Benquo finds important for having a healthy soul, but unsure. In any case healthy souls are more complicated and I’m avoiding making claims about them for now)