I think you’re pointing in an important direction, but your phrasing sounds off to me.
(In particular, ‘scapegoating’ feels like a very different frame than the one I’d use here)
If I think out loud, especially about something I’m uncertain about, that other people have opinions on, a few things can happen to me:
Someone who overhears part of my thought process might think (correctly, even!) that my thought process reveals that I am not very smart. Therefore, they will be less likely to hire me. This is punishment, but it’s very much not “scapegoating” style punishment.
Someone who overhears my private thought process might (correctly, or incorrectly! either) come to think that I am smart, and be more likely to hire me. This can be just as dangerous. In a world where all information is public, I have to attend to how the process by which I act and think looks. I am incentivized to think in ways that are legibly good.
“Judgment” is dangerous to me (epistemically) even if the judgment is positive, because it incentives me against exploring paths that look bad, or are good for incomprehensible reasons.
This seems like a general argument that providing evidence without trying to control the conclusions others draw is bad because it leads to errors. It doesn’t seem to take into account the cost of reduced info glow or the possibility that the gatekeeper might also introduce errors. That’s before we even consider self-serving bias!
TLDR: I literally do not understand how to interpret your comment as NOT a general endorsement of fraud and implicit declaration of intent to engage in it.
My intent was not that it’s “bad”, just, if you do not attempt to control the conclusions of others, they will predictably form conclusions of particular types, and this will have effects. (It so happens that I think most people won’t like those effects, and therefore will attempt to control the conclusions of others.)
Ah, if you literally just mean it increases variance & risk, that’s true in the very short term. In context it sounded to me like a policy argument against doing so, but on reflection it’s easy to read you as meaning the more reasonable thing. Thank you for explaining.
Hmm. I think I meant something more like your second interpretation than your first interpretation but I think I actually meant a third thing and am not confident we aren’t still misunderstanding each other.
An intended implication, (which comes with an if-then suggestion, which was not an essential part of my original claim but I think is relevant) is:
If you value being able to think freely and have epistemologically sound thoughts, it is important to be able to think thoughts that you will neither be rewarded nor punished for… [edit: or be extremely confident than you have accounted for your biases towards reward gradients]. And the rewards are only somewhat less bad than the punishments.
A followup implication is that this is not possible to maintain humanity-wide if thought-privacy is removed (which legalizing blackmail would contribution somewhat towards). And that this isn’t just a fact about our current equilibria, it’s intrinsic to human biology.
It seems plausible (although I am quite skeptical) that a small group of humans might be able to construct an epistemically sound world that includes lack-of-intellectual-privacy, but they’d have to have correctly accounted for wide variety of subtle errors.
[edit: all of this assumes you are running on human wetware. If you remove that as a constraint other things may be possible]
further update: I do think rewards are something like 10x less problematic than punishments, because humans are risk averse and fear punishment more than they desire reward. (“10x” is a stand-in for “whatever the psychological research says on how big the difference is between human response to rewards and punishments”)
[note: this subthread is far afield from the article—LW is about publication, not private thoughts (unless there’s a section I don’t know about where only specifically invited people can see things) . And LW karma is far from the sanctions under discussion in the rest of the post.]
Have you considered things to reduce the assymetric impact of up- and down-votes? Cap karma value at −5? Use downvotes as a divisor for upvotes (say, score is upvotes / (1 + 0.25 * downvotes)) rather than simple subtraction?
We’ve thought about things in that space, although any of the ideas would be a fairly major change, and we haven’t come up with anything we feel good enough about to commit to.
(We have done some subtle things to avoid making downvotes feel worse than they need to, such as not including the explicit number of downvotes)
Do you think that thoughts are too incentivised or not incentivised enough on the margin, for the purpose of epistemically sound thinking? If they’re too incentivised, have you considered dampening LWs karma system? If they’re not incentivised enough, what makes you believe that legalising blackmail will worsen the epistemic quality of thoughts?
The LW karma obviously has its flaws, per Goodhart’s law. It is used anyway, because the alternative is having other problems, and for the moment this seems like a reasonable trade-off.
The punishment for “heresies” is actually very mild. As long as one posts respected content in general, posting a “heretical” comment every now and then does not ruin their karma. (Compare to people having their lives changed dramatically because of one tweet.) The punishment accumulates mostly for people whose only purpose here is to post “heresies”. Also, LW karma does not prevent anyone from posting “heresies” on a different website. Thus, people can keep positive LW karma even if their main topic is talking how LW is fundamentally wrong as long as they can avoid being annoying (for example by posting hundred LW-critical posts on their personal website, posting a short summary with hyperlinks on LW, and afterwards using LW mostly to debate other topics).
Blackmail typically attacks you in real life, i.e. you can’t limit the scope of impact. If losing an online account on a website X would be the worst possible outcome of one’s behavior at the website X, life would be easy. (You would only need to keep your accounts on different websites separated from each other.) It was already mentioned somewhere in this debate that blackmail often uses the difference between norms in different communities, i.e. that your local-norm-following behavior in one context can be local-norm-breaking in another context. This is quite unlike LW karma.
I’d say thoughts aren’t incentivized enough on the margin, but:
1. A major bottleneck is how fine-tuned and useful the incentives are. (i.e. I’d want to make LW karma more closely track “reward good epistemic processes” before I made the signal stronger. I think it currently tracks that well enough that I prefer it over no-karma).
2. It’s important that people can still have private thoughts separate from the LW karma system. LW is where you come when you have thoughts that seem good enough to either contribute to the commons, or to get feedback on so you can improve your thought process… after having had time to mull things over privately without worrying about what anyone will think of you.
(But, I also think, on the margin, people should be much less scared about sharing their private thoughts than they currently are. Many people seem to be scared about sharing unfinished thoughts at all, and my actual model of what is “threatening” says that there’s a much narrower domain where you need to be worried in the current environment)
3. One conscious decision we made was not not display “number of downvotes” on a post (we tried it out privately for admins for awhile). Instead we just included “total number of votes”. Explicitly knowing how much one’s post got downvoted felt much worse than having a vague sense of how good it was overall + a rough sense of how many people *may* have downvoted it. This created a stronger punishment signal than seemed actually appropriate.
(Separately, I am right now making arguments in terms that I’m fairly confident both of us value, but I also think there are reasons to want private thoughts that are more like “having a Raemon_healthy soul”, than like being able to contribute usefully to the intellectual commons.
(I noticed while writing this that the latter might be most of what a Benquo finds important for having a healthy soul, but unsure. In any case healthy souls are more complicated and I’m avoiding making claims about them for now)
If privacy in general is reduced, then they get to see others’ thoughts too [EDIT: this sentence isn’t critical, the rest works even if they can only see your thoughts]. If they’re acting justly, then they will take into account that others might modify their thoughts to look smarter, and make basically well-calibrated (if not always accurate) judgments about how smart different people are. (People who are trying can detect posers a lot of the time, even without mind-reading). So, them having more information means they are more likely to make a correct judgment, hiring the smarter person (or, generally, whoever can do the job better). At worst, even if they are very bad at detecting posers, they can see everyone’s thoughts and choose to ignore them, making the judgment they would make without having this information (But, they were probably already vulnerable to posers, it’s just that seeing people’s thoughts doesn’t have to make them more vulnerable).
If privacy in general is reduced, then they get to see others’ thoughts too.
This response seems mostly orthogonal to what I was worried about. It is quite plausible that most hiring decisions would become better in fully transparent (and also just?) world. But, fully-and-justly-transparent-world can still mean that fewer people think original or interesting thoughts because doing so is too risky.
And I might think this is bad, not only because of fewer-objectively-useful thoughts get thunk, but also because… it just kinda sucks and I don’t get to be myself?
(As well as, fully-transparent-and-just-world might still be a more stressful world to live in, and/or involve more cognitive overhead because I need to model how others will think about me all the time. Hypothetically we could come to an equilibrium wherein we *don’t* put extra effort into signaling legibly good thought processes. This is plausible, but it is indeed a background assumption of mine that this is not possible to run on human wetware)
Regarding that sentence, I edited my comment at about the same time you posted this.
But, fully-and-justly-transparent-world can still mean that fewer people think original or interesting thoughts because doing so is too risky.
If someone taking a risk is good with respect to the social good, then the justice process should be able to see that they did that and reward them (or at least not punish them) for it, right? This gets easier the more information is available to the justice process.
So, much of my thread was respond to this sentence:
Implication: “judge” means to use information against someone.
The point being, you can have entirely positive judgment, and have it still produce distortions. All that has to be true is that some forms of thought are more legibly good and get more rewarded, for a fully transparent system to start producing warped incentives on what sort of thoughts get thought.
i.e. say I have three options of what to think about today:
some random innocuous status quo thought (neither gets me rewarded nor punished)
some weird thought that seems kind of dumb, which most of the time is evidence about being dumb, which occasionally pays off with something creative and neat. (I’m not sure what kind of world we’re stipulating here. In some “just”-worlds, this sort of thought gets punished (because it’s usually dumb). In some “just worlds” it gets rewarded (because everyone has cooperated on some kind of long term strategy). In some just-worlds it’s hit or miss because there’s a collection of people trying different strategies with their rewards.
some heretical thought that seems actively dangerous, and only occasionally produces novel usefulness if I turn out to be real good at being contrarian.
a thought that is clearly, legibly good, almost certainly net positive, either by following well worn paths, or being “creatively out of the box” in a set of ways that are known to have pretty good returns.
Even in one of the possible-just-worlds, it seems like you’re going to incentivize the last one much more than the 2nd or 3rd.
This isn’t that different from the status quo – it’s a hard problem that VC funders have an easier time investing in people doing something that seems obviously good, then someone with a genuinely weird, new idea. But I think this would crank that problem up to 11, even if we stipulate a just-world.
...
Most importantly: the key implication I believe in, is that humans are not nearly smart enough at present to coordinate on anything like a just world, even if everyone were incredibly well intentioned. This whole conversation is in fact probably not possible for the average person to follow. (And this implication in this sentence right here right now is something that could get me punished in many circles, even by people trying hard to do the right thing. For reasons related to Overconfident talking down, humble or hostile talking up)
Even in one of the possible-just-worlds, it seems like you’re going to incentivize the last one much more than the 2nd or 3rd.
This is not responsive to what I said! If you can see (or infer) the process by which someone decided to have one thought or another, you can reward them for doing things that have higher expected returns, e.g. having heretical thoughts when heresy is net positive in expectation. If you can’t implement a process that complicated, you can just stop punishing people for heresy, entirely ignoring their thoughts if necessary.
the key implication I believe in, is that humans are not nearly smart enough at present to coordinate on anything like a just world, even if everyone were incredibly well intentioned. This whole conversation is in fact probably not possible for the average person to follow.
Average people don’t need to do it, someone needs to do it. The first target isn’t “make the whole world just”, it’s “make some local context just”. Actually, before that, it’s “produce common knowledge in some local context that the world is unjust but that justice is desirable”, which might actually be accomplished in this very thread, I’m not sure.
And this implication in this sentence right here right now is something that could get me punished in many circles, even by people trying hard to do the right thing.
Thanks for adding this information. I appreciate that you’re making these parts of your worldview clear.
This is not responsive to what I said! If you can see (or infer) the process by which someone decided to have one thought or another, you can reward them for doing things that have higher expected returns, e.g. having heretical thoughts when heresy is net positive in expectation.
This was most of what I meant to imply. I am mostly talking about rewards, not punishments.
I am claiming that rewards distort thoughts similarly to punishments, although somewhat more weakly because humans seem to respond more strongly to punishment than reward.
You’re continuing to miss the completely obvious point that a just process does no worse (in expectation) by having more information potentially available to it, which it can decide what to do with. Like, either you are missing really basic decision theory stuff covered in the Sequences or you are trolling.
(Agree that rewards affect thoughts too, and that these can cause distortions when done unjustly)
Your comments don’t seem to be acknowledging that, so from my perspective you seem to be describing an Impossible Utopia (capitalized because I intend to write a post that encapsulates the concept of Which Utopias Are Possible), and so it doesn’t seem very relevant.
(I recall claims on LessWrong that a decision process can do no worse with more information, but I don’t recall a compelling case that this was true on bounded human agents. Though I am interested if you have a post that responds to Zvi’s claims in the Choices are Bad series, and/or a post that articulates what exactly you mean by “just” since it sounds like you’re using it as a jargon term that’s meant to encapsulate more information than I’m receiving right now).
I’ve periodically mentioned that my arguments about “just worlds implemented on humans”. “Just worlds implemented on non-humans or augmented humans” might be quite different, and I think it’s worth talking about too.
But the topic here is legalizing blackmail in a human world. So it matters how this will be implemented on the median human, who are responsible for most actions.
Notice that in this conversation, where you are and I are both smarter than average, it is not obvious to both of us what the correct answer is here, and we have spent some time arguing about it. When I imagine the average human town, or company, or community, attempting to implement a just world that includes blackmail and full transparency, I am imagining either a) lots more time being spent trying to figure out the right answer, b) people getting wrong answers all the time.
The two posts you linked are not even a little relevant to the question of whether, in general, bounded agents do better or worse by having more information (Yes, choice paralysis might make some information about what choices you have costly, but more info also reduces choice paralysis by increasing certainty about how good the different options are, and overall the posts make no claim about the overall direction of info being good or bad for bounded agents). To avoid feeding the trolls, I’m going to stop responding here.
I’m not trolling. I have some probability on me being the confused one here. But given the downvote record above, it seems like the claims you’re making are at least less obvious than you think they are.
If you value those claims being treated as obvious-things-to-build-off-of by the LW commentariat, you may want to expand on the details or address confusions about them at some point.
But, I do think it is generally important for people to be able to tap out of conversations whenever the conversation is seeming low value, and seems reasonable for this thread to terminate.
I have some probability on me being the confused one here.
In conversations like this, both sides are confused, that is don’t understand the other’s point, so “who is the confused one” is already an incorrect framing. One of you may be factually correct, but that doesn’t really matter for making a conversation work, understanding each other is more relevant.
(In this particular case, I think both of you are correct and fail to see what the other means, but Jessica’s point is harder to follow and pattern-matches misleading things, hence the balance of votes.)
(I downvoted some of Jessica’s comments, mostly only in the cases where I thought she was not putting in a good faith effort to try to understand what her interlocutor is trying to say, like her comment upstream in the thread. Saying that talking to someone is equivalent to feeding trolls is rarely a good move, and seems particularly bad in situations where you are talking about highly subjective and fuzzy concepts. I upvoted all of her comments that actually made points without dismissing other people’s perspectives, so in my case, I don’t really think that the voting patterns are a result of her ideas being harder to follow, and more the result of me perceiving her to be violating certain conversational norms)
In conversations like this, both sides are confused,
Nod. I did actually consider a more accurate version of the comment that said something like “at least one of us is at least somewhat confused about something”, but by the time we got to this comment I was just trying to disengage while saying the things that seemed most important to wrap up with.
Nod. I did actually consider a more accurate version of the comment that said something like “at least one of us is at least somewhat confused about something” [...]
The clarification doesn’t address what I was talking about, or else disagrees with my point, so I don’t see how that can be characterised with a “Nod”. The confusion I refer to is about what the other means, with the question of whether anyone is correct about the world irrelevant. And this confusion is significant on both sides, otherwise a conversation doesn’t go off the rails in this way. Paying attention to truth is counterproductive when intended meaning is not yet established, and you seem to be talking about truth, while I was commenting about meaning.
Hmm. Well I am now somewhat confused what you mean. Say more? (My intention was for ‘at least one of us is confused’ to be casting a fairly broad net that included ‘confused about the world’, or ‘confused about what each other meant by our words’, or ‘confused… on some other level that I couldn’t predict easily.’)
(In particular, ‘scapegoating’ feels like a very different frame than the one I’d use here)
Having read Zvi’s post and my comment, do you think the norm-enforcement process is just, or even not very unjust? If not, what makes it not scapegoating?
I think scapegoating has a particular definition – blaming someone for something that they didn’t do because your social environment demands someone get blamed. And that this isn’t relevant to most of my concerns here. You can get unjustly punished for things that have nothing to do with scapegoating.
Good point. I think there is a lot of scapegoating (in the sense you mean here) but that’s a further claim than that it’s unjust punishment, and I don’t believe this strongly enough to argue it right now.
I think you’re pointing in an important direction, but your phrasing sounds off to me.
(In particular, ‘scapegoating’ feels like a very different frame than the one I’d use here)
If I think out loud, especially about something I’m uncertain about, that other people have opinions on, a few things can happen to me:
Someone who overhears part of my thought process might think (correctly, even!) that my thought process reveals that I am not very smart. Therefore, they will be less likely to hire me. This is punishment, but it’s very much not “scapegoating” style punishment.
Someone who overhears my private thought process might (correctly, or incorrectly! either) come to think that I am smart, and be more likely to hire me. This can be just as dangerous. In a world where all information is public, I have to attend to how the process by which I act and think looks. I am incentivized to think in ways that are legibly good.
“Judgment” is dangerous to me (epistemically) even if the judgment is positive, because it incentives me against exploring paths that look bad, or are good for incomprehensible reasons.
This seems like a general argument that providing evidence without trying to control the conclusions others draw is bad because it leads to errors. It doesn’t seem to take into account the cost of reduced info glow or the possibility that the gatekeeper might also introduce errors. That’s before we even consider self-serving bias!
Related: http://benjaminrosshoffman.com/humility-argument-honesty/
TLDR: I literally do not understand how to interpret your comment as NOT a general endorsement of fraud and implicit declaration of intent to engage in it.
My intent was not that it’s “bad”, just, if you do not attempt to control the conclusions of others, they will predictably form conclusions of particular types, and this will have effects. (It so happens that I think most people won’t like those effects, and therefore will attempt to control the conclusions of others.)
(I feel somewhat confused by the above comment, actually. Can you taboo “bad” and try saying it in different words?)
Ah, if you literally just mean it increases variance & risk, that’s true in the very short term. In context it sounded to me like a policy argument against doing so, but on reflection it’s easy to read you as meaning the more reasonable thing. Thank you for explaining.
Hmm. I think I meant something more like your second interpretation than your first interpretation but I think I actually meant a third thing and am not confident we aren’t still misunderstanding each other.
An intended implication, (which comes with an if-then suggestion, which was not an essential part of my original claim but I think is relevant) is:
If you value being able to think freely and have epistemologically sound thoughts, it is important to be able to think thoughts that you will neither be rewarded nor punished for… [edit: or be extremely confident than you have accounted for your biases towards reward gradients]. And the rewards are only somewhat less bad than the punishments.
A followup implication is that this is not possible to maintain humanity-wide if thought-privacy is removed (which legalizing blackmail would contribution somewhat towards). And that this isn’t just a fact about our current equilibria, it’s intrinsic to human biology.
It seems plausible (although I am quite skeptical) that a small group of humans might be able to construct an epistemically sound world that includes lack-of-intellectual-privacy, but they’d have to have correctly accounted for wide variety of subtle errors.
[edit: all of this assumes you are running on human wetware. If you remove that as a constraint other things may be possible]
further update: I do think rewards are something like 10x less problematic than punishments, because humans are risk averse and fear punishment more than they desire reward. (“10x” is a stand-in for “whatever the psychological research says on how big the difference is between human response to rewards and punishments”)
[note: this subthread is far afield from the article—LW is about publication, not private thoughts (unless there’s a section I don’t know about where only specifically invited people can see things) . And LW karma is far from the sanctions under discussion in the rest of the post.]
Have you considered things to reduce the assymetric impact of up- and down-votes? Cap karma value at −5? Use downvotes as a divisor for upvotes (say, score is upvotes / (1 + 0.25 * downvotes)) rather than simple subtraction?
We’ve thought about things in that space, although any of the ideas would be a fairly major change, and we haven’t come up with anything we feel good enough about to commit to.
(We have done some subtle things to avoid making downvotes feel worse than they need to, such as not including the explicit number of downvotes)
Do you think that thoughts are too incentivised or not incentivised enough on the margin, for the purpose of epistemically sound thinking? If they’re too incentivised, have you considered dampening LWs karma system? If they’re not incentivised enough, what makes you believe that legalising blackmail will worsen the epistemic quality of thoughts?
The LW karma obviously has its flaws, per Goodhart’s law. It is used anyway, because the alternative is having other problems, and for the moment this seems like a reasonable trade-off.
The punishment for “heresies” is actually very mild. As long as one posts respected content in general, posting a “heretical” comment every now and then does not ruin their karma. (Compare to people having their lives changed dramatically because of one tweet.) The punishment accumulates mostly for people whose only purpose here is to post “heresies”. Also, LW karma does not prevent anyone from posting “heresies” on a different website. Thus, people can keep positive LW karma even if their main topic is talking how LW is fundamentally wrong as long as they can avoid being annoying (for example by posting hundred LW-critical posts on their personal website, posting a short summary with hyperlinks on LW, and afterwards using LW mostly to debate other topics).
Blackmail typically attacks you in real life, i.e. you can’t limit the scope of impact. If losing an online account on a website X would be the worst possible outcome of one’s behavior at the website X, life would be easy. (You would only need to keep your accounts on different websites separated from each other.) It was already mentioned somewhere in this debate that blackmail often uses the difference between norms in different communities, i.e. that your local-norm-following behavior in one context can be local-norm-breaking in another context. This is quite unlike LW karma.
I’d say thoughts aren’t incentivized enough on the margin, but:
1. A major bottleneck is how fine-tuned and useful the incentives are. (i.e. I’d want to make LW karma more closely track “reward good epistemic processes” before I made the signal stronger. I think it currently tracks that well enough that I prefer it over no-karma).
2. It’s important that people can still have private thoughts separate from the LW karma system. LW is where you come when you have thoughts that seem good enough to either contribute to the commons, or to get feedback on so you can improve your thought process… after having had time to mull things over privately without worrying about what anyone will think of you.
(But, I also think, on the margin, people should be much less scared about sharing their private thoughts than they currently are. Many people seem to be scared about sharing unfinished thoughts at all, and my actual model of what is “threatening” says that there’s a much narrower domain where you need to be worried in the current environment)
3. One conscious decision we made was not not display “number of downvotes” on a post (we tried it out privately for admins for awhile). Instead we just included “total number of votes”. Explicitly knowing how much one’s post got downvoted felt much worse than having a vague sense of how good it was overall + a rough sense of how many people *may* have downvoted it. This created a stronger punishment signal than seemed actually appropriate.
(Separately, I am right now making arguments in terms that I’m fairly confident both of us value, but I also think there are reasons to want private thoughts that are more like “having a Raemon_healthy soul”, than like being able to contribute usefully to the intellectual commons.
(I noticed while writing this that the latter might be most of what a Benquo finds important for having a healthy soul, but unsure. In any case healthy souls are more complicated and I’m avoiding making claims about them for now)
If privacy in general is reduced, then they get to see others’ thoughts too [EDIT: this sentence isn’t critical, the rest works even if they can only see your thoughts]. If they’re acting justly, then they will take into account that others might modify their thoughts to look smarter, and make basically well-calibrated (if not always accurate) judgments about how smart different people are. (People who are trying can detect posers a lot of the time, even without mind-reading). So, them having more information means they are more likely to make a correct judgment, hiring the smarter person (or, generally, whoever can do the job better). At worst, even if they are very bad at detecting posers, they can see everyone’s thoughts and choose to ignore them, making the judgment they would make without having this information (But, they were probably already vulnerable to posers, it’s just that seeing people’s thoughts doesn’t have to make them more vulnerable).
This response seems mostly orthogonal to what I was worried about. It is quite plausible that most hiring decisions would become better in fully transparent (and also just?) world. But, fully-and-justly-transparent-world can still mean that fewer people think original or interesting thoughts because doing so is too risky.
And I might think this is bad, not only because of fewer-objectively-useful thoughts get thunk, but also because… it just kinda sucks and I don’t get to be myself?
(As well as, fully-transparent-and-just-world might still be a more stressful world to live in, and/or involve more cognitive overhead because I need to model how others will think about me all the time. Hypothetically we could come to an equilibrium wherein we *don’t* put extra effort into signaling legibly good thought processes. This is plausible, but it is indeed a background assumption of mine that this is not possible to run on human wetware)
Regarding that sentence, I edited my comment at about the same time you posted this.
If someone taking a risk is good with respect to the social good, then the justice process should be able to see that they did that and reward them (or at least not punish them) for it, right? This gets easier the more information is available to the justice process.
So, much of my thread was respond to this sentence:
The point being, you can have entirely positive judgment, and have it still produce distortions. All that has to be true is that some forms of thought are more legibly good and get more rewarded, for a fully transparent system to start producing warped incentives on what sort of thoughts get thought.
i.e. say I have three options of what to think about today:
some random innocuous status quo thought (neither gets me rewarded nor punished)
some weird thought that seems kind of dumb, which most of the time is evidence about being dumb, which occasionally pays off with something creative and neat. (I’m not sure what kind of world we’re stipulating here. In some “just”-worlds, this sort of thought gets punished (because it’s usually dumb). In some “just worlds” it gets rewarded (because everyone has cooperated on some kind of long term strategy). In some just-worlds it’s hit or miss because there’s a collection of people trying different strategies with their rewards.
some heretical thought that seems actively dangerous, and only occasionally produces novel usefulness if I turn out to be real good at being contrarian.
a thought that is clearly, legibly good, almost certainly net positive, either by following well worn paths, or being “creatively out of the box” in a set of ways that are known to have pretty good returns.
Even in one of the possible-just-worlds, it seems like you’re going to incentivize the last one much more than the 2nd or 3rd.
This isn’t that different from the status quo – it’s a hard problem that VC funders have an easier time investing in people doing something that seems obviously good, then someone with a genuinely weird, new idea. But I think this would crank that problem up to 11, even if we stipulate a just-world.
...
Most importantly: the key implication I believe in, is that humans are not nearly smart enough at present to coordinate on anything like a just world, even if everyone were incredibly well intentioned. This whole conversation is in fact probably not possible for the average person to follow. (And this implication in this sentence right here right now is something that could get me punished in many circles, even by people trying hard to do the right thing. For reasons related to Overconfident talking down, humble or hostile talking up)
This is not responsive to what I said! If you can see (or infer) the process by which someone decided to have one thought or another, you can reward them for doing things that have higher expected returns, e.g. having heretical thoughts when heresy is net positive in expectation. If you can’t implement a process that complicated, you can just stop punishing people for heresy, entirely ignoring their thoughts if necessary.
Average people don’t need to do it, someone needs to do it. The first target isn’t “make the whole world just”, it’s “make some local context just”. Actually, before that, it’s “produce common knowledge in some local context that the world is unjust but that justice is desirable”, which might actually be accomplished in this very thread, I’m not sure.
Thanks for adding this information. I appreciate that you’re making these parts of your worldview clear.
This was most of what I meant to imply. I am mostly talking about rewards, not punishments.
I am claiming that rewards distort thoughts similarly to punishments, although somewhat more weakly because humans seem to respond more strongly to punishment than reward.
You’re continuing to miss the completely obvious point that a just process does no worse (in expectation) by having more information potentially available to it, which it can decide what to do with. Like, either you are missing really basic decision theory stuff covered in the Sequences or you are trolling.
(Agree that rewards affect thoughts too, and that these can cause distortions when done unjustly)
Yes, I disagree with that point, and I feel like you’ve been missing the completely obvious point that bounded agents have limited capabilities.
Choices are costly.
Choices are really costly.
Your comments don’t seem to be acknowledging that, so from my perspective you seem to be describing an Impossible Utopia (capitalized because I intend to write a post that encapsulates the concept of Which Utopias Are Possible), and so it doesn’t seem very relevant.
(I recall claims on LessWrong that a decision process can do no worse with more information, but I don’t recall a compelling case that this was true on bounded human agents. Though I am interested if you have a post that responds to Zvi’s claims in the Choices are Bad series, and/or a post that articulates what exactly you mean by “just” since it sounds like you’re using it as a jargon term that’s meant to encapsulate more information than I’m receiving right now).
I’ve periodically mentioned that my arguments about “just worlds implemented on humans”. “Just worlds implemented on non-humans or augmented humans” might be quite different, and I think it’s worth talking about too.
But the topic here is legalizing blackmail in a human world. So it matters how this will be implemented on the median human, who are responsible for most actions.
Notice that in this conversation, where you are and I are both smarter than average, it is not obvious to both of us what the correct answer is here, and we have spent some time arguing about it. When I imagine the average human town, or company, or community, attempting to implement a just world that includes blackmail and full transparency, I am imagining either a) lots more time being spent trying to figure out the right answer, b) people getting wrong answers all the time.
The two posts you linked are not even a little relevant to the question of whether, in general, bounded agents do better or worse by having more information (Yes, choice paralysis might make some information about what choices you have costly, but more info also reduces choice paralysis by increasing certainty about how good the different options are, and overall the posts make no claim about the overall direction of info being good or bad for bounded agents). To avoid feeding the trolls, I’m going to stop responding here.
I’m not trolling. I have some probability on me being the confused one here. But given the downvote record above, it seems like the claims you’re making are at least less obvious than you think they are.
If you value those claims being treated as obvious-things-to-build-off-of by the LW commentariat, you may want to expand on the details or address confusions about them at some point.
But, I do think it is generally important for people to be able to tap out of conversations whenever the conversation is seeming low value, and seems reasonable for this thread to terminate.
In conversations like this, both sides are confused, that is don’t understand the other’s point, so “who is the confused one” is already an incorrect framing. One of you may be factually correct, but that doesn’t really matter for making a conversation work, understanding each other is more relevant.
(In this particular case, I think both of you are correct and fail to see what the other means, but Jessica’s point is harder to follow and pattern-matches misleading things, hence the balance of votes.)
(I downvoted some of Jessica’s comments, mostly only in the cases where I thought she was not putting in a good faith effort to try to understand what her interlocutor is trying to say, like her comment upstream in the thread. Saying that talking to someone is equivalent to feeding trolls is rarely a good move, and seems particularly bad in situations where you are talking about highly subjective and fuzzy concepts. I upvoted all of her comments that actually made points without dismissing other people’s perspectives, so in my case, I don’t really think that the voting patterns are a result of her ideas being harder to follow, and more the result of me perceiving her to be violating certain conversational norms)
Nod. I did actually consider a more accurate version of the comment that said something like “at least one of us is at least somewhat confused about something”, but by the time we got to this comment I was just trying to disengage while saying the things that seemed most important to wrap up with.
The clarification doesn’t address what I was talking about, or else disagrees with my point, so I don’t see how that can be characterised with a “Nod”. The confusion I refer to is about what the other means, with the question of whether anyone is correct about the world irrelevant. And this confusion is significant on both sides, otherwise a conversation doesn’t go off the rails in this way. Paying attention to truth is counterproductive when intended meaning is not yet established, and you seem to be talking about truth, while I was commenting about meaning.
Hmm. Well I am now somewhat confused what you mean. Say more? (My intention was for ‘at least one of us is confused’ to be casting a fairly broad net that included ‘confused about the world’, or ‘confused about what each other meant by our words’, or ‘confused… on some other level that I couldn’t predict easily.’)
Having read Zvi’s post and my comment, do you think the norm-enforcement process is just, or even not very unjust? If not, what makes it not scapegoating?
I think scapegoating has a particular definition – blaming someone for something that they didn’t do because your social environment demands someone get blamed. And that this isn’t relevant to most of my concerns here. You can get unjustly punished for things that have nothing to do with scapegoating.
Good point. I think there is a lot of scapegoating (in the sense you mean here) but that’s a further claim than that it’s unjust punishment, and I don’t believe this strongly enough to argue it right now.