I probably don’t understand the risks! Like I have some similar example institutions in mind, but none that are super similar to what I’m doing or which have consequences as bad as you are implying, so I assume maybe there’s a significant history that you are aware of and which I have missed. What examples do you have in mind?
I plan to write my idea in much greater detail later before actually launching it. More information now about the risks could be convenient if it shows how it is a waste of time to pursue, though alternatively if I disagree I might address the disagreements in later writeups.
One risk is that similar-sounding institutions can and do occasionally get taken over precisely by the people they’re setup to prevent, and then those people have institutional backing and are even harder to dislodge.
E.g. see the section on Legible Signals from a podcast / interview with habryka from early this year:
Like in the context of the FTX situation, a proposal that I’ve discussed with a number of people is, “Fuck, man, why did people trust Sam? I didn’t trust Sam.” “What we should have done is,we should have just created a number, like, there’s 10 cards and these are the ‘actually trustworthy people’ and ‘high-integrity people’ cards, and we should have given them to 10 people in the EA community who we actually think are [highly] trustworthy and high-integrity people, so that it actually makes sense for you to trust them. And we should just have [high-legibility] signals of trust and judgment that are clearly legible to the world.”
To which my response was “Lol, that would have made this whole situation much worse.” I can guarantee you that if you [had] handed a number of people—in this ecosystem or any other ecosystem—the “This person has definitely good judgment and you should trust what they say.” [card. Then] in the moment somebody has that card and has that official role in the ecosystem, of course they will be [under a] shitton of adversarial pressure, for [them to] now endorse people who really care about getting additional resources, who really care about stuff.
...
And then I’m like, “Well, there are two worlds. Either nobody cares about who you think is high-integrity and trustworthy, or people *do* care and now you’ve made the lives of everyone who you gave a high-integrity / trustworthy card a lot worse. Because now they’re just an obvious giant target, that if you successfully get one of the people of the high-integrity, high-trustworthy cards to endorse you, you have free reign and now challenging you becomes equivalent to challenging the “high-integrity, [high-trust] people” institution. Which sure seems like one of the hardest institutions to object to.
And I think we’ve seen this in a number of other places… There was a specific board set up by the Center for Applied Rationality, [where] CFAR kept being asked to navigate various community disputes, and they were like, “Look, man, we would like to run workshops, can we please do anything else?” And then they set up a board to be like, “Look, if you have community disputes in the Bay Area, go to this board. They will maybe do some investigative stuff, and then they will try to figure out what should happen, like, do mediation, maybe [speak about] who was actually in the right, who was in the wrong.”
And approximately the first thing that happened is that, like, one of the people who I consider most abusive in the EA community basically just captured that board, and [got] all the board members to endorse him quite strongly. And then when a bunch of people who were hurt by him came out, the board was like, “Oh, we definitely don’t think these [people who were abused] are saying anything correct. We trust the guy who abused everyone.”
Which is a specific example of, if you have an institution that is being given the power to [blame] and speak judgment on people, and try to create common knowledge about [what] is trustworthy and what is non-trustworthy, that institution is under a lot of pressure...
[We] can see similar things happening with HR departments all around the world. Where the official [purpose] of the HR department is to, you know, somehow make your staff happy and give them [ways to] escalate to management if their manager is bad. But most HR departments around the world are actually, like, a trap, where if you go to the HR department, probably the person you complained about is one of the first people to find out, and then you can be kicked out of the organization before your complaint can travel anywhere else.
It’s not true in all HR departments but it’s a common enough occurrence in HR departments that if you look at Hacker News and are like, “Should I complain to HR about my problems?”, like half of the commenters will be like, “Never talk to HR.” HR is the single most corrupt part of any organization. And I think this is the case because it also tends to be the place where hiring and firing [decisions get] made and therefore is under a lot of pressure.
This is definitely something I’ve thought about and have multiple layers of plans to reduce, though my plans are admittedly of questionable strength, so there are pretty legitimate reasons to doubt my idea. I will probably research this some more before writing it up. That said my idea is very different from just handing out “you can trust this person” cards.
I think if you want to pursue a project in the “justice” space, you should write up the problems you see and your planned incentive structures in a way that is legible. Then people can decide if they should trust your justice procedure.
My crux around this is something like; conflict resolution is an iterated and adversarial game, where failure can cost a lot. “Creating an institution to work on [uncovering injustices and predators]” looks to me a lot like “Creating an institution to keep secrets on computers connected to the internet.” You don’t just have to outsmart the basic problem, you also need to outsmart everyone who has an incentive to subvert your system. I don’t think that’s an impossible challenge but it’s harder than it looks, and it’s harder than it looks in part because some people are actively trying to obscure the ways in which it’s hard. The system can even look like it’s working fine right up until it tries to tackle something important, in the same way that a substitution cypher will look like it’s working fine right up until I try to store a bunch of bank account information with it.
There’s also a dynamic of something like… this is one of those issues where being too interested in the problem is correlated with being bad at solving it. Obviously you have to compromise on this a little or these kinds of things never get done, but if someone’s only qualification is interest I think the EV is very negative.
That sounds really discouraging, so I want to tell @tailcalled: I think it’s great you care about people are want to prevent them from being hurt. I think the easiest, least risky way to do that is to create abundance so people have less dependence on any one entity and are thus less vulnerable. The more parties being thrown by people who aren’t creeps (or harboring creeps), the easier it is to avoid the parties that are. So I’d encourage you to start by building socially, rather than investigation.
Same with my comment. :-/ Maybe the downvoters want to point out the risk of this turning into some denunciation/witchhunting/revolution eating her own children/cancel culture scenario. I’m worried about these dangers too (which is why I mentioned autoimmune disorders), but didn’t want to turn my comment into an essay exploring pros and cons and risks and benefits and negative attractor states and ways to avoid them.
Of course, I would appreciate some explanation from the downvoters. My policy is to only downvote if I also take the time to comment.
Like ProgramCrafter I neither downvoted nor upvoted your comment.
Maybe the downvoters want to point out the risk of this turning into some denunciation/witchhunting/revolution eating her own children/cancel culture scenario.
Could be what they are worried about.
My current model of how witch-hunts/cancel-culture occurs is that when there is no legitimate way to get justice, people sometimes manage to arrange vigilante justice based on social alliances, but that vigilante justice is by its very nature going to be more chaotic and less accurate than a proper system.
So one consequence of my idea, if it works properly, is that it would reduce witchhunts by providing victims with effective means of achieving justice.
Possibly. I would expect it to be very difficult to build a legitimate, independent and just institution for that. There is a reason we have checks and balances in government.
I think that this idea came from a thought process that generally generates good ideas (e.g. an LLM API that predicts whether someone has read the Sequences), but this time, due to bad luck, it ended up outputting an extremely bad idea. (I didn’t downvote)
I guess that people who downvoted this would like to see more details why this “court” would work and how won’t it be sued when it misjudges (and the more cases there are, the higher probability of misjudge is).
(meta: I neither downvoted nor upvoted the proposal)
For a while I have been thinking about how one can best come to useful truth with respect to controversial subjects. I have developed a theoretical framework that I need to write up in detail, but for now here’s the short version:
There is no executive which can deal out punishment or fine to pay for damages, so neither the accuser nor the accused have sufficient motivation to unravel the truth; instead most of the value will have to come from informing the community, and we suffer a commons problem because each community member is not sufficiently motivated.
Different parties have different questions that they are most interested in. One of the biggest jobs of the court is to distinguish all of the relevant questions so that things don’t get falsely generalized.
Most people do not have time to work through the details, so the court needs to provide an easy-to-digest summary.
The court needs to dig up novel evidence, because most evidence is not publicly available, and it needs to come up with novel theory of social interactions to understand the significance of the events, making use of rationalist skills.
I’m of course open to input for how to do things differently than this, though you should expect some pushback because I do currently have some theory and observations backing the above strategy, so the discussion will have to clarify the alternatives.
Why the downvotes?
the median outcome for projects like this is doing far more harm than good
you haven’t given any indication you understand the risks, much less are likely to beat them.
I probably don’t understand the risks! Like I have some similar example institutions in mind, but none that are super similar to what I’m doing or which have consequences as bad as you are implying, so I assume maybe there’s a significant history that you are aware of and which I have missed. What examples do you have in mind?
I plan to write my idea in much greater detail later before actually launching it. More information now about the risks could be convenient if it shows how it is a waste of time to pursue, though alternatively if I disagree I might address the disagreements in later writeups.
One risk is that similar-sounding institutions can and do occasionally get taken over precisely by the people they’re setup to prevent, and then those people have institutional backing and are even harder to dislodge.
E.g. see the section on Legible Signals from a podcast / interview with habryka from early this year:
This is definitely something I’ve thought about and have multiple layers of plans to reduce, though my plans are admittedly of questionable strength, so there are pretty legitimate reasons to doubt my idea. I will probably research this some more before writing it up. That said my idea is very different from just handing out “you can trust this person” cards.
I think if you want to pursue a project in the “justice” space, you should write up the problems you see and your planned incentive structures in a way that is legible. Then people can decide if they should trust your justice procedure.
I plan on doing that.
I think that that Habryka podcast has a lot of potential for projects, it just needs a wide variety of people to build off of it.
Haven’t downvoted but was considering it.
My crux around this is something like; conflict resolution is an iterated and adversarial game, where failure can cost a lot. “Creating an institution to work on [uncovering injustices and predators]” looks to me a lot like “Creating an institution to keep secrets on computers connected to the internet.” You don’t just have to outsmart the basic problem, you also need to outsmart everyone who has an incentive to subvert your system. I don’t think that’s an impossible challenge but it’s harder than it looks, and it’s harder than it looks in part because some people are actively trying to obscure the ways in which it’s hard. The system can even look like it’s working fine right up until it tries to tackle something important, in the same way that a substitution cypher will look like it’s working fine right up until I try to store a bunch of bank account information with it.
It seems like you aren’t noticing some skulls.
There’s also a dynamic of something like… this is one of those issues where being too interested in the problem is correlated with being bad at solving it. Obviously you have to compromise on this a little or these kinds of things never get done, but if someone’s only qualification is interest I think the EV is very negative.
That sounds really discouraging, so I want to tell @tailcalled: I think it’s great you care about people are want to prevent them from being hurt. I think the easiest, least risky way to do that is to create abundance so people have less dependence on any one entity and are thus less vulnerable. The more parties being thrown by people who aren’t creeps (or harboring creeps), the easier it is to avoid the parties that are. So I’d encourage you to start by building socially, rather than investigation.
Same with my comment. :-/ Maybe the downvoters want to point out the risk of this turning into some denunciation/witchhunting/revolution eating her own children/cancel culture scenario. I’m worried about these dangers too (which is why I mentioned autoimmune disorders), but didn’t want to turn my comment into an essay exploring pros and cons and risks and benefits and negative attractor states and ways to avoid them.
Of course, I would appreciate some explanation from the downvoters. My policy is to only downvote if I also take the time to comment.
Like ProgramCrafter I neither downvoted nor upvoted your comment.
Could be what they are worried about.
My current model of how witch-hunts/cancel-culture occurs is that when there is no legitimate way to get justice, people sometimes manage to arrange vigilante justice based on social alliances, but that vigilante justice is by its very nature going to be more chaotic and less accurate than a proper system.
So one consequence of my idea, if it works properly, is that it would reduce witchhunts by providing victims with effective means of achieving justice.
Possibly. I would expect it to be very difficult to build a legitimate, independent and just institution for that. There is a reason we have checks and balances in government.
I think that this idea came from a thought process that generally generates good ideas (e.g. an LLM API that predicts whether someone has read the Sequences), but this time, due to bad luck, it ended up outputting an extremely bad idea. (I didn’t downvote)
I guess that people who downvoted this would like to see more details why this “court” would work and how won’t it be sued when it misjudges (and the more cases there are, the higher probability of misjudge is).
(meta: I neither downvoted nor upvoted the proposal)
Ah.
I briefly mentioned this in the discord:
The good news is I’ve strongly upvoted this back to positive territory.