One risk is that similar-sounding institutions can and do occasionally get taken over precisely by the people they’re setup to prevent, and then those people have institutional backing and are even harder to dislodge.
E.g. see the section on Legible Signals from a podcast / interview with habryka from early this year:
Like in the context of the FTX situation, a proposal that I’ve discussed with a number of people is, “Fuck, man, why did people trust Sam? I didn’t trust Sam.” “What we should have done is,we should have just created a number, like, there’s 10 cards and these are the ‘actually trustworthy people’ and ‘high-integrity people’ cards, and we should have given them to 10 people in the EA community who we actually think are [highly] trustworthy and high-integrity people, so that it actually makes sense for you to trust them. And we should just have [high-legibility] signals of trust and judgment that are clearly legible to the world.”
To which my response was “Lol, that would have made this whole situation much worse.” I can guarantee you that if you [had] handed a number of people—in this ecosystem or any other ecosystem—the “This person has definitely good judgment and you should trust what they say.” [card. Then] in the moment somebody has that card and has that official role in the ecosystem, of course they will be [under a] shitton of adversarial pressure, for [them to] now endorse people who really care about getting additional resources, who really care about stuff.
...
And then I’m like, “Well, there are two worlds. Either nobody cares about who you think is high-integrity and trustworthy, or people *do* care and now you’ve made the lives of everyone who you gave a high-integrity / trustworthy card a lot worse. Because now they’re just an obvious giant target, that if you successfully get one of the people of the high-integrity, high-trustworthy cards to endorse you, you have free reign and now challenging you becomes equivalent to challenging the “high-integrity, [high-trust] people” institution. Which sure seems like one of the hardest institutions to object to.
And I think we’ve seen this in a number of other places… There was a specific board set up by the Center for Applied Rationality, [where] CFAR kept being asked to navigate various community disputes, and they were like, “Look, man, we would like to run workshops, can we please do anything else?” And then they set up a board to be like, “Look, if you have community disputes in the Bay Area, go to this board. They will maybe do some investigative stuff, and then they will try to figure out what should happen, like, do mediation, maybe [speak about] who was actually in the right, who was in the wrong.”
And approximately the first thing that happened is that, like, one of the people who I consider most abusive in the EA community basically just captured that board, and [got] all the board members to endorse him quite strongly. And then when a bunch of people who were hurt by him came out, the board was like, “Oh, we definitely don’t think these [people who were abused] are saying anything correct. We trust the guy who abused everyone.”
Which is a specific example of, if you have an institution that is being given the power to [blame] and speak judgment on people, and try to create common knowledge about [what] is trustworthy and what is non-trustworthy, that institution is under a lot of pressure...
[We] can see similar things happening with HR departments all around the world. Where the official [purpose] of the HR department is to, you know, somehow make your staff happy and give them [ways to] escalate to management if their manager is bad. But most HR departments around the world are actually, like, a trap, where if you go to the HR department, probably the person you complained about is one of the first people to find out, and then you can be kicked out of the organization before your complaint can travel anywhere else.
It’s not true in all HR departments but it’s a common enough occurrence in HR departments that if you look at Hacker News and are like, “Should I complain to HR about my problems?”, like half of the commenters will be like, “Never talk to HR.” HR is the single most corrupt part of any organization. And I think this is the case because it also tends to be the place where hiring and firing [decisions get] made and therefore is under a lot of pressure.
This is definitely something I’ve thought about and have multiple layers of plans to reduce, though my plans are admittedly of questionable strength, so there are pretty legitimate reasons to doubt my idea. I will probably research this some more before writing it up. That said my idea is very different from just handing out “you can trust this person” cards.
I think if you want to pursue a project in the “justice” space, you should write up the problems you see and your planned incentive structures in a way that is legible. Then people can decide if they should trust your justice procedure.
One risk is that similar-sounding institutions can and do occasionally get taken over precisely by the people they’re setup to prevent, and then those people have institutional backing and are even harder to dislodge.
E.g. see the section on Legible Signals from a podcast / interview with habryka from early this year:
This is definitely something I’ve thought about and have multiple layers of plans to reduce, though my plans are admittedly of questionable strength, so there are pretty legitimate reasons to doubt my idea. I will probably research this some more before writing it up. That said my idea is very different from just handing out “you can trust this person” cards.
I think if you want to pursue a project in the “justice” space, you should write up the problems you see and your planned incentive structures in a way that is legible. Then people can decide if they should trust your justice procedure.
I plan on doing that.
I think that that Habryka podcast has a lot of potential for projects, it just needs a wide variety of people to build off of it.