> To be clear, while there is obviously some fun intended in this tradition, I don’t think describing it as “just a game” feels appropriate to me. I do actually really care about people being able to coordinate to not take the site down. It’s an actual hard thing to do that actually is trying to reinforce a bunch of the real and important values that I care about in Petrov day. Of course, I can’t force you to feel a certain way, but like, I do sure feel a pretty high level of disappointment reading this response.
[Epistemic-emotional status: reporting an emotional reaction, which may or may not have correct reasoning; attempting to honor the emotional tone of that reaction without actually wanting to be confrontational about this. Important note: when I say that it feels like people are being shamed, I do not mean think that anyone actually intends to shame people; it’s that I feel that that’s what the tone will communicate to many, intentionally or not. I could be wrong about that too. Again, trying to fairly report the implicit model that one part of me has, without asserting that the model is necessarily fully correct.]
So I notice that there’s a part of me which gets annoyed, borderline upset when I hear this Petrov Day ritual discussed in this manner.
The upset feels similar to what I’ve previously experienced when something that’s obviously a purely symbolic gesture is treated as a Big Important Thing That’s Actually Making A Difference. As it feels like the symbolic thing is distracting from things that actually matter.
That by itself wouldn’t trigger the reaction; the world is full of purely symbolic gestures that are claiming to make a difference, but they mostly haven’t upset me in a long time. Some of the communication around Petrov Day has. I think it’s because of a sense that this idea is being pushed on people-that-I-care-about as something important despite not actually being in accordance to their values, and that there’s social pressure for people to be quiet about it and give in to the social pressure at a cost to their epistemics.
I feel like Oliver’s comment is basically saying “people should have taken this seriously and people who treat this light-heartedly are in the wrong”. It’s spoken from a position of authority, and feels like it’s shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant, as no compelling reason for this ritual actually being significant has ever been presented.
As Chris notes, the communication around this has been contradictory, with some of it being presented as “this is just a game is good fun” and some “this is actually a thing to be taken seriously”. That also feels like a catch-22, where it’s implied that it’s shameworthy to not have taken the game seriously, when people were simultaneously also given the signal that it’s not serious.
I think that to a large extent, I stopped caring about meaningless rituals being pushed within my ingroup because I stopped considering my ingroup to include the kinds of groups that were pushing the rituals. But Chris feels like part of my ingroup, and now I feel upset again because I feel he’s being punished unfairly.
Chris wrote a comment saying this
Well, it was just a game and I had other things to do.
Before I strong-upvoted it, the comment was down at −11 karma. I read that as Chris being punished, for making a statement that feels like exactly correct to the part of my that feels upset.
Now it may be the case—a more agreeable part of me wants to interject—that this ritual actually is important, and that it should be treated as more than just a game.
But.
If so, I have never seen a particularly strong case being made for it. (Nor even the intention being communicated clearly.) It feels like it was just arbitrarily decided, then imposed on the site from a position of authority, using that position of authority to suppress dissent and strongarm people into acting in accordance with the ritual. There’s a sense of “maybe this is important and maybe not, but you don’t get to just declare it important and require people to act accordingly”.
And that’s why that part of me feels upset about this.
Yeah, I think this is a reasonable reaction, and I really appreciate you going in-depth on your reaction here. And maybe the right call is to basically not have any shared rituals and traditions that have a shared sense of importance, which feels to me like the western secular default.
But, I don’t know, that does leave me feeling pretty empty and sad, and I notice that if I don’t have an active and strong culture around me, that I just default to whatever other random culture around me does have any content, even if I don’t really like the ideas of that culture, and it feels like a pretty major loss to me. I do think that culture is really important, and shared rituals and traditions and games like these feel like how you actually build a culture that has any substantial content. And I really like Petrov Day. I consider it and Solstice to be the two primary holidays we have that we get to shape to reinforce our shared values and ideals, and want us to make use of them.
Like, I do think it’s important that we try our best to only send the invites to people who are up for taking this seriously, and it should be easy to opt-out. I think we should improve the communication to make it clearer that “yes, I do think this is a serious challenge. It’s easy for you to opt out by just ignoring this message, but please don’t destroy what I care about because you just want to have some fun. Building trust that I can have things I care about without other people destroying them just for fun, or because they dislike me or want to actively hurt me, is like half of the point of the whole exercise”.
Like, I don’t want this to be a somber serious thing. I think traditions need some fun to stay alive and not be super draining for everyone. But I also don’t want it to just be a thing that people don’t take seriously at all. Getting the communication for that right is difficult. Like, I want it to be serious play. A bunch of people who do actually try to do the right thing and win, but with an understanding that yes, this isn’t fully the real thing, this is practice, and the stakes are lower, but also not nothing, because we are doing this because we care about the bigger thing. It feels to me that most competitive sports get a substantial part of this tone right, and wargaming also has some of this feeling.
Like, I was really proud when last year we successfully didn’t blow up the site. I was proud when people considered blowing it up, and then people talked about it, and made arguments, and overall the, what seemed to me, right considerations prevailed. I think it was a pretty real and substantial achievement, and it gave me a sense of trust that everyone we sent the codes to shared at least some values and virtues with me, or at the very least wasn’t trying to actively hurt me. That trust was worth a lot. I would likely give you $5k of my own money for that trust.
And like, maybe other people don’t care much about that trust, but overall, I feel like I am in a world where trust is very scarce. Most people don’t take anything seriously and live life in what feels to me like a postmodernist careless perspective on the world, or alternatively are mindkilled and overwhelmed with values I don’t share and feel scary to me. And I when I think of a world where we successfully have 500 people over the course of 10 years, who every year successfully not blow up the site, and use it as an opportunity to remember the fragility of civilization, then that makes me pretty happy, and makes me more excited to invest in LessWrong and all the associated communities.
And like, I don’t know, what is authority for if not for coordinating exactly this kind of cultural institution? Of course, we should lose some of that authority as we do things that people don’t like, and if we repeatedly do stuff that lots of people don’t like, the authority we have should be taken away from us. But, I think, overall, my sense is that the community is happier if we use our authority to sometimes try to establish traditions like this. It is of course important that what we do isn’t a catch-22, nobody wants that. But I don’t think what happened here is a catch-22. Some of what happened here is a misunderstanding, and some of what happened here is simply the cost that you sometimes pay if you try to send a costly signal, but can’t quite get it right.
Like, I do think we should have done better at communicating the degree of seriousness of this thing. I have some ideas of how we can do that better in the future. I also sure really think Chris should have done a better job at reading the intention that was behind the email, and I did lose some trust in him and others that they will take things I care about seriously, and that they won’t just fall prey to relatively low-effort phishing attacks when it actually matters that they don’t.
Hey, I’m relieved and grateful that you took this so well. :) I hesitated for a while before posting my comment. I get that this ritual was important for you and didn’t want to disrespect that; probably also didn’t speak up last year because I wasn’t sure I could communicate it in a good way.
I totally get the desire for rituals, and think it’s an important one; I haven’t been to a Solstice but I appreciate what they’re doing. I also don’t have a problem with them, maybe because they don’t feel like they are trying to claim anything that they’re not.
Generally most of my problem with this ritual was a) some aspects of its execution, such as the communication, which is fixable, and b) the feeling that it’s actually not very analogous to the dilemma it’s trying to be symbolic for, and which it claims to be training people in. (I said a few words about that in the final paragraph of my response to lionhearted.) I think that if it really felt to me like it was teaching people to be more trustworthy and coordinate better in situations-like-the-one-Petrov faced, then I’d probably be very happy to have it around. I just don’t feel like it’s there… yet. :)
Yeah, I do think we are definitely still figuring out a bunch of rough edges around this tradition.
I do think that what we set up is reasonably analogous of what I think happened to Petrov, and the aims I have for it, though of course I would think that, but here is maybe one more attempt at synthesizing what values I care about reinforcing with Petrov Day:
Remembering that humanity is fragile, and that we have come close to destruction in the past, and will likely come close to destruction in the future.
This feels like it really straightforwardly resonates with the setup. You need to have some chance that things will go wrong to create a real sense of tension, but of course you don’t actually want to have stakes that are so high that they destroy lots of value if you mess it up, in particular while you are still dealing with a lot of uncertainty about the setup.
Practicing the virtue of not taking unilateral action and being mindful that your actions can have large negative consequences, and that you will act responsibly with the power you are given
I think this is just a really valuable virtue for a community of people to have, and I do think it’s what distinguished Petrov from most people in similar reference classes to him. Like, as I said in the other comment, he could have just been a bureaucrat who didn’t care about his job, didn’t pay much attention to what was happening, and just accidentally contributed to destroying the world. And similarly, I want the people around me to take responsibility for their actions, in particular if they have a large potential downside.
I think of these two as the core virtues of Petrov day, and I think our current ritual does a pretty decent job at reinforcing them. Like, last year when the LessWrong frontpage had a chance to go down any minute, it really felt very analogous to how I imagine people must have felt like during the cold war, where a catastrophe could have happened any minute, and suddenly destroy something I really care about. This year, the way it failed did actually really make me think that there were probably at least a couple of nearby counterfactual worlds where the cold war happened because some opsec protocols weren’t good enough, and some third party in the cold war just did some social engineering to cause a major war between the U.S. and Russia, hoping that they would end up on top in the resulting chaos.
On a meta level, I think next year’s email and post should link pretty prominently to a post just titled “Why LessWrong Petrov Day?” that explains the reasoning here pretty straightforwardly.
Remembering that humanity is fragile, and that we have come close to destruction in the past, and will likely come close to destruction in the future.
This sounds like it’s pretty well captured by current Petrov Day ritual, yeah, though I feel like it only being the front page rather than all of LW makes it feel much less serious.
Practicing the virtue of not taking unilateral action
Doesn’t Petrov’s choice actually get closer to taking than not taking unilateral action, though? The current ritual captures “think carefully about your actions”, yes, but as I understand it Petrov was supposed to report a missile launch to his superiors. who could in principle also have used their judgment to dismiss it as a false alarm.
He did the right choice, no doubt, but it feels weird to use “saw an event that could have led to the end of the world, made a choice that involved going against his standing orders and the previous planning that many others had participated in, ultimately making the decision purely himself rather than communicating it to the people with the pre-designated authority to deal with it” as a symbol of coordination and avoiding unilateral action.
Sorry, after thinking about this, I basically think that “unilateral action” is just a confusing choice of words. Let’s replace it with “being given substantial purely destructive power, and wielding that power responsibly,”, because I think while there was a substantial unilateral component to the cold war, I don’t think Petrov’s choice in particular was that reliant on unilateral considerations.
This sounds like it’s pretty well captured by current Petrov Day ritual, yeah, though I feel like it only being the front page rather than all of LW makes it feel much less serious.
I mean, I think taking all of LessWrong down would be a bit of a dick move. Like, the frontpage is what matters most to the people who participate and is a resource that feels fair and reasonably to destroy, because it being down mostly just costs the people who participate in the ritual.
But I feel like as a developer on LessWrong I have a pretty serious responsibility to be a good shepherd of content, and to make sure that you can reliably link to LessWrong content, and that you can reliably read the sequences, without it breaking. Most of the people who read that content aren’t regular users, they are people who got linked here from some other blogpost on the internet, and I don’t want to externalize our bad decisions into giving them a bad experience.
I think in general, I wouldn’t want to run rituals like this that randomly damage some public infrastructure. Like, I wouldn’t want to make it so that when someone presses a button, we barricade a random road in Berkeley. Making all the content on LessWrong inaccessible feels similar to that. It’s not my right to remove people’s access to that.
I don’t have a strong opinion on how serious Petrov Day should be. Just that if you wanted it to be taken more seriously then it should have been set up differently.
Great comment, naturally. I appreciate your epistemic status quite a bit.
I think I want to respond to the idea that it’s contradictory and bad to signal that the Petrov Day button is serious and signal that it’s fun. A few examples:
HPMOR is a book about growing up, failure, and death. It’s also hilarious and riveting.
Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It’s also a cute and colorful cartoon.
Previously warring countries often come together and have their sports teams play. It really matters that they don’t cheat and play honorably, even if it’s “fun” and “play”. It’s a game, but it’s not “just a game”.
Some animals sheath their claws for dominance fights, where the losing player loses real status but isn’t physically harmed. Again, it’s a game, and it’s in some ways play. And it’s also serious.
I’m here trying to build a community around the art of rationality. We also do an April Fools’ joke every year, like that time we made everyone’s font size proportional to their total karma for a day.
People have birthday parties, and their friends show up. And it means something for your friends to show up. It’s a party, and it’s also a true signal of friendship and being there for your friend, and you can be disappointed in them not showing up, even if it’s ‘fun’ and ‘just a party’.
I don’t think it’s contradictory to care about something deeply and to be playful with it. As with Feynman and physics, Eliezer and HPMOR, Scott and Unsong, Abram/Scott and Embedded Agency; also, LessWrong and Petrov Day.
Relatedly, (part of) you said that sometimes I signaled that this was serious, and sometimes I signaled that it was ‘just a game’. I think this is incorrect. I signaled it was playful, but never unserious, never ‘just’ or ‘merely’ a game. Everything I wrote was about trust, honor, extinction, gratitude, mourning, and being Beyond the Reach of God. I didn’t write anything that suggested you should consider pressing the button, or was secretly winking at the audience. Taking down the site is symbolic, but that doesn’t mean it’s ironic or a joke, they’re totally different. We’re symbolizing world-ending destructive technology, with a much lesser but still destructive technology. I care about this tradition a lot, and it was part of what was involved and in everything I wrote.
I feel like it takes a very cynical prior to read at everything I wrote, consider that I actually cared, then go “Yeah, he probably doesn’t mean this, why would someone actually care about this, he’s probably joking?” I don’t think anyone has that prior… I think they more have a prior that people rarely actually care about things, and so when they look at something that was meant straightforwardly and un-ironically, the hypothesis isn’t even brought up to attention.
But a lot of people got it. Most, I think. I got a bunch of short response messages to getting the codes (and some long ones) saying things like “I’m honored to be entrusted with the launch codes.” and “Roger that, general. I won’t let you down!” and “I’m honored that you gave me the opportunity” and “Awesome, thanks! I love the warm glow of not burning the commons.” I love getting these messages. And I love that it worked out last year.
It’s a distinct argument to say that I was unsuccessful at assuring that was communicated before someone took action, which this is some evidence of. Like, last year someone visited the site, pressed the button, and entered a string of zeros then hit ‘submit’, before finding out what the button even did, and without reading the announcement post. (We responded to that this year by having two sentences right underneath the button saying what was happening.)
I take seriously the charge that users like Chris would’ve gotten it if I had re-written the email and post in some ways, and I will definitely user-test it more next year. You say it’d be good to write down the case for the tradition; I can also do that, write a post called “Why the Petrov Day Big Red Button?”, and link to it from everywhere next year.
But… well, there’s more to say, but I have to go for now. I’ll add that it was my responsibility to pick the people and write the announcements. I entrusted 125 people with codes last year, and succeeded. I tried to do 270 this year, and I failed. I’m writing a postmortem, and I will work hard to ensure the site doesn’t go down next year.
We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
You’ve all been given the opportunity to show yourselves capable and trustworthy.
in 2019 was changed in 2020 to:
You’ve all been given the opportunity to not destroy LessWrong.
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
feedback: it was viscerally annoying not having access to the site today. That’s why I checked in the wee hours of the morning to see if it was back up, and that’s when I figured out what had happened today
I could feel the sense of “something valuable was destroyed” this year. That was unexpected and I think it’ll make future years feel more real to me even if we succeed
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
Hey—to preface—obviously I’m a great admirer of yours Kaj and I’ve been grateful to learn a lot from you, particularly in some of the exceptional research papers you’ve shared with me.
With that said, of course your emotions are your own but in terms of group ethics and standards, I’m very much in disagreement.
The upset feels similar to what I’ve previously experienced when something that’s obviously a purely symbolic gesture is treated as a Big Important Thing That’s Actually Making A Difference.
On the one hand, you’re totally right. On the other hand, basically the entire world is made up of abstractions along these lines. What can the Supreme Court opinion in Marbury vs Madison be recognized as other than a purely symbolic gesture? Madison wasn’t going to deliver the commissions, Justice Marshall (no relation) knew that for sure, and he made a largely symbolic gesture in how he navigated the thing. It had no practical importance for a long time but now forms one of the foundations of American jurisprudence effecting, indirectly, billions of lives. But at the time, if you dig into the history, it really was largely symbolic at the time.
The world is built out of all sorts of abstract symbolism and intersubjective convention.
That by itself wouldn’t trigger the reaction; the world is full of purely symbolic gestures that are claiming to make a difference, but they mostly haven’t upset me in a long time. Some of the communication around Petrov Day has. I think it’s because of a sense that this idea is being pushed on people-that-I-care-about as something important despite not actually being in accordance to their values, and that there’s social pressure for people to be quiet about it and give in to the social pressure at a cost to their epistemics.
(“Canonical” was intentionally chosen, incidentally.)
I feel like Oliver’s comment is basically saying “people should have taken this seriously and people who treat this light-heartedly are in the wrong”. It’s spoken from a position of authority, and feels like it’s shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant, as no compelling reason for this ritual actually being significant has ever been presented.
In any case the light didn’t go on in my head about egalitarian instincts (instincts to prevent leaders from exercising power) killing online communities until just recently. [...] I have seen rationalist communities die because they trusted their moderators too little.
Honestly, for anything that wasn’t clearly egregiously wrong, I’d support the leadership team on here even if my feelings ran in a different direction. Like, leadership is hard. Really really really hard. If there was something I didn’t believe in, I’d just quietly opt out.
Now, I fully understand I’m in the minority on this position — but I’m against both ‘every interpretation is valid’ type thinking (why would every interpretation be valid as it relates to a group activity where your behavior effects the whole group?).
Likewise, pushing back against “shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant” — isn’t that actually both good and necessary if we want to be able to coordinate and actually solve problems?
There’s a dozen or so Yudkowsky citations about this. Here’s another:
Let’s say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy.
Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?
In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.
And finally,
Now it may be the case—a more agreeable part of me wants to interject—that this ritual actually is important, and that it should be treated as more than just a game.
But.
If so, I have never seen a particularly strong case being made for it.
I even did, like, math and stuff. The “shut up and multiply” thing.
Long story short — I think shared trust and demonstrated cooperation are super valuable, good leadership is incredibly underappreciated, and whimsical defection is really bad.
Again though — all written respectfully, etc etc, and I know I’m in the minority position here in terms of many subjective personal values, especially harm/care and seriousness/fun.
Finally, it’s undoubtedly true my estimate of the potential utility of building out a base of successfully navigated low-stakes cooperative endeavors is undoubtedly multiple orders of magnitude higher than others. I put the dollar-value of that as, actually, pretty high. Reasonable minds can differ on many of these points, but that’s my logic.
Thanks for engaging :) My upset part feels much calmer now that it has been spoken for, so I’m actually pretty chill about this right now. You’ve had a lot of stuff that I’ve gotten value from, too.
But note also that that post contains a lengthy excerpt about how the “Dark Side” descends into cultishness and insanity in situations where the word of leaders is accepted without question. That was clearly also depicted as the opposite failure mode.
I agree that rationalists don’t cooperate enough, and that often just offer criticism when it’s not warranted. But… it feels like a Fully General Counterargument if you take to that the point of “no coordination may be criticized, ever, including situation where people are arguably being shamed for having good epistemics”. That sounds like this bit from the post:
How do things work on the Dark Side?
The respected leader speaks, and there comes a chorus of pure agreement: if there are any who harbor inward doubts, they keep them to themselves. So all the individual members of the audience see this atmosphere of pure agreement, and they feel more confident in the ideas presented—even if they, personally, harbored inward doubts, why, everyone else seems to agree with it.
If anyone is still unpersuaded after that, they leave the group (or in some places, are executed)—and the remainder are more in agreement, and reinforce each other with less interference.
Re: Well-Kept Gardens—again, that feels like a Fully General Counterargument. Yes, certainly there should be moderator action when necessary… like, I am on the mod team, I have seen discussions about banning users etc. and all of that’s been fine to me.
But we’re not even talking about banning users here. This isn’t about keeping the garden, it’s about one particular ritual being picked to be important. Does Well-Kept Gardens imply that everything the admins do should be treated as correct?
Likewise, pushing back against “shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant” — isn’t that actually both good and necessary if we want to be able to coordinate and actually solve problems?
Back when I banned Eugine Nier, there were people who disagreed with that decision… but if they didn’t find my argument for the ban to be particularly persuasive, it never even crossed my mind that I should shame them because I had made an argument that didn’t compel them. They disagreed, the coordination necessary for saving the site still happened, no reason not to let them disagree.
So I read this to be a central premise that much of the rest of your comment builds on:
For Ben at least, the button thing was a symbolic exercise analogous to not nuking another country and he specifically asked you not to and said he’s trusting you.
Which to me feels like it’s assuming thing that I was asking someone to prove. Yes, Ben feels that the button was symbolic and analogous to not nuking another country. But it does not feel at all analogous to me; I feel like the ritual is picking a surface-level aspect of what happened to Petrov (something like the general idea of “you should not push the red button, doing so would have bad consequences”), and that’s about it when it comes to having a resemblance to it. The outcome matrix and psychological situation for the users is something completely different than what it was for Petrov. If the ritual actually did have a clear structural resemblance to Petrov’s dilemma, then I would have much less of a problem with it. But as it is, it does not feel like it’s training the site’s users in the thing that it claims to be training them in.
I’ll review and think more carefully later — out at dinner with a friend now — but my quick thought is that the proper venue, time, and place for expressing discontent with a cooperative community project is probably afterwards, possibly beforehand, and certainly not during… I don’t believe in immunity from criticism, obviously, but I am against defection when one doesn’t agree with a choice of norms.
That’s the quick take, will review more closely later.
[Epistemic-emotional status: reporting an emotional reaction, which may or may not have correct reasoning; attempting to honor the emotional tone of that reaction without actually wanting to be confrontational about this. Important note: when I say that it feels like people are being shamed, I do not mean think that anyone actually intends to shame people; it’s that I feel that that’s what the tone will communicate to many, intentionally or not. I could be wrong about that too. Again, trying to fairly report the implicit model that one part of me has, without asserting that the model is necessarily fully correct.]
So I notice that there’s a part of me which gets annoyed, borderline upset when I hear this Petrov Day ritual discussed in this manner.
The upset feels similar to what I’ve previously experienced when something that’s obviously a purely symbolic gesture is treated as a Big Important Thing That’s Actually Making A Difference. As it feels like the symbolic thing is distracting from things that actually matter.
That by itself wouldn’t trigger the reaction; the world is full of purely symbolic gestures that are claiming to make a difference, but they mostly haven’t upset me in a long time. Some of the communication around Petrov Day has. I think it’s because of a sense that this idea is being pushed on people-that-I-care-about as something important despite not actually being in accordance to their values, and that there’s social pressure for people to be quiet about it and give in to the social pressure at a cost to their epistemics.
I feel like Oliver’s comment is basically saying “people should have taken this seriously and people who treat this light-heartedly are in the wrong”. It’s spoken from a position of authority, and feels like it’s shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant, as no compelling reason for this ritual actually being significant has ever been presented.
As Chris notes, the communication around this has been contradictory, with some of it being presented as “this is just a game is good fun” and some “this is actually a thing to be taken seriously”. That also feels like a catch-22, where it’s implied that it’s shameworthy to not have taken the game seriously, when people were simultaneously also given the signal that it’s not serious.
I think that to a large extent, I stopped caring about meaningless rituals being pushed within my ingroup because I stopped considering my ingroup to include the kinds of groups that were pushing the rituals. But Chris feels like part of my ingroup, and now I feel upset again because I feel he’s being punished unfairly.
Chris wrote a comment saying this
Before I strong-upvoted it, the comment was down at −11 karma. I read that as Chris being punished, for making a statement that feels like exactly correct to the part of my that feels upset.
Now it may be the case—a more agreeable part of me wants to interject—that this ritual actually is important, and that it should be treated as more than just a game.
But.
If so, I have never seen a particularly strong case being made for it. (Nor even the intention being communicated clearly.) It feels like it was just arbitrarily decided, then imposed on the site from a position of authority, using that position of authority to suppress dissent and strongarm people into acting in accordance with the ritual. There’s a sense of “maybe this is important and maybe not, but you don’t get to just declare it important and require people to act accordingly”.
And that’s why that part of me feels upset about this.
Yeah, I think this is a reasonable reaction, and I really appreciate you going in-depth on your reaction here. And maybe the right call is to basically not have any shared rituals and traditions that have a shared sense of importance, which feels to me like the western secular default.
But, I don’t know, that does leave me feeling pretty empty and sad, and I notice that if I don’t have an active and strong culture around me, that I just default to whatever other random culture around me does have any content, even if I don’t really like the ideas of that culture, and it feels like a pretty major loss to me. I do think that culture is really important, and shared rituals and traditions and games like these feel like how you actually build a culture that has any substantial content. And I really like Petrov Day. I consider it and Solstice to be the two primary holidays we have that we get to shape to reinforce our shared values and ideals, and want us to make use of them.
Like, I do think it’s important that we try our best to only send the invites to people who are up for taking this seriously, and it should be easy to opt-out. I think we should improve the communication to make it clearer that “yes, I do think this is a serious challenge. It’s easy for you to opt out by just ignoring this message, but please don’t destroy what I care about because you just want to have some fun. Building trust that I can have things I care about without other people destroying them just for fun, or because they dislike me or want to actively hurt me, is like half of the point of the whole exercise”.
Like, I don’t want this to be a somber serious thing. I think traditions need some fun to stay alive and not be super draining for everyone. But I also don’t want it to just be a thing that people don’t take seriously at all. Getting the communication for that right is difficult. Like, I want it to be serious play. A bunch of people who do actually try to do the right thing and win, but with an understanding that yes, this isn’t fully the real thing, this is practice, and the stakes are lower, but also not nothing, because we are doing this because we care about the bigger thing. It feels to me that most competitive sports get a substantial part of this tone right, and wargaming also has some of this feeling.
Like, I was really proud when last year we successfully didn’t blow up the site. I was proud when people considered blowing it up, and then people talked about it, and made arguments, and overall the, what seemed to me, right considerations prevailed. I think it was a pretty real and substantial achievement, and it gave me a sense of trust that everyone we sent the codes to shared at least some values and virtues with me, or at the very least wasn’t trying to actively hurt me. That trust was worth a lot. I would likely give you $5k of my own money for that trust.
And like, maybe other people don’t care much about that trust, but overall, I feel like I am in a world where trust is very scarce. Most people don’t take anything seriously and live life in what feels to me like a postmodernist careless perspective on the world, or alternatively are mindkilled and overwhelmed with values I don’t share and feel scary to me. And I when I think of a world where we successfully have 500 people over the course of 10 years, who every year successfully not blow up the site, and use it as an opportunity to remember the fragility of civilization, then that makes me pretty happy, and makes me more excited to invest in LessWrong and all the associated communities.
And like, I don’t know, what is authority for if not for coordinating exactly this kind of cultural institution? Of course, we should lose some of that authority as we do things that people don’t like, and if we repeatedly do stuff that lots of people don’t like, the authority we have should be taken away from us. But, I think, overall, my sense is that the community is happier if we use our authority to sometimes try to establish traditions like this. It is of course important that what we do isn’t a catch-22, nobody wants that. But I don’t think what happened here is a catch-22. Some of what happened here is a misunderstanding, and some of what happened here is simply the cost that you sometimes pay if you try to send a costly signal, but can’t quite get it right.
Like, I do think we should have done better at communicating the degree of seriousness of this thing. I have some ideas of how we can do that better in the future. I also sure really think Chris should have done a better job at reading the intention that was behind the email, and I did lose some trust in him and others that they will take things I care about seriously, and that they won’t just fall prey to relatively low-effort phishing attacks when it actually matters that they don’t.
Hey, I’m relieved and grateful that you took this so well. :) I hesitated for a while before posting my comment. I get that this ritual was important for you and didn’t want to disrespect that; probably also didn’t speak up last year because I wasn’t sure I could communicate it in a good way.
I totally get the desire for rituals, and think it’s an important one; I haven’t been to a Solstice but I appreciate what they’re doing. I also don’t have a problem with them, maybe because they don’t feel like they are trying to claim anything that they’re not.
Generally most of my problem with this ritual was a) some aspects of its execution, such as the communication, which is fixable, and b) the feeling that it’s actually not very analogous to the dilemma it’s trying to be symbolic for, and which it claims to be training people in. (I said a few words about that in the final paragraph of my response to lionhearted.) I think that if it really felt to me like it was teaching people to be more trustworthy and coordinate better in situations-like-the-one-Petrov faced, then I’d probably be very happy to have it around. I just don’t feel like it’s there… yet. :)
Yeah, I do think we are definitely still figuring out a bunch of rough edges around this tradition.
I do think that what we set up is reasonably analogous of what I think happened to Petrov, and the aims I have for it, though of course I would think that, but here is maybe one more attempt at synthesizing what values I care about reinforcing with Petrov Day:
Remembering that humanity is fragile, and that we have come close to destruction in the past, and will likely come close to destruction in the future.
This feels like it really straightforwardly resonates with the setup. You need to have some chance that things will go wrong to create a real sense of tension, but of course you don’t actually want to have stakes that are so high that they destroy lots of value if you mess it up, in particular while you are still dealing with a lot of uncertainty about the setup.
Practicing the virtue of not taking unilateral action and being mindful that your actions can have large negative consequences, and that you will act responsibly with the power you are given
I think this is just a really valuable virtue for a community of people to have, and I do think it’s what distinguished Petrov from most people in similar reference classes to him. Like, as I said in the other comment, he could have just been a bureaucrat who didn’t care about his job, didn’t pay much attention to what was happening, and just accidentally contributed to destroying the world. And similarly, I want the people around me to take responsibility for their actions, in particular if they have a large potential downside.
I think of these two as the core virtues of Petrov day, and I think our current ritual does a pretty decent job at reinforcing them. Like, last year when the LessWrong frontpage had a chance to go down any minute, it really felt very analogous to how I imagine people must have felt like during the cold war, where a catastrophe could have happened any minute, and suddenly destroy something I really care about. This year, the way it failed did actually really make me think that there were probably at least a couple of nearby counterfactual worlds where the cold war happened because some opsec protocols weren’t good enough, and some third party in the cold war just did some social engineering to cause a major war between the U.S. and Russia, hoping that they would end up on top in the resulting chaos.
On a meta level, I think next year’s email and post should link pretty prominently to a post just titled “Why LessWrong Petrov Day?” that explains the reasoning here pretty straightforwardly.
This sounds like it’s pretty well captured by current Petrov Day ritual, yeah, though I feel like it only being the front page rather than all of LW makes it feel much less serious.
Doesn’t Petrov’s choice actually get closer to taking than not taking unilateral action, though? The current ritual captures “think carefully about your actions”, yes, but as I understand it Petrov was supposed to report a missile launch to his superiors. who could in principle also have used their judgment to dismiss it as a false alarm.
He did the right choice, no doubt, but it feels weird to use “saw an event that could have led to the end of the world, made a choice that involved going against his standing orders and the previous planning that many others had participated in, ultimately making the decision purely himself rather than communicating it to the people with the pre-designated authority to deal with it” as a symbol of coordination and avoiding unilateral action.
Sorry, after thinking about this, I basically think that “unilateral action” is just a confusing choice of words. Let’s replace it with “being given substantial purely destructive power, and wielding that power responsibly,”, because I think while there was a substantial unilateral component to the cold war, I don’t think Petrov’s choice in particular was that reliant on unilateral considerations.
I mean, I think taking all of LessWrong down would be a bit of a dick move. Like, the frontpage is what matters most to the people who participate and is a resource that feels fair and reasonably to destroy, because it being down mostly just costs the people who participate in the ritual.
But I feel like as a developer on LessWrong I have a pretty serious responsibility to be a good shepherd of content, and to make sure that you can reliably link to LessWrong content, and that you can reliably read the sequences, without it breaking. Most of the people who read that content aren’t regular users, they are people who got linked here from some other blogpost on the internet, and I don’t want to externalize our bad decisions into giving them a bad experience.
I think in general, I wouldn’t want to run rituals like this that randomly damage some public infrastructure. Like, I wouldn’t want to make it so that when someone presses a button, we barricade a random road in Berkeley. Making all the content on LessWrong inaccessible feels similar to that. It’s not my right to remove people’s access to that.
I don’t have a strong opinion on how serious Petrov Day should be. Just that if you wanted it to be taken more seriously then it should have been set up differently.
Great comment, naturally. I appreciate your epistemic status quite a bit.
I think I want to respond to the idea that it’s contradictory and bad to signal that the Petrov Day button is serious and signal that it’s fun. A few examples:
HPMOR is a book about growing up, failure, and death. It’s also hilarious and riveting.
Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It’s also a cute and colorful cartoon.
Previously warring countries often come together and have their sports teams play. It really matters that they don’t cheat and play honorably, even if it’s “fun” and “play”. It’s a game, but it’s not “just a game”.
Some animals sheath their claws for dominance fights, where the losing player loses real status but isn’t physically harmed. Again, it’s a game, and it’s in some ways play. And it’s also serious.
I’m here trying to build a community around the art of rationality. We also do an April Fools’ joke every year, like that time we made everyone’s font size proportional to their total karma for a day.
People have birthday parties, and their friends show up. And it means something for your friends to show up. It’s a party, and it’s also a true signal of friendship and being there for your friend, and you can be disappointed in them not showing up, even if it’s ‘fun’ and ‘just a party’.
I don’t think it’s contradictory to care about something deeply and to be playful with it. As with Feynman and physics, Eliezer and HPMOR, Scott and Unsong, Abram/Scott and Embedded Agency; also, LessWrong and Petrov Day.
Relatedly, (part of) you said that sometimes I signaled that this was serious, and sometimes I signaled that it was ‘just a game’. I think this is incorrect. I signaled it was playful, but never unserious, never ‘just’ or ‘merely’ a game. Everything I wrote was about trust, honor, extinction, gratitude, mourning, and being Beyond the Reach of God. I didn’t write anything that suggested you should consider pressing the button, or was secretly winking at the audience. Taking down the site is symbolic, but that doesn’t mean it’s ironic or a joke, they’re totally different. We’re symbolizing world-ending destructive technology, with a much lesser but still destructive technology. I care about this tradition a lot, and it was part of what was involved and in everything I wrote.
I feel like it takes a very cynical prior to read at everything I wrote, consider that I actually cared, then go “Yeah, he probably doesn’t mean this, why would someone actually care about this, he’s probably joking?” I don’t think anyone has that prior… I think they more have a prior that people rarely actually care about things, and so when they look at something that was meant straightforwardly and un-ironically, the hypothesis isn’t even brought up to attention.
But a lot of people got it. Most, I think. I got a bunch of short response messages to getting the codes (and some long ones) saying things like “I’m honored to be entrusted with the launch codes.” and “Roger that, general. I won’t let you down!” and “I’m honored that you gave me the opportunity” and “Awesome, thanks! I love the warm glow of not burning the commons.” I love getting these messages. And I love that it worked out last year.
It’s a distinct argument to say that I was unsuccessful at assuring that was communicated before someone took action, which this is some evidence of. Like, last year someone visited the site, pressed the button, and entered a string of zeros then hit ‘submit’, before finding out what the button even did, and without reading the announcement post. (We responded to that this year by having two sentences right underneath the button saying what was happening.)
I take seriously the charge that users like Chris would’ve gotten it if I had re-written the email and post in some ways, and I will definitely user-test it more next year. You say it’d be good to write down the case for the tradition; I can also do that, write a post called “Why the Petrov Day Big Red Button?”, and link to it from everywhere next year.
But… well, there’s more to say, but I have to go for now. I’ll add that it was my responsibility to pick the people and write the announcements. I entrusted 125 people with codes last year, and succeeded. I tried to do 270 this year, and I failed. I’m writing a postmortem, and I will work hard to ensure the site doesn’t go down next year.
We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
in 2019 was changed in 2020 to:
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
There’s a lot of stuff I agree with here and some stuff I’d push back on but probably worth waiting for the post-mortem before going deeper.
That’s actually a pretty big difference.
Hey—to preface—obviously I’m a great admirer of yours Kaj and I’ve been grateful to learn a lot from you, particularly in some of the exceptional research papers you’ve shared with me.
With that said, of course your emotions are your own but in terms of group ethics and standards, I’m very much in disagreement.
On the one hand, you’re totally right. On the other hand, basically the entire world is made up of abstractions along these lines. What can the Supreme Court opinion in Marbury vs Madison be recognized as other than a purely symbolic gesture? Madison wasn’t going to deliver the commissions, Justice Marshall (no relation) knew that for sure, and he made a largely symbolic gesture in how he navigated the thing. It had no practical importance for a long time but now forms one of the foundations of American jurisprudence effecting, indirectly, billions of lives. But at the time, if you dig into the history, it really was largely symbolic at the time.
The world is built out of all sorts of abstract symbolism and intersubjective convention.
Canonical reply is this one:
https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/7FzD7pNm9X68Gp5ZC
(“Canonical” was intentionally chosen, incidentally.)
https://www.lesswrong.com/posts/tscc3e5eujrsEeFN4/well-kept-gardens-die-by-pacifism
From Well-Kept Gardens:
Honestly, for anything that wasn’t clearly egregiously wrong, I’d support the leadership team on here even if my feelings ran in a different direction. Like, leadership is hard. Really really really hard. If there was something I didn’t believe in, I’d just quietly opt out.
Now, I fully understand I’m in the minority on this position — but I’m against both ‘every interpretation is valid’ type thinking (why would every interpretation be valid as it relates to a group activity where your behavior effects the whole group?).
Likewise, pushing back against “shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant” — isn’t that actually both good and necessary if we want to be able to coordinate and actually solve problems?
There’s a dozen or so Yudkowsky citations about this. Here’s another:
https://www.lesswrong.com/posts/KsHmn6iJAEr9bACQW/bayesians-vs-barbarians
And finally,
I made that case last year extensively:
https://www.lesswrong.com/posts/vvzfFcbmKgEsDBRHh/honoring-petrov-day-on-lesswrong-in-2019?commentId=ZZ87dbYiGDu6uMtF8
I even did, like, math and stuff. The “shut up and multiply” thing.
Long story short — I think shared trust and demonstrated cooperation are super valuable, good leadership is incredibly underappreciated, and whimsical defection is really bad.
Again though — all written respectfully, etc etc, and I know I’m in the minority position here in terms of many subjective personal values, especially harm/care and seriousness/fun.
Finally, it’s undoubtedly true my estimate of the potential utility of building out a base of successfully navigated low-stakes cooperative endeavors is undoubtedly multiple orders of magnitude higher than others. I put the dollar-value of that as, actually, pretty high. Reasonable minds can differ on many of these points, but that’s my logic.
Thanks for engaging :) My upset part feels much calmer now that it has been spoken for, so I’m actually pretty chill about this right now. You’ve had a lot of stuff that I’ve gotten value from, too.
But note also that that post contains a lengthy excerpt about how the “Dark Side” descends into cultishness and insanity in situations where the word of leaders is accepted without question. That was clearly also depicted as the opposite failure mode.
I agree that rationalists don’t cooperate enough, and that often just offer criticism when it’s not warranted. But… it feels like a Fully General Counterargument if you take to that the point of “no coordination may be criticized, ever, including situation where people are arguably being shamed for having good epistemics”. That sounds like this bit from the post:
Re: Well-Kept Gardens—again, that feels like a Fully General Counterargument. Yes, certainly there should be moderator action when necessary… like, I am on the mod team, I have seen discussions about banning users etc. and all of that’s been fine to me.
But we’re not even talking about banning users here. This isn’t about keeping the garden, it’s about one particular ritual being picked to be important. Does Well-Kept Gardens imply that everything the admins do should be treated as correct?
Back when I banned Eugine Nier, there were people who disagreed with that decision… but if they didn’t find my argument for the ban to be particularly persuasive, it never even crossed my mind that I should shame them because I had made an argument that didn’t compel them. They disagreed, the coordination necessary for saving the site still happened, no reason not to let them disagree.
So I read this to be a central premise that much of the rest of your comment builds on:
Which to me feels like it’s assuming thing that I was asking someone to prove. Yes, Ben feels that the button was symbolic and analogous to not nuking another country. But it does not feel at all analogous to me; I feel like the ritual is picking a surface-level aspect of what happened to Petrov (something like the general idea of “you should not push the red button, doing so would have bad consequences”), and that’s about it when it comes to having a resemblance to it. The outcome matrix and psychological situation for the users is something completely different than what it was for Petrov. If the ritual actually did have a clear structural resemblance to Petrov’s dilemma, then I would have much less of a problem with it. But as it is, it does not feel like it’s training the site’s users in the thing that it claims to be training them in.
Good points.
I’ll review and think more carefully later — out at dinner with a friend now — but my quick thought is that the proper venue, time, and place for expressing discontent with a cooperative community project is probably afterwards, possibly beforehand, and certainly not during… I don’t believe in immunity from criticism, obviously, but I am against defection when one doesn’t agree with a choice of norms.
That’s the quick take, will review more closely later.