Great comment, naturally. I appreciate your epistemic status quite a bit.
I think I want to respond to the idea that it’s contradictory and bad to signal that the Petrov Day button is serious and signal that it’s fun. A few examples:
HPMOR is a book about growing up, failure, and death. It’s also hilarious and riveting.
Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It’s also a cute and colorful cartoon.
Previously warring countries often come together and have their sports teams play. It really matters that they don’t cheat and play honorably, even if it’s “fun” and “play”. It’s a game, but it’s not “just a game”.
Some animals sheath their claws for dominance fights, where the losing player loses real status but isn’t physically harmed. Again, it’s a game, and it’s in some ways play. And it’s also serious.
I’m here trying to build a community around the art of rationality. We also do an April Fools’ joke every year, like that time we made everyone’s font size proportional to their total karma for a day.
People have birthday parties, and their friends show up. And it means something for your friends to show up. It’s a party, and it’s also a true signal of friendship and being there for your friend, and you can be disappointed in them not showing up, even if it’s ‘fun’ and ‘just a party’.
I don’t think it’s contradictory to care about something deeply and to be playful with it. As with Feynman and physics, Eliezer and HPMOR, Scott and Unsong, Abram/Scott and Embedded Agency; also, LessWrong and Petrov Day.
Relatedly, (part of) you said that sometimes I signaled that this was serious, and sometimes I signaled that it was ‘just a game’. I think this is incorrect. I signaled it was playful, but never unserious, never ‘just’ or ‘merely’ a game. Everything I wrote was about trust, honor, extinction, gratitude, mourning, and being Beyond the Reach of God. I didn’t write anything that suggested you should consider pressing the button, or was secretly winking at the audience. Taking down the site is symbolic, but that doesn’t mean it’s ironic or a joke, they’re totally different. We’re symbolizing world-ending destructive technology, with a much lesser but still destructive technology. I care about this tradition a lot, and it was part of what was involved and in everything I wrote.
I feel like it takes a very cynical prior to read at everything I wrote, consider that I actually cared, then go “Yeah, he probably doesn’t mean this, why would someone actually care about this, he’s probably joking?” I don’t think anyone has that prior… I think they more have a prior that people rarely actually care about things, and so when they look at something that was meant straightforwardly and un-ironically, the hypothesis isn’t even brought up to attention.
But a lot of people got it. Most, I think. I got a bunch of short response messages to getting the codes (and some long ones) saying things like “I’m honored to be entrusted with the launch codes.” and “Roger that, general. I won’t let you down!” and “I’m honored that you gave me the opportunity” and “Awesome, thanks! I love the warm glow of not burning the commons.” I love getting these messages. And I love that it worked out last year.
It’s a distinct argument to say that I was unsuccessful at assuring that was communicated before someone took action, which this is some evidence of. Like, last year someone visited the site, pressed the button, and entered a string of zeros then hit ‘submit’, before finding out what the button even did, and without reading the announcement post. (We responded to that this year by having two sentences right underneath the button saying what was happening.)
I take seriously the charge that users like Chris would’ve gotten it if I had re-written the email and post in some ways, and I will definitely user-test it more next year. You say it’d be good to write down the case for the tradition; I can also do that, write a post called “Why the Petrov Day Big Red Button?”, and link to it from everywhere next year.
But… well, there’s more to say, but I have to go for now. I’ll add that it was my responsibility to pick the people and write the announcements. I entrusted 125 people with codes last year, and succeeded. I tried to do 270 this year, and I failed. I’m writing a postmortem, and I will work hard to ensure the site doesn’t go down next year.
We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
You’ve all been given the opportunity to show yourselves capable and trustworthy.
in 2019 was changed in 2020 to:
You’ve all been given the opportunity to not destroy LessWrong.
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
feedback: it was viscerally annoying not having access to the site today. That’s why I checked in the wee hours of the morning to see if it was back up, and that’s when I figured out what had happened today
I could feel the sense of “something valuable was destroyed” this year. That was unexpected and I think it’ll make future years feel more real to me even if we succeed
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
Great comment, naturally. I appreciate your epistemic status quite a bit.
I think I want to respond to the idea that it’s contradictory and bad to signal that the Petrov Day button is serious and signal that it’s fun. A few examples:
HPMOR is a book about growing up, failure, and death. It’s also hilarious and riveting.
Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It’s also a cute and colorful cartoon.
Previously warring countries often come together and have their sports teams play. It really matters that they don’t cheat and play honorably, even if it’s “fun” and “play”. It’s a game, but it’s not “just a game”.
Some animals sheath their claws for dominance fights, where the losing player loses real status but isn’t physically harmed. Again, it’s a game, and it’s in some ways play. And it’s also serious.
I’m here trying to build a community around the art of rationality. We also do an April Fools’ joke every year, like that time we made everyone’s font size proportional to their total karma for a day.
People have birthday parties, and their friends show up. And it means something for your friends to show up. It’s a party, and it’s also a true signal of friendship and being there for your friend, and you can be disappointed in them not showing up, even if it’s ‘fun’ and ‘just a party’.
I don’t think it’s contradictory to care about something deeply and to be playful with it. As with Feynman and physics, Eliezer and HPMOR, Scott and Unsong, Abram/Scott and Embedded Agency; also, LessWrong and Petrov Day.
Relatedly, (part of) you said that sometimes I signaled that this was serious, and sometimes I signaled that it was ‘just a game’. I think this is incorrect. I signaled it was playful, but never unserious, never ‘just’ or ‘merely’ a game. Everything I wrote was about trust, honor, extinction, gratitude, mourning, and being Beyond the Reach of God. I didn’t write anything that suggested you should consider pressing the button, or was secretly winking at the audience. Taking down the site is symbolic, but that doesn’t mean it’s ironic or a joke, they’re totally different. We’re symbolizing world-ending destructive technology, with a much lesser but still destructive technology. I care about this tradition a lot, and it was part of what was involved and in everything I wrote.
I feel like it takes a very cynical prior to read at everything I wrote, consider that I actually cared, then go “Yeah, he probably doesn’t mean this, why would someone actually care about this, he’s probably joking?” I don’t think anyone has that prior… I think they more have a prior that people rarely actually care about things, and so when they look at something that was meant straightforwardly and un-ironically, the hypothesis isn’t even brought up to attention.
But a lot of people got it. Most, I think. I got a bunch of short response messages to getting the codes (and some long ones) saying things like “I’m honored to be entrusted with the launch codes.” and “Roger that, general. I won’t let you down!” and “I’m honored that you gave me the opportunity” and “Awesome, thanks! I love the warm glow of not burning the commons.” I love getting these messages. And I love that it worked out last year.
It’s a distinct argument to say that I was unsuccessful at assuring that was communicated before someone took action, which this is some evidence of. Like, last year someone visited the site, pressed the button, and entered a string of zeros then hit ‘submit’, before finding out what the button even did, and without reading the announcement post. (We responded to that this year by having two sentences right underneath the button saying what was happening.)
I take seriously the charge that users like Chris would’ve gotten it if I had re-written the email and post in some ways, and I will definitely user-test it more next year. You say it’d be good to write down the case for the tradition; I can also do that, write a post called “Why the Petrov Day Big Red Button?”, and link to it from everywhere next year.
But… well, there’s more to say, but I have to go for now. I’ll add that it was my responsibility to pick the people and write the announcements. I entrusted 125 people with codes last year, and succeeded. I tried to do 270 this year, and I failed. I’m writing a postmortem, and I will work hard to ensure the site doesn’t go down next year.
We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
in 2019 was changed in 2020 to:
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
There’s a lot of stuff I agree with here and some stuff I’d push back on but probably worth waiting for the post-mortem before going deeper.
That’s actually a pretty big difference.