We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
You’ve all been given the opportunity to show yourselves capable and trustworthy.
in 2019 was changed in 2020 to:
You’ve all been given the opportunity to not destroy LessWrong.
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
feedback: it was viscerally annoying not having access to the site today. That’s why I checked in the wee hours of the morning to see if it was back up, and that’s when I figured out what had happened today
I could feel the sense of “something valuable was destroyed” this year. That was unexpected and I think it’ll make future years feel more real to me even if we succeed
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
in 2019 was changed in 2020 to:
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
There’s a lot of stuff I agree with here and some stuff I’d push back on but probably worth waiting for the post-mortem before going deeper.
That’s actually a pretty big difference.