I deliberately refrained from mentioning this in public yesterday out of respect for the spirit of the game, but I was disappointed that the SHA-256 hashes of the launch codes were publicly visible in the source code (such that someone could try to crack them offline) rather than being stored in the database (such that all guesses have to go online through the server).
Okay, yes, I understand that this wasn’t a “serious” vulnerability because the launch codes were sufficiently high-entropy (actually random, not chosen by a human) that no one is going to be able to crack the hash. (I didn’t make the list this year, but my launch code from last year was 16 alphanumeric characters, which is 6216≈95.2 bits of entropy, which I think is 6216/1011/(86400⋅365.25)/2 being an expectation of 7 billion years at 100 billion hashes per second? Oh, except knock off a couple orders of magnitude, because there were a hundred codes.)
Still, in principle, as a matter of security mindset, putting the hashes in the database lets you depend on one fewer assumption (about how much computing power the adversary has). It seems like the sort of thing to do as part of the spirit of the game, given that this kind of thing is how the world is actually going to end. (Someone works out that this-and-such AI design will perform safely as long as such-and-these assumptions are true, and then one of the assumptions turns out not to be true.)
I made this decision a while ago, mostly because it was something like an hour faster to get things going this way. I did the fermi, did some research into SHA-256 to make sure it was sufficiently pre-image resistant, and decided to go ahead. I stand by this decision, and think your comment is a straightforward misapplication of security mindset.
I think going through the motions of making sure the hashes weren’t publicly available would have been just virtue signaling, and the real security risks almost certainly live somewhere else in our stack (some out-of-date-library that has a security vulnerability, some OS-level issue, some configuration issue in our VPN, a social engineering attack on someone in our team or a past collaborator). There is no point in pursuing a security mindset if you are virtually certain that the thing you would be investing resources into would not be your weakest attack point. I know that LessWrong is not robust against a sophisticated dedicated attacker. I think it would be dumb of me to look at an insulated part of our stack and harden that to withstand a sophisticated attacker, when there are many other attack vectors that are much more fragile.
I sort of disagree. Not necessarily that it was the wrong choice to invest your security resources elsewhere—I think your threat model is approximately correct—but I disagree that it’s wrong to invest in that part of your stack.
My argument here is that following best practices is a good principle, and that you can and should make exceptions sometimes, but Zack is right to point it out as a vulnerability. Security best practices exist to help you reduce attack surface without having to be aware of every attack vector. You might look at this instance and rightly think “okay but SHA-256 is very hard to crack with keys this long, and there is more low hanging fruit”. But sometimes you’re going to make a wrong assumption when evaluating things like this, and best practices help protect you from limitations of your ability to model this. Maybe your SHA-256 implementation has a known vulnerability that you didn’t check, maybe your secret generation wasn’t actually random, etc. I don’t think any of these apply in this case, but I think sometimes you’re likely to be mistaken about your assumptions. The question becomes a more general one about when it makes sense to follow best practices and when it makes sense to make exceptions. In this case I would think an hour of work would be worth it to follow the best practice. If it were more like five hours, I might agree with with you.
I wouldn’t discount the signaling value either. Allies and attackers can notice whether you’re following best practices and use that info to decide whether they should invest resources in trusting your infrastructure or attacking it.
I agree that this is a correct application of security mindset; exposures like these can compound with, for example, someone’s automatic search of the 100 most common ways to screw up secure random number generation such as by using the current time as a seed. Deep security is about reducing the amount of thinking you have to do and your exposure to wrong models and stuff you didn’t think of.
Furthermore, It is also not inconceivable to me that an adversary might be able to use the hash itself without cracking it. For example, the sha256 hash of some information is commonly used to prove that someone has that information without revealing it, so an adversary, using the hash, could credibly lie that he already possesses a launch code and in a possible counterfactual world where no one found about the client side leaking the hash except this adversary, use this lie to acquire an actual code with some social engineering.
Like:
“Attention Lesswrong! With trickery I have acquired a launch code capable of destroying your site. As proof here is the sha256 hash of it: <hash>.
This is not a trick, I will leave plenty of time for you to check with your EA buddies that the hash is valid before you need to meet my demands.
I demand a launch code capable of destroying the EA forum sent to me until <time> or I will nuke this site and to this I precommitted. I won’t reveal what I plan to do with the launch code you will send to me, but by basic game theory your interest is in sending it to me as your site’s destruction is not certain that way.
I can’t prove it to you, but irl I precommitted to nuking the site if my demands are not met and also that I won’t send any more messages to prevent useless debating.
I sort of disagree. Not necessarily that it was the wrong choice to invest your security resources elsewhere—I think your threat model is approximately correct—but I disagree that it’s wrong to invest in that part of your stack.
I do not know what the difference here is. Presumably one implies the other?
Tale: Today it may be best, to invest elsewhere. But not forever. Security against a dedicated adversary is not always and forever impossible, but a result of continuous investment that is a) eventually achieved, b) requires continuing work going forward (bugs are discovered in resources once thought secure and need patching, malware and threats change, etc.)
In other words, just because it’s not your top priority, doesn’t mean it shouldn’t be improved a little bit, now and then. i.e. the difference between 0% and 5% of invested effort, compounds over time.
going through the motions of making sure the hashes weren’t publicly available would have been just virtue signaling
Yes. That’s what I meant by “the sort of thing to do as part of the spirit of the game”: in an actual nuclear (or AI) application, you’d want to pick the straightforwardly best design, not the design which was “something like an hour faster to get things going this way”, right?
So as part of the wargame ritual, maybe you should expect people to leave annoying nitpicky comments in the genre of “Your hashes are visible”, even if you don’t think there’s any real risk?
Does that seem weird? For more context on why I’m thinking this way, I thought last year’s phishing attack provided us with a very valuable and educational “red team” service that it’d be fun to see continued in some form. (“Coordinate to not destroy the world” is an appropriate premise for an existential-risk-reduction community ritual, but so is intelligent adversaries making that difficult.) I’m not personally vicious enough to try to get the site nuked, but putting on a white hat and thinking about how it could be done feels on-theme.
your comment is a straightforward misapplication of security mindset. [...] There is no point in pursuing a security mindset if you are virtually certain that the thing you would be investing resources into would not be your weakest attack point.
“Misapplication” meaning you think I’m doing the security thinking wrong, or “misapplication” meaning you think the security thinking isn’t appropriate given the context and costs? I think it’s desirable to separate the security analysis (this-and-such design is safe against such-and-this class of attacks) from the project-management analysis (this-and-such design would cost an extra hour of dev time which our team doesn’t have); external critics are often in a position to say something useful about the former but not the latter. (Although, unfortunately, my comment as written didn’t successfully separate them: “the sort of thing to do” is most naturally read as an implied policy recommendation, not a just-the-facts threat analysis. Sorry!)
I think your conclusion is reasonable that the investment of effort in security improvements is not justified by the risk of it being exploited, but I want to pull out a tiny part of your post and suggest refining:
“There is no point in pursuing a security mindset if you are virtually certain that the thing you would be investing resources into would not be your weakest attack point.”
Different attackers will target different points depending on their capability and what they care about, and which attacker will go after you depends on their motivations. Your weakest point may be lower real risk than others simply because the type of attackers who would exploit that point don’t care about you.
Organisations will regularly invest resources into what is not necessarily the weakest attack point but based on their assessment of the most cost effective way to reduce overall risk. This plays into defence in depth, where multiple layers of overall security features can provide better risk reduction, especially where the weakest attack points are expensive or impossible to address.
This may seem like a inconsequential point as it doesn’t make any difference to your conclusions, but I do see people focussing on weak attack points without considering whether their money is being well spent.
To me, a better framing would be:
You shouldn’t invest resources into measures where there are alternatives that are more effective at reducing risk.
Yep, this seems right to me. In this case, the set of attackers is also quite narrow, since the codes are only relevant for 24 hours, and then only in a somewhat bounded way that’s easy to revert.
I deliberately refrained from mentioning this in public yesterday out of respect for the spirit of the game, but I was disappointed that the SHA-256 hashes of the launch codes were publicly visible in the source code (such that someone could try to crack them offline) rather than being stored in the database (such that all guesses have to go online through the server).
Okay, yes, I understand that this wasn’t a “serious” vulnerability because the launch codes were sufficiently high-entropy (actually random, not chosen by a human) that no one is going to be able to crack the hash. (I didn’t make the list this year, but my launch code from last year was 16 alphanumeric characters, which is 6216≈95.2 bits of entropy, which I think is 6216/1011/(86400⋅365.25)/2 being an expectation of 7 billion years at 100 billion hashes per second? Oh, except knock off a couple orders of magnitude, because there were a hundred codes.)
Still, in principle, as a matter of security mindset, putting the hashes in the database lets you depend on one fewer assumption (about how much computing power the adversary has). It seems like the sort of thing to do as part of the spirit of the game, given that this kind of thing is how the world is actually going to end. (Someone works out that this-and-such AI design will perform safely as long as such-and-these assumptions are true, and then one of the assumptions turns out not to be true.)
I made this decision a while ago, mostly because it was something like an hour faster to get things going this way. I did the fermi, did some research into SHA-256 to make sure it was sufficiently pre-image resistant, and decided to go ahead. I stand by this decision, and think your comment is a straightforward misapplication of security mindset.
I think going through the motions of making sure the hashes weren’t publicly available would have been just virtue signaling, and the real security risks almost certainly live somewhere else in our stack (some out-of-date-library that has a security vulnerability, some OS-level issue, some configuration issue in our VPN, a social engineering attack on someone in our team or a past collaborator). There is no point in pursuing a security mindset if you are virtually certain that the thing you would be investing resources into would not be your weakest attack point. I know that LessWrong is not robust against a sophisticated dedicated attacker. I think it would be dumb of me to look at an insulated part of our stack and harden that to withstand a sophisticated attacker, when there are many other attack vectors that are much more fragile.
I sort of disagree. Not necessarily that it was the wrong choice to invest your security resources elsewhere—I think your threat model is approximately correct—but I disagree that it’s wrong to invest in that part of your stack.
My argument here is that following best practices is a good principle, and that you can and should make exceptions sometimes, but Zack is right to point it out as a vulnerability. Security best practices exist to help you reduce attack surface without having to be aware of every attack vector. You might look at this instance and rightly think “okay but SHA-256 is very hard to crack with keys this long, and there is more low hanging fruit”. But sometimes you’re going to make a wrong assumption when evaluating things like this, and best practices help protect you from limitations of your ability to model this. Maybe your SHA-256 implementation has a known vulnerability that you didn’t check, maybe your secret generation wasn’t actually random, etc. I don’t think any of these apply in this case, but I think sometimes you’re likely to be mistaken about your assumptions. The question becomes a more general one about when it makes sense to follow best practices and when it makes sense to make exceptions. In this case I would think an hour of work would be worth it to follow the best practice. If it were more like five hours, I might agree with with you.
I wouldn’t discount the signaling value either. Allies and attackers can notice whether you’re following best practices and use that info to decide whether they should invest resources in trusting your infrastructure or attacking it.
I agree that this is a correct application of security mindset; exposures like these can compound with, for example, someone’s automatic search of the 100 most common ways to screw up secure random number generation such as by using the current time as a seed. Deep security is about reducing the amount of thinking you have to do and your exposure to wrong models and stuff you didn’t think of.
Furthermore, It is also not inconceivable to me that an adversary might be able to use the hash itself without cracking it. For example, the sha256 hash of some information is commonly used to prove that someone has that information without revealing it, so an adversary, using the hash, could credibly lie that he already possesses a launch code and in a possible counterfactual world where no one found about the client side leaking the hash except this adversary, use this lie to acquire an actual code with some social engineering.
Like:
“Attention Lesswrong! With trickery I have acquired a launch code capable of destroying your site. As proof here is the sha256 hash of it: <hash>.
This is not a trick, I will leave plenty of time for you to check with your EA buddies that the hash is valid before you need to meet my demands.
I demand a launch code capable of destroying the EA forum sent to me until <time> or I will nuke this site and to this I precommitted. I won’t reveal what I plan to do with the launch code you will send to me, but by basic game theory your interest is in sending it to me as your site’s destruction is not certain that way.
I can’t prove it to you, but irl I precommitted to nuking the site if my demands are not met and also that I won’t send any more messages to prevent useless debating.
I hope you will make the correct choice!”
I do not know what the difference here is. Presumably one implies the other?
Epistemic note: I’m not Taleuntum.
I think the difference is:
Tale: Today it may be best, to invest elsewhere. But not forever. Security against a dedicated adversary is not always and forever impossible, but a result of continuous investment that is a) eventually achieved, b) requires continuing work going forward (bugs are discovered in resources once thought secure and need patching, malware and threats change, etc.)
In other words, just because it’s not your top priority, doesn’t mean it shouldn’t be improved a little bit, now and then. i.e. the difference between 0% and 5% of invested effort, compounds over time.
Yes. That’s what I meant by “the sort of thing to do as part of the spirit of the game”: in an actual nuclear (or AI) application, you’d want to pick the straightforwardly best design, not the design which was “something like an hour faster to get things going this way”, right?
So as part of the wargame ritual, maybe you should expect people to leave annoying nitpicky comments in the genre of “Your hashes are visible”, even if you don’t think there’s any real risk?
Does that seem weird? For more context on why I’m thinking this way, I thought last year’s phishing attack provided us with a very valuable and educational “red team” service that it’d be fun to see continued in some form. (“Coordinate to not destroy the world” is an appropriate premise for an existential-risk-reduction community ritual, but so is intelligent adversaries making that difficult.) I’m not personally vicious enough to try to get the site nuked, but putting on a white hat and thinking about how it could be done feels on-theme.
“Misapplication” meaning you think I’m doing the security thinking wrong, or “misapplication” meaning you think the security thinking isn’t appropriate given the context and costs? I think it’s desirable to separate the security analysis (this-and-such design is safe against such-and-this class of attacks) from the project-management analysis (this-and-such design would cost an extra hour of dev time which our team doesn’t have); external critics are often in a position to say something useful about the former but not the latter. (Although, unfortunately, my comment as written didn’t successfully separate them: “the sort of thing to do” is most naturally read as an implied policy recommendation, not a just-the-facts threat analysis. Sorry!)
I think your conclusion is reasonable that the investment of effort in security improvements is not justified by the risk of it being exploited, but I want to pull out a tiny part of your post and suggest refining:
“There is no point in pursuing a security mindset if you are virtually certain that the thing you would be investing resources into would not be your weakest attack point.”
Different attackers will target different points depending on their capability and what they care about, and which attacker will go after you depends on their motivations. Your weakest point may be lower real risk than others simply because the type of attackers who would exploit that point don’t care about you.
Organisations will regularly invest resources into what is not necessarily the weakest attack point but based on their assessment of the most cost effective way to reduce overall risk. This plays into defence in depth, where multiple layers of overall security features can provide better risk reduction, especially where the weakest attack points are expensive or impossible to address.
This may seem like a inconsequential point as it doesn’t make any difference to your conclusions, but I do see people focussing on weak attack points without considering whether their money is being well spent.
To me, a better framing would be:
You shouldn’t invest resources into measures where there are alternatives that are more effective at reducing risk.
Yep, this seems right to me. In this case, the set of attackers is also quite narrow, since the codes are only relevant for 24 hours, and then only in a somewhat bounded way that’s easy to revert.