I mostly endorse the “Them” as a representation of my views (not necessarily everything, e.g. idk anything specific about the author of the Doomsday Machine, though I do endorse the heuristic of “stories you hear are likely sensationalized”).
Me: Why would that be an update? We already know that state bioweapons programs have killed thousands of people with accidental releases, and there’s no particular reason that they couldn’t cause worse disasters, and that international regulation has failed to control that.
Them: [inaudible. I don’t know how to rephrase the thing that people say at this point in the conversation.]
In my case, it’s something like “the lack of really bad failures so far implies there’s some filter that prevents really bad failures from being likely. We both agree that if such a filter exists, we don’t know what it is. When you say ‘there’s no particular reason why they couldn’t cause worse disasters’, I say that that’s a reflection of our map, but in the territory there probably is a filter that is a particular reason why they couldn’t cause worse disasters, even if we don’t know what it is. If we found out that COVID was a failure of biosecurity, that would imply the filter is much less strong than we might otherwise have guessed, leading to a substantial update.”
the lack of really bad failures so far implies there’s some filter that prevents really bad failures from being likely. We both agree that if such a filter exists, we don’t know what it is.
I can think of two broad categories here: selection effects (“if almost everyone dies, I probably died too and am not around to observe it”) and all other effects. I think the selection effect filter is significant enough that I would be reluctant to think ‘all other effects’ are large without doing some very careful math.
There also haven’t been anthropogenic risks that killed 10% of humans. The selection effect update on “10% of people killed” is pretty small. (World War 2 killed ~2% and feels like the strongest example against the “Them” position.)
You could believe that most risks are all-or-nothing, in which case I agree the “Them” position is considerably weaker due to selection effects.
I agree that selection effects from smaller risks are much smaller; I also suspect that most risks so far haven’t been all-or-nothing risks. I think my main objection was that there’s a potentially big difference between “likely (in a forward-looking sense)” and “likely (in a backward-looking sense)” and was worried that the two quoted sentences don’t make that clear to the reader.
Okay, fair enough, though I want to note that the entire disagreement in the post is about the backward-looking sense (if I’m understanding you correctly). Like, the question is how to interpret the fact that there were a lot of near misses, but no catastrophes (and for the class of nukes / bio etc. a 10% catastrophe seems way more likely than a 90-100% catastrophe).
Okay, fair enough, though I want to note that the entire disagreement in the post is about the backward-looking sense (if I’m understanding you correctly).
Oh interesting! I suspect you understand me correctly and we disagree. To elaborate:
If it means something for humans to be “good at coordination”, it’s that there’s some underlying features that cause humans to succeed rather than fail at coordination challenges. If I said someone was “good at winning poker games”, I don’t just mean that they happened to win once, but that there’s some underlying fact that caused them to win in the past and also likely to win in the future. If I just want to observe that someone won last week’s poker game, but I ascribe this to chance instead of skill, I say that person is lucky.
But, of course, we can only infer whether or not someone is good at poker, rather than being able to observe it directly. So discussions about the inference necessarily have a backward-looking quality to them, because it’s about what observations led us to the epistemic state we’re at.
That all sounds right and I’m not sure where you expected me to disagree.
discussions about the inference necessarily have a backward-looking quality to them
This was exactly my point, I think? Since we’re looking backward when making inferences, and since we didn’t expect full extinction or even 90% extinction in the past, our inferences don’t need to take selection effects into account (or more accurately selection effects would have a relatively small effect on the final answer).
When I read the ‘Buck’ points, most of them feel like they’re trying to be about ‘how humans are’, or the forward-likeliness. Like, this here:
But I still feel that my overall worldview of “people will do wild and reckless things” loses fewer Bayes points than yours does.
Importantly, “wild and reckless” is describing the properties of the actions / underlying cognitive processes, not the outcomes. And later:
Why would that be an update? We already know that state bioweapons programs have killed thousands of people with accidental releases, and there’s no particular reason that they couldn’t cause worse disasters, and that international regulation has failed to control that.
At least in this presentation of Buck vs. Them, there’s a disagreement over something like “whether scope matters”; Buck thinks no (‘what damage happens to a toddler depends on how dangerous their environment is, since the toddler doesn’t know what to avoid and so can’t be scope-sensitive’) and Them thinks yes (‘sure, humanity has screwed up lots of things that don’t matter, but that’s because effort is proportional to how much the thing matters, and so they’re rationally coping with lots of fires that would be expensive to put out.’).
This feels like it’s mostly not about bets on whether X happened or not, and mostly about counterfactuals / reference class tennis (“would people have taken climate change more seriously if it were a worse problem?” / “is climate change a thing that people are actually trying to coordinate on, or a distraction?”).
At least in this presentation of Buck vs. Them, there’s a disagreement over something like “whether scope matters”
I agree this could be a disagreement, but how do selection effects matter for it?
This feels like it’s mostly not about bets on whether X happened or not, and mostly about counterfactuals / reference class tennis
Seems plausible, but again why do selection effects matter for it?
----
I may have been a bit too concise when saying
the entire disagreement in the post is about the backward-looking sense
To expand on it, I expect that if we fix a particular model of the world (e.g. coordination of the type discussed here is hard, we have basically never succeeded at it, the lack of accidents so far is just luck), Buck and I would agree much more on the forward-looking consequences of that model for AI alignment (perhaps I’d be at like 30% x-risk, idk). The disagreement is about what model of the world we should have (or perhaps what distribution over models). For that, we look at what happens in the past (both in reality and counterfactually), which is “backward-looking”.
Note that I have no idea what math to do here. The actual thing I’d do is try to figure out the reference class of ‘things that could be major disasters’, look how well the situation around them was handled (carefully, coordinated, sloppily, clumsily, etc) and then after getting close to the territory in that way, reflect loads on anthropics and wtf to update about it. I don’t know how to really do math on either.
I mostly endorse the “Them” as a representation of my views (not necessarily everything, e.g. idk anything specific about the author of the Doomsday Machine, though I do endorse the heuristic of “stories you hear are likely sensationalized”).
In my case, it’s something like “the lack of really bad failures so far implies there’s some filter that prevents really bad failures from being likely. We both agree that if such a filter exists, we don’t know what it is. When you say ‘there’s no particular reason why they couldn’t cause worse disasters’, I say that that’s a reflection of our map, but in the territory there probably is a filter that is a particular reason why they couldn’t cause worse disasters, even if we don’t know what it is. If we found out that COVID was a failure of biosecurity, that would imply the filter is much less strong than we might otherwise have guessed, leading to a substantial update.”
I can think of two broad categories here: selection effects (“if almost everyone dies, I probably died too and am not around to observe it”) and all other effects. I think the selection effect filter is significant enough that I would be reluctant to think ‘all other effects’ are large without doing some very careful math.
There also haven’t been anthropogenic risks that killed 10% of humans. The selection effect update on “10% of people killed” is pretty small. (World War 2 killed ~2% and feels like the strongest example against the “Them” position.)
You could believe that most risks are all-or-nothing, in which case I agree the “Them” position is considerably weaker due to selection effects.
I agree that selection effects from smaller risks are much smaller; I also suspect that most risks so far haven’t been all-or-nothing risks. I think my main objection was that there’s a potentially big difference between “likely (in a forward-looking sense)” and “likely (in a backward-looking sense)” and was worried that the two quoted sentences don’t make that clear to the reader.
Okay, fair enough, though I want to note that the entire disagreement in the post is about the backward-looking sense (if I’m understanding you correctly). Like, the question is how to interpret the fact that there were a lot of near misses, but no catastrophes (and for the class of nukes / bio etc. a 10% catastrophe seems way more likely than a 90-100% catastrophe).
Oh interesting! I suspect you understand me correctly and we disagree. To elaborate:
If it means something for humans to be “good at coordination”, it’s that there’s some underlying features that cause humans to succeed rather than fail at coordination challenges. If I said someone was “good at winning poker games”, I don’t just mean that they happened to win once, but that there’s some underlying fact that caused them to win in the past and also likely to win in the future. If I just want to observe that someone won last week’s poker game, but I ascribe this to chance instead of skill, I say that person is lucky.
But, of course, we can only infer whether or not someone is good at poker, rather than being able to observe it directly. So discussions about the inference necessarily have a backward-looking quality to them, because it’s about what observations led us to the epistemic state we’re at.
That all sounds right and I’m not sure where you expected me to disagree.
This was exactly my point, I think? Since we’re looking backward when making inferences, and since we didn’t expect full extinction or even 90% extinction in the past, our inferences don’t need to take selection effects into account (or more accurately selection effects would have a relatively small effect on the final answer).
When I read the ‘Buck’ points, most of them feel like they’re trying to be about ‘how humans are’, or the forward-likeliness. Like, this here:
Importantly, “wild and reckless” is describing the properties of the actions / underlying cognitive processes, not the outcomes. And later:
At least in this presentation of Buck vs. Them, there’s a disagreement over something like “whether scope matters”; Buck thinks no (‘what damage happens to a toddler depends on how dangerous their environment is, since the toddler doesn’t know what to avoid and so can’t be scope-sensitive’) and Them thinks yes (‘sure, humanity has screwed up lots of things that don’t matter, but that’s because effort is proportional to how much the thing matters, and so they’re rationally coping with lots of fires that would be expensive to put out.’).
This feels like it’s mostly not about bets on whether X happened or not, and mostly about counterfactuals / reference class tennis (“would people have taken climate change more seriously if it were a worse problem?” / “is climate change a thing that people are actually trying to coordinate on, or a distraction?”).
I agree this could be a disagreement, but how do selection effects matter for it?
Seems plausible, but again why do selection effects matter for it?
----
I may have been a bit too concise when saying
To expand on it, I expect that if we fix a particular model of the world (e.g. coordination of the type discussed here is hard, we have basically never succeeded at it, the lack of accidents so far is just luck), Buck and I would agree much more on the forward-looking consequences of that model for AI alignment (perhaps I’d be at like 30% x-risk, idk). The disagreement is about what model of the world we should have (or perhaps what distribution over models). For that, we look at what happens in the past (both in reality and counterfactually), which is “backward-looking”.
Note that I have no idea what math to do here. The actual thing I’d do is try to figure out the reference class of ‘things that could be major disasters’, look how well the situation around them was handled (carefully, coordinated, sloppily, clumsily, etc) and then after getting close to the territory in that way, reflect loads on anthropics and wtf to update about it. I don’t know how to really do math on either.