I think given that we didn’t suppress COVID, mitigating its damage probably involved new problems that we didn’t have solutions for before.
Hmm. This just doesn’t seem like what was going on to me at all. I think I disagree a lot about this, and it seems less about “how things will shake out in Slow AI Takeoff” and more about “how badly and obviously-in-advance and easily-preventably did we screw up our covid response.”
(I expect we also disagree about how Slow Takeoff would look, but I don’t think that’s the cruxy bit for me here).
I’m sort of hesitant to jump into the “why covid obviously looks like mass institutional failure, given a very straightforward, well understood scenario” argument because I feel like it’s been hashed out a lot in the past 3 months and I’m not sure where to go with it – I’m assuming you’ve read the relevant arguments and didn’t find them convincing.
The sort of things I have in mind include:
FDA actively hampers efforts to scale up testing
Hospitals don’t start re-using PPE, when it was clear they were going to have to start doing so in a month
Everyone delays 3 weeks before declaring lockdowns, at a time where the simple math clearly indicated we needed to lock down promptly if we wanted to have a chance at squashing.
Media actively downplays risk and attributes it to racism
CDC and WHO making actively misleading statements
These problems all seemed fairly straightforward and understood. There might also be novel problems going on but they don’t seem necessary to hypothesize given the above types of failure.
Ah, I see. I agree with this and do think it cuts against my point #1, but not points #2 and #3. Edited the top-level comment to note this.
I’m sort of hesitant to jump into the “why covid obviously looks like mass institutional failure, given a very straightforward, well understood scenario” argument because I feel like it’s been hashed out a lot in the past 3 months and I’m not sure where to go with it – I’m assuming you’ve read the relevant arguments and didn’t find them convincing.
Tbc, I find it quite likely that there was mass institutional failure with COVID; I’m mostly arguing that soft takeoff is sufficiently different from COVID that we shouldn’t necessarily expect the same mass institutional failure in the case of soft takeoff. (This is similar to Matthew’s argument that the pandemic shares more properties with fast takeoff than with slow takeoff.)
I do definitely expect different institutional failure in the case of Soft Takeoff. But it sort of depends on what level of abstraction you’re looking at the institutional failure through. Like, the FDA won’t be involved. But there’s a decent chance that some other regulatory will be involved, which is following the underlying FDA impulse of “Wield the one hammer we know how to wield to justify our jobs.” (In a large company, it’s possible that regulatory body could be a department inside the org, rather than a government agency)
In reasonably good outcomes, the decisions are mostly being made by tech companies full of specialists who well understand the problem. In that case the institutional failures will look more like “what ways do tech companies normally screw up due to internal politics?”
There’s a decent chance the military or someone will try to commandeer the project, in which case more typical government institutional failures will become more relevant.
One thing that seems significant is that 2 years prior to The Big Transition, you’ll have multiple companies with similar-ish tech. And some of them will be appropriately cautious (like New Zealand, Singapore), and others will not have the political wherewithal to slow down and think carefully and figure out what inconvenient things they need to do and do them (like many other countries in covid)
Yeah, these sorts of stories seem possible, and it also seems possible that institutions try some terrible policies, notice that they’re terrible, and then fix them. Like, this description:
But there’s a decent chance that some other regulatory will be involved, which is following the underlying FDA impulse of “Wield the one hammer we know how to wield to justify our jobs.” (In a large company, it’s possible that regulatory body could be a department inside the org, rather than a government agency)
just doesn’t seem to match my impression of non-EAs-or-rationalists working on AI governance. It’s possible that people in government are much less competent than people at think tanks, but this would be fairly surprising to me. In addition, while I can’t explain FDA decisions, I still pretty strongly penalize views that ascribe huge very-consequential-by-their-goals irrationality to small groups of humans working full time on something.
(Note I would defend the claim that institutions work well enough that in a slow takeoff world the probability of extinction is < 80%, and probably < 50%, just on the basis that if AI alignment turned out to be impossible, we can coordinate not to build powerful AI.)
Are you saying you think that wasn’t a fair characterization of the FDA, or that the hypothetical AI Governance bodies would be different from the FDA?
(The statement was certainly not very fair to the FDA, and I do expect there was more going on under the hood than that motivation. But, I do broadly think governing bodies do what they are incentivized to do, which includes justifying themselves, especially after being around a couple decades and gradually being infiltrated by careerists)
Hmm. This just doesn’t seem like what was going on to me at all. I think I disagree a lot about this, and it seems less about “how things will shake out in Slow AI Takeoff” and more about “how badly and obviously-in-advance and easily-preventably did we screw up our covid response.”
(I expect we also disagree about how Slow Takeoff would look, but I don’t think that’s the cruxy bit for me here).
I’m sort of hesitant to jump into the “why covid obviously looks like mass institutional failure, given a very straightforward, well understood scenario” argument because I feel like it’s been hashed out a lot in the past 3 months and I’m not sure where to go with it – I’m assuming you’ve read the relevant arguments and didn’t find them convincing.
The sort of things I have in mind include:
FDA actively hampers efforts to scale up testing
Hospitals don’t start re-using PPE, when it was clear they were going to have to start doing so in a month
Everyone delays 3 weeks before declaring lockdowns, at a time where the simple math clearly indicated we needed to lock down promptly if we wanted to have a chance at squashing.
Media actively downplays risk and attributes it to racism
CDC and WHO making actively misleading statements
These problems all seemed fairly straightforward and understood. There might also be novel problems going on but they don’t seem necessary to hypothesize given the above types of failure.
Ah, I see. I agree with this and do think it cuts against my point #1, but not points #2 and #3. Edited the top-level comment to note this.
Tbc, I find it quite likely that there was mass institutional failure with COVID; I’m mostly arguing that soft takeoff is sufficiently different from COVID that we shouldn’t necessarily expect the same mass institutional failure in the case of soft takeoff. (This is similar to Matthew’s argument that the pandemic shares more properties with fast takeoff than with slow takeoff.)
Ah, okay. I think I need to at least think a bit harder to figure out if I still disagree in that case.
I do definitely expect different institutional failure in the case of Soft Takeoff. But it sort of depends on what level of abstraction you’re looking at the institutional failure through. Like, the FDA won’t be involved. But there’s a decent chance that some other regulatory will be involved, which is following the underlying FDA impulse of “Wield the one hammer we know how to wield to justify our jobs.” (In a large company, it’s possible that regulatory body could be a department inside the org, rather than a government agency)
In reasonably good outcomes, the decisions are mostly being made by tech companies full of specialists who well understand the problem. In that case the institutional failures will look more like “what ways do tech companies normally screw up due to internal politics?”
There’s a decent chance the military or someone will try to commandeer the project, in which case more typical government institutional failures will become more relevant.
One thing that seems significant is that 2 years prior to The Big Transition, you’ll have multiple companies with similar-ish tech. And some of them will be appropriately cautious (like New Zealand, Singapore), and others will not have the political wherewithal to slow down and think carefully and figure out what inconvenient things they need to do and do them (like many other countries in covid)
Yeah, these sorts of stories seem possible, and it also seems possible that institutions try some terrible policies, notice that they’re terrible, and then fix them. Like, this description:
just doesn’t seem to match my impression of non-EAs-or-rationalists working on AI governance. It’s possible that people in government are much less competent than people at think tanks, but this would be fairly surprising to me. In addition, while I can’t explain FDA decisions, I still pretty strongly penalize views that ascribe huge very-consequential-by-their-goals irrationality to small groups of humans working full time on something.
(Note I would defend the claim that institutions work well enough that in a slow takeoff world the probability of extinction is < 80%, and probably < 50%, just on the basis that if AI alignment turned out to be impossible, we can coordinate not to build powerful AI.)
Are you saying you think that wasn’t a fair characterization of the FDA, or that the hypothetical AI Governance bodies would be different from the FDA?
(The statement was certainly not very fair to the FDA, and I do expect there was more going on under the hood than that motivation. But, I do broadly think governing bodies do what they are incentivized to do, which includes justifying themselves, especially after being around a couple decades and gradually being infiltrated by careerists)
I am mostly confused, but I expect that if I learned more I would say that it wasn’t a fair characterization of the FDA.