I’ve become somewhat pessimistic about encouraging regulatory power over AI development recently after reading this Bismarck Analysis case study on the level of influence (or lack of it) that scientists had over nuclear policy.
The impression I got from some other secondary/tertiary sources (specifically the book Organizing Genius) was that General Groves, the military man who was the interface between the military and Oppenheimer and the Manhattan Project, did his best to shield the Manhattan Project scientists from military and bureaucratic drudgery, and that Vannevar Bush was someone who served as an example of a scientist successfully steering policy.
This case study seems to show that Groves was significantly less of a value add than I thought given the likelihood of him having destroyed Leo Szilard’s political influence (and therefore Leo’s ability to influence nuclear policy in a direction of preventing an arms race or using it in war). Bush also seems like a disappointment—he waited months for information to pass through ‘official channels’ before he attempted to persuade people like FDR to begin a nuclear weapons development program. On top of that, Bush seemed like he internalized the bureaucratic norms of the political and military hierarchy he worked in—when a scientist named Ernest Lawrence tried to reach the relevant government officials to talk about the importance of nuclear weapons development, Bush (according to this paper) got annoyed by him seemingly bypassing the ‘chain of command’ (I assume by focusing on talking to people Bush would report to, instead of to Bush himself) that he threatened to politically marginalize Ernest.
Finally, I see clear parallels between the ineffective attempts by these physicists at influencing nuclear weapons policy via contributing technically and trying to build ‘political capital’, and the ineffective attempts by AI safety engineers and researchers who decide to go work at frontier labs (OpenAI is the clearest example) with the intention of building influence with the people in there so that they can steer things in the future. I’m pretty sure at this point that such a strategy is a pretty bad idea, given that it seems better to do nothing than to contribute to accelerating towards ASI.
There are galaxy-brained counter-arguments to this claim, such as davidad’s supposed game-theoretic model that (AFAICT) involves accelerating to AGI powerful enough to make the provable safety agenda viable, or Paul Christiano’s (again, AFAICT) plan that’s basically ‘given intense economic pressure for better capabilities, we shall see a steady and continuous improvement, so the danger actually is in discontinuities that make it harder for humanity to react to changes, and therefore we should accelerate to reduce compute overhang’. I remain unconvinced by them.
I’m not actually seeing where deep expertise on nuclear weapons technology would qualify anybody to have much special input into nuclear weapons policy in general. There just don’t seem to be that many technical issues compared to the vast number of political ones.
I don’t know if that applies to AI, but tend to think the two are different.
I agree with your argument here, especially your penultimate paragraph, but I’ll nitpick that framing your disagreements with Groves as him being “less of a value add” seems wrong. The value that Groves added was building the bomb, not setting diplomatic policy.
given intense economic pressure for better capabilities, we shall see a steady and continuous improvement, so the danger actually is in discontinuities that make it harder for humanity to react to changes, and therefore we should accelerate to reduce compute overhang
I don’t feel like this is actually a counterargument? You could agree with both arguments, concluding that we shouldn’t work for OpenAI but a outfit better-aligned to your values is okay.
I’ve become somewhat pessimistic about encouraging regulatory power over AI development recently after reading this Bismarck Analysis case study on the level of influence (or lack of it) that scientists had over nuclear policy.
The impression I got from some other secondary/tertiary sources (specifically the book Organizing Genius) was that General Groves, the military man who was the interface between the military and Oppenheimer and the Manhattan Project, did his best to shield the Manhattan Project scientists from military and bureaucratic drudgery, and that Vannevar Bush was someone who served as an example of a scientist successfully steering policy.
This case study seems to show that Groves was significantly less of a value add than I thought given the likelihood of him having destroyed Leo Szilard’s political influence (and therefore Leo’s ability to influence nuclear policy in a direction of preventing an arms race or using it in war). Bush also seems like a disappointment—he waited months for information to pass through ‘official channels’ before he attempted to persuade people like FDR to begin a nuclear weapons development program. On top of that, Bush seemed like he internalized the bureaucratic norms of the political and military hierarchy he worked in—when a scientist named Ernest Lawrence tried to reach the relevant government officials to talk about the importance of nuclear weapons development, Bush (according to this paper) got annoyed by him seemingly bypassing the ‘chain of command’ (I assume by focusing on talking to people Bush would report to, instead of to Bush himself) that he threatened to politically marginalize Ernest.
Finally, I see clear parallels between the ineffective attempts by these physicists at influencing nuclear weapons policy via contributing technically and trying to build ‘political capital’, and the ineffective attempts by AI safety engineers and researchers who decide to go work at frontier labs (OpenAI is the clearest example) with the intention of building influence with the people in there so that they can steer things in the future. I’m pretty sure at this point that such a strategy is a pretty bad idea, given that it seems better to do nothing than to contribute to accelerating towards ASI.
There are galaxy-brained counter-arguments to this claim, such as davidad’s supposed game-theoretic model that (AFAICT) involves accelerating to AGI powerful enough to make the provable safety agenda viable, or Paul Christiano’s (again, AFAICT) plan that’s basically ‘given intense economic pressure for better capabilities, we shall see a steady and continuous improvement, so the danger actually is in discontinuities that make it harder for humanity to react to changes, and therefore we should accelerate to reduce compute overhang’. I remain unconvinced by them.
I’m not actually seeing where deep expertise on nuclear weapons technology would qualify anybody to have much special input into nuclear weapons policy in general. There just don’t seem to be that many technical issues compared to the vast number of political ones.
I don’t know if that applies to AI, but tend to think the two are different.
I agree with your argument here, especially your penultimate paragraph, but I’ll nitpick that framing your disagreements with Groves as him being “less of a value add” seems wrong. The value that Groves added was building the bomb, not setting diplomatic policy.
I don’t feel like this is actually a counterargument? You could agree with both arguments, concluding that we shouldn’t work for OpenAI but a outfit better-aligned to your values is okay.