Roko’s post is not about the general problem of incumbents facing internal conflict (which happens all the time for many reasons) but about a specific class of situation where something is ripe to be taken over by a person or group with new capabilities, and those capabilities come along and it happens.
While nuclear weapons represented a dramatic gain in capabilities for the United States, what you’re talking about isn’t the US using its new nuclear capabilities to overturn the world order, but internal politicking within the US military. The arrival of nuclear weapons didn’t represent an unprecedented gain in politicking capabilities for one set of US military officials over another. It is not helpful to think about the “revolt of the admirals” in terms of a “(US military officials who could lose influence if a new kind of weapon comes along) overhang”, so far as I can tell. There’s no analogy here, just two situations that happen to be describable using the words “incumbents” and “new capabilities”.
My thinking on your theories about psychological manipulation is that they don’t belong in this thread, and I will not be drawn into an attempt to make this thread be about them. You’ve already made four posts about them in the last ~2 weeks and those are perfectly good places for that discussion.
In this 11-paragraph post, the last two paragraphs turn the focus to incumbents, so I thought that expanding the topic on incumbents was reasonable, especially because nuclear weapons history is highly relevant to the overall theme, and the Revolt of the Admirals was both 1) incumbent-focused, and 2) an interesting fact about the emergence of the nuclear weapons paradigm that few people are aware of.
My thinking on the current situation with AI-powered supermanipulation is that it’s highly relevant to AI safety because it’s reasonably likely to end up in a world that’s hostile to the AI safety community, and it’s also an excellent and relevant example of technological overhang. Anyone with a SWE background can, if they look, immediately verify that SOTA systems are orders of magnitude more powerful than needed to automatically and continuously track and research deep structures like belief causality, and I’m arguing that things like social media scrolling data are the main bottleneck for intense manipulation capabilities, strongly indicating that they are already prevalent and therefore relevant.
I haven’t been using Lesswrong for a seventh of the amount of time that you have, so I don’t know what kinds of bizarre pretenders and galaxy-brained social status grabs have happened over the years. But I still think that talking about technological change, and sorting bad technological forecasts from good forecasts, is a high-value activity that LW has prioritized and gotten good at but can also do better.
Roko’s post is not about the general problem of incumbents facing internal conflict (which happens all the time for many reasons) but about a specific class of situation where something is ripe to be taken over by a person or group with new capabilities, and those capabilities come along and it happens.
While nuclear weapons represented a dramatic gain in capabilities for the United States, what you’re talking about isn’t the US using its new nuclear capabilities to overturn the world order, but internal politicking within the US military. The arrival of nuclear weapons didn’t represent an unprecedented gain in politicking capabilities for one set of US military officials over another. It is not helpful to think about the “revolt of the admirals” in terms of a “(US military officials who could lose influence if a new kind of weapon comes along) overhang”, so far as I can tell. There’s no analogy here, just two situations that happen to be describable using the words “incumbents” and “new capabilities”.
My thinking on your theories about psychological manipulation is that they don’t belong in this thread, and I will not be drawn into an attempt to make this thread be about them. You’ve already made four posts about them in the last ~2 weeks and those are perfectly good places for that discussion.
In this 11-paragraph post, the last two paragraphs turn the focus to incumbents, so I thought that expanding the topic on incumbents was reasonable, especially because nuclear weapons history is highly relevant to the overall theme, and the Revolt of the Admirals was both 1) incumbent-focused, and 2) an interesting fact about the emergence of the nuclear weapons paradigm that few people are aware of.
My thinking on the current situation with AI-powered supermanipulation is that it’s highly relevant to AI safety because it’s reasonably likely to end up in a world that’s hostile to the AI safety community, and it’s also an excellent and relevant example of technological overhang. Anyone with a SWE background can, if they look, immediately verify that SOTA systems are orders of magnitude more powerful than needed to automatically and continuously track and research deep structures like belief causality, and I’m arguing that things like social media scrolling data are the main bottleneck for intense manipulation capabilities, strongly indicating that they are already prevalent and therefore relevant.
I haven’t been using Lesswrong for a seventh of the amount of time that you have, so I don’t know what kinds of bizarre pretenders and galaxy-brained social status grabs have happened over the years. But I still think that talking about technological change, and sorting bad technological forecasts from good forecasts, is a high-value activity that LW has prioritized and gotten good at but can also do better.