I never claimed that AI safety is more X than “any other advocacy group”; I specifically said “most other advocacy groups”. And of course I’m not sure about this, asking for that is an isolated demand for rigor. It feels like your objection is the thing that’s vibe-based here.
On the object level: these are good examples, but because movements vary on so many axes, it’s hard to weigh up two of them against each other. That’s why I identified the three features of AI safety which seem to set it apart from most other movements. (Upon reflection I’d also add a fourth: the rapid growth.)
I’m curious if there are specific features which some of the movements you named have that you think contribute significantly to their power-seeking-ness, which AI safety doesn’t have.
I agree that the word “any” is wrong here. I used “sure” in the sense of “reasonably everyday I-won’t-get-hit-by-a-car-if-I-cross-the-road-now sure,” not in the “100% math-proof sure” sense.
By “vibe-based,” I refer to the features that you mention. Yes, there is a lot of talk about consequentialism and efficiency in rationalist/EA-adjacent AIS circles, and consequentialism gives a base for power-seeking actions, but it’s unclear how much this talk leads to actual power-seeking actions and how much it’s just local discourse framing.
The same applies to the feel of urgency.
Funnily, consequentialism can lead to less power-seeking if we define the problem in a less open-ended manner. If your task is to “maximize the number of good things in the world,” you benefit from power-seeking. If your task is to “design a safe system capable of producing superintelligent work,” you are, in fact, interested in completing this task with minimal effort and, therefore, minimal resources.
I think that the broad environmentalist movement is at least in the same tier as the AIS movement for all of the mentioned features.
Power-seeking inferred from consequentialism? Environmentalism is a movement about political control from the very start. The Club of Rome was initially founded inside the OECD and tried to influence its policy towards degrowth.
Consequentialism leading to questionable practices? Mass sterilizations in the 1980s are far beyond whatever AI Safety has done to date.
Urgency? You can look at all the people claiming that 2030 is a point of no return for climate change.
Focus of elites? The Club of Rome and the Sierra Club are literally what is written in the name—they are elite clubs.
I think I would agree if you say that there are a lot of nuances and the chance of AIS being more power-seeking than the environmentalism movement is non-negligible, but to measure power-seeking with necessary accuracy, we would need not a blog post written by one person, but the work of an army of sociologists.
I never claimed that AI safety is more X than “any other advocacy group”; I specifically said “most other advocacy groups”. And of course I’m not sure about this, asking for that is an isolated demand for rigor. It feels like your objection is the thing that’s vibe-based here.
On the object level: these are good examples, but because movements vary on so many axes, it’s hard to weigh up two of them against each other. That’s why I identified the three features of AI safety which seem to set it apart from most other movements. (Upon reflection I’d also add a fourth: the rapid growth.)
I’m curious if there are specific features which some of the movements you named have that you think contribute significantly to their power-seeking-ness, which AI safety doesn’t have.
I agree that the word “any” is wrong here. I used “sure” in the sense of “reasonably everyday I-won’t-get-hit-by-a-car-if-I-cross-the-road-now sure,” not in the “100% math-proof sure” sense.
By “vibe-based,” I refer to the features that you mention. Yes, there is a lot of talk about consequentialism and efficiency in rationalist/EA-adjacent AIS circles, and consequentialism gives a base for power-seeking actions, but it’s unclear how much this talk leads to actual power-seeking actions and how much it’s just local discourse framing.
The same applies to the feel of urgency.
Funnily, consequentialism can lead to less power-seeking if we define the problem in a less open-ended manner. If your task is to “maximize the number of good things in the world,” you benefit from power-seeking. If your task is to “design a safe system capable of producing superintelligent work,” you are, in fact, interested in completing this task with minimal effort and, therefore, minimal resources.
I think that the broad environmentalist movement is at least in the same tier as the AIS movement for all of the mentioned features.
Power-seeking inferred from consequentialism? Environmentalism is a movement about political control from the very start. The Club of Rome was initially founded inside the OECD and tried to influence its policy towards degrowth.
Consequentialism leading to questionable practices? Mass sterilizations in the 1980s are far beyond whatever AI Safety has done to date.
Urgency? You can look at all the people claiming that 2030 is a point of no return for climate change.
Focus of elites? The Club of Rome and the Sierra Club are literally what is written in the name—they are elite clubs.
I think I would agree if you say that there are a lot of nuances and the chance of AIS being more power-seeking than the environmentalism movement is non-negligible, but to measure power-seeking with necessary accuracy, we would need not a blog post written by one person, but the work of an army of sociologists.