I think there’s a difference between “Steelmanning something to learn the most you can from it” (for your own benefit), and accurately engaging with what people actually think and mean.
(For example, I think it’s common for consequentialists to “steelman” a deontological argument into consequentialism… but the actual reasons for a deontologists beliefs just have nothing to do with consequentialism)
In the case of people saying “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” and then leaving it at that, I honestly justdon’t think they’re thinking about it that hard, and rounding things off to vague plausibilities without a model. (Somes they don’t just leave it at that – I think Robin Hanson generally has some kind of model, maybe closer to what you’re saying here. But in that case you can engage with whatever they’re actually saying without as much need to steelman it yourself)
(I also think my OP here is roughly as good an answer to your steelman here – the issue still remains that there doesn’t have to be a sharply treacherous turn, to result in things just eventually snowballing in a way similar to how powerful empires snowball, long after it’s too late to do anything about it)
I like arguing with myself. So it is fun to make the best case. But yup I was going beyond what people might. I think I find arguments against naive views less interesting so spice them up some.
In accelerando the participants in Economy 2.0 had a treacherous turn because they had the pressure of being in a sharply competitive, resource hungry environment. This could have happened if they were EM or even aligned AGI to a subset of humanity, if they don’t solve co-ordination problems.
This kind of evolutionary problem has not been talked about for a bit (everyone seems focussed on corrigibility etc), so maybe people have forgotten? I think it worth making it explicit that that is what you need to worry about. But the question then becomes should we worry about it now or when we have cheaper intelligence and a greater understanding of how intelligences might co-ordinate?
Edit: One might even make the case we should focus our thought on short term existential risks, like avoiding nuclear war during the start of AGI, because if we don’t pass that test we won’t get to worry about super intelligence. And you can’t use the cheaper later intelligence to solve that problem.
I think there’s a difference between “Steelmanning something to learn the most you can from it” (for your own benefit), and accurately engaging with what people actually think and mean.
(For example, I think it’s common for consequentialists to “steelman” a deontological argument into consequentialism… but the actual reasons for a deontologists beliefs just have nothing to do with consequentialism)
In the case of people saying “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” and then leaving it at that, I honestly just don’t think they’re thinking about it that hard, and rounding things off to vague plausibilities without a model. (Somes they don’t just leave it at that – I think Robin Hanson generally has some kind of model, maybe closer to what you’re saying here. But in that case you can engage with whatever they’re actually saying without as much need to steelman it yourself)
(I also think my OP here is roughly as good an answer to your steelman here – the issue still remains that there doesn’t have to be a sharply treacherous turn, to result in things just eventually snowballing in a way similar to how powerful empires snowball, long after it’s too late to do anything about it)
I like arguing with myself. So it is fun to make the best case. But yup I was going beyond what people might. I think I find arguments against naive views less interesting so spice them up some.
In accelerando the participants in Economy 2.0 had a treacherous turn because they had the pressure of being in a sharply competitive, resource hungry environment. This could have happened if they were EM or even aligned AGI to a subset of humanity, if they don’t solve co-ordination problems.
This kind of evolutionary problem has not been talked about for a bit (everyone seems focussed on corrigibility etc), so maybe people have forgotten? I think it worth making it explicit that that is what you need to worry about. But the question then becomes should we worry about it now or when we have cheaper intelligence and a greater understanding of how intelligences might co-ordinate?
Edit: One might even make the case we should focus our thought on short term existential risks, like avoiding nuclear war during the start of AGI, because if we don’t pass that test we won’t get to worry about super intelligence. And you can’t use the cheaper later intelligence to solve that problem.