I feel that this post is straw-manning “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” a bit.
A steel man might be,
I don’t feel super intelligence is worth worrying at this point, as in a soft takeoff scenario we will have lots of small AGI related accidents (people wire heading themselves with AI). This will provide both financial incentives to companies to concentrate of safety to stop themselves getting sued and if they are using it themselves, stopping the damages caused by it to themselves. It will also provide government incentives to introduce regulation to make them safe, from political pressure. AGI Scientists on the cusp of creating AGI have incentives to not be associated with the bad consequences of AGI, they are also on the best position to understand what safe guards are needed.
Also there will be a general selective pressure towards safe AGI as we would destroy the unaligned ones with the safer/most alignable ones. There is no reason to expect a treacherous turn when the machines get to a decisive strategic advantage, as we will have seen treacherous behaviour in AGIs that are not super rational or good at hiding their treachery and then designed against it.
It is only when there is the chance of foom do we the current generation need to worry about super intelligence right now.
As such it would be better to save money now and use the componded interest to then buy safer AGI from safety focused AGI companies to distribute to needy people. The safety focused company will have greater knowledge of AGI and be able to make more a lot more AGI safety for the dollar than we currently can with our knowledge.
If you want to make AGI, then worrying about the super intelligence case is probably a good exercise in seeing where the cracks are in your system to avoid the small accidents.
I’m not sure I believe it. But it is worth seeing that incentives for safety are there.
I think there’s a difference between “Steelmanning something to learn the most you can from it” (for your own benefit), and accurately engaging with what people actually think and mean.
(For example, I think it’s common for consequentialists to “steelman” a deontological argument into consequentialism… but the actual reasons for a deontologists beliefs just have nothing to do with consequentialism)
In the case of people saying “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” and then leaving it at that, I honestly justdon’t think they’re thinking about it that hard, and rounding things off to vague plausibilities without a model. (Somes they don’t just leave it at that – I think Robin Hanson generally has some kind of model, maybe closer to what you’re saying here. But in that case you can engage with whatever they’re actually saying without as much need to steelman it yourself)
(I also think my OP here is roughly as good an answer to your steelman here – the issue still remains that there doesn’t have to be a sharply treacherous turn, to result in things just eventually snowballing in a way similar to how powerful empires snowball, long after it’s too late to do anything about it)
I like arguing with myself. So it is fun to make the best case. But yup I was going beyond what people might. I think I find arguments against naive views less interesting so spice them up some.
In accelerando the participants in Economy 2.0 had a treacherous turn because they had the pressure of being in a sharply competitive, resource hungry environment. This could have happened if they were EM or even aligned AGI to a subset of humanity, if they don’t solve co-ordination problems.
This kind of evolutionary problem has not been talked about for a bit (everyone seems focussed on corrigibility etc), so maybe people have forgotten? I think it worth making it explicit that that is what you need to worry about. But the question then becomes should we worry about it now or when we have cheaper intelligence and a greater understanding of how intelligences might co-ordinate?
Edit: One might even make the case we should focus our thought on short term existential risks, like avoiding nuclear war during the start of AGI, because if we don’t pass that test we won’t get to worry about super intelligence. And you can’t use the cheaper later intelligence to solve that problem.
I feel that this post is straw-manning “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” a bit.
A steel man might be,
I don’t feel super intelligence is worth worrying at this point, as in a soft takeoff scenario we will have lots of small AGI related accidents (people wire heading themselves with AI). This will provide both financial incentives to companies to concentrate of safety to stop themselves getting sued and if they are using it themselves, stopping the damages caused by it to themselves. It will also provide government incentives to introduce regulation to make them safe, from political pressure. AGI Scientists on the cusp of creating AGI have incentives to not be associated with the bad consequences of AGI, they are also on the best position to understand what safe guards are needed.
Also there will be a general selective pressure towards safe AGI as we would destroy the unaligned ones with the safer/most alignable ones. There is no reason to expect a treacherous turn when the machines get to a decisive strategic advantage, as we will have seen treacherous behaviour in AGIs that are not super rational or good at hiding their treachery and then designed against it.
It is only when there is the chance of foom do we the current generation need to worry about super intelligence right now.
As such it would be better to save money now and use the componded interest to then buy safer AGI from safety focused AGI companies to distribute to needy people. The safety focused company will have greater knowledge of AGI and be able to make more a lot more AGI safety for the dollar than we currently can with our knowledge.
If you want to make AGI, then worrying about the super intelligence case is probably a good exercise in seeing where the cracks are in your system to avoid the small accidents.
I’m not sure I believe it. But it is worth seeing that incentives for safety are there.
I think there’s a difference between “Steelmanning something to learn the most you can from it” (for your own benefit), and accurately engaging with what people actually think and mean.
(For example, I think it’s common for consequentialists to “steelman” a deontological argument into consequentialism… but the actual reasons for a deontologists beliefs just have nothing to do with consequentialism)
In the case of people saying “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” and then leaving it at that, I honestly just don’t think they’re thinking about it that hard, and rounding things off to vague plausibilities without a model. (Somes they don’t just leave it at that – I think Robin Hanson generally has some kind of model, maybe closer to what you’re saying here. But in that case you can engage with whatever they’re actually saying without as much need to steelman it yourself)
(I also think my OP here is roughly as good an answer to your steelman here – the issue still remains that there doesn’t have to be a sharply treacherous turn, to result in things just eventually snowballing in a way similar to how powerful empires snowball, long after it’s too late to do anything about it)
I like arguing with myself. So it is fun to make the best case. But yup I was going beyond what people might. I think I find arguments against naive views less interesting so spice them up some.
In accelerando the participants in Economy 2.0 had a treacherous turn because they had the pressure of being in a sharply competitive, resource hungry environment. This could have happened if they were EM or even aligned AGI to a subset of humanity, if they don’t solve co-ordination problems.
This kind of evolutionary problem has not been talked about for a bit (everyone seems focussed on corrigibility etc), so maybe people have forgotten? I think it worth making it explicit that that is what you need to worry about. But the question then becomes should we worry about it now or when we have cheaper intelligence and a greater understanding of how intelligences might co-ordinate?
Edit: One might even make the case we should focus our thought on short term existential risks, like avoiding nuclear war during the start of AGI, because if we don’t pass that test we won’t get to worry about super intelligence. And you can’t use the cheaper later intelligence to solve that problem.