I have to say I am a little puzzled. I’m not sure who you and cousin_it are talking to with these moderate take off posts. I don’t see anyone arguing that a moderate take off would be okay by default.
Even more mainstream places like mit, seem to be saying it is too early to focus on AI safety, rather than never focus on AI safety. I hope that there would conversation around when to focus on AI safety. While there is no default fire alarm it doesn’t mean you can’t construct one. Get people working on AGI science to say what they expect their creations to be capable of and formulate a plan for what to do if it is vastly more capabale than they expect.
This was primarily a response to an in person conversation, but it was also (in part), an answer to Calvin Ho on the “Taking AI Seriously” thread. They said:
I read the sequences and superintelligence, but I still don’t see how an AI would proliferate and advance faster than our ability to kill it—a year to get from baby to einstein level intelligence is plenty long enough to react.
And I guess I’ll take this slot to answer them directly:
This post isn’t precisely an answer to this question, but points at how you could get an AI who looked pretty safe, and that honestly was pretty safe – as safe as an empathetic human who makes a reasonable effort to avoid killing bugs – and so during the year when you could have done something, it didn’t look like you needed to. And then a couple decades later you find that everything is computronium and only minds that are optimized for controlling the solar system get to control the solar system.
Ah, makes sense. I saw something on facebook by Robert Wiblin arguing against unnamed people in the “evidence-based optimist” group. And thought I was missing something important going on, for both you and cousin_it to react to. You have not been vocal on take off scenarios before. But it seems it is just conincidence.
Thanks for the post, it was a viewpoint I hadn’t closely considered (that a friendly but “technically unsafe” AI would be the singularity and its lack of safety would not be addressed in time due to its benefits) and is worth thinking about more.
Also, cousin_it is specifically talking to RobinHanson, so as far as “who are we talking to”, anyone who takes him seriously. (Although Robin Hanson also has some additional things going on like “the Accelerando scenario seems good [or something?”]. I’m not sure I understand that, it’s just my vague impression)
Like Raemon’s comment suggests, people don’t necessarily say it explicitly, but it’s implicit whenever someone just says something along the lines of “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic”, and just leaves it at that. And I at least have seen a lot of people say something like that.
I feel that this post is straw-manning “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” a bit.
A steel man might be,
I don’t feel super intelligence is worth worrying at this point, as in a soft takeoff scenario we will have lots of small AGI related accidents (people wire heading themselves with AI). This will provide both financial incentives to companies to concentrate of safety to stop themselves getting sued and if they are using it themselves, stopping the damages caused by it to themselves. It will also provide government incentives to introduce regulation to make them safe, from political pressure. AGI Scientists on the cusp of creating AGI have incentives to not be associated with the bad consequences of AGI, they are also on the best position to understand what safe guards are needed.
Also there will be a general selective pressure towards safe AGI as we would destroy the unaligned ones with the safer/most alignable ones. There is no reason to expect a treacherous turn when the machines get to a decisive strategic advantage, as we will have seen treacherous behaviour in AGIs that are not super rational or good at hiding their treachery and then designed against it.
It is only when there is the chance of foom do we the current generation need to worry about super intelligence right now.
As such it would be better to save money now and use the componded interest to then buy safer AGI from safety focused AGI companies to distribute to needy people. The safety focused company will have greater knowledge of AGI and be able to make more a lot more AGI safety for the dollar than we currently can with our knowledge.
If you want to make AGI, then worrying about the super intelligence case is probably a good exercise in seeing where the cracks are in your system to avoid the small accidents.
I’m not sure I believe it. But it is worth seeing that incentives for safety are there.
I think there’s a difference between “Steelmanning something to learn the most you can from it” (for your own benefit), and accurately engaging with what people actually think and mean.
(For example, I think it’s common for consequentialists to “steelman” a deontological argument into consequentialism… but the actual reasons for a deontologists beliefs just have nothing to do with consequentialism)
In the case of people saying “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” and then leaving it at that, I honestly justdon’t think they’re thinking about it that hard, and rounding things off to vague plausibilities without a model. (Somes they don’t just leave it at that – I think Robin Hanson generally has some kind of model, maybe closer to what you’re saying here. But in that case you can engage with whatever they’re actually saying without as much need to steelman it yourself)
(I also think my OP here is roughly as good an answer to your steelman here – the issue still remains that there doesn’t have to be a sharply treacherous turn, to result in things just eventually snowballing in a way similar to how powerful empires snowball, long after it’s too late to do anything about it)
I like arguing with myself. So it is fun to make the best case. But yup I was going beyond what people might. I think I find arguments against naive views less interesting so spice them up some.
In accelerando the participants in Economy 2.0 had a treacherous turn because they had the pressure of being in a sharply competitive, resource hungry environment. This could have happened if they were EM or even aligned AGI to a subset of humanity, if they don’t solve co-ordination problems.
This kind of evolutionary problem has not been talked about for a bit (everyone seems focussed on corrigibility etc), so maybe people have forgotten? I think it worth making it explicit that that is what you need to worry about. But the question then becomes should we worry about it now or when we have cheaper intelligence and a greater understanding of how intelligences might co-ordinate?
Edit: One might even make the case we should focus our thought on short term existential risks, like avoiding nuclear war during the start of AGI, because if we don’t pass that test we won’t get to worry about super intelligence. And you can’t use the cheaper later intelligence to solve that problem.
I have to say I am a little puzzled. I’m not sure who you and cousin_it are talking to with these moderate take off posts. I don’t see anyone arguing that a moderate take off would be okay by default.
Even more mainstream places like mit, seem to be saying it is too early to focus on AI safety, rather than never focus on AI safety. I hope that there would conversation around when to focus on AI safety. While there is no default fire alarm it doesn’t mean you can’t construct one. Get people working on AGI science to say what they expect their creations to be capable of and formulate a plan for what to do if it is vastly more capabale than they expect.
This was primarily a response to an in person conversation, but it was also (in part), an answer to Calvin Ho on the “Taking AI Seriously” thread. They said:
And I guess I’ll take this slot to answer them directly:
This post isn’t precisely an answer to this question, but points at how you could get an AI who looked pretty safe, and that honestly was pretty safe – as safe as an empathetic human who makes a reasonable effort to avoid killing bugs – and so during the year when you could have done something, it didn’t look like you needed to. And then a couple decades later you find that everything is computronium and only minds that are optimized for controlling the solar system get to control the solar system.
Ah, makes sense. I saw something on facebook by Robert Wiblin arguing against unnamed people in the “evidence-based optimist” group. And thought I was missing something important going on, for both you and cousin_it to react to. You have not been vocal on take off scenarios before. But it seems it is just conincidence.
Thanks for the explanation.
Thanks for the post, it was a viewpoint I hadn’t closely considered (that a friendly but “technically unsafe” AI would be the singularity and its lack of safety would not be addressed in time due to its benefits) and is worth thinking about more.
Also, cousin_it is specifically talking to Robin Hanson, so as far as “who are we talking to”, anyone who takes him seriously. (Although Robin Hanson also has some additional things going on like “the Accelerando scenario seems good [or something?”]. I’m not sure I understand that, it’s just my vague impression)
Like Raemon’s comment suggests, people don’t necessarily say it explicitly, but it’s implicit whenever someone just says something along the lines of “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic”, and just leaves it at that. And I at least have seen a lot of people say something like that.
I feel that this post is straw-manning “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” a bit.
A steel man might be,
I don’t feel super intelligence is worth worrying at this point, as in a soft takeoff scenario we will have lots of small AGI related accidents (people wire heading themselves with AI). This will provide both financial incentives to companies to concentrate of safety to stop themselves getting sued and if they are using it themselves, stopping the damages caused by it to themselves. It will also provide government incentives to introduce regulation to make them safe, from political pressure. AGI Scientists on the cusp of creating AGI have incentives to not be associated with the bad consequences of AGI, they are also on the best position to understand what safe guards are needed.
Also there will be a general selective pressure towards safe AGI as we would destroy the unaligned ones with the safer/most alignable ones. There is no reason to expect a treacherous turn when the machines get to a decisive strategic advantage, as we will have seen treacherous behaviour in AGIs that are not super rational or good at hiding their treachery and then designed against it.
It is only when there is the chance of foom do we the current generation need to worry about super intelligence right now.
As such it would be better to save money now and use the componded interest to then buy safer AGI from safety focused AGI companies to distribute to needy people. The safety focused company will have greater knowledge of AGI and be able to make more a lot more AGI safety for the dollar than we currently can with our knowledge.
If you want to make AGI, then worrying about the super intelligence case is probably a good exercise in seeing where the cracks are in your system to avoid the small accidents.
I’m not sure I believe it. But it is worth seeing that incentives for safety are there.
I think there’s a difference between “Steelmanning something to learn the most you can from it” (for your own benefit), and accurately engaging with what people actually think and mean.
(For example, I think it’s common for consequentialists to “steelman” a deontological argument into consequentialism… but the actual reasons for a deontologists beliefs just have nothing to do with consequentialism)
In the case of people saying “I don’t think superintelligence is worth worrying about because I don’t think that a hard takeoff is realistic” and then leaving it at that, I honestly just don’t think they’re thinking about it that hard, and rounding things off to vague plausibilities without a model. (Somes they don’t just leave it at that – I think Robin Hanson generally has some kind of model, maybe closer to what you’re saying here. But in that case you can engage with whatever they’re actually saying without as much need to steelman it yourself)
(I also think my OP here is roughly as good an answer to your steelman here – the issue still remains that there doesn’t have to be a sharply treacherous turn, to result in things just eventually snowballing in a way similar to how powerful empires snowball, long after it’s too late to do anything about it)
I like arguing with myself. So it is fun to make the best case. But yup I was going beyond what people might. I think I find arguments against naive views less interesting so spice them up some.
In accelerando the participants in Economy 2.0 had a treacherous turn because they had the pressure of being in a sharply competitive, resource hungry environment. This could have happened if they were EM or even aligned AGI to a subset of humanity, if they don’t solve co-ordination problems.
This kind of evolutionary problem has not been talked about for a bit (everyone seems focussed on corrigibility etc), so maybe people have forgotten? I think it worth making it explicit that that is what you need to worry about. But the question then becomes should we worry about it now or when we have cheaper intelligence and a greater understanding of how intelligences might co-ordinate?
Edit: One might even make the case we should focus our thought on short term existential risks, like avoiding nuclear war during the start of AGI, because if we don’t pass that test we won’t get to worry about super intelligence. And you can’t use the cheaper later intelligence to solve that problem.