This was primarily a response to an in person conversation, but it was also (in part), an answer to Calvin Ho on the “Taking AI Seriously” thread. They said:
I read the sequences and superintelligence, but I still don’t see how an AI would proliferate and advance faster than our ability to kill it—a year to get from baby to einstein level intelligence is plenty long enough to react.
And I guess I’ll take this slot to answer them directly:
This post isn’t precisely an answer to this question, but points at how you could get an AI who looked pretty safe, and that honestly was pretty safe – as safe as an empathetic human who makes a reasonable effort to avoid killing bugs – and so during the year when you could have done something, it didn’t look like you needed to. And then a couple decades later you find that everything is computronium and only minds that are optimized for controlling the solar system get to control the solar system.
Ah, makes sense. I saw something on facebook by Robert Wiblin arguing against unnamed people in the “evidence-based optimist” group. And thought I was missing something important going on, for both you and cousin_it to react to. You have not been vocal on take off scenarios before. But it seems it is just conincidence.
Thanks for the post, it was a viewpoint I hadn’t closely considered (that a friendly but “technically unsafe” AI would be the singularity and its lack of safety would not be addressed in time due to its benefits) and is worth thinking about more.
This was primarily a response to an in person conversation, but it was also (in part), an answer to Calvin Ho on the “Taking AI Seriously” thread. They said:
And I guess I’ll take this slot to answer them directly:
This post isn’t precisely an answer to this question, but points at how you could get an AI who looked pretty safe, and that honestly was pretty safe – as safe as an empathetic human who makes a reasonable effort to avoid killing bugs – and so during the year when you could have done something, it didn’t look like you needed to. And then a couple decades later you find that everything is computronium and only minds that are optimized for controlling the solar system get to control the solar system.
Ah, makes sense. I saw something on facebook by Robert Wiblin arguing against unnamed people in the “evidence-based optimist” group. And thought I was missing something important going on, for both you and cousin_it to react to. You have not been vocal on take off scenarios before. But it seems it is just conincidence.
Thanks for the explanation.
Thanks for the post, it was a viewpoint I hadn’t closely considered (that a friendly but “technically unsafe” AI would be the singularity and its lack of safety would not be addressed in time due to its benefits) and is worth thinking about more.