A common misconception is that STEM-level AGI is dangerous because of something murky about “agents” or about self-awareness. Instead, I’d say that the danger is inherent to the nature of action sequences that push the world toward some sufficiently-hard-to-reach state.[8]
Call such sequences “plans”.
If you sampled a random plan from the space of all writable plans (weighted by length, in any extant formal language), and all we knew about the plan is that executing it would successfully achieve some superhumanly ambitious technological goal like “invent fast-running whole-brain emulation”, then hitting a button to execute the plan would kill all humans, with very high probability.
Many plans have been executed, and none have Killed All Humans, so far. In fact, when humans executed a plan to build the most destructive weapon in history, they carefully checked that it wouldn’t ignite the atmosphere and kill everybody.
I wouldn’t expect a mixed group of humans and slightly-above-human AIs , with the usual reviews and checks that go into science , to be much more dangerous than all human science.
So where’s the problem? There’s a hint here:
then hitting a button to execute the plan
If you envision an all-in-one science ASI, that plans the research and also executes it in some kind of automated lab, without any reviews or checks...pushbutton science....that could be dangerous. But for the rather uninteresting reason that you have removed everything that makes human science safe.
Many plans have been executed, and none have Killed All Humans, so far. In fact, when humans executed a plan to build the most destructive weapon in history, they carefully checked that it wouldn’t ignite the atmosphere and kill everybody.
I wouldn’t expect a mixed group of humans and slightly-above-human AIs , with the usual reviews and checks that go into science , to be much more dangerous than all human science.
So where’s the problem? There’s a hint here:
If you envision an all-in-one science ASI, that plans the research and also executes it in some kind of automated lab, without any reviews or checks...pushbutton science....that could be dangerous. But for the rather uninteresting reason that you have removed everything that makes human science safe.