You think building a machine that can be stopped is the same level of difficulty as building a machine that reflects the desires of one or more humans while it is left on?
I beg to differ—stopping on schedule or on demand is one of the simplest possible problems for a machine—while doing what humans want you to do while you are switched on is much trickier.
Only the former problem needs to be solved to eliminate the spectre of a runaway superintelligence that fills the universe with its idea of utility against the wishes of its creator.
You think building a machine that can be stopped is the same level of difficulty as building a machine that reflects the desires of one or more humans while it is left on?
I beg to differ—stopping on schedule or on demand is one of the simplest possible problems for a machine—while doing what humans want you to do while you are switched on is much trickier.
Only the former problem needs to be solved to eliminate the spectre of a runaway superintelligence that fills the universe with its idea of utility against the wishes of its creator.
Beware simple seeming wishes.