But my disagreements are mainly over details. I foresee the most likely AGI designs and goal systems as being vaguely human-like, which entails a different type of risk. Basically I’m worried about AGI’s with human inspired motivational systems taking off and taking control (peacefully/economically) or outcompeting us before we can upload in numbers, and a resulting sub-optimal amount of uploading, rather than paperclippers.
Yes, human-like AGI’s are really scary. I think a fabulous fictional treatment here is ‘Blindsight’ by Peter Watts, where humanity managed to resurrect vampires. More: Gurl ner qrcvpgrq nf angheny uhzna cerqngbef, n fhcreuhzna cflpubcnguvp Ubzb trahf jvgu zvavzny pbafpvbhfarff (zber enj cebprffvat cbjre vafgrnq) gung pna sbe rknzcyr ubyq obgu nfcrpgf bs n Arpxre phor va gurve urnqf ng gur fnzr gvzr. Uhznaf erfheerpgrq gurz jvgu n qrsvpvg gung jnf fhccbfrq gb znxr gurz pbagebyynoyr naq qrcraqrag ba gurve uhzna znfgref. Ohg bs pbhefr gung’f yvxr n zbhfr gelvat gb ubyq n png nf crg. V guvax gung abiry fubjf zber guna nal bgure yvgrengher ubj qnatrebhf whfg n yvggyr zber vagryyvtrapr pna or. Vg dhvpxyl orpbzrf pyrne gung uhznaf ner whfg yvxr yvggyr Wrjvfu tveyf snpvat n Jnssra FF fdhnqeba juvyr oryvrivat gurl’yy tb njnl vs gurl bayl pybfr gurve rlrf.
Yes, human-like AGI’s are really scary. I think a fabulous fictional treatment here is ‘Blindsight’ by Peter Watts, where humanity managed to resurrect vampires. More: Gurl ner qrcvpgrq nf angheny uhzna cerqngbef, n fhcreuhzna cflpubcnguvp Ubzb trahf jvgu zvavzny pbafpvbhfarff (zber enj cebprffvat cbjre vafgrnq) gung pna sbe rknzcyr ubyq obgu nfcrpgf bs n Arpxre phor va gurve urnqf ng gur fnzr gvzr. Uhznaf erfheerpgrq gurz jvgu n qrsvpvg gung jnf fhccbfrq gb znxr gurz pbagebyynoyr naq qrcraqrag ba gurve uhzna znfgref. Ohg bs pbhefr gung’f yvxr n zbhfr gelvat gb ubyq n png nf crg. V guvax gung abiry fubjf zber guna nal bgure yvgrengher ubj qnatrebhf whfg n yvggyr zber vagryyvtrapr pna or. Vg dhvpxyl orpbzrf pyrne gung uhznaf ner whfg yvxr yvggyr Wrjvfu tveyf snpvat n Jnssra FF fdhnqeba juvyr oryvrivat gurl’yy tb njnl vs gurl bayl pybfr gurve rlrf.
That fictional treatment is interesting to the point of me actually looking up the book. But ..
The future is scary. Human-like AGI’s should not intrinsically be more scary than the future, accelerated.