Imagine a man who really likes the color green. Maybe he’s obsessed with it, to the exclusion of everything else, at a pathological extreme. This man doesn’t seem too dangerous to us. However, what if the man were a genius? Then, due to his bizarre preferences, he becomes dangerous. Maybe he wants to turn the entire sky green, so he invents a smog generator that blots out the sun. Maybe he wants to turn people green, so he engineers a bioweapon that turns its victims green. High intelligence, plus a simple goal, is a recipe for disaster. Right now, companies and governments are racing to build AI programs, smarter than any human genius… dedicated to simple goals, like maximizing some metric. Without safeguards in place, super-human AI would be as dangerous as the man obsessed with green. (Policymakers)
Imagine a man who really likes the color green. Maybe he’s obsessed with it, to the exclusion of everything else, at a pathological extreme. This man doesn’t seem too dangerous to us. However, what if the man were a genius? Then, due to his bizarre preferences, he becomes dangerous. Maybe he wants to turn the entire sky green, so he invents a smog generator that blots out the sun. Maybe he wants to turn people green, so he engineers a bioweapon that turns its victims green. High intelligence, plus a simple goal, is a recipe for disaster. Right now, companies and governments are racing to build AI programs, smarter than any human genius… dedicated to simple goals, like maximizing some metric. Without safeguards in place, super-human AI would be as dangerous as the man obsessed with green. (Policymakers)