Presumably “curing cancer” would usually be relatively simple if you had a superintelligence and some humans—so the issue seems mostly down to the “wiping out humanity” business.
A practical superintelligence with so little instrumental concern about its origins and history that it fails to preserve some humans seems pretty unlikely to me.
The idea is that the superintelligence will preserve humans as part of a general instrumental strategy of preserving the past—provided its goal is a long-term, open-ended one.
Why would such a superintelligence care about preserving its origins? Simply because that is a critically-important clue to the form of any aliens it might meet in the alien race—and it needs to be as prepared for that as possible. Not only is it likely to preserve humans, it will probably simulate and experiment with many close variants of us.
One possible screw-up is excessive discounting—but that seems fairly simple to avoid—we already know not to build a myopic machine.
So, going by this definition of the term, friendliness seems practically automatic—and we should probably be setting our sights on something higher.
Here’s one attempt at a definition:
Presumably “curing cancer” would usually be relatively simple if you had a superintelligence and some humans—so the issue seems mostly down to the “wiping out humanity” business.
A practical superintelligence with so little instrumental concern about its origins and history that it fails to preserve some humans seems pretty unlikely to me.
The idea is that the superintelligence will preserve humans as part of a general instrumental strategy of preserving the past—provided its goal is a long-term, open-ended one.
Why would such a superintelligence care about preserving its origins? Simply because that is a critically-important clue to the form of any aliens it might meet in the alien race—and it needs to be as prepared for that as possible. Not only is it likely to preserve humans, it will probably simulate and experiment with many close variants of us.
One possible screw-up is excessive discounting—but that seems fairly simple to avoid—we already know not to build a myopic machine.
So, going by this definition of the term, friendliness seems practically automatic—and we should probably be setting our sights on something higher.