Software that initially appears to care what you mean will be selected by market forces. But nearly all software that superficially looks Friendly isn’t Friendly. If there are seasoned AI researchers who can’t wrap their heads around the five theses, then how can I be confident that the Invisible Hand will both surpass them intellectually and recurrently sacrifice short-term gains on this basis?
Software that looks friendly isn’t really friendly in the sense that it really understands what we want. It isn’t dangerously unfriendly because we’re still here. If its commercially successful, it’s friendly enough for us to want it in our lives.
Human beings aren’t friendly, in the Friendly-AI sense. If a random human acquired immense power, it would probably result in an existential catastrophe. Humans do have a better sense of human value than, say, a can-opener does; they have more power and autonomy than a can-opener, so they need fuller access to human values in order to reach similar safety levels. A superintelligent AI would require even more access to human values to reach comparable safety levels.
If you grafted absolute power onto a human with average ethical insight, you might get absolute corruption. But what is that analogous to in .AI terms? Why assume asymmetric development by default?
If you assume top down singleton AI with a walled of ethics module, things look difficult. If you reverse this assumptions, FAI is already happening.
Software that cares what you mean will be selected for by market forces.
Software that initially appears to care what you mean will be selected by market forces. But nearly all software that superficially looks Friendly isn’t Friendly. If there are seasoned AI researchers who can’t wrap their heads around the five theses, then how can I be confident that the Invisible Hand will both surpass them intellectually and recurrently sacrifice short-term gains on this basis?
Software that looks friendly isn’t really friendly in the sense that it really understands what we want. It isn’t dangerously unfriendly because we’re still here. If its commercially successful, it’s friendly enough for us to want it in our lives.
Human beings aren’t friendly, in the Friendly-AI sense. If a random human acquired immense power, it would probably result in an existential catastrophe. Humans do have a better sense of human value than, say, a can-opener does; they have more power and autonomy than a can-opener, so they need fuller access to human values in order to reach similar safety levels. A superintelligent AI would require even more access to human values to reach comparable safety levels.
There is more than one sense to friendly .AI.
If you grafted absolute power onto a human with average ethical insight, you might get absolute corruption. But what is that analogous to in .AI terms? Why assume asymmetric development by default?
If you assume top down singleton AI with a walled of ethics module, things look difficult. If you reverse this assumptions, FAI is already happening.