you implicitly assume that it is given the incentive to develop the cognitive flexibility and comprehension to act in a real-world environment and do those things but at the same time you propose that the same people who are capable of giving it such extensive urges fail on another goal in such a blatant and obvious way. How does that make sense?
Because people make stupid mistakes, especially when programming. And telling your fully-programmed AI what you want it to do still counts as programming.
At this point, I am going to stop my reply, because the remainder of your comment consists of taking things I said out of context and turning them into irrelevancies:
I didn’t say an AI would try to convert people to Christianity—I said that humans without sufficient shared background will interpret things literally, and so would AIs.
I didn’t say the AI needed to be creative or smart, I said you wouldn’t need to be creative or smart to make a list of ways those three simple instructions could be given a disastrous literal interpretation.
you seem to believe that there are many pathways that lead to superhuman abilities yet all of them fail to comprehend some goals while still being able to self-improve on them.
There are many paths to superhuman ability, as humans really aren’t that smart.
This also means that you can easily be superhuman in ability, and still really dumb—in terms of comprehending what humans mean… but don’t actually say.
Because people make stupid mistakes, especially when programming. And telling your fully-programmed AI what you want it to do still counts as programming.
At this point, I am going to stop my reply, because the remainder of your comment consists of taking things I said out of context and turning them into irrelevancies:
I didn’t say an AI would try to convert people to Christianity—I said that humans without sufficient shared background will interpret things literally, and so would AIs.
I didn’t say the AI needed to be creative or smart, I said you wouldn’t need to be creative or smart to make a list of ways those three simple instructions could be given a disastrous literal interpretation.
There are many paths to superhuman ability, as humans really aren’t that smart.
This also means that you can easily be superhuman in ability, and still really dumb—in terms of comprehending what humans mean… but don’t actually say.