superhuman with dumb mistakes’ − 4 brilliant games, one stupid loose.
dumb with some superhuman skills—dumb in one game, unbeatble in another.
better at some things and worse at others—different performance in different domains.
I think that if superhuman AI with bugs will start to self-improve, the bugs will start to accumulate. This will ruin or AIs power, or AIs goal system. The first is good and the second is bad. I also could suggest that first AI which will try to self improve will still have some bugs. The open question is if AI will be able to debug itself.
Some bugs may prevent seeing them as bugs, so they are reccurent. The closest thing is human bias of overconfidence. Overconfident human can’t understand that there is something wrong with him.
I think the difference here is distribution.
superhuman with dumb mistakes’ − 4 brilliant games, one stupid loose.
dumb with some superhuman skills—dumb in one game, unbeatble in another.
better at some things and worse at others—different performance in different domains.
I think that if superhuman AI with bugs will start to self-improve, the bugs will start to accumulate. This will ruin or AIs power, or AIs goal system. The first is good and the second is bad. I also could suggest that first AI which will try to self improve will still have some bugs. The open question is if AI will be able to debug itself. Some bugs may prevent seeing them as bugs, so they are reccurent. The closest thing is human bias of overconfidence. Overconfident human can’t understand that there is something wrong with him.