But how could a seed AI be able to make itself superhuman powerful if it did not care about avoiding mistakes such as autocoreccting “meditating” to “masturbating”?
As Robb said you’re confusing mistake in the sense of “The program is doing something we don’t want to do” with mistake in the sense of “The program has wrong beliefs about reality”.
I suppose a different way of thinking about these is “A mistaken human belief about the program” vs “A mistaken computer belief about the human”. We keep talking about the former (the program does something we didn’t know it would do), and you keep treating it as if it’s the latter.
Let’s say we have a program (not an AI, just a program) which uses Newton’s laws in order to calculate the trajectory of a ball. We want it to calculate this in order to have it move a tennis racket and hit the ball back.
When it finally runs, we observe that the program always avoids the ball rather than hit it back. Is it because it’s calculating the trajectory of the ball wrongly? No, it calculates the trajectory very well indeed, it’s just that an instruction in the program was wrongly inserted so that the end result is “DO NOT hit the ball back”.
It knows what the “trajectory of the ball” is. It knows what “hit the ball” is. But it’s program is “DO NOT hit the ball” rather than “hit the ball”. Why? Because of a human mistaken belief on what the program would do, not the program’s mistaken belief.
As Robb said you’re confusing mistake in the sense of “The program is doing something we don’t want to do” with mistake in the sense of “The program has wrong beliefs about reality”.
I suppose a different way of thinking about these is “A mistaken human belief about the program” vs “A mistaken computer belief about the human”. We keep talking about the former (the program does something we didn’t know it would do), and you keep treating it as if it’s the latter.
Let’s say we have a program (not an AI, just a program) which uses Newton’s laws in order to calculate the trajectory of a ball. We want it to calculate this in order to have it move a tennis racket and hit the ball back. When it finally runs, we observe that the program always avoids the ball rather than hit it back. Is it because it’s calculating the trajectory of the ball wrongly? No, it calculates the trajectory very well indeed, it’s just that an instruction in the program was wrongly inserted so that the end result is “DO NOT hit the ball back”.
It knows what the “trajectory of the ball” is. It knows what “hit the ball” is. But it’s program is “DO NOT hit the ball” rather than “hit the ball”. Why? Because of a human mistaken belief on what the program would do, not the program’s mistaken belief.
And you are confusing self-improving AIs with conventional programmes.