(3) What happens when it fails to behave in accordance with human intention?
NAI: It crashes, freezes or halts. It generally fails in a way that is harmful to its own functioning. If for example an autonomous car fails at driving autonomously it usually means that it will either go into safe-mode and halt or crash.
GAI: It works perfectly well. Superhumanly well. All its intended capabilities are intact except that it completely fails at working as intended in such a way as to destroy all human value in the universe. It will be able to improve itself and capable of obtaining a perfect encoding of human values. It will use those intended capabilities in order to deceive and overpower humans rather than doing what it was intended to do.
Firstly, “fails in a way that is harmful to its own functioning” appears to be tautological.
Secondly, you seem to be listing things that apply to any kind of AI in the NAI section—is this intentional? (This happens throughout your comment, in fact.)
Two points:
Firstly, “fails in a way that is harmful to its own functioning” appears to be tautological.
Secondly, you seem to be listing things that apply to any kind of AI in the NAI section—is this intentional? (This happens throughout your comment, in fact.)