I have written plenty of mere desktop applications that are a major pain for a human mind to understand in a debugger. And have you ever written programs that generate machine code for another program? And then tried to inspect that machine code when something went wrong to figure out why?
Well, that stuff is nothing compared to the difficulty of debugging or otherwise understanding an AI, even given full access to inspect its implementation. An unfriendly AI is likely to be built by throwing lots of parallel hardware together until something sticks. If the designers actually knew what they were doing, they would figure out they need to make it friendly. So, yes, you could pause the whole thing, and look at how the billions of CPU’s are interconnected, and the state of the terabytes of memory, but you are not logically omniscient, and you will not make sense of it.
It’s not like you could just inspect the evil bit.
I have written plenty of mere desktop applications that are a major pain for a human mind to understand in a debugger. And have you ever written programs that generate machine code for another program? And then tried to inspect that machine code when something went wrong to figure out why?
Well, that stuff is nothing compared to the difficulty of debugging or otherwise understanding an AI, even given full access to inspect its implementation. An unfriendly AI is likely to be built by throwing lots of parallel hardware together until something sticks. If the designers actually knew what they were doing, they would figure out they need to make it friendly. So, yes, you could pause the whole thing, and look at how the billions of CPU’s are interconnected, and the state of the terabytes of memory, but you are not logically omniscient, and you will not make sense of it.
It’s not like you could just inspect the evil bit.