You might be able to recognize the right solution (e.g. a theory) when you see it, while unable to generate it yourself (as fast). If you are sensitive enough to attempts of UFAI to confuse you into doing the wrong thing, just going with the deal may be the best option for it.
First of all, the “won’t build it” option does not make any sense. It is not like the UFAI is going to do anything before it exists.
You decide whether to build something new before it exists, based on its properties. AI’s decisions are such properties.
If you are sensitive enough to attempts of UFAI to confuse you into doing the wrong thing, just going with the deal may be the best option for it.
And if we are not sensitive enough, our molecules get reused for paperclips. You are talking about matching wits with something that is orders of magnitude smarter than us, thinks orders of magnitude faster than us, has a detailed model of our minds, and doesn’t think we are worth the utilons it can build out of our quarks. Yes, we would think we recognize the right solution, that it is obvious now that it’s been pointed out, and we would be wrong. An argument than a UFAI figured out would be persuasive to us is nowhere near as trustworthy as an argument we figure out ourselves. While we can be wrong about our own arguments, the UFAI will present arguments that we will be systematically wrong about in very dangerous ways.
You decide whether to build something based on its properties before it exists.
And for our next trick, we will just ask Omega to build an FAI for us.
No AI, friendly or unfriendly, will ever have a model of our minds as detailed as the model we have of its mind, because we can pause it and inspect its source code while it can’t do anything analogous to us.
I have written plenty of mere desktop applications that are a major pain for a human mind to understand in a debugger. And have you ever written programs that generate machine code for another program? And then tried to inspect that machine code when something went wrong to figure out why?
Well, that stuff is nothing compared to the difficulty of debugging or otherwise understanding an AI, even given full access to inspect its implementation. An unfriendly AI is likely to be built by throwing lots of parallel hardware together until something sticks. If the designers actually knew what they were doing, they would figure out they need to make it friendly. So, yes, you could pause the whole thing, and look at how the billions of CPU’s are interconnected, and the state of the terabytes of memory, but you are not logically omniscient, and you will not make sense of it.
It’s not like you could just inspect the evil bit.
You might be able to recognize the right solution (e.g. a theory) when you see it, while unable to generate it yourself (as fast). If you are sensitive enough to attempts of UFAI to confuse you into doing the wrong thing, just going with the deal may be the best option for it.
You decide whether to build something new before it exists, based on its properties. AI’s decisions are such properties.
And if we are not sensitive enough, our molecules get reused for paperclips. You are talking about matching wits with something that is orders of magnitude smarter than us, thinks orders of magnitude faster than us, has a detailed model of our minds, and doesn’t think we are worth the utilons it can build out of our quarks. Yes, we would think we recognize the right solution, that it is obvious now that it’s been pointed out, and we would be wrong. An argument than a UFAI figured out would be persuasive to us is nowhere near as trustworthy as an argument we figure out ourselves. While we can be wrong about our own arguments, the UFAI will present arguments that we will be systematically wrong about in very dangerous ways.
And for our next trick, we will just ask Omega to build an FAI for us.
No AI, friendly or unfriendly, will ever have a model of our minds as detailed as the model we have of its mind, because we can pause it and inspect its source code while it can’t do anything analogous to us.
I have written plenty of mere desktop applications that are a major pain for a human mind to understand in a debugger. And have you ever written programs that generate machine code for another program? And then tried to inspect that machine code when something went wrong to figure out why?
Well, that stuff is nothing compared to the difficulty of debugging or otherwise understanding an AI, even given full access to inspect its implementation. An unfriendly AI is likely to be built by throwing lots of parallel hardware together until something sticks. If the designers actually knew what they were doing, they would figure out they need to make it friendly. So, yes, you could pause the whole thing, and look at how the billions of CPU’s are interconnected, and the state of the terabytes of memory, but you are not logically omniscient, and you will not make sense of it.
It’s not like you could just inspect the evil bit.