Nick that’s an interesting thought, but it is possible that someone does know how to build a friendly AI, but may not know that he knows this. What I mean is that a person may be working on something unrelated to AI or is not interested in AI, but knows all the ingredients and technical things needed to make a friendly AI. Maybe he just doesn’t know the implications of his own knowledge.
I think it’s fair to say many scientists fall into this category, if only due to specialization. Maybe some inorganic chemist knows the cure to cancer. But simply due to his ignorance/uninterest in biochemistry or organic chemistry or whatever subject that’s relevant is unaware of the implications of his own specialized knowledge on topics outside his area of expertise.
Eliezer,
“[L]ogical ignorance is still ignorance; if this were not so, I would know a proof of the Riemann Hypothesis.” I understand what you mean by logical ignorance is still ignorance; as for the second part of your statement, I don’t think that’s necessarily true.
For example, if the RH were true you would need n statements in a particular order. Meaning that you have stated assumptions, then you proceed to S1->S2->...->Sn (where Si is the ith logical statement) and Sn is the conclusion of the proof. What you are saying is that you know all n statements, but not the order and you’re trying to figure that out or possibly could if you thought about it. BUT, you may not know all n statements, which is what the second part of your statement claims. Your statement, also, assumes the RH is true, which is what the fuss is all about. No one really knows if it’s true.
What I was trying to say is that you may know all n statements, even their order and have come up with them to solve some other problem. But you may never have heard of the RH, so don’t know that you can solve it. I understand that this is still a form logical ignorance, but the second part of your statement (that you know a proof of the RH) doesn’t follow from the first part.