Any answer other than “create a superintelligent friendly AI to optimize the universe” would be a waste of this particular genie, but there are some steps in between that and 3-SAT which I can’t specify yet.
Your idea seems to be a general method for solving unformalized problems using an NP-solver: generate phrases that an uploaded human would classify as insights. I’m still afraid that many such phrases will be mindhacks instead of genuine insights, though, and the risk gets worse as the problem gets harder (or more precisely, as the next inferential step becomes harder to hit).
I agree it’s still risky, but with the safety features I put in (having a small limit on the phrase length, outputting the top 100,000 (considered individually) in random order instead of just the top 1, review/discussion by a team, and we can also mix together the top 10,000 insights from each of 10 uploads for additional safety) it seems no worse than just having humans try to solve the problem by thinking about it, since we could also come up with self-mindhacks while thinking.
(Do you agree that the FAI problem has to be solved sooner or later? I think you didn’t respond to the last argument I made on that.)
An NP oracle makes AI rather easier… but I’m not sure it currently would be as much help in FAI in particular. That is, I don’t think it would help as much with the F part.
In other words, I suspect that an NP oracle is the sort of thing that if you discovered one… you should be really really quiet about it, and be very cautious about who you tell. (I may be totally wrong about this, though.)
In other words, I suspect that an NP oracle is the sort of thing that if you discovered one… you should be really really quiet about it, and be very cautious about who you tell. (I may be totally wrong about this, though.)
There are both good and bad arguments against this. If one can find such a thing it seems likely that others can too. A lot of the more mundane bad stuff that can be done with a practical solution to P=NP would be things like breaking encryption which work iff people don’t know you have such a solution (otherwise people then go back to things like one-time pads. The economy will suffer until the quantum crypto is really up and running, but the amount of long-term badness someone with the solution can do will be severely limited.) Moreover, such a solution can also help do a lot of good in mundane areas where one needs to routinely solve instances of NP complete problems. So, overall, I’d go with releasing it.
NP oracles allow easy learning by allowing one to find compact models to explain/predict available data...
Also gives the ability to do stuff like “what actions can I take which, within N inferential steps of this model, will produce an outcome I desire?” or “will produce utility > U with probability > P” or such.
Maybe I’m way off on this, but it sure does seem like a cheap NP oracle would make at least UFAI comparatively easy.
If I’m totally wrong about NP oracles → easy AI, then I’d agree with you re releasing it… with a caveat. I’d say to it with advance warning… ie, anonymously demonstrate to various banks, security institutions, and possibly the public that there exists someone with the ability to efficiently solve NP complete problems, and that the algorithm will be released in X amount of time. (1-2 years would be my first thought for how long the advanced warning should be.)
This way everyone has time to at least partly prepare for the day where all crypto other than OTP (quantum crypto would be an example of an OTP style crypto) would be dead.
Also gives the ability to do stuff like “what actions can I take which, within N inferential steps of this model, will produce an outcome I desire?” or “will produce utility > U with probability > P” or such.
Yes, but if the models aren’t well-defined then that won’t be doable. At this point we can even give rigorous notions what we mean by an intelligence, so our 3-SAT oracle won’t be able to help much. The 3-SAT oracle is only going to help for precisely defined questions.
Huh? I’m not sure I understand what your objection.
An NP oracle would let you do stuff like “given this sensory data, find a model of size N or less that within K computational steps or less will reproduce the data to within error x, given such a model exists”
Then one can run “which sequence of actions, given this model, will, within S steps, produce outcome A with probability P?”
Whether or not we can give a rigorous definition of intelligence, seems like the above is sufficient to act like an intelligence, right? Yeah, there’re a few tricky parts re reflective decision theory, but even without that. Even if we let the thing be non-self-modifying… giving a nice chunk of computing power to the above, given an efficient NP oracle, would be enough to potentially cause trouble. Or so I’d imagine.
Just some property of the outcome that you’re interested in. ie, all of the above, with the question being “blah blah blah, with f(outcome) = blah with probability blah blah blah blah blah”
Huh? I don’t follow. (Note, the whole point is that I was claiming that an NP oracle would make, say, a UFAI potentially easy, while to achieve the rather more specific FAI would still be difficult.)
It seems that we may be talking past each other. Could you give an explicit example of what sort of question you would ask the NP oracle to help get a UFAI?
Oooh, sorry, I was unclear. I meant the NP oracle itself would be a component of the AI.
ie, give me an algorithm for efficiently solving NP complete problems, and one could then use that to perform the sort of computations I mentioned earlier.
Hmm, I’m confused. Why do you think that such an object would be helpful as part of the AI? I see how it would be useful to an AI once one had one, but I don’t see why you would want it as a component that makes it easier to make an AGI.
It would make AI easy specifically because it would allow the sorts of computations I described above. If I had a fast way to solve NP complete problems, then I could turn that into a way of performing the previously mentioned computations. Those previously mentioned computations amount to “efficient learning” + “efficient planning”.
The oracle itself is what gives one the ability to perform this computation: Find compact model that efficiently predicts/explains with bounded error the observed sensory data. (This is rough description of the more precise version stated above)
Also gives one the ability to efficiently perform this computation: Given a generated model, determine actions that will lead to desired outcome in bounded number of steps, with reasonably good probability.
The ability to perform the former computation would amount to the ability to efficiently learn. The ability to perform the latter computation would amount to the ability to efficiently plan.
ie, if one has an algorithm for efficiently solving NP complete problems, one can be really good at doing the above two things. The above two things amount to the ability to learn and the ability to plan.
clearer version: the first type of computation would allow it, from observation and such, to determine stuff like the laws of physics, human psychology, etc...
The second computation would allow it to do stuff like… figure out what actions it needs to take to increase the rate of paperclip production or whatever.
(incidentally A slight amount of additional reflectivity, without needing to solve the really hard problems of reflective decision theory or such, would probably be sufficient to allow it to figure out what experiments it needs to do to gain data it needs to form better models.)
Any answer other than “create a superintelligent friendly AI to optimize the universe” would be a waste of this particular genie, but there are some steps in between that and 3-SAT which I can’t specify yet.
Coincidentally, I recently had an idea for making progress on FAI that would benefit greatly from a solution to 3-SAT.
Your idea seems to be a general method for solving unformalized problems using an NP-solver: generate phrases that an uploaded human would classify as insights. I’m still afraid that many such phrases will be mindhacks instead of genuine insights, though, and the risk gets worse as the problem gets harder (or more precisely, as the next inferential step becomes harder to hit).
I agree it’s still risky, but with the safety features I put in (having a small limit on the phrase length, outputting the top 100,000 (considered individually) in random order instead of just the top 1, review/discussion by a team, and we can also mix together the top 10,000 insights from each of 10 uploads for additional safety) it seems no worse than just having humans try to solve the problem by thinking about it, since we could also come up with self-mindhacks while thinking.
(Do you agree that the FAI problem has to be solved sooner or later? I think you didn’t respond to the last argument I made on that.)
...And right now, thinking about possible replies to your comment, I finally switched to agreeing with that. Thanks.
Oh hell. This changes a lot. I need to think.
This is a most excellent update. I look forward to hearing what comes out of this thinking.
An NP oracle makes AI rather easier… but I’m not sure it currently would be as much help in FAI in particular. That is, I don’t think it would help as much with the F part.
In other words, I suspect that an NP oracle is the sort of thing that if you discovered one… you should be really really quiet about it, and be very cautious about who you tell. (I may be totally wrong about this, though.)
(Declining to resist temptation to link a sci-fi story (7500 words) despite Google confirming you’re already familiar with it.)
But I already am immune to that story. ;)
How so? This isn’t obvious to me.
There are both good and bad arguments against this. If one can find such a thing it seems likely that others can too. A lot of the more mundane bad stuff that can be done with a practical solution to P=NP would be things like breaking encryption which work iff people don’t know you have such a solution (otherwise people then go back to things like one-time pads. The economy will suffer until the quantum crypto is really up and running, but the amount of long-term badness someone with the solution can do will be severely limited.) Moreover, such a solution can also help do a lot of good in mundane areas where one needs to routinely solve instances of NP complete problems. So, overall, I’d go with releasing it.
NP oracles allow easy learning by allowing one to find compact models to explain/predict available data...
Also gives the ability to do stuff like “what actions can I take which, within N inferential steps of this model, will produce an outcome I desire?” or “will produce utility > U with probability > P” or such.
Maybe I’m way off on this, but it sure does seem like a cheap NP oracle would make at least UFAI comparatively easy.
If I’m totally wrong about NP oracles → easy AI, then I’d agree with you re releasing it… with a caveat. I’d say to it with advance warning… ie, anonymously demonstrate to various banks, security institutions, and possibly the public that there exists someone with the ability to efficiently solve NP complete problems, and that the algorithm will be released in X amount of time. (1-2 years would be my first thought for how long the advanced warning should be.)
This way everyone has time to at least partly prepare for the day where all crypto other than OTP (quantum crypto would be an example of an OTP style crypto) would be dead.
Yes, but if the models aren’t well-defined then that won’t be doable. At this point we can even give rigorous notions what we mean by an intelligence, so our 3-SAT oracle won’t be able to help much. The 3-SAT oracle is only going to help for precisely defined questions.
Huh? I’m not sure I understand what your objection.
An NP oracle would let you do stuff like “given this sensory data, find a model of size N or less that within K computational steps or less will reproduce the data to within error x, given such a model exists”
Then one can run “which sequence of actions, given this model, will, within S steps, produce outcome A with probability P?”
Whether or not we can give a rigorous definition of intelligence, seems like the above is sufficient to act like an intelligence, right? Yeah, there’re a few tricky parts re reflective decision theory, but even without that. Even if we let the thing be non-self-modifying… giving a nice chunk of computing power to the above, given an efficient NP oracle, would be enough to potentially cause trouble. Or so I’d imagine.
Ok, but how are you specifying the outcome? And are you going to specify each input and output?
Just some property of the outcome that you’re interested in. ie, all of the above, with the question being “blah blah blah, with f(outcome) = blah with probability blah blah blah blah blah”
Then you will need to specify your AI for every single output-input pair you are interested in.
Huh? I don’t follow. (Note, the whole point is that I was claiming that an NP oracle would make, say, a UFAI potentially easy, while to achieve the rather more specific FAI would still be difficult.)
It seems that we may be talking past each other. Could you give an explicit example of what sort of question you would ask the NP oracle to help get a UFAI?
Oooh, sorry, I was unclear. I meant the NP oracle itself would be a component of the AI.
ie, give me an algorithm for efficiently solving NP complete problems, and one could then use that to perform the sort of computations I mentioned earlier.
Hmm, I’m confused. Why do you think that such an object would be helpful as part of the AI? I see how it would be useful to an AI once one had one, but I don’t see why you would want it as a component that makes it easier to make an AGI.
It would make AI easy specifically because it would allow the sorts of computations I described above. If I had a fast way to solve NP complete problems, then I could turn that into a way of performing the previously mentioned computations. Those previously mentioned computations amount to “efficient learning” + “efficient planning”.
I’m sorry but I’m still not following what learning and planning you would do. Are you attaching the oracle to some sort of reward mechanism?
The oracle itself is what gives one the ability to perform this computation: Find compact model that efficiently predicts/explains with bounded error the observed sensory data. (This is rough description of the more precise version stated above)
Also gives one the ability to efficiently perform this computation: Given a generated model, determine actions that will lead to desired outcome in bounded number of steps, with reasonably good probability.
The ability to perform the former computation would amount to the ability to efficiently learn. The ability to perform the latter computation would amount to the ability to efficiently plan.
ie, if one has an algorithm for efficiently solving NP complete problems, one can be really good at doing the above two things. The above two things amount to the ability to learn and the ability to plan.
clearer version: the first type of computation would allow it, from observation and such, to determine stuff like the laws of physics, human psychology, etc...
The second computation would allow it to do stuff like… figure out what actions it needs to take to increase the rate of paperclip production or whatever.
(incidentally A slight amount of additional reflectivity, without needing to solve the really hard problems of reflective decision theory or such, would probably be sufficient to allow it to figure out what experiments it needs to do to gain data it needs to form better models.)