Assuming we’ve conquered overfitting, the optimizer should perform very well on new optimization problems, especially after self-improvement.
This is a huge assumption. Hell, this is the entire question of AI. In Godel, Escher, and Bach, he describes consciousness as the ability to overcome local maxima by thinking outside the system. Your system is a hill climbing problem, and you’re saying “assume we’ve already invented eyes for machines, hill climbing is easy.”
Once we have a specification for friendly AI, it should be straightforward to feed it into the optimizer and get a satisfactory program back.
The act of software engineering is the creation of a specification. The act of coding is translating your specifications into a language the computer can understand (and discovering holes in your specs). If you’ve already got an airtight specification for Friendly AI, then you’ve already got Friendly AI and don’t need any optimizer in the first place.
Other problems which arise are that the problems you’re asking your optimization machine to work on are NP-hard. We’ve also already got something that can take inputted computer programs and optimize them as much as possible without changing their essential structure; it’s called an optimizing compiler. Oh, and the biggest one being that your plan to create friendly AI is to build and run a billion AIs and keep the best one. Lets just hope none of the evil ones FOOM during the testing phase.
More theory here is required. I think it’s at least plausible that some tradeoff between complexity and performance is possible that allows the system to generalize to new problems.
In Godel, Escher, and Bach, he describes consciousness as the ability to overcome local maxima by thinking outside the system.
If a better optimizer according to program 3 exists, the current optimizer will eventually find it, at least through brute force search. The relevant questions are 1. will this better optimizer generalize to new problems? and 2. how fast? I don’t see any kind of “thinking outside the system” that is not possible by writing a better optimizer.
The act of software engineering is the creation of a specification. The act of coding is translating your specifications into a language the computer can understand (and discovering holes in your specs).
Right, this system can do “coding” according to your definition but “software engineering” is harder. Perhaps software engineering can be defined in terms of induction: given English description/software specification pairs, induce a simple function from English to software specification.
If you’ve already got an airtight specification for Friendly AI, then you’ve already got Friendly AI and don’t need any optimizer in the first place.
It’s not that straightforward. If we replace “friendly AI” with “paperclip maximizer”, I think we can see that knowing what it means to maximize paperclips does not imply supreme ability to do so. This system solves the second part and might provide some guidance to the first part.
We’ve also already got something that can take inputted computer programs and optimize them as much as possible without changing their essential structure; it’s called an optimizing compiler.
A sufficiently smart optimizing compiler can solve just about any clearly specified problem. No such optimizing compiler exists today.
Oh, and the biggest one being that your plan to create friendly AI is to build and run a billion AIs and keep the best one. Lets just hope none of the evil ones FOOM during the testing phase.
Not sure what you’re talking about here. I’ve addressed safety concerns.
There is no reason to believe a non-sentient program will ever escape it’s local maxima. We have not yet devised the optimization process that will provably not get stuck in a local maxima in bounded time. If you give this optimizer the MU Puzzle (aka 2^n mod 3 = 0) it will never figure it out, even though most children will come to the right answer in minutes. That’s what’s so great about consciousness that we don’t understand yet. Creating a program which can solve this class of problems is the creation of artificial consciousness, full stop.
“Well it self improves so it’ll improve to the point it solves it” How? And don’t say complexity or emergence. And how can you prove that it’s more likely to self-improve into having artificial consciousness within, say, 10 billion years. Theoretically, a program that randomly put down characters into a text file and tried to compile it would eventually create an AI too. But there’s no reason to think it would do so before the heat death of the universe came knocking.
The words “paperclip maximizer is not a specification, just like “friendly AI” is not a specification. Those are both suggestively named LISP tokens. An actual specification for friendly AI is a blueprint for it, the same way that human DNA is a specification for the human body. “Featherless biped with two arms, two legs, a head with two eyes, two ears, a nose, and the ability to think.” Is not a specification for humans, it’s a description. You could come up with any number of creatures from that description. The base sequence of our DNA which will create a human and nothing but a human is a specification. Until you have a set of directions that create a friendly AI and nothing but a friendly AI, you haven’t got specs for them. And by the time you have that, you can just build a friendly AI.
I hope jacobt doesn’t think something like this can be implemented easily; I see it as a proposal for safely growing a seed AI if we had the relevant GAI insights to make a suitable seed (with simple initial goals). I agree with you that we don’t currently have the conceptual background needed to write such a seed.
I think we disagree on what a specification is. By specification I mean a verifier: if you had something fitting the specification, you could tell if it did. For example we have a specification for “proof that P != NP” because we have a system in which that proof could be written and verified. Similarly, this system contains a specification for general optimization. You seem to be interpreting specification as knowing how to make the thing.
If you give this optimizer the MU Puzzle (aka 2^n mod 3 = 0) it will never figure it out, even though most children will come to the right answer in minutes.
If you define the problem as “find n such that 2^n mod 3 = 0” then everyone will fail the problem. And I don’t see why the optimizer couldn’t have some code that monitors its own behavior. Sure it’s difficult to write, but the point of this system is to go from a seed AI to a superhuman AI safely. And such a function (“consciousness”) would help it solve many of the sample optimization problems without significantly increasing complexity.
This is a huge assumption. Hell, this is the entire question of AI. In Godel, Escher, and Bach, he describes consciousness as the ability to overcome local maxima by thinking outside the system. Your system is a hill climbing problem, and you’re saying “assume we’ve already invented eyes for machines, hill climbing is easy.”
The act of software engineering is the creation of a specification. The act of coding is translating your specifications into a language the computer can understand (and discovering holes in your specs). If you’ve already got an airtight specification for Friendly AI, then you’ve already got Friendly AI and don’t need any optimizer in the first place.
Other problems which arise are that the problems you’re asking your optimization machine to work on are NP-hard. We’ve also already got something that can take inputted computer programs and optimize them as much as possible without changing their essential structure; it’s called an optimizing compiler. Oh, and the biggest one being that your plan to create friendly AI is to build and run a billion AIs and keep the best one. Lets just hope none of the evil ones FOOM during the testing phase.
More theory here is required. I think it’s at least plausible that some tradeoff between complexity and performance is possible that allows the system to generalize to new problems.
If a better optimizer according to program 3 exists, the current optimizer will eventually find it, at least through brute force search. The relevant questions are 1. will this better optimizer generalize to new problems? and 2. how fast? I don’t see any kind of “thinking outside the system” that is not possible by writing a better optimizer.
Right, this system can do “coding” according to your definition but “software engineering” is harder. Perhaps software engineering can be defined in terms of induction: given English description/software specification pairs, induce a simple function from English to software specification.
It’s not that straightforward. If we replace “friendly AI” with “paperclip maximizer”, I think we can see that knowing what it means to maximize paperclips does not imply supreme ability to do so. This system solves the second part and might provide some guidance to the first part.
A sufficiently smart optimizing compiler can solve just about any clearly specified problem. No such optimizing compiler exists today.
Not sure what you’re talking about here. I’ve addressed safety concerns.
There is no reason to believe a non-sentient program will ever escape it’s local maxima. We have not yet devised the optimization process that will provably not get stuck in a local maxima in bounded time. If you give this optimizer the MU Puzzle (aka 2^n mod 3 = 0) it will never figure it out, even though most children will come to the right answer in minutes. That’s what’s so great about consciousness that we don’t understand yet. Creating a program which can solve this class of problems is the creation of artificial consciousness, full stop.
“Well it self improves so it’ll improve to the point it solves it” How? And don’t say complexity or emergence. And how can you prove that it’s more likely to self-improve into having artificial consciousness within, say, 10 billion years. Theoretically, a program that randomly put down characters into a text file and tried to compile it would eventually create an AI too. But there’s no reason to think it would do so before the heat death of the universe came knocking.
The words “paperclip maximizer is not a specification, just like “friendly AI” is not a specification. Those are both suggestively named LISP tokens. An actual specification for friendly AI is a blueprint for it, the same way that human DNA is a specification for the human body. “Featherless biped with two arms, two legs, a head with two eyes, two ears, a nose, and the ability to think.” Is not a specification for humans, it’s a description. You could come up with any number of creatures from that description. The base sequence of our DNA which will create a human and nothing but a human is a specification. Until you have a set of directions that create a friendly AI and nothing but a friendly AI, you haven’t got specs for them. And by the time you have that, you can just build a friendly AI.
I hope jacobt doesn’t think something like this can be implemented easily; I see it as a proposal for safely growing a seed AI if we had the relevant GAI insights to make a suitable seed (with simple initial goals). I agree with you that we don’t currently have the conceptual background needed to write such a seed.
I think we disagree on what a specification is. By specification I mean a verifier: if you had something fitting the specification, you could tell if it did. For example we have a specification for “proof that P != NP” because we have a system in which that proof could be written and verified. Similarly, this system contains a specification for general optimization. You seem to be interpreting specification as knowing how to make the thing.
If you define the problem as “find n such that 2^n mod 3 = 0” then everyone will fail the problem. And I don’t see why the optimizer couldn’t have some code that monitors its own behavior. Sure it’s difficult to write, but the point of this system is to go from a seed AI to a superhuman AI safely. And such a function (“consciousness”) would help it solve many of the sample optimization problems without significantly increasing complexity.