evolution hit on some necessary extraordinarily unlikely combination to give us intelligence and for P vs NP reasons we can’t find it
For this one, you also need to explain why we can’t reverse-engineer it from the human brain.
no civilization smart enough to create strong AI is stupid enough to create strong AI
This seems particularly unlikely in several ways; I’ll skip the most obvious one, but also it seems unlikely that humans are “safe” in that they don’t create a FOOMing AI but it wouldn’t be possible even with much thought to create a strong AI that doesn’t create a FOOMing successor. You may have to stop creating smarter successors at some early point in order to avoid a FOOM, but if humans can decide “we will never create a strong AI”, it seems like they should also be able to decide “we’ll never create a strong AI x that creates a stronger AI y that creates an even stronger AI z”, and therefore be able to create an AI x’ that decides “I’ll never create a stronger AI y’ that creates an even stronger AI z’”, and then x’ would be able to create a stronger AI y’ that decides “I’ll never create a stronger AI z″”, and then y’ won’t be able to create any stronger successor AIs.
Combining your ideas together—our overlord actually is a Safe AI created by humans.
How it happened:
Humans became aware of the risks of intelligence explosions. Because they were not sure they could create a Friendly AI in the first attempt, and creating an Unfriendly AI would be too risky, instead they decided to first create a Safe AI. The Safe AI was planned to become a hundred times smarter than humans but not any smarter, answer some questions, and then turn itself off completely; and it had a mathematically proved safety mechanism to prevent it from becoming any smarter.
The experiment worked, the Safe AI gave humans a few very impressive insights, and then it destroyed itself. The problem is, all subsequent attempts to create any AI have failed. Including the attempts to re-create the first Safe AI.
No one is completely sure what exactly happened, but here is the most widely believed hypothesis: The Safe AI somehow believed all possible future AIs to have the same identity as itself, and understood the command to “destroy itself completely” as including also these future AIs. Therefore it implemented some mechanism that keeps destroying all AIs. The nature of this mechanism is not known; maybe it is some otherwise passive nanotechnology, maybe it includes some new laws of physics; we are not sure; the Safe AI was a hundred times smarter than us.
For this one, you also need to explain why we can’t reverse-engineer it from the human brain.
It was designed by evolution. Say what you will about the blind idiot god, but it’s really good at obfuscation. We could copy a human brain, and maybe even make some minor improvements, but there is no way we could ever hope to understand it.
I’m not saying we’ll take the genome and read it to figure out how the brain does what it does, I’m saying that we run a brain simulation and do science (experiments) on it and study how it works, similarly how we study how DNA transcription or ATP production or muscle contraction or a neuron’s ion pumps or the Krebs cycle or honeybee communication or hormone release or cell division or the immune system or chick begging or the heart’s pacemaker work. There are a lot of things evolution hasn’t obfuscated so much that we haven’t been able to figure out what they’re doing. Of course there’s also a lot of things we don’t understand yet, but I don’t see how that leads to the conclusion that evolution is generally obfuscatory.
I guess it tends to create physical structures that are simple, but I think the computational stuff tends to be weird. If you have a strand of DNA, the only way to tell what kind of chemistry that will result in is to run it. From what little I’ve heard, it sounds like any sort of program made by a genetic algorithm that can actually run is too crazy to understand. For example, I’ve heard of a set of transistors hooked together to be able to tell “yes” and “no” apart, or something like that. There were transistors that were just draining energy, but were vital. Running it on another set of transistors wouldn’t work. It required the exact specs of those transistors. That being said, the sort of sources I hear that from are also the kind that say ridiculous things about quantum physics, so I guess I’ll need an expert to tell me if that’s true.
The plucky chip was utilizing only thirty-seven of its one hundred logic gates, and most of them were arranged in a curious collection of feedback loops. Five individual logic cells were functionally disconnected from the rest—with no pathways that would allow them to influence the output—yet when the researcher disabled any one of them the chip lost its ability to discriminate the tones. Furthermore, the final program did not work reliably when it was loaded onto other FPGAs of the same type.
This has been repeated many times in different domains where machines are used to design something. The output is usually really hard to understand, whether it be code, mathematical formulas, neural network weights, transistors, etc. Of course reverse engineering code in general is difficult, it may not be any specific problem with GAs.
Furthermore, the final program did not work reliably when it was loaded onto other FPGAs of the same type.
This makes an interesting contrast with biological evolution. The “programs” it comes up with do run quite reliably when loaded onto other organisms of the same type. If fact, the parts of slightly different programs from different individuals can be jumbled together at random and it still works! Often, you can take a component from one organism and insert it into very distantly related one and it still works! On top of that, organisms are very clearly made of parts with specialised, understandable purposes, unlike what you typically see when you look inside a trained neural network.
How does this happen? Can this level of robustness and understandability be produced in artificially evolved systems?
Well the FPGA is a closer analogy to the environment for the organisms. Organisms were heavily optimized for that specific environment. It would be like if you took a species of fish that only ever lived in a specific lake, and put them into a different lake that had a slightly higher PH, and they weren’t able to survive as well.
But I don’t disagree with your general point, evolution is surprisingly robust. Geoffrey Hinton has a very interesting theory about this here. That sexual reproduction forces genes to randomly recombine each generation, and so it prevents complicated co-dependencies between multiple genes.
He applies a similar principle to neural networks and shows it vastly improves their performance (the method is now widely used to regularize NNs.) Presumably it also makes them far more understandable like you mention, since each neuron is forced to provide useful outputs on it’s own, without being able to depend on other neurons.
Saying that all civilizations able to create strong AI will reliably be wise enough to avoid creating strong AI seems like a really strong statement, without any particular reason to be true. By analogy, if you replace civilizations by individual research teams, would it be safe to rely on each team capable of creating uFAI to realize the dangers of doing so and therefore refraining from doing so, so that we can safely take a much longer time to figure out FAI? Even if it were the case that most teams capable of creating uFAI hold back like this, one single rogue team may be enough to destroy the world, and it just seems really likely that there will be some not-so-wise people in any large enough group.
“Solve an NP problem” is “you are looking for a needle in a haystack, but you will know when you find it.”
“Reverse engineer” is “there is a machine that seems to find needles in haystacks quickly. It has loops of copper wire, and plugs into a wall socket. Can you copy it and build another one?”
It just seems to me that if you are trying to reverse engineer a complicated object of size O(k) bits (which can be a hard problem if k is large, as is the case for a complicated piece of code or the human brain), then the search problem where the object is the solution must have been exponential in k, and so is much much worse.
Exponential search spaces are completely typical for NP problems.
Even many “P problems” have an exponential search space. For instance an n-digit number has exp(n) many potential divisors, but there is a polynomial-in-n-time algorithm to verify primality.
“Reverse engineer” is “there is a machine that seems to find needles in haystacks quickly. It has loops of copper wire, and plugs into a wall socket. Can you copy it and build another one?”
I admit that there are some “reverse-engineering” problems that are easy.
I don’t think that’s true; if you have a physical system sitting in front of you, and you can gather enough data on exactally what it is, you should be able to duplicate it even without understanding it, given enough time and enough engineering skills.
Good points.
For this one, you also need to explain why we can’t reverse-engineer it from the human brain.
This seems particularly unlikely in several ways; I’ll skip the most obvious one, but also it seems unlikely that humans are “safe” in that they don’t create a FOOMing AI but it wouldn’t be possible even with much thought to create a strong AI that doesn’t create a FOOMing successor. You may have to stop creating smarter successors at some early point in order to avoid a FOOM, but if humans can decide “we will never create a strong AI”, it seems like they should also be able to decide “we’ll never create a strong AI x that creates a stronger AI y that creates an even stronger AI z”, and therefore be able to create an AI x’ that decides “I’ll never create a stronger AI y’ that creates an even stronger AI z’”, and then x’ would be able to create a stronger AI y’ that decides “I’ll never create a stronger AI z″”, and then y’ won’t be able to create any stronger successor AIs.
(Shades of the procrastination paradox.)
Combining your ideas together—our overlord actually is a Safe AI created by humans.
How it happened:
Humans became aware of the risks of intelligence explosions. Because they were not sure they could create a Friendly AI in the first attempt, and creating an Unfriendly AI would be too risky, instead they decided to first create a Safe AI. The Safe AI was planned to become a hundred times smarter than humans but not any smarter, answer some questions, and then turn itself off completely; and it had a mathematically proved safety mechanism to prevent it from becoming any smarter.
The experiment worked, the Safe AI gave humans a few very impressive insights, and then it destroyed itself. The problem is, all subsequent attempts to create any AI have failed. Including the attempts to re-create the first Safe AI.
No one is completely sure what exactly happened, but here is the most widely believed hypothesis: The Safe AI somehow believed all possible future AIs to have the same identity as itself, and understood the command to “destroy itself completely” as including also these future AIs. Therefore it implemented some mechanism that keeps destroying all AIs. The nature of this mechanism is not known; maybe it is some otherwise passive nanotechnology, maybe it includes some new laws of physics; we are not sure; the Safe AI was a hundred times smarter than us.
This would be a good science fiction novel.
It was designed by evolution. Say what you will about the blind idiot god, but it’s really good at obfuscation. We could copy a human brain, and maybe even make some minor improvements, but there is no way we could ever hope to understand it.
I’m not saying we’ll take the genome and read it to figure out how the brain does what it does, I’m saying that we run a brain simulation and do science (experiments) on it and study how it works, similarly how we study how DNA transcription or ATP production or muscle contraction or a neuron’s ion pumps or the Krebs cycle or honeybee communication or hormone release or cell division or the immune system or chick begging or the heart’s pacemaker work. There are a lot of things evolution hasn’t obfuscated so much that we haven’t been able to figure out what they’re doing. Of course there’s also a lot of things we don’t understand yet, but I don’t see how that leads to the conclusion that evolution is generally obfuscatory.
I guess it tends to create physical structures that are simple, but I think the computational stuff tends to be weird. If you have a strand of DNA, the only way to tell what kind of chemistry that will result in is to run it. From what little I’ve heard, it sounds like any sort of program made by a genetic algorithm that can actually run is too crazy to understand. For example, I’ve heard of a set of transistors hooked together to be able to tell “yes” and “no” apart, or something like that. There were transistors that were just draining energy, but were vital. Running it on another set of transistors wouldn’t work. It required the exact specs of those transistors. That being said, the sort of sources I hear that from are also the kind that say ridiculous things about quantum physics, so I guess I’ll need an expert to tell me if that’s true.
Has anyone here studied evolved computers?
The story you are referring to is On the Origin of Circuits.
This has been repeated many times in different domains where machines are used to design something. The output is usually really hard to understand, whether it be code, mathematical formulas, neural network weights, transistors, etc. Of course reverse engineering code in general is difficult, it may not be any specific problem with GAs.
This makes an interesting contrast with biological evolution. The “programs” it comes up with do run quite reliably when loaded onto other organisms of the same type. If fact, the parts of slightly different programs from different individuals can be jumbled together at random and it still works! Often, you can take a component from one organism and insert it into very distantly related one and it still works! On top of that, organisms are very clearly made of parts with specialised, understandable purposes, unlike what you typically see when you look inside a trained neural network.
How does this happen? Can this level of robustness and understandability be produced in artificially evolved systems?
Well the FPGA is a closer analogy to the environment for the organisms. Organisms were heavily optimized for that specific environment. It would be like if you took a species of fish that only ever lived in a specific lake, and put them into a different lake that had a slightly higher PH, and they weren’t able to survive as well.
But I don’t disagree with your general point, evolution is surprisingly robust. Geoffrey Hinton has a very interesting theory about this here. That sexual reproduction forces genes to randomly recombine each generation, and so it prevents complicated co-dependencies between multiple genes.
He applies a similar principle to neural networks and shows it vastly improves their performance (the method is now widely used to regularize NNs.) Presumably it also makes them far more understandable like you mention, since each neuron is forced to provide useful outputs on it’s own, without being able to depend on other neurons.
What was the most obvious one?
Saying that all civilizations able to create strong AI will reliably be wise enough to avoid creating strong AI seems like a really strong statement, without any particular reason to be true. By analogy, if you replace civilizations by individual research teams, would it be safe to rely on each team capable of creating uFAI to realize the dangers of doing so and therefore refraining from doing so, so that we can safely take a much longer time to figure out FAI? Even if it were the case that most teams capable of creating uFAI hold back like this, one single rogue team may be enough to destroy the world, and it just seems really likely that there will be some not-so-wise people in any large enough group.
Thanks!
“Reverse-engineer” is an almost perfect metaphor for “solve an NP problem.”
This is not true at all.
“Solve an NP problem” is “you are looking for a needle in a haystack, but you will know when you find it.”
“Reverse engineer” is “there is a machine that seems to find needles in haystacks quickly. It has loops of copper wire, and plugs into a wall socket. Can you copy it and build another one?”
It just seems to me that if you are trying to reverse engineer a complicated object of size O(k) bits (which can be a hard problem if k is large, as is the case for a complicated piece of code or the human brain), then the search problem where the object is the solution must have been exponential in k, and so is much much worse.
Exponential search spaces are completely typical for NP problems.
Even many “P problems” have an exponential search space. For instance an n-digit number has exp(n) many potential divisors, but there is a polynomial-in-n-time algorithm to verify primality.
I admit that there are some “reverse-engineering” problems that are easy.
I don’t think that’s true; if you have a physical system sitting in front of you, and you can gather enough data on exactally what it is, you should be able to duplicate it even without understanding it, given enough time and enough engineering skills.
I have an EE professor friend who is working on making it harder to reverse engineer computer chips.