My current thinking is that AI might be in the space of things we can’t understand.
While we are improving our knowledge of the brain, no one is coming up with simple theories that explains the brain as a whole rather than as bits and pieces with no coherent design, that we can see.
Under this scenario AI is still possible, but if we do make it, it will be done by semi-blindly copying the machinery we have with random tweaks. And if it does start to self-improve it will be doing so with random tweaks only, as it will have our lack of ability to comprehend itself.
Why does AI design need to have anything to do with the brain? (Third Alternative: ab initio development based on a formal normative theory of general intelligence, not a descriptive theory of human intelligence, comprehensible even to us to say nothing of itself once it gets smart enough.)
(Edit: Also, it’s a huge leap from “no one is coming up with simple theories of the brain yet” to “we may well never understand intelligence”.)
A specific AI design need be nothing like the design of the brain. However the brain is the only object we know of in mind space, so having difficulty understanding it is evidence, although very weak, that we may have difficulty understanding minds in general.
We might expect it to be a special case as we are trying to understand methods of understanding, so we are being somewhat self-referential.
If you read my comment you’ll see I only raised it as a possibility, something to try and estimate the probability of, rather than necessarily the most likely case.
What would you estimate the probability of this scenario being, and why?
There might be formal proofs, but they probably are reliant on the definition of things like what understanding is, I’ve been trying to think of mathematical formalisms to explore this question, but I haven’t come up with a satisfactory one yet.
It is trivial to say one AIXI can’t comprehend another instance of AIXI, if by comprehend you mean form an accurate model.
AIXI expects the environment to be computable and is itself incomputable. So if one AIXI comes across another, it won’t be able to form a true model of it.
However I am not sure of the value of this argument as we expect intelligence to be computable.
Seems plausible. However under this model, there’s still room for self-improvement using something like genetic algorithms; that is, it could make small, random tweaks, but find and implement the best ones in much less time than we could possibly do with humans. Then it could still be recursively self-improving.
A lot of us think this scenario is much more likely. Mostly those on the side of Chaos in a particular Grand Narrative. Plug for The Future and its Enemies—arguably one of the most important works in political philosophy from the 20th century.
That is much weaker than the type of RSI that is supposed to cause FOOM. For one you are only altering software not hardware, and secondly I don’t think a system that replaces itself with a random variation, even if it has been tested, will necessarily be better, if it doesn’t understand itself. Random alterations, may cause madness, introduce bugs or other problems a long time after the change.
My current thinking is that AI might be in the space of things we can’t understand.
While we are improving our knowledge of the brain, no one is coming up with simple theories that explains the brain as a whole rather than as bits and pieces with no coherent design, that we can see.
Under this scenario AI is still possible, but if we do make it, it will be done by semi-blindly copying the machinery we have with random tweaks. And if it does start to self-improve it will be doing so with random tweaks only, as it will have our lack of ability to comprehend itself.
Why does AI design need to have anything to do with the brain? (Third Alternative: ab initio development based on a formal normative theory of general intelligence, not a descriptive theory of human intelligence, comprehensible even to us to say nothing of itself once it gets smart enough.)
(Edit: Also, it’s a huge leap from “no one is coming up with simple theories of the brain yet” to “we may well never understand intelligence”.)
A specific AI design need be nothing like the design of the brain. However the brain is the only object we know of in mind space, so having difficulty understanding it is evidence, although very weak, that we may have difficulty understanding minds in general.
We might expect it to be a special case as we are trying to understand methods of understanding, so we are being somewhat self-referential.
If you read my comment you’ll see I only raised it as a possibility, something to try and estimate the probability of, rather than necessarily the most likely case.
What would you estimate the probability of this scenario being, and why?
There might be formal proofs, but they probably are reliant on the definition of things like what understanding is, I’ve been trying to think of mathematical formalisms to explore this question, but I haven’t come up with a satisfactory one yet.
Have you looked at AIXI?
It is trivial to say one AIXI can’t comprehend another instance of AIXI, if by comprehend you mean form an accurate model.
AIXI expects the environment to be computable and is itself incomputable. So if one AIXI comes across another, it won’t be able to form a true model of it.
However I am not sure of the value of this argument as we expect intelligence to be computable.
Seems plausible. However under this model, there’s still room for self-improvement using something like genetic algorithms; that is, it could make small, random tweaks, but find and implement the best ones in much less time than we could possibly do with humans. Then it could still be recursively self-improving.
A lot of us think this scenario is much more likely. Mostly those on the side of Chaos in a particular Grand Narrative. Plug for The Future and its Enemies—arguably one of the most important works in political philosophy from the 20th century.
That is much weaker than the type of RSI that is supposed to cause FOOM. For one you are only altering software not hardware, and secondly I don’t think a system that replaces itself with a random variation, even if it has been tested, will necessarily be better, if it doesn’t understand itself. Random alterations, may cause madness, introduce bugs or other problems a long time after the change.
Note: Deliberate alterations may cause madness or introduce bugs or other problems a long time after the change.
The idea with Eliezer style RSI is formally proved good alterations.