you could say that experts disagreed about one of the 5 theses (intelligence explosion), as only 10% thought a human level AI could reach a strongly superhuman level within 2 years
Hit the brakes on that line of reasoning! That’s not what the question asked. It asked WILL it, not COULD it. If there is any sort of caution at all in development, it’s going to take more than 2 years before that AI gets to see its own source code.
If there is any sort of caution at all in development, it’s going to take more than 2 years before that AI gets to see its own source code.
In many architectures the AI “sees” its source code as part of normal operations. It’s a required fact for how these architectures are structured. Indeed, it is the only way the benefits of AI can be applied to AI itself, and the chief mechanism for an intelligence explosion.
It’s like having a nuclear program and saying “let’s avoid using any fissionable materials.” It could happen… but it would be rather missing the point.
I’d dispute your last paragraph. An AI that doesn’t examine and modify its own source code as its core architecture can still examine and modify its own source code if you let it. That’s like keeping the fissionable materials wrapped in graphite like in a pebble-bed reactor.
I would class baking it in from the start not as simply having fissionable material, but more as juggling bare, slightly subcritical blocks of Plutonium.
That’s missing the point I’m afraid. What I meant was that the operation of the AI itself necessarily involves modifying its own “source code.” The act of (generalized!) thinking itself is self-modifying. An artificial general intelligence is capible of solving any problem, including the problem of artificial general intelligence. And the architecture of most actual AGIs involve modifying internal behavior based on the output of thinking processes in such a way that is Turing complete. So even if you didn’t explicitly program the machine to modify its own source code (although any efficient AGI would need to), it could learn or stumble upon a self-aware, self-modifying method of thinking. Even if it involves something as convoluted as using the memory database as a read/write store, and updating belief networks as gates in an emulated CPU.
Having to start over from scratch would be a very significant impediment. Don’t forget that we’re talking about the pre-super-intelligence phase, here.
Gah, no, my point wasn’t about starting over from scratch at all. It was that most AGI architectures include self-modification as a core and inseparable part of the architecture. For example, by running previously evolved thinking processes. You can’t just say “we’ll disable the self-modification for safety’s sake”—you’d be giving it a total lobotomy!
I was then only making a side point that even if you designed an architecture that didn’t self-modify—unlikely for performance reasons—it would still discover how to wire itself into self-modification eventually. So that doean’t really solve the safety issue, alone.
I don’t disagree with this comment. The scare quotes is because the AI wouldn’t literally be editing the C++ (or whatever) code directly, the sort of things that a reader might think of when I say “editing source code.” Rather it will probably manipulate encodings of thinking processes in some sort of easy to analyze recombinant programming language, as well as adjust weighting vectors as you mention. There’s a reason LISP, where code is data and data is code is the traditional or stereotypical language of artificial intelligence, although personally I think a more strongly typed concatenative language would be a better choice. Such a language is what the AI would use to represent its own thinking processes, and what it would manipulate to “edit its own source code.”
...you could say that experts disagreed about one of the 5 theses (intelligence explosion), as only 10% thought a human level AI could reach a strongly superhuman level within 2 years
Hit the brakes on that line of reasoning! That’s not what the question asked. It asked WILL it, not COULD it.
If I have a statement “X will happen”, and ask people to assign a probability to it, then if the probability is <=50% I believe it isn’t too much to a stretch to paraphrase “X will happen with a probability <=50%” as “It could be that X will happen”. Looking at the data of the survey, of 163 people who gave a probability estimate, only 15 people assigned a probability >50% to the possibility that there will be a superhuman intelligence that greatly surpasses the performance of humans within 2 years after the creation of a human level intelligence.
That said, I didn’t use the word “could” on purpose in my comment. It was just an unintentional inaccuracy. If you think that is a big deal, then I am sorry. I’ll try to be more careful in future.
Hit the brakes on that line of reasoning! That’s not what the question asked. It asked WILL it, not COULD it. If there is any sort of caution at all in development, it’s going to take more than 2 years before that AI gets to see its own source code.
In many architectures the AI “sees” its source code as part of normal operations. It’s a required fact for how these architectures are structured. Indeed, it is the only way the benefits of AI can be applied to AI itself, and the chief mechanism for an intelligence explosion.
It’s like having a nuclear program and saying “let’s avoid using any fissionable materials.” It could happen… but it would be rather missing the point.
I’d dispute your last paragraph. An AI that doesn’t examine and modify its own source code as its core architecture can still examine and modify its own source code if you let it. That’s like keeping the fissionable materials wrapped in graphite like in a pebble-bed reactor.
I would class baking it in from the start not as simply having fissionable material, but more as juggling bare, slightly subcritical blocks of Plutonium.
That’s missing the point I’m afraid. What I meant was that the operation of the AI itself necessarily involves modifying its own “source code.” The act of (generalized!) thinking itself is self-modifying. An artificial general intelligence is capible of solving any problem, including the problem of artificial general intelligence. And the architecture of most actual AGIs involve modifying internal behavior based on the output of thinking processes in such a way that is Turing complete. So even if you didn’t explicitly program the machine to modify its own source code (although any efficient AGI would need to), it could learn or stumble upon a self-aware, self-modifying method of thinking. Even if it involves something as convoluted as using the memory database as a read/write store, and updating belief networks as gates in an emulated CPU.
Having to start over from scratch would be a very significant impediment. Don’t forget that we’re talking about the pre-super-intelligence phase, here.
So, no, I don’t think I missed the point at all.
Gah, no, my point wasn’t about starting over from scratch at all. It was that most AGI architectures include self-modification as a core and inseparable part of the architecture. For example, by running previously evolved thinking processes. You can’t just say “we’ll disable the self-modification for safety’s sake”—you’d be giving it a total lobotomy!
I was then only making a side point that even if you designed an architecture that didn’t self-modify—unlikely for performance reasons—it would still discover how to wire itself into self-modification eventually. So that doean’t really solve the safety issue, alone.
I was disagreeing that that architectural change would not be helpful on the safety issue.
You put source code in scare quotes. Most AIs don’t literally modify their source cor, they just adjust weighting .ir whatever...essentially data.
I don’t disagree with this comment. The scare quotes is because the AI wouldn’t literally be editing the C++ (or whatever) code directly, the sort of things that a reader might think of when I say “editing source code.” Rather it will probably manipulate encodings of thinking processes in some sort of easy to analyze recombinant programming language, as well as adjust weighting vectors as you mention. There’s a reason LISP, where code is data and data is code is the traditional or stereotypical language of artificial intelligence, although personally I think a more strongly typed concatenative language would be a better choice. Such a language is what the AI would use to represent its own thinking processes, and what it would manipulate to “edit its own source code.”
If I have a statement “X will happen”, and ask people to assign a probability to it, then if the probability is <=50% I believe it isn’t too much to a stretch to paraphrase “X will happen with a probability <=50%” as “It could be that X will happen”. Looking at the data of the survey, of 163 people who gave a probability estimate, only 15 people assigned a probability >50% to the possibility that there will be a superhuman intelligence that greatly surpasses the performance of humans within 2 years after the creation of a human level intelligence.
That said, I didn’t use the word “could” on purpose in my comment. It was just an unintentional inaccuracy. If you think that is a big deal, then I am sorry. I’ll try to be more careful in future.
The difference here is that you considered this position to strictly imply being against the possibility of intelligence explosion.
One can consider intelligence explosion a real risk, and then take steps to prevent it, with the resulting estimate being low probability.