It is hard to tell whether anyone took this seriously—but it seems that an isomorphic argument ‘proves’ that computer programs will crash—since “almost any” computer program crashes. The “AGI Apocalypse Argument” as stated thus appears to be rather silly.
I don’t see why this makes the argument seem silly. It seems to me that the isomorphic argument is correct, and that computer programs do crash.
Some computer programs crash—just as some possible superintelligences would kill alll humans.
However, the behavior of a computer program chosen at random tells you very little about how an actual real-world computer program will behave—since computer programs are typically produced by selection processes performed by intelligent agents.
Some computer programs crash—just as some possible superintelligences would kill all humans.
No most computer programs crash, it’s just that most people never see them because said programs are repeatedly tested and modified until they no longer crash before being shown to people (this process is called “debugging”). With a self-modifying AI this is a lot harder to do.
Some computer programs crash—just as some possible superintelligences would kill all humans.
No *most” computer programs crash [...]
By “no”, you apparently mean “yes”.
With a self-modifying AI this is a lot harder to do.
Well, that is a completely different argument—and one that would appear to be in need of supporting evidence—since automated testing, linting and the ability to program in high-level languages are all improving simultaneously.
I am not aware of any evidence that real computer programs are getting more crash-prone with the passage of time.
With a self-modifying AI this is a lot harder to do.
Well, that is a completely different argument—and one that would appear to be in need of supporting evidence—since automated testing, linting and the ability to program in high-level languages are all improving simultaneously.
The point is that the first time you run the seed AI it will attempt to take over the world, so you don’t have the luxury of debugging it.
Almost certainly, the first time you run the seed AI, it’ll crash quickly. I think it’s very unlikely that you construct a successful-enough-to-be-dangerous AI without a lot of mentally crippled ones first.
Almost certainly, the first time you run the seed AI, it’ll crash quickly. I think it’s very unlikely that you construct a successful-enough-to-be-dangerous AI without a lot of mentally crippled ones first.
If so then we are all going to die. That is, if you have that level of buggy code then it is absurdly unlikely that the first time the “intelligence” part works at all it works well enough to be friendly. (And that scenario seems likely.)
I am not entirely sure I disagree with you. However, I am having difficulty modeling you.
“Achieving a goal” seems to mean, for our purposes, something along the lines of “Bringing about a world-state.” Most possible world-states do not involve human existence. Thus, it seems that for most possible goals, achieving a goal entails human extinction.
However, your mention of computer programs being produced by intelligent agents is interesting. Are you implying that most AGI’s (assume these intelligences can go FOOM) would not result in human extinction?
If this is not what you were implying, I apologize for modeling you poorly. If this is what you were implying, I would like to indicate that this post was non-hostile.
Are you implying that most AGI’s (assume these intelligences can go FOOM) would not result in human extinction?
Questions about fractions of infinite sets require an enumeration strategy to be specified—or they don’t make much sense. Assuming lexicographic ordering of their source code—and only considering the set of superintelligent programs—no: I don’t mean to imply that.
A statement which we can derive from the simple fact that the mere existence of general intelligence (apes) does not result automatically in catastrophe.
I wonder how long it’ll take before people catch onto the notion that artificial “dumbness” is in many ways a more interesting field than artificial “intelligence”? (As in, how much could an AGI no smarter than a dog, but hooked into expert systems similar to Watson, do?)
It was pretty well accepted at MIT’s Media Lab back when my orbit took me around there periodically, a decade or so ago, that there was a huge amount of low-hanging fruit in this area… not necessarily of academic interest, but damned useful (and commercial).
That’s interesting since my impression if anything is the exact opposite. There seem to be a lot of people trying to apply Bayesian learning systems and expert learning systems to all sorts of different practical problems. I wonder if this is a new thing or whether I simply don’t have a good view of the field.
I can see that for expert systems, but Bayesian learning systems seem to be a distinct category. The primary limits seem to be scalibility not architecture.
Bayesian learning systems are essentially another form of trainable neural network. That makes them very good in a narrow range of categories but also makes them insufficient to the cause of achieving general intelligence.
I do not see that scaling Bayesian learning networks would ever achieve general intelligence. No matter how big the hammer, it’ll never be a wrench. That being said, I do believe that some form of pattern recognition and ‘selective forgetting’ is important to cognition and as such Bayesian learning architecture is a good tool towards that end.
not necessarily of academic interest, but damned useful (and commercial).
Actually, I’m curious that isn’t seen as an area of significan academic interest—designing artificial systems around being efficient parsers of extraneous data. I recall that one of the major differences between Deep Blue and Deep Fritz in the Kasperov chess matches was precisely that Fritz was designed around not probing every last possible set of playable moves; that is, Deep Fritz was “learning to forget the right things”.
It seems to me that understanding this mechanism and how it behaves in humans could have huge potential for opening up the understanding of general intelligence and cognition. And that’s a very academic concern.
I don’t see why this makes the argument seem silly. It seems to me that the isomorphic argument is correct, and that computer programs do crash.
Some computer programs crash—just as some possible superintelligences would kill alll humans.
However, the behavior of a computer program chosen at random tells you very little about how an actual real-world computer program will behave—since computer programs are typically produced by selection processes performed by intelligent agents.
The “for almost any goals” argument is bunk.
No most computer programs crash, it’s just that most people never see them because said programs are repeatedly tested and modified until they no longer crash before being shown to people (this process is called “debugging”). With a self-modifying AI this is a lot harder to do.
By “no”, you apparently mean “yes”.
Well, that is a completely different argument—and one that would appear to be in need of supporting evidence—since automated testing, linting and the ability to program in high-level languages are all improving simultaneously.
I am not aware of any evidence that real computer programs are getting more crash-prone with the passage of time.
The point is that the first time you run the seed AI it will attempt to take over the world, so you don’t have the luxury of debugging it.
That is not a very impressive argument, IMHO.
We will have better test harnesses by then—allowing such machines to be debugged.
Almost certainly, the first time you run the seed AI, it’ll crash quickly. I think it’s very unlikely that you construct a successful-enough-to-be-dangerous AI without a lot of mentally crippled ones first.
If so then we are all going to die. That is, if you have that level of buggy code then it is absurdly unlikely that the first time the “intelligence” part works at all it works well enough to be friendly. (And that scenario seems likely.)
The first machine intellligences we build will be stupid ones.
By the time smarter ones are under developpment we will have other trustworthy smart machines on hand to help keep the newcomers in check.
I am not entirely sure I disagree with you. However, I am having difficulty modeling you.
“Achieving a goal” seems to mean, for our purposes, something along the lines of “Bringing about a world-state.” Most possible world-states do not involve human existence. Thus, it seems that for most possible goals, achieving a goal entails human extinction.
However, your mention of computer programs being produced by intelligent agents is interesting. Are you implying that most AGI’s (assume these intelligences can go FOOM) would not result in human extinction?
If this is not what you were implying, I apologize for modeling you poorly. If this is what you were implying, I would like to indicate that this post was non-hostile.
Questions about fractions of infinite sets require an enumeration strategy to be specified—or they don’t make much sense. Assuming lexicographic ordering of their source code—and only considering the set of superintelligent programs—no: I don’t mean to imply that.
A statement which we can derive from the simple fact that the mere existence of general intelligence (apes) does not result automatically in catastrophe.
I wonder how long it’ll take before people catch onto the notion that artificial “dumbness” is in many ways a more interesting field than artificial “intelligence”? (As in, how much could an AGI no smarter than a dog, but hooked into expert systems similar to Watson, do?)
It was pretty well accepted at MIT’s Media Lab back when my orbit took me around there periodically, a decade or so ago, that there was a huge amount of low-hanging fruit in this area… not necessarily of academic interest, but damned useful (and commercial).
That’s interesting since my impression if anything is the exact opposite. There seem to be a lot of people trying to apply Bayesian learning systems and expert learning systems to all sorts of different practical problems. I wonder if this is a new thing or whether I simply don’t have a good view of the field.
For what it’s worth, I consider Bayesian learning systems and expert learning systems to be “narrow” AI—hence the example I gave of Watson.
I think Ben Goertzel’s Novamente project is the closest extant project to a ‘general’ AI of any form that I’ve heard of.
I can see that for expert systems, but Bayesian learning systems seem to be a distinct category. The primary limits seem to be scalibility not architecture.
Bayesian learning systems are essentially another form of trainable neural network. That makes them very good in a narrow range of categories but also makes them insufficient to the cause of achieving general intelligence.
I do not see that scaling Bayesian learning networks would ever achieve general intelligence. No matter how big the hammer, it’ll never be a wrench. That being said, I do believe that some form of pattern recognition and ‘selective forgetting’ is important to cognition and as such Bayesian learning architecture is a good tool towards that end.
Actually, I’m curious that isn’t seen as an area of significan academic interest—designing artificial systems around being efficient parsers of extraneous data. I recall that one of the major differences between Deep Blue and Deep Fritz in the Kasperov chess matches was precisely that Fritz was designed around not probing every last possible set of playable moves; that is, Deep Fritz was “learning to forget the right things”.
It seems to me that understanding this mechanism and how it behaves in humans could have huge potential for opening up the understanding of general intelligence and cognition. And that’s a very academic concern.