Having read the piece I was not impressed. I became even less impressed when I read his criticism of Legg’s piece. It seems to be basically come down “computers can’t do things that humans can. And they never will be able to. So there.”
My intention for linking to it was not that I thought it featured good arguments, as you might notice by my comments over there, but that he is an educated skeptic with potential influence in the mainstream rationality community. The post is a sample of an outsiders perception and assessment of the SIAI. And right now is the time for the SIAI to hone its appearance and public relations. Because once people like PZ Myers become aware of the SIAI and portray it and LW negatively, this community will be inundated with literally thousands of mediocre rationalists and many potential donors will be lost.
this community will be inundated with literally thousands of mediocre rationalists and many potential donors will be lost.
Bad comments will get downvoted and not seen by many people. If someone isn’t getting much out of LW and LW isn’t getting much out of their presence, they’ll leave eventually. If the moderation system continues working about as well as it has been working, an influx of new users shouldn’t be a problem. (It’s probably something the site needs to be prepared for when Eliezer’s books come out, anyway.)
My intention for linking to it was not that I thought it featured good arguments...
Gee, thanks. So you basically linked and replied as a form of damage control? And by the way, the “outsiders’ perception” isn’t helped when the “insiders’” arguments seem to be based not on what computers actually do, but what they’re made to do in comic books.
Gee, thanks. So you basically linked and replied as a form of damage control?
XiXi is actually one of the people here who is more critical of the SI and the notion of run-away superintelligence. XiXi can correct me if I’m wrong here, but I suspect that XiXi’s intention in this particular instance was to do just what he said. To give an example of an outsider’s perspective on the SI of exactly the type of outsider who the SI should be trying to convince and should be able to convince if their arguments have much validity.
And by the way, the “outsiders’ perception” isn’t helped when the “insiders’” arguments seem to be based not on what computers actually do, but what they’re made to do in comic books.
Ok. This is the sort of remark that get’s the SI people correctly annoyed. Generalizations from fictional evidence are bad. But, at the same time, that something happens to have occurred in fictional settings isn’t in general a reason to assign it lower probability than you would if one weren’t aware of such fiction. (To use a silly example, there’s fiction set after the sun has become a red giant. The fact that there’s such fiction isn’t relevant to evaluating whether or not the sun will enter such a phase). It also misses one of the fundamental points that the SI people have made repeatedly: computers as they exist today are very weak entities. The SI’s argument doesn’t have to do with computers in general. It centers around what happens once machines have human level intelligence. So, ask yourself, how likely is it do you think that we’ll have general AI ever, and if we do have general AI, what buggy failure modes seem most likely?
It centers around what happens once machines have human level intelligence.
As defined by… what exactly? We have problems measuring our own intelligence or even defining it so we’re giving computers a very wide sliding scale of intelligence based on personal opinions and ideas morethan a rigirous examination. A computer today could ace just about any general knowledge test we give it if we tell it how to search for an answer or compute a problem. Does that make it as intelligent as a really academically adept human? Oh and it can do it in a tiny fraction of the time it would take us. Does that make it superhuman?
It may be a red herring to focus on the definition of “intelligence” in this context. If you prefer, taboo the words intelligent and intelligence in this context and simply refer to a computer capable of doing at least everything a regular person can do. The issue is what happens after one has a machine that reaches that point.
… the words intelligent and intelligence in this context and simply refer to a computer capable of doing at least everything a regular person can do.
But we already have things capable of doing everything a regular person can do. We call them regular people. Are we trying to build another person in digital format here, and if so, why? Just because we want to see if can? Or because we have some big plans for it?
But we already have things capable of doing everything a regular person can do. We call them regular people. Are we trying to build another person in digital format here, and if so, why? Just because we want to see if can? Or because we have some big plans for it?
Irrelevant to the question at hand, which is what would happen if a machine had such capabilities. But, if you insist on discussing this issue also, machines with human-like abilities could be very helpful. For example, one might be able to train one of them to do some task, and then make multiple copies of it, much more efficient than individually training lots of humans. Or one could send such AIs into dangerous situations where we might not ethically send a person (whether it would actually be ethical to send an AI is a distinct question.)
Or one could send such AIs into dangerous situations where we might not ethically send a person (whether it would actually be ethical to send an AI is a distinct question.)
Why is it distinct? Whether doing something is an error determines if it’s beneficial to obtain ability and willingness to do it.
It’s distinct when the question is about risk to the human, rather than about the ethics of the task itself. We could make nonsentient nonpersons that nevertheless have humanlike abilities in some broad or narrow sense, so that sacrificing them in some risky or suicidal task doesn’t impact the ethical calculation as it would if we were sending a person.
(I think that’s what JoshuaZ was getting at. The “distinct question” would presumably be that of the AI’s potential personhood.)
Um… we already do all that to a pretty high extent and we don’t need general intelligence in every single facet of human ability to do that. Just make it an expert in its task and that’s all you need.
Um… we already do all that to a pretty high extent and we don’t need general intelligence in every single facet of human ability to do that. Just make it an expert in its task and that’s all you need.
There are a large number of tasks where the expertise level needed by current technology is woefully insufficient. Anything that has a strong natural language requirement for example.
Oh fun, we’re talking about my advisers’ favorite topic! Yeah, strong natural language is a huge pain and if we had devices that understood human speech well, tech companies would jump on that ASAP.
But here’s the thing. If you want natural language processing, why build a Human 2.0? Why not just build the speech recognition system? It’s making AGI for something like that the equivalent of building a 747 to fly one person across a state? I can see various expert systems coming together as an AGI, but not starting out as such.
It would surprise me if human-level natural-language processing were possible without sitting on top of a fairly sophisticated and robust world-model.
I mean, just as an example, consider how much a system has to know about the world to realize that in your next-to-last sentence, “It’s” is most likely a typo for “Isn’t.”
Granted that one could manually construct and maintain such a model rather than build tools that maintain it automatically based on ongoing observations, but the latter seems like it would pay off over time.
I don’t think this is a good argument. Just because you cannot define something doesn’t mean it’s not a real phenomena or that you cannot reason about it at all. Before we understood fire completely, it was still real and we could reason about it somewhat (fire consumes some things, fire is hot etc.). Similarly, intelligence is a real phenomena that we don’t completely understand and we can still do some reasoning about it. It is meaningful to talk about a computer having “human-level” (I think “human-like” might be more descriptive) intelligence.
I don’t think this is a good argument. Just because you cannot define something doesn’t mean it’s not a real phenomena or that you cannot reason about it at all.
If you have no working definition for what you’re trying to discuss, you’re more than likely to be barking up the wrong tree about it. We didn’t understand fire completely, but we knew that it was hot, you couldn’t touch it, and you made it by rubbing dry sticks together really, really fast, or by making a spark with rocks and have it land on dry straw.
Also, where did I say that until I get a definition of intelligence all discussion about the concept is meaningless? I just want to know what criteria an AI must meet to be considered human and match them with what we have so far so I can see how far we might be from those benchmarks. I think it’s a perfectly reasonable way to go about this kind of discussion.
I apologize, the intent of your question was not at all clear to me from your previous post. It sounded to me like you were using this as an argument that SIAI types were clearly wrong headed.
To answer your question then, the relevant dimension of intelligence is something like “ability to design and examine itself similarly to it’s human designers”.
the relevant dimension of intelligence is something like “ability to design and examine itself similarly to it’s human designers”.
Ok, I’ll buy that. I would agree that any system that could be its own architect and hold meaningful design and code review meetings with its builders would qualify as human-level intelligent.
To clarify: I didn’t mean that such a machine is necessarily “human level intelligent” in all respects, just that that is the characteristic relevant to the idea of an “intelligence explosion”.
I just want to know what criteria an AI must meet to be considered human and match them with what we have so far so I can see how far we might be from those benchmarks.
Wow, if that’s all you got from a post trying to explain the very real difference between acing an intelligence test by figuring things out on your own and having a machine do the same after you give it all the answers and how the suggested equations only measure how many answers were right, not how that feat was accomplished, I don’t even know how to properly respond...
Oh and by the way, in the comments I suggest how to keep track of the machine doing some learning and figuring out to Dr. Legg so there’s another thing to consider. And yes, I’ve had the formal instruction in discrete math to do so.
Wow, if that’s all you got from a post trying to explain the very real difference between acing an intelligence test by figuring things out on your own and having a machine do the same after you give it all the answers and how the suggested equations only measure how many answers were right, not how that feat was accomplished, I don’t even know how to properly respond...
It is possible that I didn’t explain my point well. The problem I am referring to is your apparent insistence that there are things that machines can’t do that people can and that this is insurmountable. Most of your subclaims are completely reasonable, but the overarching premise that machines can only do what they are programmed to seems to show up in both pieces, and is simply wrong. Even today, that’s not true by most definitions of those terms. Neural nets and genetic algorithms often don’t do what they are told.
… but the overarching premise that machines can only do what they are programmed to seems to show up in both pieces, and is simply wrong.
Only if you choose to discard any thought to how machines are actually built. There’s no magic going on in that blinking box, just ciruits performing the functions they were designed to do in the order they’re told.
Neural nets and genetic algorithms often don’t do what they are told.
Actually, they do precisely what they’re told because without a fitness function which determines what problem they are to solve in their output and their level of correctness, they just crash the computer. Don’t confuse algorithms that have very generous bounds and allow us to try different possible solutions to the same problem for some sort of thinking or initiative on the computer’s part. And when computers do something weird, it’s because of a bug which sends them persuing their logic in whays programmers never intended, not because they decide to go off on their own.
I can’t tell you how many seemingly bizarre and ridiculous problems I’ve eventually tracked down to a bad loop, or a bad index value, or a missing symbol in a string...
Only if you choose to discard any thought to how machines are actually built. There’s no magic going on in that blinking box, just ciruits performing the functions they were designed to do in the order they’re told.
There’s no magic going on inside the two pounds of fatty tissue inside my skull either. Magic is apparently not required for creativity or initiative (whatever those may be).
Actually, they do precisely what they’re told because without a fitness function which determines what problem they are to solve in their output and their level of correctness, they just crash the computer. Don’t confuse algorithms that have very generous bounds and allow us to try different possible solutions to the same problem for some sort of thinking or initiative on the computer’s part.
I’m confused by what you mean by “thinking” and “initiative.” Let’s narrow the field slightly. Would the ability to come up with new definitions and conjectures in math be an example of thinking and initiative?
And when computers do something weird, it’s because of a bug which sends them persuing their logic in whays programmers never intended, not because they decide to go off on their own.
Calling something a bug doesn’t change the nature of what is happening. That’s just a label. Humans are likely as smart as they are due to runaway sexual selection for intelligence. And then humans got really smart and realized that they could have all the pleasure of sex while avoiding the hassle of reproduction. Is the use of birth-control an example of human initiative or a bug? Does it make a difference?
Would the ability to come up with new definitions and conjectures in math be an example of thinking and initiative?
Yes, but with a caveat. I could teach an ANN how to solve a problem but it would be more or less by random trial and error with a squashing function until each “neuron” has the right weight and activation function. So it will learn how to solve this generic problem, but it won’t be because it traced its way along all the steps.
(Actually I made in mistake in my previous reply, ANNs have no fitness function, that’s a genetic algorithm. ANNs are given an input and a desired output.)
So if you develop a new defintion or conjecture and can state why and how you did it, then develop a proof, you’ve shown thought. Your attempt to suddenly create a new definition or theorem just because you wanted to and were curious rather than just tasked to do it would be initiative.
Calling something a bug doesn’t change the nature of what is happening. That’s just a label.
No, you see, a bug is when a computer does something it’s not supposed to do and handles its data incorrectly. Birth control is actually another approach to reproduction most of the time, delaying progeny until we feel ready to raise them. Those who don’t have children have put their evolutionary desire to provide for themselves above the drive to reproduce and counter that urge with protected sex. So it’s not really a bug as it is a solution to some of the problems posed by reproduction. Now, celibacy is something I’d call a bug and we know from many studies that it’s almost always a really bad idea to forgo sex altogether. Mental health tends to suffer greatly.
So if you develop a new defintion or conjecture and can state why and how you did it, then develop a proof, you’ve shown thought. Your attempt to suddenly create a new definition or theorem just because you wanted to and were curious rather than just tasked to do it would be initiative.
Hmm, so would a grad student who is thinking about a thesis problem because their advisor said to think about it be showing initiative? Is a professional mathematician showing initiative? They keep thinking about math because that’s what gives them positive feedback (e.g. salary, tenure, positive remarks from their peers).
No, you see, a bug is when a computer does something it’s not supposed to do and handles its data incorrectly
Is “incorrectly” a normative or descriptive term? .How is it different than “this program didn’t do what I expected it to do” other than that you label it a bug when the program deviates more from what you wanted to accomplish? Keep in mind that what a human wants isn’t a notion that cleaves reality at the joints.
Birth control is actually another approach to reproduction most of the time, delaying progeny until we feel ready to raise them. Those who don’t have children have put their evolutionary desire to provide for themselves above the drive to reproduce and counter that urge with protected sex. So it’s not really a bug as it is a solution to some of the problems posed by reproduction.
Ok. So when someone (and I know quite a few people in this category) deliberately uses birth control because they want the pleasure of sex but don’t want to ever have kids, is that a bug in your view?
Hmm, so would a grad student who is thinking about a thesis problem because their advisor said to think about it be showing initiative?
Dis he/she volunteer to work on a problem and come to the advisor saying that this is the thesis subject? Doesn’t sound like it, so I’d say it’s not. Initiative is doing something that’s not required, but something you feel needs to be done or something you want to do.
Is “incorrectly” a normative or descriptive term?
Yes. When you need it to return “A” and it retuns “Finland,” it made a mistake which has to be fixed. How it came to that mistake can be found by tracing the logic after the bug manifests itself.
Keep in mind that what a human wants isn’t a notion that cleaves reality at the joints.
Ok, whan you build a car but the car doesn’t start, I don’t think you’re going to say that the car is just doing what it wants and we humans are just selfishly insisting that it bends to our whims. You’re probably going to take that thing to a mechanic. Same thing with computers, even AI. If you build an AI to learn a language and it doesn’t seem to be able to do so, there’s a bug in the system.
So when someone (and I know quite a few people in this category) deliberately uses birth control because they want the pleasure of sex but don’t want to ever have kids, is that a bug in your view?
That’s answered in the second sentence of the quote you chose...
Dis he/she volunteer to work on a problem and come to the advisor saying that this is the thesis subject? Doesn’t sound like it, so I’d say it’s not. Initiative is doing something that’s not required, but something you feel needs to be done or something you want to do.
Ok. Now, if said grad student did come to the thesis adviser, but their motivation was that they’ve been taught from a very young age that they should do math. Is there initiative?
Ok, whan you build a car but the car doesn’t start, I don’t think you’re going to say that the car is just doing what it wants and we humans are just selfishly insisting that it bends to our whims. You’re probably going to take that thing to a mechanic. Same thing with computers, even AI. If you build an AI to learn a language and it doesn’t seem to be able to do so, there’s a bug in the system.
It seems that a large part of the disagreement is implicit premises here. You seem to be focused on very narrow AI, when the entire issue is what happens when one doesn’t have narrow AI but have AI that has most capabilities that humans have. Let’s set aside whether or not we should build such AIs and whether or not they are possible. Assuming that such entities are possible, do you or do you not think there’s a risk of the AI getting out of control.
So when someone (and I know quite a few people in this category) deliberately uses birth control because they want the pleasure of sex but don’t want to ever have kids, is that a bug in your view?
That’s answered in the second sentence of the quote you chose...
Either there’s a miscommunication here or there’s a misunderstanding about how evolution works. An organism that puts its own survival over reproducing is an evolutionary dead end. Historically, lots of humans didn’t want any children, but they didn’t have effective birth control methods, so in the ancestral environment there was minimal evolutionary incentive to remove that preference. It has only been recently that there is widespread and effective birth control. So, what you’ve said is one evolved desire overriding another would still seem to be a bug.
Now, if said grad student did come to the thesis adviser, but their motivation was that they’ve been taught from a very young age that they should do math. Is there initiative?
Not sure. You could argue both points in this situation.
Assuming that such entities are possible, do you or do you not think there’s a risk of the AI getting out of control.
Any AI can get out of control. I never denied that. My issue is with how that should be managed, not whether it can happen.
So, what you’ve said is one evolved desire overriding another would still seem to be a bug.
Can you clarify how it’s helpful to know that my machine only does what it’s been told to do, if I can’t know what I’m telling it to do or be certain what I have told it to do?
I mean, there’s a sense in which humans only do “what they’ve been told to do”, also… we have programs embedded in DNA that manifest themselves in brains that construct minds from experience in constrained ways. (Unless you believe in some kind of magic free will in human minds, in which case this line of reasoning won’t seem sensible to you.) But so what? Knowing that doesn’t make humans harmless.
Additionally, a big part of what SIAI types emphasize is that knowing very precisely and very broadly (at the same time) what humans want is very important. Human desires are very complex, so this is not a simple task.
Can you clarify how it’s helpful to know that my machine only does what it’s been told to do, if I can’t know what I’m telling it to do or be certain what I have told it to do?
If you have no idea what you want your AI to do, why are you building it in the first place? I have never built an app that does, you know, anything and whatever. It’ll just be muddled mess that probably won’t even compile.
we have programs embedded in DNA that manifest themselves in brains...
No we do not. This is not how biology works. Brains are self-organizing structures built by a combination of cellular signals and environmental cues. All that DNA does is to regulate what proteins the cell will manufacture. Development goes well beyond that.
If you have no idea what you want your AI to do, why are you building it in the first place?
I’m not sure how you got from my question to your answer. I’m not talking at all about programmers not having intentions, and I agree with you that in pretty much all cases they do have intentions.
I’ll assume that I wasn’t clear, rather than that you’re willing to ignore what’s actually being said in favor of what lets you make a more compelling argument, and will attempt to be clearer.
You keep suggesting that there’s no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.
At the same time, you admit that computer programs sometimes do things their programmers didn’t intend for them to do. I might have written a stupid bug that causes the program to delete the contents of my hard drive, for example.
I agree completely that, in doing so, it is merely doing what I told it to do: I’m the one who wrote that stupid bug, it didn’t magically come out of nowhere, the program doesn’t have any mysterious kind of free will or anything. It’s just a program I wrote.
But I don’t see why that should be particularly reassuring. The fact remains that the contents of my hard drive are deleted, and I didn’t want them to be. That I’m the one who told the program to delete them makes no difference I care about; far more salient to me is that I didn’t intend for the program to delete them.
And the more a program is designed to flexibly construct strategies for achieving particular goals in the face of unpredictable environments, the harder it is to predict what it is that I’m actually telling my program to do, regardless of what I intend for it to do.
In other words: “I can’t know what I’m telling it to do or be certain what I have told it to do.”
Sure, once it deletes the files, I can (in principle) look back over the source code and say “Oh, I see why that happened.” But that doesn’t get me my files back.
Brains are self-organizing structures built by a combination of cellular signals and environmental cues. All that DNA does is to regulate what proteins the cell will manufacture. Development goes well beyond that.
And yet, remarkably, brains don’t “self-organize” in the absence of that regulation.
You’re right, of course, that the correct environment is also crucial; DNA won’t magically turn into a brain without a very specific environment in which to manifest.
Then again, source code won’t magically turn into a running program without a very specific environment either, and quite a lot of the information defining that running program comes from the compiler and the hardware platform rather than the source code… and yet we have no significant difficulty equating a running program with its source code.
(Sure, sometimes bugs turn out to be in the compiler or the hardware, but even halfway competent programmers don’t look there except as a matter of last resort. If the running program is doing something I didn’t intend, it’s most likely that the source code includes an instruction I didn’t intend to give.)
You keep suggesting that there’s no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.
No, I just keep saying that we don’t need to program them to “like rewards and fear punishments” and train them like we’d train dogs.
I agree completely that, in doing so, it is merely doing what I told it to do: I’m the one who wrote that stupid bug, it didn’t magically come out of nowhere, the program doesn’t have any mysterious kind of free will or anything. It’s just a program I wrote. But I don’t see why that should be particularly reassuring.
Oh no, it’s not. I have several posts on my blog detailing how bugs like that could actually turn a whole machine army against us and turn Terminator into a reality rather than a cheesy robots-take-over-the-world-for-shits-and-giggles flick.
… and yet we have no significant difficulty equating a running program with its source code.
But the source code isn’t like DNA in an organism. Source code covers so much more ground than that. Imagine having an absolute blueprint of how every cell cluster in your body will react to any stimuli through your entire life and every process it will undertake from now until your death, including how it will age. That would be source code. Your DNA is not ever nearly that complete. It’s more like a list of suggestions and blueprints for raw materials.
No, I just keep saying that we don’t need to program them to “like rewards and fear punishments” and train them like we’d train dogs.
(shrug) OK, fair enough.
I agree with you that reward/punishment conditioning of software is a goofy idea.
I was reading your comment here to indicate that we can constrain the behavior of human-level AGIs by just putting appropriate constraints in the code. (“You don’t want the machine to do something? Put in a boundry. [..] with a machine, you can just tell it not to do that.”)
I think that idea is importantly wrong, which is why I was responding to it, but if you don’t actually believe that then we apparently don’t have a disagreement.
Re: source code… if we’re talking about code that is capable of itself generating executable code as output in response to situations that arise (which seems implicit in the idea of a human-level AGI, given that humans are capable of generating executable code), it isn’t at all clear to me that its original source code comprises in any kind of useful way an absolute blueprint for how every part of it will react to any stimuli.
Again, sure, I’m not positing magic: whatever it does, it does because of the interaction between its source code and the environment in which it runs, there’s no kind of magic third factor. So, sure, given the source code and an accurate specification of its environment (including its entire relevant history), I can in principle determine precisely what it will do. Absolutely agreed. (Of course, in practice that might be so complicated that I can’t actually do it, but you aren’t claiming otherwise.)
If you don’t think the same is true of humans, then we disagree about humans, but I think that’s incidental.
… if we’re talking about code that is capable of itself generating executable code as output in response to situations that arise
Again, it really shouldn’t be doing that. It should have the capacity to learn new skills and build new neural networks to do so. That doesn’t require new code, it just requires a routine to initialize a new set of ANN objects at runtime.
If it somehow follows from that that there’s an absolute blueprint in it for how every part of it will react to any stimuli in a way that is categorically different from how human genetics specify how humans will respond to any environment, then I don’t follow the connection… sorry. I have only an interested layman’s understanding of ANNs.
Having read the piece I was not impressed. I became even less impressed when I read his criticism of Legg’s piece. It seems to be basically come down “computers can’t do things that humans can. And they never will be able to. So there.”
My intention for linking to it was not that I thought it featured good arguments, as you might notice by my comments over there, but that he is an educated skeptic with potential influence in the mainstream rationality community. The post is a sample of an outsiders perception and assessment of the SIAI. And right now is the time for the SIAI to hone its appearance and public relations. Because once people like PZ Myers become aware of the SIAI and portray it and LW negatively, this community will be inundated with literally thousands of mediocre rationalists and many potential donors will be lost.
Bad comments will get downvoted and not seen by many people. If someone isn’t getting much out of LW and LW isn’t getting much out of their presence, they’ll leave eventually. If the moderation system continues working about as well as it has been working, an influx of new users shouldn’t be a problem. (It’s probably something the site needs to be prepared for when Eliezer’s books come out, anyway.)
Gee, thanks. So you basically linked and replied as a form of damage control? And by the way, the “outsiders’ perception” isn’t helped when the “insiders’” arguments seem to be based not on what computers actually do, but what they’re made to do in comic books.
XiXi is actually one of the people here who is more critical of the SI and the notion of run-away superintelligence. XiXi can correct me if I’m wrong here, but I suspect that XiXi’s intention in this particular instance was to do just what he said. To give an example of an outsider’s perspective on the SI of exactly the type of outsider who the SI should be trying to convince and should be able to convince if their arguments have much validity.
Ok. This is the sort of remark that get’s the SI people correctly annoyed. Generalizations from fictional evidence are bad. But, at the same time, that something happens to have occurred in fictional settings isn’t in general a reason to assign it lower probability than you would if one weren’t aware of such fiction. (To use a silly example, there’s fiction set after the sun has become a red giant. The fact that there’s such fiction isn’t relevant to evaluating whether or not the sun will enter such a phase). It also misses one of the fundamental points that the SI people have made repeatedly: computers as they exist today are very weak entities. The SI’s argument doesn’t have to do with computers in general. It centers around what happens once machines have human level intelligence. So, ask yourself, how likely is it do you think that we’ll have general AI ever, and if we do have general AI, what buggy failure modes seem most likely?
As defined by… what exactly? We have problems measuring our own intelligence or even defining it so we’re giving computers a very wide sliding scale of intelligence based on personal opinions and ideas morethan a rigirous examination. A computer today could ace just about any general knowledge test we give it if we tell it how to search for an answer or compute a problem. Does that make it as intelligent as a really academically adept human? Oh and it can do it in a tiny fraction of the time it would take us. Does that make it superhuman?
It may be a red herring to focus on the definition of “intelligence” in this context. If you prefer, taboo the words intelligent and intelligence in this context and simply refer to a computer capable of doing at least everything a regular person can do. The issue is what happens after one has a machine that reaches that point.
But we already have things capable of doing everything a regular person can do. We call them regular people. Are we trying to build another person in digital format here, and if so, why? Just because we want to see if can? Or because we have some big plans for it?
Irrelevant to the question at hand, which is what would happen if a machine had such capabilities. But, if you insist on discussing this issue also, machines with human-like abilities could be very helpful. For example, one might be able to train one of them to do some task, and then make multiple copies of it, much more efficient than individually training lots of humans. Or one could send such AIs into dangerous situations where we might not ethically send a person (whether it would actually be ethical to send an AI is a distinct question.)
Why is it distinct? Whether doing something is an error determines if it’s beneficial to obtain ability and willingness to do it.
It’s distinct when the question is about risk to the human, rather than about the ethics of the task itself. We could make nonsentient nonpersons that nevertheless have humanlike abilities in some broad or narrow sense, so that sacrificing them in some risky or suicidal task doesn’t impact the ethical calculation as it would if we were sending a person.
(I think that’s what JoshuaZ was getting at. The “distinct question” would presumably be that of the AI’s potential personhood.)
Um… we already do all that to a pretty high extent and we don’t need general intelligence in every single facet of human ability to do that. Just make it an expert in its task and that’s all you need.
There are a large number of tasks where the expertise level needed by current technology is woefully insufficient. Anything that has a strong natural language requirement for example.
Oh fun, we’re talking about my advisers’ favorite topic! Yeah, strong natural language is a huge pain and if we had devices that understood human speech well, tech companies would jump on that ASAP.
But here’s the thing. If you want natural language processing, why build a Human 2.0? Why not just build the speech recognition system? It’s making AGI for something like that the equivalent of building a 747 to fly one person across a state? I can see various expert systems coming together as an AGI, but not starting out as such.
It would surprise me if human-level natural-language processing were possible without sitting on top of a fairly sophisticated and robust world-model.
I mean, just as an example, consider how much a system has to know about the world to realize that in your next-to-last sentence, “It’s” is most likely a typo for “Isn’t.”
Granted that one could manually construct and maintain such a model rather than build tools that maintain it automatically based on ongoing observations, but the latter seems like it would pay off over time.
I don’t think this is a good argument. Just because you cannot define something doesn’t mean it’s not a real phenomena or that you cannot reason about it at all. Before we understood fire completely, it was still real and we could reason about it somewhat (fire consumes some things, fire is hot etc.). Similarly, intelligence is a real phenomena that we don’t completely understand and we can still do some reasoning about it. It is meaningful to talk about a computer having “human-level” (I think “human-like” might be more descriptive) intelligence.
If you have no working definition for what you’re trying to discuss, you’re more than likely to be barking up the wrong tree about it. We didn’t understand fire completely, but we knew that it was hot, you couldn’t touch it, and you made it by rubbing dry sticks together really, really fast, or by making a spark with rocks and have it land on dry straw.
Also, where did I say that until I get a definition of intelligence all discussion about the concept is meaningless? I just want to know what criteria an AI must meet to be considered human and match them with what we have so far so I can see how far we might be from those benchmarks. I think it’s a perfectly reasonable way to go about this kind of discussion.
I apologize, the intent of your question was not at all clear to me from your previous post. It sounded to me like you were using this as an argument that SIAI types were clearly wrong headed.
To answer your question then, the relevant dimension of intelligence is something like “ability to design and examine itself similarly to it’s human designers”.
Ok, I’ll buy that. I would agree that any system that could be its own architect and hold meaningful design and code review meetings with its builders would qualify as human-level intelligent.
To clarify: I didn’t mean that such a machine is necessarily “human level intelligent” in all respects, just that that is the characteristic relevant to the idea of an “intelligence explosion”.
Interesting question, Wikipedia does list some requirements.
Wow, if that’s all you got from a post trying to explain the very real difference between acing an intelligence test by figuring things out on your own and having a machine do the same after you give it all the answers and how the suggested equations only measure how many answers were right, not how that feat was accomplished, I don’t even know how to properly respond...
Oh and by the way, in the comments I suggest how to keep track of the machine doing some learning and figuring out to Dr. Legg so there’s another thing to consider. And yes, I’ve had the formal instruction in discrete math to do so.
It is possible that I didn’t explain my point well. The problem I am referring to is your apparent insistence that there are things that machines can’t do that people can and that this is insurmountable. Most of your subclaims are completely reasonable, but the overarching premise that machines can only do what they are programmed to seems to show up in both pieces, and is simply wrong. Even today, that’s not true by most definitions of those terms. Neural nets and genetic algorithms often don’t do what they are told.
Only if you choose to discard any thought to how machines are actually built. There’s no magic going on in that blinking box, just ciruits performing the functions they were designed to do in the order they’re told.
Actually, they do precisely what they’re told because without a fitness function which determines what problem they are to solve in their output and their level of correctness, they just crash the computer. Don’t confuse algorithms that have very generous bounds and allow us to try different possible solutions to the same problem for some sort of thinking or initiative on the computer’s part. And when computers do something weird, it’s because of a bug which sends them persuing their logic in whays programmers never intended, not because they decide to go off on their own.
I can’t tell you how many seemingly bizarre and ridiculous problems I’ve eventually tracked down to a bad loop, or a bad index value, or a missing symbol in a string...
There’s no magic going on inside the two pounds of fatty tissue inside my skull either. Magic is apparently not required for creativity or initiative (whatever those may be).
I’m confused by what you mean by “thinking” and “initiative.” Let’s narrow the field slightly. Would the ability to come up with new definitions and conjectures in math be an example of thinking and initiative?
Calling something a bug doesn’t change the nature of what is happening. That’s just a label. Humans are likely as smart as they are due to runaway sexual selection for intelligence. And then humans got really smart and realized that they could have all the pleasure of sex while avoiding the hassle of reproduction. Is the use of birth-control an example of human initiative or a bug? Does it make a difference?
Yes, but with a caveat. I could teach an ANN how to solve a problem but it would be more or less by random trial and error with a squashing function until each “neuron” has the right weight and activation function. So it will learn how to solve this generic problem, but it won’t be because it traced its way along all the steps.
(Actually I made in mistake in my previous reply, ANNs have no fitness function, that’s a genetic algorithm. ANNs are given an input and a desired output.)
So if you develop a new defintion or conjecture and can state why and how you did it, then develop a proof, you’ve shown thought. Your attempt to suddenly create a new definition or theorem just because you wanted to and were curious rather than just tasked to do it would be initiative.
No, you see, a bug is when a computer does something it’s not supposed to do and handles its data incorrectly. Birth control is actually another approach to reproduction most of the time, delaying progeny until we feel ready to raise them. Those who don’t have children have put their evolutionary desire to provide for themselves above the drive to reproduce and counter that urge with protected sex. So it’s not really a bug as it is a solution to some of the problems posed by reproduction. Now, celibacy is something I’d call a bug and we know from many studies that it’s almost always a really bad idea to forgo sex altogether. Mental health tends to suffer greatly.
Hmm, so would a grad student who is thinking about a thesis problem because their advisor said to think about it be showing initiative? Is a professional mathematician showing initiative? They keep thinking about math because that’s what gives them positive feedback (e.g. salary, tenure, positive remarks from their peers).
Is “incorrectly” a normative or descriptive term? .How is it different than “this program didn’t do what I expected it to do” other than that you label it a bug when the program deviates more from what you wanted to accomplish? Keep in mind that what a human wants isn’t a notion that cleaves reality at the joints.
Ok. So when someone (and I know quite a few people in this category) deliberately uses birth control because they want the pleasure of sex but don’t want to ever have kids, is that a bug in your view?
Dis he/she volunteer to work on a problem and come to the advisor saying that this is the thesis subject? Doesn’t sound like it, so I’d say it’s not. Initiative is doing something that’s not required, but something you feel needs to be done or something you want to do.
Yes. When you need it to return “A” and it retuns “Finland,” it made a mistake which has to be fixed. How it came to that mistake can be found by tracing the logic after the bug manifests itself.
Ok, whan you build a car but the car doesn’t start, I don’t think you’re going to say that the car is just doing what it wants and we humans are just selfishly insisting that it bends to our whims. You’re probably going to take that thing to a mechanic. Same thing with computers, even AI. If you build an AI to learn a language and it doesn’t seem to be able to do so, there’s a bug in the system.
That’s answered in the second sentence of the quote you chose...
Ok. Now, if said grad student did come to the thesis adviser, but their motivation was that they’ve been taught from a very young age that they should do math. Is there initiative?
It seems that a large part of the disagreement is implicit premises here. You seem to be focused on very narrow AI, when the entire issue is what happens when one doesn’t have narrow AI but have AI that has most capabilities that humans have. Let’s set aside whether or not we should build such AIs and whether or not they are possible. Assuming that such entities are possible, do you or do you not think there’s a risk of the AI getting out of control.
Either there’s a miscommunication here or there’s a misunderstanding about how evolution works. An organism that puts its own survival over reproducing is an evolutionary dead end. Historically, lots of humans didn’t want any children, but they didn’t have effective birth control methods, so in the ancestral environment there was minimal evolutionary incentive to remove that preference. It has only been recently that there is widespread and effective birth control. So, what you’ve said is one evolved desire overriding another would still seem to be a bug.
Not sure. You could argue both points in this situation.
Any AI can get out of control. I never denied that. My issue is with how that should be managed, not whether it can happen.
I suppose it would.
Ah. In that case, there’s actually very minimal disagreement.
Can you clarify how it’s helpful to know that my machine only does what it’s been told to do, if I can’t know what I’m telling it to do or be certain what I have told it to do?
I mean, there’s a sense in which humans only do “what they’ve been told to do”, also… we have programs embedded in DNA that manifest themselves in brains that construct minds from experience in constrained ways. (Unless you believe in some kind of magic free will in human minds, in which case this line of reasoning won’t seem sensible to you.) But so what? Knowing that doesn’t make humans harmless.
Additionally, a big part of what SIAI types emphasize is that knowing very precisely and very broadly (at the same time) what humans want is very important. Human desires are very complex, so this is not a simple task.
If you have no idea what you want your AI to do, why are you building it in the first place? I have never built an app that does, you know, anything and whatever. It’ll just be muddled mess that probably won’t even compile.
No we do not. This is not how biology works. Brains are self-organizing structures built by a combination of cellular signals and environmental cues. All that DNA does is to regulate what proteins the cell will manufacture. Development goes well beyond that.
I’m not sure how you got from my question to your answer. I’m not talking at all about programmers not having intentions, and I agree with you that in pretty much all cases they do have intentions.
I’ll assume that I wasn’t clear, rather than that you’re willing to ignore what’s actually being said in favor of what lets you make a more compelling argument, and will attempt to be clearer.
You keep suggesting that there’s no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.
At the same time, you admit that computer programs sometimes do things their programmers didn’t intend for them to do. I might have written a stupid bug that causes the program to delete the contents of my hard drive, for example.
I agree completely that, in doing so, it is merely doing what I told it to do: I’m the one who wrote that stupid bug, it didn’t magically come out of nowhere, the program doesn’t have any mysterious kind of free will or anything. It’s just a program I wrote.
But I don’t see why that should be particularly reassuring. The fact remains that the contents of my hard drive are deleted, and I didn’t want them to be. That I’m the one who told the program to delete them makes no difference I care about; far more salient to me is that I didn’t intend for the program to delete them.
And the more a program is designed to flexibly construct strategies for achieving particular goals in the face of unpredictable environments, the harder it is to predict what it is that I’m actually telling my program to do, regardless of what I intend for it to do.
In other words: “I can’t know what I’m telling it to do or be certain what I have told it to do.”
Sure, once it deletes the files, I can (in principle) look back over the source code and say “Oh, I see why that happened.” But that doesn’t get me my files back.
And yet, remarkably, brains don’t “self-organize” in the absence of that regulation.
You’re right, of course, that the correct environment is also crucial; DNA won’t magically turn into a brain without a very specific environment in which to manifest.
Then again, source code won’t magically turn into a running program without a very specific environment either, and quite a lot of the information defining that running program comes from the compiler and the hardware platform rather than the source code… and yet we have no significant difficulty equating a running program with its source code.
(Sure, sometimes bugs turn out to be in the compiler or the hardware, but even halfway competent programmers don’t look there except as a matter of last resort. If the running program is doing something I didn’t intend, it’s most likely that the source code includes an instruction I didn’t intend to give.)
No, I just keep saying that we don’t need to program them to “like rewards and fear punishments” and train them like we’d train dogs.
Oh no, it’s not. I have several posts on my blog detailing how bugs like that could actually turn a whole machine army against us and turn Terminator into a reality rather than a cheesy robots-take-over-the-world-for-shits-and-giggles flick.
But the source code isn’t like DNA in an organism. Source code covers so much more ground than that. Imagine having an absolute blueprint of how every cell cluster in your body will react to any stimuli through your entire life and every process it will undertake from now until your death, including how it will age. That would be source code. Your DNA is not ever nearly that complete. It’s more like a list of suggestions and blueprints for raw materials.
(shrug) OK, fair enough.
I agree with you that reward/punishment conditioning of software is a goofy idea.
I was reading your comment here to indicate that we can constrain the behavior of human-level AGIs by just putting appropriate constraints in the code. (“You don’t want the machine to do something? Put in a boundry. [..] with a machine, you can just tell it not to do that.”)
I think that idea is importantly wrong, which is why I was responding to it, but if you don’t actually believe that then we apparently don’t have a disagreement.
Re: source code… if we’re talking about code that is capable of itself generating executable code as output in response to situations that arise (which seems implicit in the idea of a human-level AGI, given that humans are capable of generating executable code), it isn’t at all clear to me that its original source code comprises in any kind of useful way an absolute blueprint for how every part of it will react to any stimuli.
Again, sure, I’m not positing magic: whatever it does, it does because of the interaction between its source code and the environment in which it runs, there’s no kind of magic third factor. So, sure, given the source code and an accurate specification of its environment (including its entire relevant history), I can in principle determine precisely what it will do. Absolutely agreed. (Of course, in practice that might be so complicated that I can’t actually do it, but you aren’t claiming otherwise.)
If you don’t think the same is true of humans, then we disagree about humans, but I think that’s incidental.
Again, it really shouldn’t be doing that. It should have the capacity to learn new skills and build new neural networks to do so. That doesn’t require new code, it just requires a routine to initialize a new set of ANN objects at runtime.
If it somehow follows from that that there’s an absolute blueprint in it for how every part of it will react to any stimuli in a way that is categorically different from how human genetics specify how humans will respond to any environment, then I don’t follow the connection… sorry. I have only an interested layman’s understanding of ANNs.