If you have no idea what you want your AI to do, why are you building it in the first place?
I’m not sure how you got from my question to your answer. I’m not talking at all about programmers not having intentions, and I agree with you that in pretty much all cases they do have intentions.
I’ll assume that I wasn’t clear, rather than that you’re willing to ignore what’s actually being said in favor of what lets you make a more compelling argument, and will attempt to be clearer.
You keep suggesting that there’s no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.
At the same time, you admit that computer programs sometimes do things their programmers didn’t intend for them to do. I might have written a stupid bug that causes the program to delete the contents of my hard drive, for example.
I agree completely that, in doing so, it is merely doing what I told it to do: I’m the one who wrote that stupid bug, it didn’t magically come out of nowhere, the program doesn’t have any mysterious kind of free will or anything. It’s just a program I wrote.
But I don’t see why that should be particularly reassuring. The fact remains that the contents of my hard drive are deleted, and I didn’t want them to be. That I’m the one who told the program to delete them makes no difference I care about; far more salient to me is that I didn’t intend for the program to delete them.
And the more a program is designed to flexibly construct strategies for achieving particular goals in the face of unpredictable environments, the harder it is to predict what it is that I’m actually telling my program to do, regardless of what I intend for it to do.
In other words: “I can’t know what I’m telling it to do or be certain what I have told it to do.”
Sure, once it deletes the files, I can (in principle) look back over the source code and say “Oh, I see why that happened.” But that doesn’t get me my files back.
Brains are self-organizing structures built by a combination of cellular signals and environmental cues. All that DNA does is to regulate what proteins the cell will manufacture. Development goes well beyond that.
And yet, remarkably, brains don’t “self-organize” in the absence of that regulation.
You’re right, of course, that the correct environment is also crucial; DNA won’t magically turn into a brain without a very specific environment in which to manifest.
Then again, source code won’t magically turn into a running program without a very specific environment either, and quite a lot of the information defining that running program comes from the compiler and the hardware platform rather than the source code… and yet we have no significant difficulty equating a running program with its source code.
(Sure, sometimes bugs turn out to be in the compiler or the hardware, but even halfway competent programmers don’t look there except as a matter of last resort. If the running program is doing something I didn’t intend, it’s most likely that the source code includes an instruction I didn’t intend to give.)
You keep suggesting that there’s no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.
No, I just keep saying that we don’t need to program them to “like rewards and fear punishments” and train them like we’d train dogs.
I agree completely that, in doing so, it is merely doing what I told it to do: I’m the one who wrote that stupid bug, it didn’t magically come out of nowhere, the program doesn’t have any mysterious kind of free will or anything. It’s just a program I wrote. But I don’t see why that should be particularly reassuring.
Oh no, it’s not. I have several posts on my blog detailing how bugs like that could actually turn a whole machine army against us and turn Terminator into a reality rather than a cheesy robots-take-over-the-world-for-shits-and-giggles flick.
… and yet we have no significant difficulty equating a running program with its source code.
But the source code isn’t like DNA in an organism. Source code covers so much more ground than that. Imagine having an absolute blueprint of how every cell cluster in your body will react to any stimuli through your entire life and every process it will undertake from now until your death, including how it will age. That would be source code. Your DNA is not ever nearly that complete. It’s more like a list of suggestions and blueprints for raw materials.
No, I just keep saying that we don’t need to program them to “like rewards and fear punishments” and train them like we’d train dogs.
(shrug) OK, fair enough.
I agree with you that reward/punishment conditioning of software is a goofy idea.
I was reading your comment here to indicate that we can constrain the behavior of human-level AGIs by just putting appropriate constraints in the code. (“You don’t want the machine to do something? Put in a boundry. [..] with a machine, you can just tell it not to do that.”)
I think that idea is importantly wrong, which is why I was responding to it, but if you don’t actually believe that then we apparently don’t have a disagreement.
Re: source code… if we’re talking about code that is capable of itself generating executable code as output in response to situations that arise (which seems implicit in the idea of a human-level AGI, given that humans are capable of generating executable code), it isn’t at all clear to me that its original source code comprises in any kind of useful way an absolute blueprint for how every part of it will react to any stimuli.
Again, sure, I’m not positing magic: whatever it does, it does because of the interaction between its source code and the environment in which it runs, there’s no kind of magic third factor. So, sure, given the source code and an accurate specification of its environment (including its entire relevant history), I can in principle determine precisely what it will do. Absolutely agreed. (Of course, in practice that might be so complicated that I can’t actually do it, but you aren’t claiming otherwise.)
If you don’t think the same is true of humans, then we disagree about humans, but I think that’s incidental.
… if we’re talking about code that is capable of itself generating executable code as output in response to situations that arise
Again, it really shouldn’t be doing that. It should have the capacity to learn new skills and build new neural networks to do so. That doesn’t require new code, it just requires a routine to initialize a new set of ANN objects at runtime.
If it somehow follows from that that there’s an absolute blueprint in it for how every part of it will react to any stimuli in a way that is categorically different from how human genetics specify how humans will respond to any environment, then I don’t follow the connection… sorry. I have only an interested layman’s understanding of ANNs.
I’m not sure how you got from my question to your answer. I’m not talking at all about programmers not having intentions, and I agree with you that in pretty much all cases they do have intentions.
I’ll assume that I wasn’t clear, rather than that you’re willing to ignore what’s actually being said in favor of what lets you make a more compelling argument, and will attempt to be clearer.
You keep suggesting that there’s no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.
At the same time, you admit that computer programs sometimes do things their programmers didn’t intend for them to do. I might have written a stupid bug that causes the program to delete the contents of my hard drive, for example.
I agree completely that, in doing so, it is merely doing what I told it to do: I’m the one who wrote that stupid bug, it didn’t magically come out of nowhere, the program doesn’t have any mysterious kind of free will or anything. It’s just a program I wrote.
But I don’t see why that should be particularly reassuring. The fact remains that the contents of my hard drive are deleted, and I didn’t want them to be. That I’m the one who told the program to delete them makes no difference I care about; far more salient to me is that I didn’t intend for the program to delete them.
And the more a program is designed to flexibly construct strategies for achieving particular goals in the face of unpredictable environments, the harder it is to predict what it is that I’m actually telling my program to do, regardless of what I intend for it to do.
In other words: “I can’t know what I’m telling it to do or be certain what I have told it to do.”
Sure, once it deletes the files, I can (in principle) look back over the source code and say “Oh, I see why that happened.” But that doesn’t get me my files back.
And yet, remarkably, brains don’t “self-organize” in the absence of that regulation.
You’re right, of course, that the correct environment is also crucial; DNA won’t magically turn into a brain without a very specific environment in which to manifest.
Then again, source code won’t magically turn into a running program without a very specific environment either, and quite a lot of the information defining that running program comes from the compiler and the hardware platform rather than the source code… and yet we have no significant difficulty equating a running program with its source code.
(Sure, sometimes bugs turn out to be in the compiler or the hardware, but even halfway competent programmers don’t look there except as a matter of last resort. If the running program is doing something I didn’t intend, it’s most likely that the source code includes an instruction I didn’t intend to give.)
No, I just keep saying that we don’t need to program them to “like rewards and fear punishments” and train them like we’d train dogs.
Oh no, it’s not. I have several posts on my blog detailing how bugs like that could actually turn a whole machine army against us and turn Terminator into a reality rather than a cheesy robots-take-over-the-world-for-shits-and-giggles flick.
But the source code isn’t like DNA in an organism. Source code covers so much more ground than that. Imagine having an absolute blueprint of how every cell cluster in your body will react to any stimuli through your entire life and every process it will undertake from now until your death, including how it will age. That would be source code. Your DNA is not ever nearly that complete. It’s more like a list of suggestions and blueprints for raw materials.
(shrug) OK, fair enough.
I agree with you that reward/punishment conditioning of software is a goofy idea.
I was reading your comment here to indicate that we can constrain the behavior of human-level AGIs by just putting appropriate constraints in the code. (“You don’t want the machine to do something? Put in a boundry. [..] with a machine, you can just tell it not to do that.”)
I think that idea is importantly wrong, which is why I was responding to it, but if you don’t actually believe that then we apparently don’t have a disagreement.
Re: source code… if we’re talking about code that is capable of itself generating executable code as output in response to situations that arise (which seems implicit in the idea of a human-level AGI, given that humans are capable of generating executable code), it isn’t at all clear to me that its original source code comprises in any kind of useful way an absolute blueprint for how every part of it will react to any stimuli.
Again, sure, I’m not positing magic: whatever it does, it does because of the interaction between its source code and the environment in which it runs, there’s no kind of magic third factor. So, sure, given the source code and an accurate specification of its environment (including its entire relevant history), I can in principle determine precisely what it will do. Absolutely agreed. (Of course, in practice that might be so complicated that I can’t actually do it, but you aren’t claiming otherwise.)
If you don’t think the same is true of humans, then we disagree about humans, but I think that’s incidental.
Again, it really shouldn’t be doing that. It should have the capacity to learn new skills and build new neural networks to do so. That doesn’t require new code, it just requires a routine to initialize a new set of ANN objects at runtime.
If it somehow follows from that that there’s an absolute blueprint in it for how every part of it will react to any stimuli in a way that is categorically different from how human genetics specify how humans will respond to any environment, then I don’t follow the connection… sorry. I have only an interested layman’s understanding of ANNs.