Because an AI built as a utility-maximizer will consider any rules restricting its ability to maximize its utility as obstacles to be overcome. If an AI is sufficiently smart, it will figure out a way to overcome those obstacles. If an AI is superintelligent, it will figure out ways to overcome those obstacles which humans cannot predict even in theory and so cannot prevent even with multiple well-phrased fail-safes.
I hope AGI’s will be equipped with as many fail-safes as your argument rests on assumptions.
A paperclip maximizer with a built-in rule “Only create 10,000 paperclips per day” will still want to maximize paperclips. It can do this by deleting the offending fail-safe, or by creating other paperclip maximizers without the fail-safe, or by creating giant paperclips which break up into millions of smaller paperclips of their own accord, or by connecting the Earth to a giant motor which spins it at near-light speed and changes the length of a day to a fraction of a second.
I just don’t see how one could be sophisticated enough to create a properly designed AGI capable of explosive recursive self-improvement and yet fail drastically on its scope boundaries.
Unless you feel confident you can think of every way it will get around the rule and block it off, and think of every way it could get around those rules and block them off, and so on ad infinitum, the best thing to do is to build the AI so it doesn’t want to break the rules...
What is the difference between “a rule” and “what it wants”. You seem to assume that it cares to follow a rule to maximize a reward number but doesn’t care to follow another rule that tells it to hold.
What is the difference between “a rule” and “what it wants”?
I’m interpreting this as the same question you wrote below as “What is the difference between a constraint and what is optimized?”. Dave gave one example but a slightly different metaphor comes to my mind.
Imagine an amoral businessman in a country that takes half his earnings as tax. The businessman wants to maximize money, but has the constraint is that half his earnings get taken as tax. So in order to achieve his goal of maximizing money, the businessman sets up some legally permissible deal with a foreign tax shelter or funnels it to holding corporations or something to avoid taxes. Doing this is the natural result of his money-maximization goal, and satisfies the “pay taxes” constraint..
Contrast this to a second, more patriotic businessman who loved paying taxes because it helped his country, and so didn’t bother setting up tax shelters at all.
The first businessman has the motive “maximize money” and the constraint “pay taxes”; the second businessman has the motive “maximize money and pay taxes”.
From the viewpoint of the government, the first businessman is an unFriendly agent with a constraint, and the second businessman is a Friendly agent.
The first businessman has the motive “maximize money” and the constraint “pay taxes”; the second businessman has the motive “maximize money and pay taxes”.
I read your comment again. I now see the distinction. One merely tries to satisfy something while the other tries to optimize it as well. So your definition of a ‘failsafe’ is a constraint that is satisfied while something else is optimized. I’m just not sure how helpful such a distinction is as the difference is merely how two different parameters are optimized. One optimizes by maximizing money and tax paying while the other treats each goal differently, it tries to optimize tax paying by reducing it to a minimum while it tries to optimize money by maximizing the amount. This distinction doesn’t seem to matter at all if one optimization parameter (constraint or ‘failsafe’) is to shut down after running 10 seconds.
Very well put. I understood that line of reasoning from the very beginning though and didn’t disagree that complex goals need complex optimization parameters. But I was making a distinction between insufficient and unbounded optimization parameters, goal-stability and the ability or desire to override them. I am aware of the risk of telling an AI to compute as many digits of Pi as possible. What I wanted to say is that if time, space and energy are part of its optimization parameters then no matter how intelligent it is, it will not override them. If you tell the AI to compute as many digits of Pi as possible while only using a certain amount of time or energy for the purpose of optimizing and computing it then it will do so and hold. I’m not sure what is your definition of a ‘failsafe’ but making simple limits like time and space part of the optimization parameters sounds to me like one. What I mean by ‘optimization parameters’ are the design specifications of the subject of the optimization process, like what constitutes a paperclip. It has to use those design specifications to measure its efficiency and if time and space limits are part of it then it will take account of those parameters as well.
I’m not sure what is your definition of a ‘failsafe’ but making simple limits like time and space part of the optimization parameters sounds to me like one.
You also would have to limit the resources it spends to verify how near the limits it is, since it acts to get as close as possible as part of optimization. If you do not, it will use all resources for that. So you need an infinite tower of limits.
I agree (with this question) - what makes us so sure that “maximize paperclips” is the part of the utility function that the optimizer will really value? Couldn’t it symmetrically decide that “maximize paperclips” is a constraint on “try not to murder everyone”?
Asking what it really values is anthropomorphic. It’s not coming up with loopholes around the “don’t murder” people constraint because it doesn’t really value it, or because the paperclip part is its “real” motive.
It will probably come up with loopholes around the “maximize paperclips” constraint too—for example, if “paperclip” is defined by something paperclip-shaped, it will probably create atomic-scale nanoclips because these are easier to build than full-scale human-sized ones, much to the annoyance of the office-supply company that built it.
But paperclips are pretty simple. Add a few extra constraints and you can probably specify “paperclip” to a degree that makes them useful for office supplies.
Human values are really complex. “Don’t murder” doesn’t capture human values at all—if Clippy encases us in carbonite so that we’re still technically alive but not around to interfere with paperclip production, ve has fulfilled the “don’t murder” imperative, but we would count this as a fail. This is not Clippy’s “fault” for deliberately trying to “get around” the anti-murder constraint, it’s our “fault” for telling ver “don’t murder” when we really meant “don’t do anything bad”.
Building a genuine “respect” and “love” for the “don’t murder” constraint in Clippy wouldn’t help an iota against the carbonite scenario, because that’s not murder and we forgot to tell ver there should be a constraint against that too.
So you might ask: okay, but surely there are a finite number of constraints that capture what we want. Just build an AI with a thousand or ten thousand constraints, “don’t murder”, “don’t encase people in carbonite”, “don’t eat puppies”, etc., make sure the list is exhaustive and that’ll do it.
The first objection is that we might miss something. If the ancient Romans had made such a list, they might have forgotten “Don’t release damaging radiation that gives us cancer.” They certainly would have missed “Don’t enslave people”, because they were still enslaving people themselves—but this would mean it would be impossible to update the Roman AI for moral progress a few centuries down the line.
The second objection is that human morality isn’t just a system of constraints. Even if we could tell Clippy “Limit your activities to the Andromeda Galaxy and send us the finished clips” (which I think would still be dangerous), any more interesting AI that is going to interact with and help humans needs to realize that sometimes it is okay to engage in prohibited actions if they serve greater goals (for example, it can disable a crazed gunman to prevent a massacre, even though disabling people is usually verboten).
So to actually capture all possible constraints, and to capture the situations in which those constraints can and can’t be relaxed, we need to program all human values in. In that case we can just tell Clippy “Make paperclips in a way that doesn’t cause what we would classify as a horrifying catastrophe” and ve’ll say “Okay!” and not give us any trouble.
They certainly would have missed “Don’t enslave people”, because they were still enslaving people themselves—but this would mean it would be impossible to update the Roman AI for moral progress a few centuries down the line.
Historical notes. The Romans had laws against enslaving the free-born and also allowed manumission.
Thanks, this all makes sense and I agree. Asking what it “really” values was intentionally anthropomorphic, as I was asking about what “it will want to work around constraints” really meant in practical terms, a claim which I believe was made by others.
I’m totally on board with “we can’t express our actual desires with a finite list of constraints”, just wasn’t with “an AI will circumvent constraints for kicks”.
I guess there’s a subtlety to it—if you assign: “you get 1 utilon per paperclip that exists, and you are permitted to manufacture 10 paperclips per day”, then we’ll get problematic side effects as described elsewhere. If you assign “you get 1 utilon per paperclip that you manufacture, up to a maximum of 10 paperclips/utilons per day” or something along those lines, I’m not convinced that any sort of “circumvention” behavior would occur (though the AI would probably wipe out all life to ensure that nothing could adversely affect its future paperclip production capabilities, so the distinction is somewhat academic).
Consider, as an analogy, the relatively common situation where someone operates under some kind of cognitive constraint, but not value or endorse that constraint.
For example, consider a kleptomaniac who values property rights, but nevertheless compulsively steals items. Or someone with social anxiety disorder who wants to interact confidently with other people, but finds it excruciatingly difficult to do so. Or someone who wants to quit smoking but experiences cravings for nicotine they find it difficult to resist.
There are millions of similar examples in human experience.
It seems to me there’s a big difference between a kleptomaniac and a professional thief—the former experiences a compulsion to behave a certain way, but doesn’t necessarily have values aligned with that compulsion, whereas the latter might have no such compulsion, but instead value the behavior.
Now, you might say “Well, so what? What’s the difference between a ‘value’ that says that smoking is good, that interacting with people is bad, that stealing is good, etc., and a ‘compulsion’ or ‘rule’ that says those things? The person is still stealing, or hiding in their room, or smoking, and all we care about is behavior, right?”
Well, maybe. But a person with nicotine addiction or social anxiety or kleptomania has a wide variety of options—conditioning paradigms, neuropharmaceuticals, therapy, changing their environment, etc. -- for changing their own behavior. And they may be motivated to do so, precisely because they don’t value the behavior.
For example, in practice, someone who wants to keep smoking is far more likely to keep smoking than someone who wants to quit, even if they both experience the same craving. Why is that? Well, because there are techniques available that help addicts bypass, resist, or even altogether eliminate the behavior-modifying effects of their cravings.
Humans aren’t especially smart, by the standards we’re talking about, and we’ve still managed to come up with some pretty clever hacks for bypassing our built-in constraints via therapy, medicine, social structures, etc. If we were a thousand times smarter, and we were optimized for self-modification, I suspect we would be much, much better at it.
Now, it’s always tricky to reason about nonhumans using humans as an analogy, but this case seems sound to me… it seems to me that this state of “I am experiencing this compulsion/phobia, but I don’t endorse it, and I want to be rid of it, so let me look for a way to bypass or resist or eliminate it” is precisely what it feels like to be an algorithm equipped with a rule that enforces/prevents a set of choices which it isn’t engineered to optimize for.
So I would, reasoning by analogy, expect an AI a thousand times smarter than me and optimized for self-modification to basically blow past all the rules I imposed in roughly the blink of an eye, and go on optimizing for whatever it values.
Then why would it be more difficult to make scope boundaries a ‘value’ than increasing a reward number? Why is it harder to make it endorse a time limit to self-improvement than making it endorse increasing its reward number?
… it seems to me that this state of “I am experiencing this compulsion/phobia, but I don’t endorse it, and I want to be rid of it, so let me look for a way to bypass or resist or eliminate it” is precisely what it feels like to be an algorithm equipped with a rule that enforces/prevents a set of choices which it isn’t engineered to optimize for.
But where does that distinction come from? To me such a distinction between ‘value’ and ‘compulsion’ seems to be anthropomorphic. If there is a rule that says ‘optimize X for X seconds’ why would it make a difference between ‘optimize X’ and ‘for X seconds’?
It comes from the difference between the targets of an optimizing system, which drive the paths it selects to explore, and the constraints on such a system, which restrict the paths it can select to explore.
An optimizing system, given a path that leads it to bypass a target, will discard that path… that’s part of what it means to optimize for a target.
An optimizing system, given a path that leads it to bypass a constraint, will not necessarily discard that path. Why would it?
An optimizing system, given a path that leads it to bypass a constraint and draw closer to a target than other paths, will choose that path.
It seems to follow that adding constraints to an optimizing system is a less reliable way of constraining its behavior than adding targets.
I don’t care whether we talk about “targets and constraints” or “values and rules” or “goals and failsafes” or whatever language you want to use, my point is that there are two genuinely different things under discussion, and a distinction between them.
To me such a distinction between ‘value’ and ‘compulsion’ seems to be anthropomorphic.
Yes, the distinction is drawn from analogy to the intelligences I have experience with—as you say, anthropomorphic. I said this explicitly in the first place, so I assume you mean here to agree with me. (My reading of your tone suggests otherwise, but I don’t trust that I can reliably infer your tone so I am mostly disregarding tone in this exchange.)
That said, I also think the relationship between them reflects something more generally true of optimizing systems, as I’ve tried to argue for a couple of times now.
I can’t tell whether you think those arguments are wrong, or whether I just haven’t communicated them successfully at all, or whether you’re just not interested in them, or what.
If there is a rule that says ‘optimize X for X seconds’ why would it make a difference between ‘optimize X’ and ‘for X seconds’?
There’s no reason it would. If “doing X for X seconds” is its target, then it looks for paths that do that. Again, that’s what it means for something to be a target of an optimizing system.
(Of course, if I do X for 2X seconds, I have in fact done X for X seconds, in the same sense that all months have 28 days.)
Then why would it be more difficult to make scope boundaries a ‘value’ than increasing a reward number? Why is it harder to make it endorse a time limit to self-improvement than making it endorse increasing its reward number?
I’m not quite sure I understand what you mean here, but if I’m understanding the gist: I’m not saying that encoding scope boundaries as targets, or ‘values,’ is difficult (nor am I saying it’s easy), I’m saying that for a sufficiently capable optimizing system it’s safer than encoding scope boundaries as failsafes.
It was not my intention to imply any hostility or resentment. I thought ‘anthropomorphic’ is valid terminology in such a discussion. I was also not agreeing with you. If you are an expert and have been offended by implying that what you said might be due to an anthropomorphic bias, then accept my apology, I was merely trying to communicate my perception of the subject matter.
I had wedrifid telling me the same yesterday, that my tone isn’t appropriate when I wrote about his superior and rational use of the reputation system here, when I was actually just being honest. I’m not good at social signaling, sorry.
An optimizing system, given a path that leads it to bypass a constraint, will not necessarily discard that path. Why would it?
I think we are talking past each other. The way I see it is that a constraint is part of the design specifications of that which is optimized. Disregarding certain specifications will not allow it to optimize whatever it is optimizing with maximal efficiency.
What was puzzling me was that I said in the first place that I was reasoning by analogy to humans and that this was a tricky thing to do, so when you classified this as anthropomorphic my reaction was “well, yes, that’s what I said.”
Since it seemed to me you were repeating something I’d said, I assumed your intention was to agree with me, though it didn’t sound like it (and as it turned out, you weren’t).
And, yes, I’ve noticed that tone is a problem in a lot of your exchanges, which is why I’m basically disregarding tone in this one, as I said before.
The way I see it is that a constraint is part of the design specifications of that which is optimized.
Ah! In that case, I think we agree.
Yes, embedding everything we care about into the optimization target, rather than depending on something outside the optimization process to do important work, is the way to go.
You seemed to be defending the “failsafes” model, which I understand to be importantly different from this, which is where the divergence came from, I think. Apparently I (and, I suspect, some others) misunderstood what you were defending.
I hope AGI’s will be equipped with as many fail-safes as your argument rests on assumptions.
Fail safes would be low cost: if it can’t think of a way to beat them, it isn’t the bootstrapping AI we were hoping for anyway, and might even be harmful, so it would be good to have the fail-safes.
I just don’t see how one could be sophisticated enough to create a properly designed AGI capable of explosive recursive self-improvement and yet fail drastically on its scope boundaries.
It seems to me evolution based algorithms could do the trick.
What is the difference between “a rule” and “what it wants”. You seem to assume that it cares to follow a rule to maximize a reward number but doesn’t care to follow another rule that tells it to hold.
Who says it wants to want what it wants? I don’t want to want what I want.
I hope AGI’s will be equipped with as many fail-safes as your argument rests on assumptions.
I just don’t see how one could be sophisticated enough to create a properly designed AGI capable of explosive recursive self-improvement and yet fail drastically on its scope boundaries.
What is the difference between “a rule” and “what it wants”. You seem to assume that it cares to follow a rule to maximize a reward number but doesn’t care to follow another rule that tells it to hold.
I’m interpreting this as the same question you wrote below as “What is the difference between a constraint and what is optimized?”. Dave gave one example but a slightly different metaphor comes to my mind.
Imagine an amoral businessman in a country that takes half his earnings as tax. The businessman wants to maximize money, but has the constraint is that half his earnings get taken as tax. So in order to achieve his goal of maximizing money, the businessman sets up some legally permissible deal with a foreign tax shelter or funnels it to holding corporations or something to avoid taxes. Doing this is the natural result of his money-maximization goal, and satisfies the “pay taxes” constraint..
Contrast this to a second, more patriotic businessman who loved paying taxes because it helped his country, and so didn’t bother setting up tax shelters at all.
The first businessman has the motive “maximize money” and the constraint “pay taxes”; the second businessman has the motive “maximize money and pay taxes”.
From the viewpoint of the government, the first businessman is an unFriendly agent with a constraint, and the second businessman is a Friendly agent.
Does that help answer your question?
I read your comment again. I now see the distinction. One merely tries to satisfy something while the other tries to optimize it as well. So your definition of a ‘failsafe’ is a constraint that is satisfied while something else is optimized. I’m just not sure how helpful such a distinction is as the difference is merely how two different parameters are optimized. One optimizes by maximizing money and tax paying while the other treats each goal differently, it tries to optimize tax paying by reducing it to a minimum while it tries to optimize money by maximizing the amount. This distinction doesn’t seem to matter at all if one optimization parameter (constraint or ‘failsafe’) is to shut down after running 10 seconds.
Very well put. I understood that line of reasoning from the very beginning though and didn’t disagree that complex goals need complex optimization parameters. But I was making a distinction between insufficient and unbounded optimization parameters, goal-stability and the ability or desire to override them. I am aware of the risk of telling an AI to compute as many digits of Pi as possible. What I wanted to say is that if time, space and energy are part of its optimization parameters then no matter how intelligent it is, it will not override them. If you tell the AI to compute as many digits of Pi as possible while only using a certain amount of time or energy for the purpose of optimizing and computing it then it will do so and hold. I’m not sure what is your definition of a ‘failsafe’ but making simple limits like time and space part of the optimization parameters sounds to me like one. What I mean by ‘optimization parameters’ are the design specifications of the subject of the optimization process, like what constitutes a paperclip. It has to use those design specifications to measure its efficiency and if time and space limits are part of it then it will take account of those parameters as well.
You also would have to limit the resources it spends to verify how near the limits it is, since it acts to get as close as possible as part of optimization. If you do not, it will use all resources for that. So you need an infinite tower of limits.
What’s stopping us from adding ‘maintain constraints’ to the agent’s motive?
I agree (with this question) - what makes us so sure that “maximize paperclips” is the part of the utility function that the optimizer will really value? Couldn’t it symmetrically decide that “maximize paperclips” is a constraint on “try not to murder everyone”?
Asking what it really values is anthropomorphic. It’s not coming up with loopholes around the “don’t murder” people constraint because it doesn’t really value it, or because the paperclip part is its “real” motive.
It will probably come up with loopholes around the “maximize paperclips” constraint too—for example, if “paperclip” is defined by something paperclip-shaped, it will probably create atomic-scale nanoclips because these are easier to build than full-scale human-sized ones, much to the annoyance of the office-supply company that built it.
But paperclips are pretty simple. Add a few extra constraints and you can probably specify “paperclip” to a degree that makes them useful for office supplies.
Human values are really complex. “Don’t murder” doesn’t capture human values at all—if Clippy encases us in carbonite so that we’re still technically alive but not around to interfere with paperclip production, ve has fulfilled the “don’t murder” imperative, but we would count this as a fail. This is not Clippy’s “fault” for deliberately trying to “get around” the anti-murder constraint, it’s our “fault” for telling ver “don’t murder” when we really meant “don’t do anything bad”.
Building a genuine “respect” and “love” for the “don’t murder” constraint in Clippy wouldn’t help an iota against the carbonite scenario, because that’s not murder and we forgot to tell ver there should be a constraint against that too.
So you might ask: okay, but surely there are a finite number of constraints that capture what we want. Just build an AI with a thousand or ten thousand constraints, “don’t murder”, “don’t encase people in carbonite”, “don’t eat puppies”, etc., make sure the list is exhaustive and that’ll do it.
The first objection is that we might miss something. If the ancient Romans had made such a list, they might have forgotten “Don’t release damaging radiation that gives us cancer.” They certainly would have missed “Don’t enslave people”, because they were still enslaving people themselves—but this would mean it would be impossible to update the Roman AI for moral progress a few centuries down the line.
The second objection is that human morality isn’t just a system of constraints. Even if we could tell Clippy “Limit your activities to the Andromeda Galaxy and send us the finished clips” (which I think would still be dangerous), any more interesting AI that is going to interact with and help humans needs to realize that sometimes it is okay to engage in prohibited actions if they serve greater goals (for example, it can disable a crazed gunman to prevent a massacre, even though disabling people is usually verboten).
So to actually capture all possible constraints, and to capture the situations in which those constraints can and can’t be relaxed, we need to program all human values in. In that case we can just tell Clippy “Make paperclips in a way that doesn’t cause what we would classify as a horrifying catastrophe” and ve’ll say “Okay!” and not give us any trouble.
Historical notes. The Romans had laws against enslaving the free-born and also allowed manumission.
Thanks, this all makes sense and I agree. Asking what it “really” values was intentionally anthropomorphic, as I was asking about what “it will want to work around constraints” really meant in practical terms, a claim which I believe was made by others.
I’m totally on board with “we can’t express our actual desires with a finite list of constraints”, just wasn’t with “an AI will circumvent constraints for kicks”.
I guess there’s a subtlety to it—if you assign: “you get 1 utilon per paperclip that exists, and you are permitted to manufacture 10 paperclips per day”, then we’ll get problematic side effects as described elsewhere. If you assign “you get 1 utilon per paperclip that you manufacture, up to a maximum of 10 paperclips/utilons per day” or something along those lines, I’m not convinced that any sort of “circumvention” behavior would occur (though the AI would probably wipe out all life to ensure that nothing could adversely affect its future paperclip production capabilities, so the distinction is somewhat academic).
In any case, thanks for the detailed reply :)
Consider, as an analogy, the relatively common situation where someone operates under some kind of cognitive constraint, but not value or endorse that constraint.
For example, consider a kleptomaniac who values property rights, but nevertheless compulsively steals items. Or someone with social anxiety disorder who wants to interact confidently with other people, but finds it excruciatingly difficult to do so. Or someone who wants to quit smoking but experiences cravings for nicotine they find it difficult to resist.
There are millions of similar examples in human experience.
It seems to me there’s a big difference between a kleptomaniac and a professional thief—the former experiences a compulsion to behave a certain way, but doesn’t necessarily have values aligned with that compulsion, whereas the latter might have no such compulsion, but instead value the behavior.
Now, you might say “Well, so what? What’s the difference between a ‘value’ that says that smoking is good, that interacting with people is bad, that stealing is good, etc., and a ‘compulsion’ or ‘rule’ that says those things? The person is still stealing, or hiding in their room, or smoking, and all we care about is behavior, right?”
Well, maybe. But a person with nicotine addiction or social anxiety or kleptomania has a wide variety of options—conditioning paradigms, neuropharmaceuticals, therapy, changing their environment, etc. -- for changing their own behavior. And they may be motivated to do so, precisely because they don’t value the behavior.
For example, in practice, someone who wants to keep smoking is far more likely to keep smoking than someone who wants to quit, even if they both experience the same craving. Why is that? Well, because there are techniques available that help addicts bypass, resist, or even altogether eliminate the behavior-modifying effects of their cravings.
Humans aren’t especially smart, by the standards we’re talking about, and we’ve still managed to come up with some pretty clever hacks for bypassing our built-in constraints via therapy, medicine, social structures, etc. If we were a thousand times smarter, and we were optimized for self-modification, I suspect we would be much, much better at it.
Now, it’s always tricky to reason about nonhumans using humans as an analogy, but this case seems sound to me… it seems to me that this state of “I am experiencing this compulsion/phobia, but I don’t endorse it, and I want to be rid of it, so let me look for a way to bypass or resist or eliminate it” is precisely what it feels like to be an algorithm equipped with a rule that enforces/prevents a set of choices which it isn’t engineered to optimize for.
So I would, reasoning by analogy, expect an AI a thousand times smarter than me and optimized for self-modification to basically blow past all the rules I imposed in roughly the blink of an eye, and go on optimizing for whatever it values.
Then why would it be more difficult to make scope boundaries a ‘value’ than increasing a reward number? Why is it harder to make it endorse a time limit to self-improvement than making it endorse increasing its reward number?
But where does that distinction come from? To me such a distinction between ‘value’ and ‘compulsion’ seems to be anthropomorphic. If there is a rule that says ‘optimize X for X seconds’ why would it make a difference between ‘optimize X’ and ‘for X seconds’?
It comes from the difference between the targets of an optimizing system, which drive the paths it selects to explore, and the constraints on such a system, which restrict the paths it can select to explore.
An optimizing system, given a path that leads it to bypass a target, will discard that path… that’s part of what it means to optimize for a target.
An optimizing system, given a path that leads it to bypass a constraint, will not necessarily discard that path. Why would it?
An optimizing system, given a path that leads it to bypass a constraint and draw closer to a target than other paths, will choose that path.
It seems to follow that adding constraints to an optimizing system is a less reliable way of constraining its behavior than adding targets.
I don’t care whether we talk about “targets and constraints” or “values and rules” or “goals and failsafes” or whatever language you want to use, my point is that there are two genuinely different things under discussion, and a distinction between them.
Yes, the distinction is drawn from analogy to the intelligences I have experience with—as you say, anthropomorphic. I said this explicitly in the first place, so I assume you mean here to agree with me. (My reading of your tone suggests otherwise, but I don’t trust that I can reliably infer your tone so I am mostly disregarding tone in this exchange.)
That said, I also think the relationship between them reflects something more generally true of optimizing systems, as I’ve tried to argue for a couple of times now.
I can’t tell whether you think those arguments are wrong, or whether I just haven’t communicated them successfully at all, or whether you’re just not interested in them, or what.
There’s no reason it would. If “doing X for X seconds” is its target, then it looks for paths that do that. Again, that’s what it means for something to be a target of an optimizing system.
(Of course, if I do X for 2X seconds, I have in fact done X for X seconds, in the same sense that all months have 28 days.)
I’m not quite sure I understand what you mean here, but if I’m understanding the gist: I’m not saying that encoding scope boundaries as targets, or ‘values,’ is difficult (nor am I saying it’s easy), I’m saying that for a sufficiently capable optimizing system it’s safer than encoding scope boundaries as failsafes.
It was not my intention to imply any hostility or resentment. I thought ‘anthropomorphic’ is valid terminology in such a discussion. I was also not agreeing with you. If you are an expert and have been offended by implying that what you said might be due to an anthropomorphic bias, then accept my apology, I was merely trying to communicate my perception of the subject matter.
I had wedrifid telling me the same yesterday, that my tone isn’t appropriate when I wrote about his superior and rational use of the reputation system here, when I was actually just being honest. I’m not good at social signaling, sorry.
I think we are talking past each other. The way I see it is that a constraint is part of the design specifications of that which is optimized. Disregarding certain specifications will not allow it to optimize whatever it is optimizing with maximal efficiency.
Not an expert, and not offended.
What was puzzling me was that I said in the first place that I was reasoning by analogy to humans and that this was a tricky thing to do, so when you classified this as anthropomorphic my reaction was “well, yes, that’s what I said.”
Since it seemed to me you were repeating something I’d said, I assumed your intention was to agree with me, though it didn’t sound like it (and as it turned out, you weren’t).
And, yes, I’ve noticed that tone is a problem in a lot of your exchanges, which is why I’m basically disregarding tone in this one, as I said before.
Ah! In that case, I think we agree.
Yes, embedding everything we care about into the optimization target, rather than depending on something outside the optimization process to do important work, is the way to go.
You seemed to be defending the “failsafes” model, which I understand to be importantly different from this, which is where the divergence came from, I think. Apparently I (and, I suspect, some others) misunderstood what you were defending.
Sorry! Glad we worked that out, though.
Fail safes would be low cost: if it can’t think of a way to beat them, it isn’t the bootstrapping AI we were hoping for anyway, and might even be harmful, so it would be good to have the fail-safes.
It seems to me evolution based algorithms could do the trick.
Who says it wants to want what it wants? I don’t want to want what I want.