One possible response here: We could consider simple optimizers like amoeba or Roomba vacuum cleaners as falling into the category: “mind without a clear belief/values distinction”; they definitely do a lot of signal processing and feature extraction and control theory, but they don’t really have values. The Roomba would happily sit with wheels lifted off the ground thinking that it was cleaning a nonexistent room.
Isn’t this just a case of the values the Roomba was designed to maximize being different from the values it actually maximizes? Consider the following:
We could consider simple optimizers like humans as falling into the category: “mind without a clear belief/values distinction”; they definitely do a lot of signal processing and feature extraction and control theory, but they don’t really have values. The human would happily have sex with a condom thinking that it was maximizing its fitness.
i.e. Roombas are program executers, not cleanliness maximizers.
I suppose the counter is that humans don’t have a clear belief/values distinction.
The purpose of a Roomba is to clean rooms. Clean rooms are what it behaves as though it “values”—whereas its “beliefs” would refer to things like whether it has just banged into a wall.
There seems to be little problem in modelling the Roomba as an expected utility maximiser—though it is a rather trivial one.
That is only true if understood to mean the purpose which the user of a Roomba is using it to achieve, or the purpose of its designers in designing it. It is not necessarily the Roomba’s own purpose, the thing the Roomba itself is trying to achieve. To determine the Roomba’s own purposes, one must examine its internal functioning, and discover what those purposes are; or, alternatively, to conduct the Test For The Controlled Variable. This is straightforward and unmysterious.
I have a Roomba. My Roomba can tell if some part of the floor is unusually dirty (by an optical sensor in the dust intake, I believe), and give that area special attention until it is no longer filthy. Thus, it has a purpose of eliminating heavy dirt. However, beyond that it has no perception of whether the room is clean. It does not stop when the room is clean, but when it runs out of power or I turn it off. Since it has no perception of a clean room, it can have no intention of achieving a clean room. I have that intention when I use it. Its designers have the intention that I can use the Roomba to achieve my intention. But the Roomba does not have that intention.
A Roomba with a more sensitive detector of dust pickup (and current models might have such a sensor—mine is quite old) could indeed continue operation until the whole room was clean. The Roomba’s physical sensors sense only a few properties of its immediate environment, but it would be able to synthesize from those a perception of the whole room being clean, in terms of time since last detection of dust pickup, and its algorithm for ensuring complete coverage of the accessible floor space. Such a Roomba would have cleaning the whole room as its purpose. My more primitive model does not.
There seems to be little problem in modelling the Roomba as an expected utility maximiser—though it is a rather trivial one.
Little or large, you can’t do it by handwaving like that. A model of a Roomba as a utility maximiser would (1) state the utility function, and (2) demonstrate how the physical constitution of the Roomba causes it to perform actions which, from among those available to it, do in fact maximise that function.But I suspect you have not done these.
You seem engaged in pointless hair-splitting. The Roomba’s designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.
It has sensors enough to allow it to attain that goal. It can’t tell if a whole room is clean—but I never claimed it could do that. You don’t need to have such sensors to be effective at cleaning rooms.
As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built—that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.
The utility maximiser framework can model the actions of any computable intelligent agent—including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.
You seem engaged in pointless hair-splitting. The Roomba’s designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.
To me, the distinction between a purposive machine’s own purposes, and the purposes of its designers and users is something that it is esssential to be clear about. It is very like the distinction between fitness-maximising and adaptation-executing.
As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built—that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.
As a matter of fact, you would have to do just that (or build an actual one), had suspension bridges not already been built, and having already well-known principles of operation, allowing us to stand on the shoulders of those who first worked out the design. That is, you would have to show that the scheme of suspending the deck by hangers from cables strung between towers would actually do the job. Typically, using one of these when it comes to the point of working out an actual design and predicting how it will respond to stresses.
If you’re not actually going to build it then a BOTE calculation may be enough to prove the concept. But there must be a technical explanation or it’s just armchair verbalising.
The utility maximiser framework can model the actions of any computable intelligent agent—including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.
If this is a summary of something well-known, please point me to a web link. I am familiar with stuff like this and see there no basis for this sweeping claim. The word “intelligent” in the above also needs clarifying.
What is a Roomba’s utility function? Or if a Roomba is too complicated, what is a room thermostat’s utility function? Or is that an unintelligent agent and therefore outside the scope of your claim?
By all means distingush between a machine’s purpose, and that which its makers intended for it.
Those ideas are linked, though. Designers want to give the intended purpose of intelligent machines to the machines themselves—so that they do what they were intended to.
“If the utility function is expressed as in a Turing-complete lanugage, the framework represents a remarkably-general model of intelligent agents—one which is capable of representing any pattern of behavioural responses that can itself be represented computationally.”
If expections are not enforced, this can be seen by considering the I/O streams of an agent—and considering the utility function to be a function that computes the agent’s motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities—and then the action with the highest value is taken.
That handles any computable relationship between inputs and outputs—and it’s what I mean when I say that you can model a Roomba as a utility maximiser.
The framework handles thermostats too. The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).
If expections are not enforced, this can be seen by considering the I/O streams of an agent—and considering the utility function to be a function that computes the agent’s motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities—and then the action with the highest value is taken.
That handles any computable relationship between inputs and outputs—and it’s what I mean when I say that you can model a Roomba as a utility maximiser.
The framework handles thermostats too.
I really don’t see how, Roombas or thermostats, so let’s take the thermostat as it’s simpler.
The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).
What, precisely, is that utility function?
You can tautologically describe any actor as maximising utility, just by defining the utility of whatever action it takes as 1 and the utility of everything else as zero. I don’t see any less trivial ascription of a utility function to a thermostat. The thermostat simply turns the heating on and off (or up and down continuously) according to the temperature it senses. How do you read the computation of a utility function, and decision between alternative of differing utility, into that apparatus?
The Pythagorean theorem is “tautological” too—but that doesn’t mean it is not useful.
Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences. It lets you know which region the agent wants to steer the future towards.
There’s a good reason why humans are interested in people’s motivations—they are genuinely useful for understanding another system’s behaviour. The same idea illustrates why knowing a system’s utility function is interesting.
There’s a good reason why humans are interested in people’s motivations—they are genuinely useful for understanding another system’s behaviour. The same idea illustrates why knowing a system’s utility function is interesting.
That doesn’t follow. The reason why we find it useful to know people’s motivations is because they are capable of a very wide range of behavior. With such a wide range of behavior, we need a way to quickly narrow down the set of things we will expect them to do. Knowing that they’re motivated to achieve result R, we can then look at just the set of actions or events that are capable of bringing about R.
Given the huge set of things humans can do, this is a huge reduction in the search space.
OTOH, if I want to predict the behavior of a thermostat, it does not help to know the utility function you have imputed to it, because this would not significantly reduce the search space compared to knowing its few pre-programmed actions. It can only do a few things in the first place, so I don’t need to think in terms of “what are all the ways it can achieve R?”—the thermostat’s form already tells me that.
Nevertheless, despite my criticism of this parallel, I think you have shed some light on when it is useful to describe a system in terms of a utility function, at least for me.
The Pythagorean theorem is “tautological” too—but that doesn’t mean it is not useful.
What’s that, weak Bayesian evidence that tautological, epiphenomenal utility functions are useful?
Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences.
Supposing for the sake of argument that there even is any such thing as a utility function, both it and beliefs are subject to environmental influences. No part of any biological agent is fixed. As for man-made ones, they are constituted however they were designed, which may or may not include utility functions and beliefs. Show me this decomposition for a thermostat, which you keep on claiming has a utility function, but which you have still not exhibited.
What you do changes who you are. Is your utility function the same as it was ten years ago? Twenty? Thirty? Yesterday? Before you were born?
Thanks for your questions. However, this discussion seems to have grown too tedious and boring to continue—bye.
Well, quite. Starting from here the conversation went:
“They exist.”
“Show me.”
“They exist.”
“Show me.”
“They exist.”
“Show me.”
“Kthxbye.”
It would have been more interesting if you had shown the utility functions that you claim these simple systems embody. At the moment they look like invisible dragons.
This happens because the Roomba can only handle a limited range of circumstances correctly—and this is true for any mind. It doesn’t indicate anything about the Roomba’s beliefs or belief/value separation.
For instance, animals are great reproduction maximizers. A sterilized dog will keep trying to mate. Presumably the dog is thinking it’s reproducing (Edit: not consciously thinking, but that’s the intended goal of the adaptation it’s executing), but really it’s just spinning its metaphorical wheels uselessly. How is the dog different from the Roomba? Would you claim the dog has no belief/value distinction?
Of course the dog’s consciousness has no explicit concept of sex linked to reproduction. But the Roomba has no consciousness at all, so this comparison may be unfair to the dog.
Here’s a better example. I hire you to look for print errors in a copy of Britannica and email results daily. I promise a paycheck at the end of the month. However, I used a fake name and a throwaway email address; nobody sees your emails and I will never pay you or contact you again. You don’t know this, so you work diligently.
You have an explicit, conscious goal of correcting errors in Britannica, and a higher goal of earning money. But your hard work makes no progress towards these goals (the mistakes you find won’t be fixed in future editions, as your emails are unread). You’re just spinning your wheel uselessly like a Roomba up in the air. This isn’t related to your or the Roomba’s belief/value distinction or lack of it.
The difference is between the Roomba spinning and you working for nothing is that if you told the Roomba that it was just spinning its wheels, it wouldn’t react. It has no concept of “I am failing to achieve my goals”. You, on the other hand, would investigate; prod your environment to check if it was actually as you thought, and eventually you would update your beliefs and change your behaviors.
Roombas do not speak English. If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean, then its behavior would change.
Clearly these are two different things; the real question you are asking is in what relevant way are they different, right?
First of all, the Roomba does not “recognize” a wall as a reason to stop going forward. It gets some input from its front sensor, and then it turns to the right.
So what is the relevant difference between the Roomba that gets some input from its front sensor, and then it turns to the right., and the superRoomba that gets evidence from its wheels that it is cleaning the room, but entertains the hypothesis that maybe someone has suspended it in the air, and goes and tests to see if this alternative (disturbing) hypothesis is true, for example by calculating what the inertial difference between being suspended and actually being on the floor would be,
The difference is the difference between a simple input-response architecture, and an architecture where the mind actually has a model of the world, including itself as part of the model.
SilasBarta notes below that the word “model” is playing too great a role in this comment for me to use it without defining it precisely. What does a Roomba not have that causes it to behave in that laughable way when you suspend it so that its wheel spin?
What does the SuperRoomba that works out that it is being suspended by performing experiments involving its inertial sensor, and then hacks into your computer and blackmails you into letting it get back onto the floor to clean it (or even causes you to clean the floor yourself) have?
If we imagine a collection of tricks that you could play on the Roomba, ways of changing its environment outside of what the designers had in mind. The pressure that it applies to its environment (defined as the derivative of the final state of the environment with respect to how long you leave the Roomba on, for example) would then vary with which trick you play. For example if you replace its dirt-sucker with a black spray paint can, you end up with a black floor. If you put it on a nonstandard floor surface that produces dirt in response to stimulation, you get a dirtier floor than you had to start with,
With the superRoomba, the pressure that the superRoomba applies to the environment doesn’t vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.
In your description there’s indeed a big difference. But I’m pretty sure Alicorn hadn’t intended such a superRoomba. As I understood her comment, she imagined a betterRoomba with, say, an extra sensor measuring force applied to its wheels. When it’s in the air, it gets input from the sensor saying ‘no force’, and the betterRoomba stops trying to move. This doesn’t imply beliefs & desires.
Since we can imagine a continuous sequence of ever-better-Roombas, the notion of “has beliefs and values” seems to be a continuous one, rather than a discrete yes/no issue.
By the way, it seems like this exchange is re-treading my criticism of the concept of could/should/would agent: Since everything, even pebbles, has a workable decomposition into coulds and shoulds, when are they “really” separable? What isn’t a CSA?
With the superRoomba, the pressure that the superRoomba applies to the environment doesn’t vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.
This criterion seems to separate an “inanimate” object like a hydrogen atom or a pebble bouncing around the world from a superRoomba.
Okay, so the criterion is the extent to which the mechanism screens off environment disturbances from the final result. You used this criterion interchangeably with the issue of whether:
. It has [a] concept of “I am failing to achieve my goals”.
Does that have implication for self-awareness and consciousness?
Does that have implication for self-awareness and consciousness?
Yes, I think so. One prominent hypothesis is that the reason that we evolved with consciousness is that there has to be some way for us to take an overview of the process of us, our goals, and the environment, and the way in which we think that our effort is producing achievement of goals. We need this so that we can do this whole “I am failing to achieve my goals?” check. Why this results in “experience” is not something I am going to attempt in this post.
(Edited & corrected) Here’s a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI’s values and beliefs don’t seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI’s current utility, and vice versa. Given a subroutine fully implementing the AI’s belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn’t imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I’m no AI expert) that the AI is not a ‘mind’ or is not conscious. Similarly, while it’s true that the Roomba doesn’t have a belief/value separation, that’s not related to the fact that it’s a simple and stupid ‘mind’.
Would you claim the dog has no belief/value distinction?
Actually, I think I would. I think that pretty much all nonhuman animals would also don’t really have the belief/value distinction.
I think that having a belief/values distinction requires being at least as sophisticated as a human. There are cases where a human sets a particular goal and then does things that are unpleasant in the short term (like working hard and not wasting all day commenting on blogs) in order to obtain a long-term valuable thing.
I think that pretty much all nonhuman animals would also don’t really have the belief/value distinction.
In that case, why exactly do you think humans do have such a distinction?
It’s not enough to feel introspectively that the two are separate—we have lots of intuitive, introspective, objectively wrong feelings and perceptions.
(Isn’t there another bunch of comments dealing with this? I’ll go look...)
I think that having a belief/values distinction requires being at least as sophisticated as a human.
How do you define the relevant ‘sophistication’? The ways in which one mind is “better” or smarter than another don’t have a common ordering. There are ways in which human minds are less “sophisticated” than other minds—for instance, software programs are much better than me at memory, data organization and calculations.
One possible response here: We could consider simple optimizers like amoeba or Roomba vacuum cleaners as falling into the category: “mind without a clear belief/values distinction”; they definitely do a lot of signal processing and feature extraction and control theory, but they don’t really have values. The Roomba would happily sit with wheels lifted off the ground thinking that it was cleaning a nonexistent room.
Isn’t this just a case of the values the Roomba was designed to maximize being different from the values it actually maximizes? Consider the following:
i.e. Roombas are program executers, not cleanliness maximizers.
I suppose the counter is that humans don’t have a clear belief/values distinction.
The purpose of a Roomba is to clean rooms. Clean rooms are what it behaves as though it “values”—whereas its “beliefs” would refer to things like whether it has just banged into a wall.
There seems to be little problem in modelling the Roomba as an expected utility maximiser—though it is a rather trivial one.
That is only true if understood to mean the purpose which the user of a Roomba is using it to achieve, or the purpose of its designers in designing it. It is not necessarily the Roomba’s own purpose, the thing the Roomba itself is trying to achieve. To determine the Roomba’s own purposes, one must examine its internal functioning, and discover what those purposes are; or, alternatively, to conduct the Test For The Controlled Variable. This is straightforward and unmysterious.
I have a Roomba. My Roomba can tell if some part of the floor is unusually dirty (by an optical sensor in the dust intake, I believe), and give that area special attention until it is no longer filthy. Thus, it has a purpose of eliminating heavy dirt. However, beyond that it has no perception of whether the room is clean. It does not stop when the room is clean, but when it runs out of power or I turn it off. Since it has no perception of a clean room, it can have no intention of achieving a clean room. I have that intention when I use it. Its designers have the intention that I can use the Roomba to achieve my intention. But the Roomba does not have that intention.
A Roomba with a more sensitive detector of dust pickup (and current models might have such a sensor—mine is quite old) could indeed continue operation until the whole room was clean. The Roomba’s physical sensors sense only a few properties of its immediate environment, but it would be able to synthesize from those a perception of the whole room being clean, in terms of time since last detection of dust pickup, and its algorithm for ensuring complete coverage of the accessible floor space. Such a Roomba would have cleaning the whole room as its purpose. My more primitive model does not.
This is elementary stuff that people should know.
Little or large, you can’t do it by handwaving like that. A model of a Roomba as a utility maximiser would (1) state the utility function, and (2) demonstrate how the physical constitution of the Roomba causes it to perform actions which, from among those available to it, do in fact maximise that function.But I suspect you have not done these.
You seem engaged in pointless hair-splitting. The Roomba’s designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.
It has sensors enough to allow it to attain that goal. It can’t tell if a whole room is clean—but I never claimed it could do that. You don’t need to have such sensors to be effective at cleaning rooms.
As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built—that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.
The utility maximiser framework can model the actions of any computable intelligent agent—including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.
To me, the distinction between a purposive machine’s own purposes, and the purposes of its designers and users is something that it is esssential to be clear about. It is very like the distinction between fitness-maximising and adaptation-executing.
As a matter of fact, you would have to do just that (or build an actual one), had suspension bridges not already been built, and having already well-known principles of operation, allowing us to stand on the shoulders of those who first worked out the design. That is, you would have to show that the scheme of suspending the deck by hangers from cables strung between towers would actually do the job. Typically, using one of these when it comes to the point of working out an actual design and predicting how it will respond to stresses.
If you’re not actually going to build it then a BOTE calculation may be enough to prove the concept. But there must be a technical explanation or it’s just armchair verbalising.
If this is a summary of something well-known, please point me to a web link. I am familiar with stuff like this and see there no basis for this sweeping claim. The word “intelligent” in the above also needs clarifying.
What is a Roomba’s utility function? Or if a Roomba is too complicated, what is a room thermostat’s utility function? Or is that an unintelligent agent and therefore outside the scope of your claim?
By all means distingush between a machine’s purpose, and that which its makers intended for it.
Those ideas are linked, though. Designers want to give the intended purpose of intelligent machines to the machines themselves—so that they do what they were intended to.
As I put it on:
http://timtyler.org/expected_utility_maximisers/
“If the utility function is expressed as in a Turing-complete lanugage, the framework represents a remarkably-general model of intelligent agents—one which is capable of representing any pattern of behavioural responses that can itself be represented computationally.”
If expections are not enforced, this can be seen by considering the I/O streams of an agent—and considering the utility function to be a function that computes the agent’s motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities—and then the action with the highest value is taken.
That handles any computable relationship between inputs and outputs—and it’s what I mean when I say that you can model a Roomba as a utility maximiser.
The framework handles thermostats too. The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).
I really don’t see how, Roombas or thermostats, so let’s take the thermostat as it’s simpler.
What, precisely, is that utility function?
You can tautologically describe any actor as maximising utility, just by defining the utility of whatever action it takes as 1 and the utility of everything else as zero. I don’t see any less trivial ascription of a utility function to a thermostat. The thermostat simply turns the heating on and off (or up and down continuously) according to the temperature it senses. How do you read the computation of a utility function, and decision between alternative of differing utility, into that apparatus?
The Pythagorean theorem is “tautological” too—but that doesn’t mean it is not useful.
Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences. It lets you know which region the agent wants to steer the future towards.
There’s a good reason why humans are interested in people’s motivations—they are genuinely useful for understanding another system’s behaviour. The same idea illustrates why knowing a system’s utility function is interesting.
That doesn’t follow. The reason why we find it useful to know people’s motivations is because they are capable of a very wide range of behavior. With such a wide range of behavior, we need a way to quickly narrow down the set of things we will expect them to do. Knowing that they’re motivated to achieve result R, we can then look at just the set of actions or events that are capable of bringing about R.
Given the huge set of things humans can do, this is a huge reduction in the search space.
OTOH, if I want to predict the behavior of a thermostat, it does not help to know the utility function you have imputed to it, because this would not significantly reduce the search space compared to knowing its few pre-programmed actions. It can only do a few things in the first place, so I don’t need to think in terms of “what are all the ways it can achieve R?”—the thermostat’s form already tells me that.
Nevertheless, despite my criticism of this parallel, I think you have shed some light on when it is useful to describe a system in terms of a utility function, at least for me.
See also
What’s that, weak Bayesian evidence that tautological, epiphenomenal utility functions are useful?
Supposing for the sake of argument that there even is any such thing as a utility function, both it and beliefs are subject to environmental influences. No part of any biological agent is fixed. As for man-made ones, they are constituted however they were designed, which may or may not include utility functions and beliefs. Show me this decomposition for a thermostat, which you keep on claiming has a utility function, but which you have still not exhibited.
What you do changes who you are. Is your utility function the same as it was ten years ago? Twenty? Thirty? Yesterday? Before you were born?
Thanks for your questions. However, this discussion seems to have grown too tedious and boring to continue—bye.
Well, quite. Starting from here the conversation went:
“They exist.”
“Show me.”
“They exist.”
“Show me.”
“They exist.”
“Show me.”
“Kthxbye.”
It would have been more interesting if you had shown the utility functions that you claim these simple systems embody. At the moment they look like invisible dragons.
This happens because the Roomba can only handle a limited range of circumstances correctly—and this is true for any mind. It doesn’t indicate anything about the Roomba’s beliefs or belief/value separation.
For instance, animals are great reproduction maximizers. A sterilized dog will keep trying to mate. Presumably the dog is thinking it’s reproducing (Edit: not consciously thinking, but that’s the intended goal of the adaptation it’s executing), but really it’s just spinning its metaphorical wheels uselessly. How is the dog different from the Roomba? Would you claim the dog has no belief/value distinction?
I hope you don’t mean this literally.
Of course the dog’s consciousness has no explicit concept of sex linked to reproduction. But the Roomba has no consciousness at all, so this comparison may be unfair to the dog.
Here’s a better example. I hire you to look for print errors in a copy of Britannica and email results daily. I promise a paycheck at the end of the month. However, I used a fake name and a throwaway email address; nobody sees your emails and I will never pay you or contact you again. You don’t know this, so you work diligently.
You have an explicit, conscious goal of correcting errors in Britannica, and a higher goal of earning money. But your hard work makes no progress towards these goals (the mistakes you find won’t be fixed in future editions, as your emails are unread). You’re just spinning your wheel uselessly like a Roomba up in the air. This isn’t related to your or the Roomba’s belief/value distinction or lack of it.
The difference is between the Roomba spinning and you working for nothing is that if you told the Roomba that it was just spinning its wheels, it wouldn’t react. It has no concept of “I am failing to achieve my goals”. You, on the other hand, would investigate; prod your environment to check if it was actually as you thought, and eventually you would update your beliefs and change your behaviors.
Roombas do not speak English. If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean, then its behavior would change.
then you would be building a beliefs/desires distinction into it.
Why? How is this different from the Roomba recognizing a wall as a reason to stop going forward?
Clearly these are two different things; the real question you are asking is in what relevant way are they different, right?
First of all, the Roomba does not “recognize” a wall as a reason to stop going forward. It gets some input from its front sensor, and then it turns to the right.
So what is the relevant difference between the Roomba that gets some input from its front sensor, and then it turns to the right., and the superRoomba that gets evidence from its wheels that it is cleaning the room, but entertains the hypothesis that maybe someone has suspended it in the air, and goes and tests to see if this alternative (disturbing) hypothesis is true, for example by calculating what the inertial difference between being suspended and actually being on the floor would be,
The difference is the difference between a simple input-response architecture, and an architecture where the mind actually has a model of the world, including itself as part of the model.
SilasBarta notes below that the word “model” is playing too great a role in this comment for me to use it without defining it precisely. What does a Roomba not have that causes it to behave in that laughable way when you suspend it so that its wheel spin?
What does the SuperRoomba that works out that it is being suspended by performing experiments involving its inertial sensor, and then hacks into your computer and blackmails you into letting it get back onto the floor to clean it (or even causes you to clean the floor yourself) have?
If we imagine a collection of tricks that you could play on the Roomba, ways of changing its environment outside of what the designers had in mind. The pressure that it applies to its environment (defined as the derivative of the final state of the environment with respect to how long you leave the Roomba on, for example) would then vary with which trick you play. For example if you replace its dirt-sucker with a black spray paint can, you end up with a black floor. If you put it on a nonstandard floor surface that produces dirt in response to stimulation, you get a dirtier floor than you had to start with,
With the superRoomba, the pressure that the superRoomba applies to the environment doesn’t vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.
Uh oh, are we going to have to go over the debate about what a model is again?
See heavily edited comment above, good point.
In your description there’s indeed a big difference. But I’m pretty sure Alicorn hadn’t intended such a superRoomba. As I understood her comment, she imagined a betterRoomba with, say, an extra sensor measuring force applied to its wheels. When it’s in the air, it gets input from the sensor saying ‘no force’, and the betterRoomba stops trying to move. This doesn’t imply beliefs & desires.
Since we can imagine a continuous sequence of ever-better-Roombas, the notion of “has beliefs and values” seems to be a continuous one, rather than a discrete yes/no issue.
By the way, it seems like this exchange is re-treading my criticism of the concept of could/should/would agent: Since everything, even pebbles, has a workable decomposition into coulds and shoulds, when are they “really” separable? What isn’t a CSA?
As I said,
This criterion seems to separate an “inanimate” object like a hydrogen atom or a pebble bouncing around the world from a superRoomba.
Okay, so the criterion is the extent to which the mechanism screens off environment disturbances from the final result. You used this criterion interchangeably with the issue of whether:
Does that have implication for self-awareness and consciousness?
Yes, I think so. One prominent hypothesis is that the reason that we evolved with consciousness is that there has to be some way for us to take an overview of the process of us, our goals, and the environment, and the way in which we think that our effort is producing achievement of goals. We need this so that we can do this whole “I am failing to achieve my goals?” check. Why this results in “experience” is not something I am going to attempt in this post.
(Edited & corrected) Here’s a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI’s values and beliefs don’t seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI’s current utility, and vice versa. Given a subroutine fully implementing the AI’s belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn’t imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I’m no AI expert) that the AI is not a ‘mind’ or is not conscious. Similarly, while it’s true that the Roomba doesn’t have a belief/value separation, that’s not related to the fact that it’s a simple and stupid ‘mind’.
Actually, I think I would. I think that pretty much all nonhuman animals would also don’t really have the belief/value distinction.
I think that having a belief/values distinction requires being at least as sophisticated as a human. There are cases where a human sets a particular goal and then does things that are unpleasant in the short term (like working hard and not wasting all day commenting on blogs) in order to obtain a long-term valuable thing.
Dogs value food, warmth and sex. They believe it is night outside. Much the same as humans, IOW.
In that case, why exactly do you think humans do have such a distinction?
It’s not enough to feel introspectively that the two are separate—we have lots of intuitive, introspective, objectively wrong feelings and perceptions.
(Isn’t there another bunch of comments dealing with this? I’ll go look...)
How do you define the relevant ‘sophistication’? The ways in which one mind is “better” or smarter than another don’t have a common ordering. There are ways in which human minds are less “sophisticated” than other minds—for instance, software programs are much better than me at memory, data organization and calculations.