The difference is between the Roomba spinning and you working for nothing is that if you told the Roomba that it was just spinning its wheels, it wouldn’t react. It has no concept of “I am failing to achieve my goals”. You, on the other hand, would investigate; prod your environment to check if it was actually as you thought, and eventually you would update your beliefs and change your behaviors.
Roombas do not speak English. If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean, then its behavior would change.
Clearly these are two different things; the real question you are asking is in what relevant way are they different, right?
First of all, the Roomba does not “recognize” a wall as a reason to stop going forward. It gets some input from its front sensor, and then it turns to the right.
So what is the relevant difference between the Roomba that gets some input from its front sensor, and then it turns to the right., and the superRoomba that gets evidence from its wheels that it is cleaning the room, but entertains the hypothesis that maybe someone has suspended it in the air, and goes and tests to see if this alternative (disturbing) hypothesis is true, for example by calculating what the inertial difference between being suspended and actually being on the floor would be,
The difference is the difference between a simple input-response architecture, and an architecture where the mind actually has a model of the world, including itself as part of the model.
SilasBarta notes below that the word “model” is playing too great a role in this comment for me to use it without defining it precisely. What does a Roomba not have that causes it to behave in that laughable way when you suspend it so that its wheel spin?
What does the SuperRoomba that works out that it is being suspended by performing experiments involving its inertial sensor, and then hacks into your computer and blackmails you into letting it get back onto the floor to clean it (or even causes you to clean the floor yourself) have?
If we imagine a collection of tricks that you could play on the Roomba, ways of changing its environment outside of what the designers had in mind. The pressure that it applies to its environment (defined as the derivative of the final state of the environment with respect to how long you leave the Roomba on, for example) would then vary with which trick you play. For example if you replace its dirt-sucker with a black spray paint can, you end up with a black floor. If you put it on a nonstandard floor surface that produces dirt in response to stimulation, you get a dirtier floor than you had to start with,
With the superRoomba, the pressure that the superRoomba applies to the environment doesn’t vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.
In your description there’s indeed a big difference. But I’m pretty sure Alicorn hadn’t intended such a superRoomba. As I understood her comment, she imagined a betterRoomba with, say, an extra sensor measuring force applied to its wheels. When it’s in the air, it gets input from the sensor saying ‘no force’, and the betterRoomba stops trying to move. This doesn’t imply beliefs & desires.
Since we can imagine a continuous sequence of ever-better-Roombas, the notion of “has beliefs and values” seems to be a continuous one, rather than a discrete yes/no issue.
By the way, it seems like this exchange is re-treading my criticism of the concept of could/should/would agent: Since everything, even pebbles, has a workable decomposition into coulds and shoulds, when are they “really” separable? What isn’t a CSA?
With the superRoomba, the pressure that the superRoomba applies to the environment doesn’t vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.
This criterion seems to separate an “inanimate” object like a hydrogen atom or a pebble bouncing around the world from a superRoomba.
Okay, so the criterion is the extent to which the mechanism screens off environment disturbances from the final result. You used this criterion interchangeably with the issue of whether:
. It has [a] concept of “I am failing to achieve my goals”.
Does that have implication for self-awareness and consciousness?
Does that have implication for self-awareness and consciousness?
Yes, I think so. One prominent hypothesis is that the reason that we evolved with consciousness is that there has to be some way for us to take an overview of the process of us, our goals, and the environment, and the way in which we think that our effort is producing achievement of goals. We need this so that we can do this whole “I am failing to achieve my goals?” check. Why this results in “experience” is not something I am going to attempt in this post.
(Edited & corrected) Here’s a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI’s values and beliefs don’t seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI’s current utility, and vice versa. Given a subroutine fully implementing the AI’s belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn’t imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I’m no AI expert) that the AI is not a ‘mind’ or is not conscious. Similarly, while it’s true that the Roomba doesn’t have a belief/value separation, that’s not related to the fact that it’s a simple and stupid ‘mind’.
The difference is between the Roomba spinning and you working for nothing is that if you told the Roomba that it was just spinning its wheels, it wouldn’t react. It has no concept of “I am failing to achieve my goals”. You, on the other hand, would investigate; prod your environment to check if it was actually as you thought, and eventually you would update your beliefs and change your behaviors.
Roombas do not speak English. If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean, then its behavior would change.
then you would be building a beliefs/desires distinction into it.
Why? How is this different from the Roomba recognizing a wall as a reason to stop going forward?
Clearly these are two different things; the real question you are asking is in what relevant way are they different, right?
First of all, the Roomba does not “recognize” a wall as a reason to stop going forward. It gets some input from its front sensor, and then it turns to the right.
So what is the relevant difference between the Roomba that gets some input from its front sensor, and then it turns to the right., and the superRoomba that gets evidence from its wheels that it is cleaning the room, but entertains the hypothesis that maybe someone has suspended it in the air, and goes and tests to see if this alternative (disturbing) hypothesis is true, for example by calculating what the inertial difference between being suspended and actually being on the floor would be,
The difference is the difference between a simple input-response architecture, and an architecture where the mind actually has a model of the world, including itself as part of the model.
SilasBarta notes below that the word “model” is playing too great a role in this comment for me to use it without defining it precisely. What does a Roomba not have that causes it to behave in that laughable way when you suspend it so that its wheel spin?
What does the SuperRoomba that works out that it is being suspended by performing experiments involving its inertial sensor, and then hacks into your computer and blackmails you into letting it get back onto the floor to clean it (or even causes you to clean the floor yourself) have?
If we imagine a collection of tricks that you could play on the Roomba, ways of changing its environment outside of what the designers had in mind. The pressure that it applies to its environment (defined as the derivative of the final state of the environment with respect to how long you leave the Roomba on, for example) would then vary with which trick you play. For example if you replace its dirt-sucker with a black spray paint can, you end up with a black floor. If you put it on a nonstandard floor surface that produces dirt in response to stimulation, you get a dirtier floor than you had to start with,
With the superRoomba, the pressure that the superRoomba applies to the environment doesn’t vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.
Uh oh, are we going to have to go over the debate about what a model is again?
See heavily edited comment above, good point.
In your description there’s indeed a big difference. But I’m pretty sure Alicorn hadn’t intended such a superRoomba. As I understood her comment, she imagined a betterRoomba with, say, an extra sensor measuring force applied to its wheels. When it’s in the air, it gets input from the sensor saying ‘no force’, and the betterRoomba stops trying to move. This doesn’t imply beliefs & desires.
Since we can imagine a continuous sequence of ever-better-Roombas, the notion of “has beliefs and values” seems to be a continuous one, rather than a discrete yes/no issue.
By the way, it seems like this exchange is re-treading my criticism of the concept of could/should/would agent: Since everything, even pebbles, has a workable decomposition into coulds and shoulds, when are they “really” separable? What isn’t a CSA?
As I said,
This criterion seems to separate an “inanimate” object like a hydrogen atom or a pebble bouncing around the world from a superRoomba.
Okay, so the criterion is the extent to which the mechanism screens off environment disturbances from the final result. You used this criterion interchangeably with the issue of whether:
Does that have implication for self-awareness and consciousness?
Yes, I think so. One prominent hypothesis is that the reason that we evolved with consciousness is that there has to be some way for us to take an overview of the process of us, our goals, and the environment, and the way in which we think that our effort is producing achievement of goals. We need this so that we can do this whole “I am failing to achieve my goals?” check. Why this results in “experience” is not something I am going to attempt in this post.
(Edited & corrected) Here’s a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI’s values and beliefs don’t seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI’s current utility, and vice versa. Given a subroutine fully implementing the AI’s belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn’t imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I’m no AI expert) that the AI is not a ‘mind’ or is not conscious. Similarly, while it’s true that the Roomba doesn’t have a belief/value separation, that’s not related to the fact that it’s a simple and stupid ‘mind’.