A “utility function” that can only be calculated—even by the agent itself—in hindsight is not a utility function. The agent is not using it to make choices and no observer can use it to make predictions about the agent.
Er, the idea is that you can make a utility-maximising model of the agent—using the specified utility function—that does the same things the agent does if you put it in the same environment.
Can people please stop dissing the concept of a human utility function. Correcting these people is getting tedious—and I don’t want to be boring.
Er, the idea is that you can make a utility-maximising model of the agent—using the specified utility function—that does the same things the agent does if you put it in the same environment.
Doesn’t work. The Texas Sharpshooter utility function described by Dewar cannot be used to make a utility-maximising model of the agent, except by putting a copy of the actual agent into the box, seeing what it does, declaring that to have utility 1, and doing it. The step of declaring it to have utility 1 plays no role in deciding the actions. It is a uselessly spinning cog doing no more work than a suggestive name on a Lisp symbol.
Can people please stop dissing the concept of a human utility function. Correcting these people is getting tedious—and I don’t want to be boring.
I was thinking a similar thought about you. You’re the only person here that I’ve seen taking these trivial utility functions seriously.
Er, the idea is that you can make a utility-maximising model of the agent—using the specified utility function—that does the same things the agent does if you put it in the same environment.
Doesn’t work. The Texas Sharpshooter utility function described by Dewar cannot be used to make a utility-maximising model of the agent, except by putting a copy of the actual agent into the box, seeing what it does, declaring that to have utility 1, and doing it. The step of declaring it to have utility 1 plays no role in deciding the actions. It is a uselessly spinning cog doing no more work than a suggestive name on a Lisp symbol.
The idea here is that—if the agent is computable—then it can be simulated by any other computable system. So, if the map between its inputs and state, and its motor output is computable then we can make another computable system which produces the same map—since all universal computing systems can simulate each other by virtue of being Turing complete (and systems made of e.g. partial recursive functions can simulate each other too—if they are given enough memory to do so).
I mentioned computability at the top, by saying: “any computable agent has a utility function”.
The idea here is that—if the agent is computable—then it can be simulated by any other computable system. So, if the map between its inputs and state, and its motor output is computable then we can make another computable system which produces the same map—since all universal computing systems can simulate each other by virtue of being Turing complete (and systems made of e.g. partial recursive functions can simulate each other too—if they are given enough memory to do so).
I don’t see how this bears on the possibility of modelling every agent by a utility-maximising agent. Dewar’s construction doesn’t work. Its simulation of an agent by a utility-maximising agent just uses the agent to simulate itself and attaches the label “utility=1” to its actions.
Dewey says pretty plainly: “any agents can be written in O-maximizer form”.
I know that he says that. I am saying, I thought pretty plainly, that I disagree with him.
He makes an O-maximiser from an agent, A. Once you have the corresponding O-maximiser, the agent A could be discarded.
He only does that in the earlier paper. His construction is as I described it: define O as doing whatever A does and label the result with utility 1. A is a part of O and cannot be discarded. He even calls this construction trivial himself, but underrates its triviality.
I don’t really understand which problem you are raising. If the O eventually contains a simulated copy of A—so what? O is still a utililty-maximiser that behaves the same way that A does if placed in the same environment.
The idea of a utility maximiser as used here is that it assigns utilities to all its possible actions and then chooses the action with the highest utility. O does that—so it qualifies as a utililty-maximiser.
The idea of a utility maximiser as used here is that it assigns utilities to all its possible actions and then chooses the action with the highest utility. O does that—so it qualifies as a utililty-maximiser.
O doesn’t assign utilities to its actions and then choose the best. It chooses its action (by simulating A), labels it with utility 1, and chooses to perform the action it just chose. The last two steps are irrelevant.
O doesn’t assign utilities to its actions and then choose the best. It chooses its action (by simulating A), labels it with utility 1, and chooses to perform the action it just chose. The last two steps are irrelevant.
“Irrelevant”? If it didin’t perform those steps, it wouldn’t be a utility maximiser, and then the proof that you can build a utility maximiser which behaves like any computable agent wouldn’t go through. Those steps are an important part of the reason for exhibiting this construction in the first place.
I think that everyone understands the point you’re trying to make—you can usefully model people as having a utility function in a wide variety of cases—but very often people use such models unskillfully, and it causes people like me to facepalm. If you want to model a lot of humans, for instance, it’s simple and decently accurate to model them as having utility functions. Economics, say. And if you have something like AIXI, or as Dawkins might argue a gene, then a utility function isn’t even a model, it’s right there in front of you.
I hypothesize that the real trouble starts when a person confuses the two; he sees or imagines a Far model of humans with utility functions, zooms in on an individual human or zooms in on himself, and thinks he can see the real utility function sitting right there in front of him, like he could with AIXI. Yeah, he knows in the abstract that he doesn’t have direct access to it, but it feels Near. This can lead to a lot of confusion, and it leads people like me to think folk shouldn’t talk about a person’s “utility function” except in cases where it obviously applies.
Even where you can say “Person A has a utility function that assigns 4 utility to getting cheesecake and 2 utility to getting paperlips”, why not say “Agent A”? But that’s not what I facepalm at. I only facepalm when people say they got their “utility function” from natural selection (i.e. ignoring memes), or say they wish they could modify their utility function, et cetera. In many cases it works as an abstraction, but if you’re not at all thinking about EU, why not talk directly about your preferences/values? It’s simpler and less misleading.
This seems like a bit of a different issue—and one that I am not so interested in.
A couple of comments about your examples, though:
I only facepalm when people say they got their “utility function” from natural selection (i.e. ignoring memes), or say they wish they could modify their utility function, et cetera.
For someone like me it is pretty accurate to say that I got my utility function from natural selection acting on DNA genes. Memes influence me, but I try not to let them influnce my goals. I regard them as symbiotes: mutualists and pathogens. In principle they could do deals with me that might make me change my goals—but currently I have a powerful bargaining position, their bargaining position is typically weak—and so I just get my way. They don’t get to affect my goals. Those that try get rejected by my memetic immune system. I do not want to become the victim of a memetic hijacking.
As for the implied idea that natural selection does not apply to memes, I’ll try to bite my tongue there.
why not talk directly about your preferences/values? It’s simpler and less misleading.
That seems closely equivalent to me. The cases where people talk about utility functions are mostly those where you want to compare with machines, or conjour up the idea of an expected utility maximiser for some reason. Sometimes even having “utility” in the context is enough for the conversation to wander on to utility functions.
My council would be something like: “Don’t like it? Get used to it!” There is not, in fact, anything wrong with it.
As for the implied idea that natural selection does not apply to memes, I’ll try to bite my tongue there.
That totally wasn’t what I meant to imply. I am definitely a universal Darwinist. (You can view pretty much any optimization process as “evolution”, though, so in some cases it’s questionably useful. Bayesian updating is just like population genetics. But with memes it’s obviously a good description.)
For someone like me it is pretty accurate to say that I got my utility function from natural selection acting on DNA genes.
Yes, but I think you’re rather unusual in this regard; most people aren’t so wary of memes. Might I ask why you prefer genes to memes? This seems odd to me. Largely because humans evolved for memes and with memes. Archetypes, for example. But also because the better memes seem to have done a lot of good in the world. (My genetically evolved cognitive algorithms—that is, the algorithms in my brain that I think aren’t the result of culture, but instead are universal machinery—stare in appreciation at the beauty of cathedrals, and are grateful that economies make my life easier.)
As for the implied idea that natural selection does not apply to memes, I’ll try to bite my tongue there.
That totally wasn’t what I meant to imply.
’s why I tried to bite my tongue—but it was difficult to completely let it go by...
For someone like me it is pretty accurate to say that I got my utility function from natural selection acting on DNA genes.
Yes, but I think you’re rather unusual in this regard; most people aren’t so wary of memes. Might I ask why you prefer genes to memes? This seems odd to me.
Well, I love memes, but DNA-genes built 99% of my ancestors unassisted, and are mostly responsible for building me. They apparently equipped me with a memetic immune system, for weeding out undesirable memes, to allow me to defend myself in those cases where there is a conflict of interests.
Why should I side with the memes? They aren’t even related to me. The best of them are beneficial human symbionts—rather like lettuces and strawberries. I care for them some—but don’t exactly embrace their optimisation targets as my own.
I don’t dispute memes have done a lot of good things in the world. So has Mother Teresa—but that doesn’t mean I have to adopt her goals as my own either.
Er, the idea is that you can make a utility-maximising model of the agent—using the specified utility function—that does the same things the agent does if you put it in the same environment.
Can people please stop dissing the concept of a human utility function. Correcting these people is getting tedious—and I don’t want to be boring.
Doesn’t work. The Texas Sharpshooter utility function described by Dewar cannot be used to make a utility-maximising model of the agent, except by putting a copy of the actual agent into the box, seeing what it does, declaring that to have utility 1, and doing it. The step of declaring it to have utility 1 plays no role in deciding the actions. It is a uselessly spinning cog doing no more work than a suggestive name on a Lisp symbol.
I was thinking a similar thought about you. You’re the only person here that I’ve seen taking these trivial utility functions seriously.
The idea here is that—if the agent is computable—then it can be simulated by any other computable system. So, if the map between its inputs and state, and its motor output is computable then we can make another computable system which produces the same map—since all universal computing systems can simulate each other by virtue of being Turing complete (and systems made of e.g. partial recursive functions can simulate each other too—if they are given enough memory to do so).
I mentioned computability at the top, by saying: “any computable agent has a utility function”.
As far as anyone can tell, the whole universe is computable.
I don’t see how this bears on the possibility of modelling every agent by a utility-maximising agent. Dewar’s construction doesn’t work. Its simulation of an agent by a utility-maximising agent just uses the agent to simulate itself and attaches the label “utility=1” to its actions.
Dewey says pretty plainly: “any agents can be written in O-maximizer form”.
O-maximisers are just plain old utility maximisers. Dewey rechristens them “Observation-Utility Maximizers” in his reworked paper.
He makes an O-maximiser from an agent, A. Once you have the corresponding O-maximiser, the agent A could be discarded.
I know that he says that. I am saying, I thought pretty plainly, that I disagree with him.
He only does that in the earlier paper. His construction is as I described it: define O as doing whatever A does and label the result with utility 1. A is a part of O and cannot be discarded. He even calls this construction trivial himself, but underrates its triviality.
I don’t really understand which problem you are raising. If the O eventually contains a simulated copy of A—so what? O is still a utililty-maximiser that behaves the same way that A does if placed in the same environment.
The idea of a utility maximiser as used here is that it assigns utilities to all its possible actions and then chooses the action with the highest utility. O does that—so it qualifies as a utililty-maximiser.
O doesn’t assign utilities to its actions and then choose the best. It chooses its action (by simulating A), labels it with utility 1, and chooses to perform the action it just chose. The last two steps are irrelevant.
“Irrelevant”? If it didin’t perform those steps, it wouldn’t be a utility maximiser, and then the proof that you can build a utility maximiser which behaves like any computable agent wouldn’t go through. Those steps are an important part of the reason for exhibiting this construction in the first place.
I think that everyone understands the point you’re trying to make—you can usefully model people as having a utility function in a wide variety of cases—but very often people use such models unskillfully, and it causes people like me to facepalm. If you want to model a lot of humans, for instance, it’s simple and decently accurate to model them as having utility functions. Economics, say. And if you have something like AIXI, or as Dawkins might argue a gene, then a utility function isn’t even a model, it’s right there in front of you.
I hypothesize that the real trouble starts when a person confuses the two; he sees or imagines a Far model of humans with utility functions, zooms in on an individual human or zooms in on himself, and thinks he can see the real utility function sitting right there in front of him, like he could with AIXI. Yeah, he knows in the abstract that he doesn’t have direct access to it, but it feels Near. This can lead to a lot of confusion, and it leads people like me to think folk shouldn’t talk about a person’s “utility function” except in cases where it obviously applies.
Even where you can say “Person A has a utility function that assigns 4 utility to getting cheesecake and 2 utility to getting paperlips”, why not say “Agent A”? But that’s not what I facepalm at. I only facepalm when people say they got their “utility function” from natural selection (i.e. ignoring memes), or say they wish they could modify their utility function, et cetera. In many cases it works as an abstraction, but if you’re not at all thinking about EU, why not talk directly about your preferences/values? It’s simpler and less misleading.
This seems like a bit of a different issue—and one that I am not so interested in.
A couple of comments about your examples, though:
For someone like me it is pretty accurate to say that I got my utility function from natural selection acting on DNA genes. Memes influence me, but I try not to let them influnce my goals. I regard them as symbiotes: mutualists and pathogens. In principle they could do deals with me that might make me change my goals—but currently I have a powerful bargaining position, their bargaining position is typically weak—and so I just get my way. They don’t get to affect my goals. Those that try get rejected by my memetic immune system. I do not want to become the victim of a memetic hijacking.
As for the implied idea that natural selection does not apply to memes, I’ll try to bite my tongue there.
That seems closely equivalent to me. The cases where people talk about utility functions are mostly those where you want to compare with machines, or conjour up the idea of an expected utility maximiser for some reason. Sometimes even having “utility” in the context is enough for the conversation to wander on to utility functions.
My council would be something like: “Don’t like it? Get used to it!” There is not, in fact, anything wrong with it.
That totally wasn’t what I meant to imply. I am definitely a universal Darwinist. (You can view pretty much any optimization process as “evolution”, though, so in some cases it’s questionably useful. Bayesian updating is just like population genetics. But with memes it’s obviously a good description.)
Yes, but I think you’re rather unusual in this regard; most people aren’t so wary of memes. Might I ask why you prefer genes to memes? This seems odd to me. Largely because humans evolved for memes and with memes. Archetypes, for example. But also because the better memes seem to have done a lot of good in the world. (My genetically evolved cognitive algorithms—that is, the algorithms in my brain that I think aren’t the result of culture, but instead are universal machinery—stare in appreciation at the beauty of cathedrals, and are grateful that economies make my life easier.)
’s why I tried to bite my tongue—but it was difficult to completely let it go by...
Well, I love memes, but DNA-genes built 99% of my ancestors unassisted, and are mostly responsible for building me. They apparently equipped me with a memetic immune system, for weeding out undesirable memes, to allow me to defend myself in those cases where there is a conflict of interests.
Why should I side with the memes? They aren’t even related to me. The best of them are beneficial human symbionts—rather like lettuces and strawberries. I care for them some—but don’t exactly embrace their optimisation targets as my own.
I don’t dispute memes have done a lot of good things in the world. So has Mother Teresa—but that doesn’t mean I have to adopt her goals as my own either.