While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn’t be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we’re talking about practical purposes, this is a relevant point.
Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I’m generally more repulsed by the ones in which my map says it’s fine where it’s not than by the opposite.
any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
This something of a nitpick, but this isn’t strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
If you install an alarm system that uses a video camera to recognize movement and calls the police if it’s armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we’ve resorted to arguing about the semantics of “map” versus “territory”, as I don’t see a scenario where we’d predict or decide differently from each other on account of this disagreement. As such, I’m willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
The [alarm] system decides what to do by itself, according to the aspect of your values it implements.
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system’s controller—an implementation of values that are not our own.
If there are any “preferences about reality” in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker’s values.
By analogy, if our preferences are the implementation of any values, they are the “values” of natural selection, not our own.
If now you say that natural selection doesn’t have any preferences or values, then we are left with no preferences anywhere—merely isomorphism between control systems and their environments. Saying this isomorphism is “about” something is saying that a mental entity (the “about” relationship) exists in the real world, i.e., supernaturalism.
In short, what I’m saying is that anybody who argues human preferences are “about” reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of “preference”, and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are “about” reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its “preferences” still “about” reality in that case?
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
So an alarm system has preferences? That is not most people’s understanding of the word “preference”, which requires a degree of agency that most rationalists wouldn’t attribute to an alarm system.
Nonetheless, let us say an alarm system has preferences. You didn’t answer any of my follow-on questions for that case.
As for explaining away the rainbow, you seem to have me confused with an anti-reductionist. See Explaining vs. Explaining Away, in particular:
If you don’t distinguish between the multi-level map and the mono-level territory, then when someone tries to explain to you that the rainbow is not a fundamental thing in physics, acceptance of this will feel like erasing rainbows from your multi-level map, which feels like erasing rainbows from the world.
At this point, I am attempting to show that the very concept of a “preference” existing in the first place is something projected onto the world by an inbuilt bias in human perception. Reality does not have preferences, it has behaviors.
This is not erasing the rainbow from the world, it’s attempting to erase the projection of a mind-modeling variable (“preference”) from the world, in much the same way as Eliezer broke down the idea of “possible” actions in one of his series.
So, if you are claiming that preference actually exists, please give your definition of a preference, such that alarm systems and humans both have them.
Note that probability is also in the mind, but yet your see all the facts through it, and you can’t ever revoke it, each mind is locked in its subjectively objective character. What do you think of that?
I think that those things have already been very well explained by Eliezer—so much so that I assumed that you (and the others participating in this discussion) would have already internalized them to the same degree as I have, such that asserting “preferences” to be “about” things would be a blatantly obvious instance of the mind projection fallacy.
That’s why, early on, I tended to just speak as though it was bloody obvious, and why I haven’t been painstakingly breaking it all out piece by piece, and why I’ve been baffled by the argument, confusion, and downvoting from people for whom this sort of basic reductionism ought to be a bloody simple matter.
Oh, and finally, I think that you still haven’t given your definition of “preference”, such that humans and alarm systems both have it, so that we can then discuss how it can then be “about” something… and whether that “aboutness” exists in the thing having the preference, or merely in your mental model of the thing.
I think that those things have already been very well explained by Eliezer
That in reply to a comment full of links to Eliezer’s articles. You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument. I wasn’t even talking about preference.
I know. That’s the problem. See this comment and this one, where I asked for your definition of preference, which you still haven’t given.
You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument.
That’s because you also “didn’t answer my comment, but wrote some text that doesn’t help me in our argument.” I was attempting to redirect you to answering the question which you’ve now ducked twice in a row.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation. Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation. I’m currently trying to understand your position in terms of my position, not to explain to you my position.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation.
We reached a point in the discussion where it appears the only way we could disagree is if we had a different definition of “preference”. Since I believe I’ve made my definition quite clear, I wanted to know what yours is.
It might not help you, but it would certainly help me to understand your position, if you are not using the common definition of preference.
Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation.
I asked you first, and you responded with (AFAICT) a non-answer. You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
So, I actually answered your question (i.e. “what do you think?”), even though you still haven’t answered mine.
You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
I chose the articles in the comment above because they were in surface-match with what you are talking about, as a potential point on establishing understanding. I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
And I answered that I agree with them, and that I considered it foundational material to what I’m talking about.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
Indeed, which is why I’d now like to have the answer to my question, please. What definition of “preferences” are you using, such that an alarm system, thermostat, and human all have them? (Since this is not the common, non-metaphorical usage of “preference”.)
Preference is order on the lotteries of possible worlds (ideally established by expected utility), usually with agent a part of the world. Computations about this structure are normally performed by a mind inside the mind. The agent tries to find actions that determine the world to be as high as possible on the preference order, given the knowledge about it. Now, does it really help?
Yes, as it makes clear that what you’re talking about is a useful reduction of “preference”, unrelated to the common, “felt” meaning of “preference”. That alleviates the need to further discuss that portion of the reduction.
The next step of reduction would be to unpack your phrase “determine the world”… because that’s where you’re begging the question that the agent is determining the world, rather than determining the thing it models as “the world”.
So far, I have seen no-one explain how an agent can go beyond its own model of the world, except as perceived by another agent modeling the relationship between that agent and the world. It is simply repeatedly asserted (as you have effectively just done) as an obvious fact.
But if it is an obvious fact, it should be reducible, as “preference” is reducible, should it not?
While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn’t be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we’re talking about practical purposes, this is a relevant point.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I’m generally more repulsed by the ones in which my map says it’s fine where it’s not than by the opposite.
This something of a nitpick, but this isn’t strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
Oh, good point. I should have restricted it to “any calculation you personally do”, in which case I believe it holds.
At which point it becomes trivial: any calculation that is done on your map is done using your map, just Markovity of computation...
A related point is that you can create tools that make decisions themselves, in situations only of possibility of which you are aware.
Right. It’s trivial, but relevant when discussing in what sense our decision algorithms refer to territory versus map.
I can’t parse this. What do you mean?
If you install an alarm system that uses a video camera to recognize movement and calls the police if it’s armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we’ve resorted to arguing about the semantics of “map” versus “territory”, as I don’t see a scenario where we’d predict or decide differently from each other on account of this disagreement. As such, I’m willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
It appears to lead nowhere: your comments are clear, while his are all smoke and mirrors, in many many words.
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system’s controller—an implementation of values that are not our own.
If there are any “preferences about reality” in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker’s values.
By analogy, if our preferences are the implementation of any values, they are the “values” of natural selection, not our own.
If now you say that natural selection doesn’t have any preferences or values, then we are left with no preferences anywhere—merely isomorphism between control systems and their environments. Saying this isomorphism is “about” something is saying that a mental entity (the “about” relationship) exists in the real world, i.e., supernaturalism.
In short, what I’m saying is that anybody who argues human preferences are “about” reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of “preference”, and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are “about” reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its “preferences” still “about” reality in that case?
Both. You are now trying to explain away the rainbow, by insisting that it consists of atoms, which can’t in themselves possess the properties of a rainbow.
So an alarm system has preferences? That is not most people’s understanding of the word “preference”, which requires a degree of agency that most rationalists wouldn’t attribute to an alarm system.
Nonetheless, let us say an alarm system has preferences. You didn’t answer any of my follow-on questions for that case.
As for explaining away the rainbow, you seem to have me confused with an anti-reductionist. See Explaining vs. Explaining Away, in particular:
At this point, I am attempting to show that the very concept of a “preference” existing in the first place is something projected onto the world by an inbuilt bias in human perception. Reality does not have preferences, it has behaviors.
This is not erasing the rainbow from the world, it’s attempting to erase the projection of a mind-modeling variable (“preference”) from the world, in much the same way as Eliezer broke down the idea of “possible” actions in one of his series.
So, if you are claiming that preference actually exists, please give your definition of a preference, such that alarm systems and humans both have them.
A good reply, if only you approached the discussion this constructively more often.
Note that probability is also in the mind, but yet your see all the facts through it, and you can’t ever revoke it, each mind is locked in its subjectively objective character. What do you think of that?
I think that those things have already been very well explained by Eliezer—so much so that I assumed that you (and the others participating in this discussion) would have already internalized them to the same degree as I have, such that asserting “preferences” to be “about” things would be a blatantly obvious instance of the mind projection fallacy.
That’s why, early on, I tended to just speak as though it was bloody obvious, and why I haven’t been painstakingly breaking it all out piece by piece, and why I’ve been baffled by the argument, confusion, and downvoting from people for whom this sort of basic reductionism ought to be a bloody simple matter.
Oh, and finally, I think that you still haven’t given your definition of “preference”, such that humans and alarm systems both have it, so that we can then discuss how it can then be “about” something… and whether that “aboutness” exists in the thing having the preference, or merely in your mental model of the thing.
That in reply to a comment full of links to Eliezer’s articles. You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument. I wasn’t even talking about preference.
I know. That’s the problem. See this comment and this one, where I asked for your definition of preference, which you still haven’t given.
That’s because you also “didn’t answer my comment, but wrote some text that doesn’t help me in our argument.” I was attempting to redirect you to answering the question which you’ve now ducked twice in a row.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation. Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation. I’m currently trying to understand your position in terms of my position, not to explain to you my position.
We reached a point in the discussion where it appears the only way we could disagree is if we had a different definition of “preference”. Since I believe I’ve made my definition quite clear, I wanted to know what yours is.
It might not help you, but it would certainly help me to understand your position, if you are not using the common definition of preference.
I asked you first, and you responded with (AFAICT) a non-answer. You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
So, I actually answered your question (i.e. “what do you think?”), even though you still haven’t answered mine.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
I chose the articles in the comment above because they were in surface-match with what you are talking about, as a potential point on establishing understanding. I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
And I answered that I agree with them, and that I considered it foundational material to what I’m talking about.
Indeed, which is why I’d now like to have the answer to my question, please. What definition of “preferences” are you using, such that an alarm system, thermostat, and human all have them? (Since this is not the common, non-metaphorical usage of “preference”.)
Preference is order on the lotteries of possible worlds (ideally established by expected utility), usually with agent a part of the world. Computations about this structure are normally performed by a mind inside the mind. The agent tries to find actions that determine the world to be as high as possible on the preference order, given the knowledge about it. Now, does it really help?
Yes, as it makes clear that what you’re talking about is a useful reduction of “preference”, unrelated to the common, “felt” meaning of “preference”. That alleviates the need to further discuss that portion of the reduction.
The next step of reduction would be to unpack your phrase “determine the world”… because that’s where you’re begging the question that the agent is determining the world, rather than determining the thing it models as “the world”.
So far, I have seen no-one explain how an agent can go beyond its own model of the world, except as perceived by another agent modeling the relationship between that agent and the world. It is simply repeatedly asserted (as you have effectively just done) as an obvious fact.
But if it is an obvious fact, it should be reducible, as “preference” is reducible, should it not?
Hmm… Okay, this should’ve been easier if the possibility of this agreement was apparent to you. This thread is thereby merged here.