Where X is some set. What should that set be? Certainly it shouldn’t be the set of states of the universe, because then you can’t say that you enjoy certain processes (such as bringing up a child, as opposed to the child just appearing). Perhaps the set of possible histories of the universe is a better candidate.
Part of the problem is the X is necessarily based in the map, not the territory. There will always be the chance that one will learn something that radically changes the map, so it seems like an explicit statement of f will have to involve all possible maps that one might have.
Part of the problem is the X is necessarily based in the map, not the territory.
Necessarily? If I place value on e.g. “my friends actually liking and respecting me”, rather than just “the subjective sense that my friends like and respect me” then my utility function seems to be responding directly to the territory rather than the map. (It also means that I won’t ever really know my utility, but that’s true of lots of things.) Some people argue that things can’t affect one’s well-being unless one somehow experiences them, but that’s a contentious position.
You’re right—I should have written “necessarily based in the map, not just the territory.”
My intended meaning has to do with fundamental shifts in one’s understanding of how reality works that make some previous apparent question of fact become a “wrong question” or category error or similar non-issue.
If I place value on e.g. “my friends actually liking and respecting me”, rather than just “the subjective sense that my friends like and respect me” then my utility function seems to be responding directly to the territory rather than the map.
In practice, your definition of what “liking and respecting me” means—i.e., what evidence you expect to see in the world of that—is part of the map, not the territory.
Suppose, for example, that your friends really and truly like and respect you… but they have to beat you up and call you names, for some other reason. Does that match what you actually value? It’s out there in the territory, after all.
That is, is merely knowing that they “like and respect you” enough? Or is that phrase really just a shorthand in your map for a set of behaviors and non-behaviors that you actually value?
Note that if you argue that, “if they really liked and respected me, they wouldn’t do that”, then you are now back to talking about your map of what that phrase means, as opposed to what someone else’s map is.
System 2 thinking is very tricky this way—it’s prone to manipulating symbols as if they were the things they’re merely pointing at, as though the map were the territory… when the only things that exist in its perceptual sphere are the labels on the map.
Most of the time, when we think we’re talking about the territory, we’re talking about the shapes on the map, but words aren’t even the shapes on the map!
We have only access to our current map to tell us about the territory, yes. But we have strong intuitions about how we would act if we could explicitly choose that our future map permanently diverge from our current map (which we currently see as the territory). If we (again by our current map) believe that this divergence would conform less to the territory (as opposed to a new map created by learning information), many of us would oppose that change even against pretty high stakes.
I mean, if Omega told me that I had to choose between
(A) my sister on Mars being well but cut off from all contact with me, or
(B) my sister being killed but a nonsentient chatbot impersonating her to me in happy weekly chats,
and that in either case my memory of this choice would be wiped when I made it, I would choose (A) without hesitation.
I understand that calling our current map “the territory” looks like a categorical error, but rejecting conchis’ point entirely is the wrong response. There’s a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.
Of course, because the immediate pain of the thought of choosing B would outweigh the longer-term lesser pain of the thought of losing contact with your sister.
This has nothing to do with whether the events actually occur, and everything to do with your mapping of the experience of the conditions, as you imagine them for purposes of making a decision.
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences
It so happens that there is some correspondence between this (real) process and the way we would prefer to think we establish and evaluate our preferences. Specifically, both models will generate similar results, most of the time. It’s just that the reasons we end up with for the responses are quite different.
There’s a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.
But calling that latter concept “territory” is still a category error, because what you are using to evaluate it is still your perception of how you would experience the change.
We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have “rational” preferences it is because they will ultimately lead to some desired emotion or sensation.
However, our brains are constructed in such a way so as to allow us to plausibly overlook and deny this fact, so that we can be honestly “sincere” in our altruism… specifically by claiming that our responses are “really” about things outside ourselves.
For example, your choice of “A” allows you to self-signal altruism, even if your sister would actually prefer death to being imprisoned on Mars for the rest of her life! Your choice isn’t about making her life better, it’s about you feeling better for the brief moment that you’re aware you did something.
(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences.
I affirm this, but it does not follow that:
This has nothing to do with whether the events actually occur...
Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.
Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.
And a thermostat’s map is also “entangled” with the territory, but as loqi pointed out, what it really prefers is only that its input sensor match its temperature setting!
I am not saying there are no isomorphisms between the shape of our preferences and the shape of reality, I am saying that assuming this isomorphism means the preferences are therefore “about” the territory is mind projection.
If you look at a thermostat, you can project that it was made by an optimizing process that “wanted” it to do certain things by responding to the territory, and that thus, the thermostat’s map is “about” the territory. And in the same way, you can look at a human and project that it was made by an optimizing process (evolution) that “wanted” it to do certain thing by responding to the territory.
However, the “aboutness” of the thermostat does not reside in the thermostat; it resides in the maker of the thermostat, if it can be said to exist at all! (In fact, this “aboutness” cannot exist, because it is not a material entity; it’s a mental entity—the idea of aboutness.)
So despite the existence of inputs and outputs, both the human and the thermostat do their “preference” calculations inside the closed box of their respective models of the world.
It just so happens that humans’ model of the world also includes a Mind Projection device, that causes humans to see intention and purpose everywhere they look. And when they look through this lens at themselves, they imagine that their preferences are about the territory… which then keeps them from noticing various kinds of erroneous reasoning and subgoal stomps.
For that matter, it keeps them from noticing things like the idea that if you practice being a pessimist, nothing good can last for you, because you’ve trained yourself to find bad things about anything. (And vice versa for optimists.)
Ostensibly, optimism and pessimism are “about” the outside world, but in fact, they’re simply mechanical, homeostatic processes very much like a thermostat.
I am not a solipsist nor do I believe people “create your own reality”, with respect to the actual territory. What I’m saying is that people are deluded about the degree of isomorphism between their preferences and reality, because they confuse the map with the territory. And even with maximal isomorphism between preference and reality, they are still living in the closed box of their model.
It is reasonable to assume that existence actually exists, but all we can actually reason about is our experience of it, “inside the box”.
That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.
Then, as I said, he cares about something closer to the reality.
The major point I’ve been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations. And not asking the question is a perfect example of the sort of decision error this can produce!
However, asking the question doesn’t magically make the preference about the territory either; in order to prefer the future include his sister’s best interests, he must first have an experience of the sister and a reason to wish well of her. But it’s still better than not asking, which is basically wireheading.
The irony I find in this discussion is that people seem to think I’m in favor of wireheading because I point out that we’re all doing it, all the time. When in fact, the usefulness of being aware that it’s all wireheading, is that it makes you better at noticing when you’re doing it less-usefully.
The fact that he hadn’t asked his sister, or about his sister’s actual well-being instantly jumped off the screen at me, because it was (to me) obvious wireheading.
So, you could say that I’m biased by my belief to notice wireheading more, but that’s an advantage for a rationalist, not a disadvantage.
The major point I’ve been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations.
Is human knowledge also not just in the map, but exclusively of the map? If not, what’s the difference?
Is human knowledge also not just in the map, but exclusively of the map? If not, what’s the difference?
Any knowledge about the actual territory can in principle be reduced to mechanical form without the presence of a human being in the system.
To put it another way, a preference is not a procedure, process, or product. The very use of the word “preference” is a mind projection—mechanical systems do not have “preferences”—they just have behavior.
The only reason we even think we have preferences in the first place (let alone that they’re about the territory!) is because we have inbuilt mind projection. The very idea of having preferences is hardwired into the model we use for thinking about other animals and people.
So, knowledge exists in the structure of map and is about the territory, while preference can’t be implemented in natural artifacts. Preference is a magical property of subjective experience, and it is over maps, or about subjective experience, but not, for example, about the brain. Saying that preference exists in the structure of map or that it is about the territory is a confusion, that you call “mind projection” Does that summarize your position? What are the specific errors in this account?
Preference is a magical property of subjective experience
No, “preference” is an illusory magical property projected by brains onto reality, which contains only behaviors.
Our brains infer “preferences” as a way of modeling expected behaviors of other agents: humans, animals, and anything else we perceive as having agency (e.g. gods, spirits, monsters). When a thing has a behavior, our brains conclude that the thing “prefers” to have either the behavior or the outcome of the behavior, in a particular circumstance. In other words, “preference” is a label attached to a clump of behavior-tendency observations and predictions in the brain—not a statement about the nature of the thing being observed.
Thus, presuming that these “preferences” actually exist in the territory is supernaturalism, i.e., acting as though basic mental entities exist.
My original point had more to do with the types of delusion that occur when we reason on the basis of preferences actually existing, rather than the idea simply being a projection of our own minds. However, the above will do for a start, as I believe my other conclusions can be easily reached from this point.
Thus, presuming that these “preferences” actually exist in the territory is supernaturalism, i.e., acting as though basic mental entities exist.
Do you think someone is advocating the position that goodness of properties of the territory is an inherent property of territory (that sounds like a kind of moral realism)? This looks like the lack of distinction between 1-place and 2-place words. You could analogize preference (and knowledge) as a relation between the mind and the (possible states of the) territory, that is neither a property of the mind alone, nor of the territory alone, but a property of them being involved in a certain interaction.
Do you think someone is advocating the position that goodness of properties of the territory is an inherent property of territory?
No, I assume that everybody who’s been seriously participating has at least got that part straight.
This looks like the lack of distinction between 1-place and 2-place words. You could analogize preference (and knowledge) as a relation between the mind and the (possible states of the) territory, that is neither a property of the mind alone, nor of the territory alone, but a property of them being involved in a certain interaction.
Now you’re getting close to what I’m saying, but on the wrong logical level. What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
If I assert that my preferences are “about” the real world, I am making a category error because my preferences are relationships between portions of my map, some of which I have labeled as representing the territory.
The fact that there is a limited isomorphism between that portion of my map, and the actual territory, does not make my preferences “about” the territory, unless you represent that idea in another map.
That is, I can represent the idea that “your” preferences are about the territory in my map… in that I can posit a relationship between the part of my map referring to “you”, and the part of my map referring to “the territory”. But that “aboutness” relationship is only contained in my map; it doesn’t exist in reality either.
That’s why it’s always a mind projection fallacy to assert that preferences are “about” territory: one cannot assert it of one’s own preferences, because that implies the territory is inside the map. And if one asserts it of another person’s preferences, then that one is projecting their own map onto the territory.
I initially only picked on the specific case of self-applied projection, because understanding that case can be very practically useful for mind hacking. In particular, it helps to dissolve certain irrational fears that changing one’s preferences will necessarily result in undesirable futures. (That is, these fears are worrying that the gnomes and fairies will be destroyed by the truth, when in fact they were never there to start with.)
Now you’re getting close to what I’m saying, but on the wrong logical level. What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
How’s that? You can write Newton’s law of universal gravitation describing the orbit of the Earth around the Sun on a piece of paper located on the surface of a table standing in a house on the surface of the Earth. Where does this analogy break from your point of view?
″...but, you can’t fold up the territory and put it in your glove compartment”
How’s that? You can write Newton’s law of universal gravitation describing the orbit of the Earth around the Sun on a piece of paper located on the surface of a table standing in a house on the surface of the Earth. Where does this analogy break from your point of view?
The “aboutness” relationship between the written version of Newton’s law and the actual instances of it is something that lives in the map in your head.
IOW, the aboutness is not on the piece of paper. Nor does it exist in some supernatural link between the piece of paper and the objects acting in accordance with the expressed law.
The “aboutness” relationship between the written version of Newton’s law and the actual instances of it is something that lives in the map in your head.
Your head describes how your head rotates around the Sun.
What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
Your head describes how your head rotates around the Sun.
No, your head is rotating around the Sun, and it contains a description relating the ideas of “head” and “Sun”. You are confusing head 1 (the real head) with head 2 (the “head” pictured inside head 1), as well as Sun 1 (the real Sun) and Sun 2 (the “Sun” pictured inside head 1).
No, I’m not confusing them. They are different things. Yet the model simulates the real thing, which means the following (instead of magical aboutness): By examining the model it’s possible to discover new properties of its real counterpart, that were not apparent when the model was being constructed, and that can’t be observed directly (or it’s just harder to do), yet can be computed from the model.
By examining the model it’s possible to discover new properties of its real counterpart, that were not apparent when the model was being constructed, and that can’t be observed directly (or it’s just harder to do), yet can be computed from the model.
Indeed. Although more precisely, examining the model merely suggests or predicts these “new” (rather, previously undiscovered, unnoticed, or unobservable) properties.
That is what I mean by isomorphism between model and territory. The common usage of “about”, however, projects an intention onto this isomorphism—a link that can only exist in the mind of the observer, not the similarity of shapes between one physical process and another.
Since agent’s possible actions are one of the things in the territory captured by the model, it’s possible to use the model to select an action leading to a preferable outcome, and to perform thus selected action, determining the territory to conform with the plan. The correspondence between the preferred state of the world in the mind and the real world is ensured by this mechanism for turning plans into actuality. Pathologies aside, or course.
I don’t disagree with anything you’ve just said, but it does nothing to support the idea of an isomorphism inherently meaning that one thing is “about” another.
If I come across a near-spherical rock that resembles the moon, does this make the rock “about” the moon? If I find another rock that is shaped the same, does that mean it is about the moon? The first rock? Something else entirely?
The :”aboutness” of a thing can’t be in the thing, and that applies equally to thermostats and humans.
The (external) aboutness of a thermostat’s actions don’t reside in the thermostat’s map, and humans are deluded when they project that the (external) aboutness of their own actions actually resides within the same map they’re using to decide those actions. It is merely a sometimes-useful (but often harmful) fiction.
Taboo “aboutness” already. However unfathomably confused the philosophic and folk usage of this word is doesn’t interest me much. What I mean by this word I described in thesecomments, and this usage seems reasonably close to the usual one, which justifies highjacking the word for the semi-technical meaning rather than inventing a new one. This is also the way meaning/aboutness is developed in formal theories of semantics.
What I mean by this word I described in these comments, and this usage seems reasonably close to the usual one, which justifies highjacking the word for the semi-technical meaning rather than inventing a new one.
So, you are saying that you have no argument with my position, because you have not been using either “about” or “preference” with their common usage?
If that is the case, why couldn’t you simply say that, instead of continued argument and posturing about your narrower definition of the words? ISTM you could have pointed that out days ago and saved us all a lot of time.
This is also not the first time where I have been reducing the common usage of a word (e.g. “should”) and then had you argue that I was wrong, based on your own personal meaning of the word.
Since I have no way of knowing in advance all of the words you have chosen to redefine in your specialized vocabulary, would it be too much to ask if you point out which words you are treating as specialized when you argue that my objection to (or reduction of) the common meaning of the word is incorrect, because it does not apply to your already-reduced personal version of the word?
Then, I could simply nod, and perhaps ask for your reduction in the case where I do not have a good one already, and we would not need to have an extended argument where we are using utterly incompatible definitions for such words as “about”, “preference”, and “should.”
Actually, the point is that most of the other usages of these words are meaningless confusion, and the argument is that this particular semi-technical sense is what the word actually means, when you get the nonsense out of it. It’s not how it’s used, but it’s the only meaningful thing that fits the idea.
Since you don’t just describe the usage of the word, but argue for the confusion behind it, we have a disagreement. Presenting a clear definition is the easy part. Showing that ten volumes of the Encyclopedia of Astrology is utter nonsense is harder, and arguing with each point made in its chapters is a wrong approach. It should be debunked on meta-level, with an argument that doesn’t require the object-level details, but that requires the understanding of the general shape of the confusion.
Actually, the point is that most of the other usages of these words are meaningless confusion
Yes, but ones which most people do not understand to be confusion, and the only reason I started this discussion in the first place was because I was trying to clear up one point in that confusion.
Since you don’t just describe the usage of the word, but argue for the confusion behind it, we have a disagreement
I am arguing against the confusion, not for the confusion. So, as far as I can tell, there should be no disagreement.
In practice, however, you have been making arguments that sound like you are still confusing map and territory in your own thinking, despite seeming to agree with my reasoning on the surface. You are consistently treating “about” as a 2-way relationship, when to be minimally cohesive, it requires 3 entities: the 2 entities that have an isomorphism, and the third entity whose map ascribes some significance to this isomorphism.
You’ve consistently omitted the presence of the third entity, making it sound as though you do not believe it to be required, and thereby committing the mind projection fallacy.
Also, please stop using “mind projection fallacy”, you are misapplying the term.
How is that, precisely? My understanding is that it is mind projection when you mistakenly believe a property of an object to be intrinsic, rather than merely attributed.
I am pointing out that “aboutness” (whose definition I never agreed on, because you handwaved away the subject by saying it is I who should define it), is not an intrinsic property of isomorphic relationships.
Rather, it is a property being attributed to that relationship, a label that is being expressed in some map.
(Prediction: your next reply will still not address this point, nor clarify your definition of “about”, but simply handwave again why it is that I am doing something else wrong. Anything but actually admitting that you have been using a mind-projecting definition of “about” since the beginning of this conversation, right up until the point where you ducked the question by asking me to taboo it, rather than defend the imprecise definition you’ve been using, or clear up any of the other handwaving you’ve been using to beg the question. I base this prediction on the rapid increase in non-responsive replies that, instead of defending the weak points of your position, represent attempts to find new ways to attack me and/or my position. A rationalist should be able to attack their own weak points, let alone being able to defend them, without resorting to distraction, subject-changing, and playing to the gallery.)
There are natural categories, like “tigers”, that don’t require much of a mind to define. It’s not mind projection fallacy to say that something is a tiger.
P.S. I’m correcting self-censoring threshold, so expect silence where before I’d say something for the fifth time.
There are natural categories, like “tigers”, that don’t require much of a mind to define. It’s not mind projection fallacy to say that something is a tiger.
Is that actually an argument? ’cause it sounds like a random sentence injected into the conversation, perhaps as an invitation for me to waste time tearing “natural categories” to shreds, while leaving you still able to deny that your statement actually relates in any substantial way to your point… thereby once again relieving you of any need to actually defend your position.
That is, are you actually claiming aboutness to be a natural category? Or just trying to get me to treat your argument as if you were doing so?
P.S. I’m correcting self-censoring threshold, so expect silence
I already did and do expect it; see my “prediction” in the parent to your comment. I predicted that you would remain silent on any substantive issues, and avoid admitting anywhere where you were mistaken or incorrect. (I notice, for example, that you went back and deleted the comment where you said I was using “mind projection fallacy” incorrectly, rather than admit your attack was in error.)
And, as predicted, you avoided directly addressing the actual point of contention, instead choosing to enter a new piece of handwaving to imply that I am doing something else wrong.
That is, you appear to now be implying that I am using an overbroad definition of the MPF, without actually saying that I am doing it, or that your statement is in any way connected to your own position. This is a nice double bind, since either way I interpret the statement, you can retreat… and throw in more irrelevancies.
I don’t know if “troll” is a natural category, but you’re sure getting close to where I’d mind-project your behavior as matching that of one. ;-)
For the record, I thought it obvious that my argument above implied that I claim aboutness to be a natural category (although I’m not perfectly sure it’s a sound argument). I deleted my comment because I deemed it low-quality, before knowing you responded to it.
I claim aboutness to be a natural category (although I’m not perfectly sure it’s a sound argument)
It’s not.
First, the only way it can be one is if “natural category” has the reductionist meaning of “a category based on distinctions that humans are biased towards using as discriminators”, rather than “a category that ‘naturally’ exists in the territory”. (Categories are abstractions, not physical entities, after all.)
And second, even if you do use the reductionist meaning of “natural category”, then this does not in any way undermine the conclusion that “aboutness” is mind projection when you omit the entity mapping that aboutness from the description.
In other words, this argument appears to result in only one of two possibilities: either “aboutness” is not a natural category per the reductionist definition, and thus inherently a mind projection when the attribution source is omitted, or “aboutness” is a natural category per the reductionist definition… in which case the attribution source has to be a human brain (i.e., in another map).
Finally, if we entirely reject the reductionist definition of “natural category”, then “natural category” is itself an instance of the mind projection fallacy, since the description omits any definition of for whom the category is “natural”.
In short, QED: the argument is not sound. (I just didn’t want to bother typing all this if you were going to retreat to a claim this was never your argument.)
To the (unknowable*) extent that the portion of my map labelled “territory” is an accurate reflection of the relevant portion of the territory, do I get to say that my preferences are “about” the territory (implicitly including disclaimers like “as mediated by the map”)?
* due at the very least to Matrix/simulation scenarios
To the (unknowable*) extent that the portion of my map labelled “territory” is an accurate reflection of the relevant portion of the territory, do I get to say that my preferences are “about” the territory (implicitly including disclaimers like “as mediated by the map”)?
You can say it all you want, it just won’t make it true. ;-) Your preference is “about” your experience, just as the thermostat’s heating and cooling preferences are “about” the temperature of its sensor, relative to its setting.
For there to be an “about”, there has to be another observer, projecting a relationship of intention onto the two things. It’s a self-applied mind projection—a “strange loop” in your model—to assert that you can make such statements about your own preferences, like a drawing of Escher wherein Escher is pictured, making the drawing. The whole thing only makes sense within the surface of the paper.
(Heck, it’s probably a similar strange loop to make statements about one’s self in general, but this probably doesn’t lead to the same kind of confusion and behavioral problems that result from making assertions about one’s preferences.… No, wait, actually, yes it does! Self-applied nominalizations, like “I’m bad at math” are an excellent example. Huh. I keep learning interesting new things in this discussion.)
I feel your frustration, but throwing the word “magical” in there is just picking a fight, IMO. Anyway, I too would like to see P.J. Eby summarize his position in this format.
I have a certain technical notion of magic in mind. This particular comment wasn’t about frustration (some of the others were), I’m trying out something different of which I might write a post later.
(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)
Be charitable in your interpretation, and remember the Least Convenient Possible World principle. I was presuming that the setup was such that being alive on Mars wouldn’t be a ‘fate worse than death’ for her; if it were, I’d choose differently. If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences.
I would say rather that my reaction is my evaluation of an imagined future world. The essence of many decision algorithms is to model possible futures and compare them to some criteria. In this case, I have complicated unconscious affective criteria for imagined futures (which dovetail well with my affective criteria for states of affairs I directly experience), and my affective reaction generally determines my actions.
We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have “rational” preferences it is because they will ultimately lead to some desired emotion or sensation.
To the extent this is true (as in the sense of my previous sentence), it is a tautology. I understand what you’re arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals. I am not asserting this; I believe that our affective algorithms do often operate under more selfish and basic criteria, and that they fixate on the most salient possibilities instead of weighing probabilities properly, among other things.
However, these affective algorithms do appear to respond more strongly to certain facets of “how I expect the world to be” than to facets of “how I expect to think the world is” when the two conflict (with an added penalty for the expectation of being deceived), and I don’t find that problematic on any level.
If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.
As I said, it’s still going to be about your experience during the moments until your memory is erased.
I understand what you’re arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals.
I took that as a given, actually. ;-) What I’m really arguing against is the naive self-applied mind projection fallacy that causes people to see themselves as decision-making agents—i.e., beings with “souls”, if you will. Asserting that your preferences are “about” the territory is the same sort of error as saying that the thermostat “wants” it to be a certain temperature. The “wanting” is not in the thermostat, it’s in the thermostat’s maker.
Of course, it makes for convenient language to say it wants, but we should not confuse this with thinking the thermostat can really “want” anything but for its input and setting to match. And the same goes for humans.
(This is not a mere fine point of tautological philosophy; human preferences in general suffer from high degrees of subgoal stomp, chaotic loops, and other undesirable consequences arising as a direct result of this erroneous projection. Understanding the actual nature of preferences makes it easier to dissolve these confusions.)
What features of that comment made it communicate something new to you? What was it that got communicated?
The comment restated a claim that a certain relationship is desirable as a claim that given that it’s desirable, there is a process that establishes it to be true. It’s interesting how this restatement could pierce inferential distance: is preference less trustworthy than a fact, and so demonstrating the conversion of preference into a fact strengthens the case?
Given the length of the thread I branched from, it looks like you and P.J. Eby ended up talking past each other to some extent, and I think that you both failed to distinguish explicitly between the current map (which is what you calculate the territory to be) and a hypothetical future map.
P.J. Eby was (correctly) insisting that your utility function is only in contact with your current map, not the territory directly. You were (correctly) insisting that your utility function cares about (what it calculates to be) the future territory, and not just the future map.
Utility function is no more “in contact” with your current map than the actual truth of 2+2=4 is in contact with display of a calculator that displays the statement. Utility function may care about past territory (and even counterfactual territory) as well as future territory, with map being its part. Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn’t be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we’re talking about practical purposes, this is a relevant point.
Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I’m generally more repulsed by the ones in which my map says it’s fine where it’s not than by the opposite.
any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
This something of a nitpick, but this isn’t strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
If you install an alarm system that uses a video camera to recognize movement and calls the police if it’s armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we’ve resorted to arguing about the semantics of “map” versus “territory”, as I don’t see a scenario where we’d predict or decide differently from each other on account of this disagreement. As such, I’m willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
The [alarm] system decides what to do by itself, according to the aspect of your values it implements.
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system’s controller—an implementation of values that are not our own.
If there are any “preferences about reality” in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker’s values.
By analogy, if our preferences are the implementation of any values, they are the “values” of natural selection, not our own.
If now you say that natural selection doesn’t have any preferences or values, then we are left with no preferences anywhere—merely isomorphism between control systems and their environments. Saying this isomorphism is “about” something is saying that a mental entity (the “about” relationship) exists in the real world, i.e., supernaturalism.
In short, what I’m saying is that anybody who argues human preferences are “about” reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of “preference”, and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are “about” reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its “preferences” still “about” reality in that case?
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
So an alarm system has preferences? That is not most people’s understanding of the word “preference”, which requires a degree of agency that most rationalists wouldn’t attribute to an alarm system.
Nonetheless, let us say an alarm system has preferences. You didn’t answer any of my follow-on questions for that case.
As for explaining away the rainbow, you seem to have me confused with an anti-reductionist. See Explaining vs. Explaining Away, in particular:
If you don’t distinguish between the multi-level map and the mono-level territory, then when someone tries to explain to you that the rainbow is not a fundamental thing in physics, acceptance of this will feel like erasing rainbows from your multi-level map, which feels like erasing rainbows from the world.
At this point, I am attempting to show that the very concept of a “preference” existing in the first place is something projected onto the world by an inbuilt bias in human perception. Reality does not have preferences, it has behaviors.
This is not erasing the rainbow from the world, it’s attempting to erase the projection of a mind-modeling variable (“preference”) from the world, in much the same way as Eliezer broke down the idea of “possible” actions in one of his series.
So, if you are claiming that preference actually exists, please give your definition of a preference, such that alarm systems and humans both have them.
Note that probability is also in the mind, but yet your see all the facts through it, and you can’t ever revoke it, each mind is locked in its subjectively objective character. What do you think of that?
I think that those things have already been very well explained by Eliezer—so much so that I assumed that you (and the others participating in this discussion) would have already internalized them to the same degree as I have, such that asserting “preferences” to be “about” things would be a blatantly obvious instance of the mind projection fallacy.
That’s why, early on, I tended to just speak as though it was bloody obvious, and why I haven’t been painstakingly breaking it all out piece by piece, and why I’ve been baffled by the argument, confusion, and downvoting from people for whom this sort of basic reductionism ought to be a bloody simple matter.
Oh, and finally, I think that you still haven’t given your definition of “preference”, such that humans and alarm systems both have it, so that we can then discuss how it can then be “about” something… and whether that “aboutness” exists in the thing having the preference, or merely in your mental model of the thing.
I think that those things have already been very well explained by Eliezer
That in reply to a comment full of links to Eliezer’s articles. You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument. I wasn’t even talking about preference.
I know. That’s the problem. See this comment and this one, where I asked for your definition of preference, which you still haven’t given.
You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument.
That’s because you also “didn’t answer my comment, but wrote some text that doesn’t help me in our argument.” I was attempting to redirect you to answering the question which you’ve now ducked twice in a row.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation. Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation. I’m currently trying to understand your position in terms of my position, not to explain to you my position.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation.
We reached a point in the discussion where it appears the only way we could disagree is if we had a different definition of “preference”. Since I believe I’ve made my definition quite clear, I wanted to know what yours is.
It might not help you, but it would certainly help me to understand your position, if you are not using the common definition of preference.
Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation.
I asked you first, and you responded with (AFAICT) a non-answer. You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
So, I actually answered your question (i.e. “what do you think?”), even though you still haven’t answered mine.
You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
I chose the articles in the comment above because they were in surface-match with what you are talking about, as a potential point on establishing understanding. I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
And I answered that I agree with them, and that I considered it foundational material to what I’m talking about.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
Indeed, which is why I’d now like to have the answer to my question, please. What definition of “preferences” are you using, such that an alarm system, thermostat, and human all have them? (Since this is not the common, non-metaphorical usage of “preference”.)
Preference is order on the lotteries of possible worlds (ideally established by expected utility), usually with agent a part of the world. Computations about this structure are normally performed by a mind inside the mind. The agent tries to find actions that determine the world to be as high as possible on the preference order, given the knowledge about it. Now, does it really help?
Yes, as it makes clear that what you’re talking about is a useful reduction of “preference”, unrelated to the common, “felt” meaning of “preference”. That alleviates the need to further discuss that portion of the reduction.
The next step of reduction would be to unpack your phrase “determine the world”… because that’s where you’re begging the question that the agent is determining the world, rather than determining the thing it models as “the world”.
So far, I have seen no-one explain how an agent can go beyond its own model of the world, except as perceived by another agent modeling the relationship between that agent and the world. It is simply repeatedly asserted (as you have effectively just done) as an obvious fact.
But if it is an obvious fact, it should be reducible, as “preference” is reducible, should it not?
I’d been following this topic and getting frustrated with my inability to put my opinion on the whole preferences-about-the-territory thing into words, and I thought that orthonomal’s comment accomplished it very nicely. I don’t think I understand your other question.
Utility is about the territory in the same sense that the map is about the territory; the map tells us the way the territory is, utility tells us the way we want the territory to be. Us non-wireheaders want an accurate map because it’s the territory we care about.
Supposing utility is not about the territory but about the map, we get people who want nothing more than to sabotage their own mapmaking capabilities. If the territory is not what we care about, maintaining the correspondence of map to territory would be a pointless waste of effort. Wireheading would look like an unambiguously good idea, not just to some people but to everyone.
Conchis’ example of wanting his friends to really like and respect him is correct. He may have failed to explicitly point out that he also has an unrelated preference for not being beaten up. He’s also in the unfortunate position of valuing something he can’t know about without using long chains of messy inductive inferences. But his values are still about the territory, and he wants his map to accurately reflect the territory because it’s the territory he cares about.
Utility is about the territory in the same sense that the map is about the territory; the map tells us the way the territory is, utility tells us the way we want the territory to be. Us non-wireheaders want an accurate map because it’s the territory we care about.
I am only saying that the entire stack of concepts you have just mentioned exists only in your map.
Supposing utility is not about the territory but about the map, we get people who want nothing more than to sabotage their own mapmaking capabilities.
Permit me to translate: supposing utility is not about the (portion of map labeled) territory but about the (portion of map labeled) map, we get people who want nothing more than to sabotage their own mapmaking capabilities.
Does that make it any clearer what I’m saying?
This is a “does the tree make a sound” argument, and I’m on the, “no it doesn’t” side, due to using a definition of “sound” that means “the representation of audio waves within a human nervous system”. You are on the “of course it makes a sound” side, because your definition of sound is “pressure waves in the air.”
I am only saying that the entire stack of concepts you have just mentioned exists only in your map.
As far as I can tell, you’re saying that there is no territory, or that the territory is irrelevant. In other words, solipsism. You’ve overcome the naive map/territory confusion, but only to wind up with a more sophisticated form of confusion.
This isn’t a “does the tree make a sound” argument. It’s more like a “dude… how do we even really know reality is really real” argument. Rationality is entirely pointless if all we’re doing is manipulating completely arbitrary map-symbols. But in that case, why not leave us poor, deluded believers in reality to define the words “map”, “territory”, and “utility” the way we have always done?
Even though “this is not a pipe”, the form of a depiction of a pipe is nevertheless highly constrained by the physical properties of actual pipes. Do you deny that? If not, how do you explain it?
This is a “does the tree make a sound” argument, and I’m on the, “no it doesn’t” side, due to using a definition of “sound” that means “the representation of audio waves within a human nervous system”. You are on the “of course it makes a sound” side, because your definition of sound is “pressure waves in the air.”
I’ve been trying to be on the “it depends on your definition and my definition sits within the realm of acceptable definitions” side. Unfortunately, whether this is what you intend or not, most of your comments come across as though you’re on the “it depends on the definition, and my (PJ’s) defintion is right and yours is wrong” side, which is what seems to be getting people’s backs up.
There’s two things to say in response to this: first, I can define “liking and respecting me” as “experiencing analogous brain states to mine when I like and respect someone else”. That’s in the territory (modulo some assumptions about the cognitive unity of humankind): I could verify it in principle, although not in practice.
The second thing is that even if we grant that the example was poor, the point was still valid. For example, one might prefer that one’s spouse never cheat to one’s spouse cheating but never being aware of that fact. (ETA: but maybe you weren’t arguing against the point, only the example.)
There’s two things to say in response to this: first, I can define “liking and respecting me” as “experiencing analogous brain states to mine when I like and respect someone else”. That’s in the territory (modulo some assumptions about the cognitive unity of humankind): I could verify it in principle, although not in practice.
But what if they experience that state, and still, say, beat you up and treat you like jerks, because that’s what their map says you should do when you feel that way?
This isn’t about the example being poor, it’s about people thinking things in the map actually exist in the territory. Everything you perceive is mediated by your maps, even if only in the minimal sense of being reduced to human sensory-pattern recognition symbols first, let alone all the judgments about the symbols that we add on top.
For example, one might prefer that one’s spouse never cheat to one’s spouse cheating but never being aware of that fact. (ETA: but maybe you weren’t arguing against the point, only the example.)
How about the case where you absolutely believe the spouse is cheating, but they really aren’t?
This is certainly a better example, in that it’s easier to show that it’s not reality that you value, but the part of your map that you label “reality”. If you really truly believe the spouse is cheating, then you will feel exactly the same as if they really are.
IOW, when you say that you value something “in the territory”, all you are ever really talking about is the part of your map that you label “the territory”, whether that portion of the map actually corresponds to the territory or not.
This is not some sort of hypothetical word-argument, btw. (I have no use for them, which is why I mostly avoid the Omega discussions.) This is a practical point for minimizing one’s suffering and unwanted automatic responses to events in the world. To the extent you believe that your map is the territory, you will suffer when it is out-of-sync.
But what if they experience that state, and still, say, beat you up and treat you like jerks, because that’s what their map says you should do when you feel that way?
It’s still possible to prefer this state of affairs to one where they are beating me because they are contemptuous of me. Remember, we’re talking about a function from some set X to the real numbers, and we’re trying to figure out what sorts of things are members of X. In general, people do have preferences about the way things actually are.
How about the case where you absolutely believe the spouse is cheating, but they really aren’t?… If you really truly believe the spouse is cheating, then you will feel exactly the same as if they really are.
But my spouse won’t, and I have preferences about that fact. All other things being equal, my preference ordering is “my spouse never cheats and I believe my spouse never cheats” > “my spouse cheats and I find out” > “my spouse cheats and I believe my spouse never cheats” > “my spouse never cheats but I believe she does”. If a utility function exists that captures this preference, it will be a function that takes both reality and my map as arguments.
It’s still possible to prefer this state of affairs to one where they are beating me because they are contemptuous of me. Remember, we’re talking about a function from some set X to the real numbers, and we’re trying to figure out what sorts of things are members of X. In general, people do have preferences about the way things actually are.
Right, which is where this veers off into “hypothetical word-arguments” for me, because the entire point I’m making is that all your preferences are still about the map, no matter how many times you point to a region of the map marked, “the way things actually are”, and distinguish it from another part of the map labeled, “my map”.
You’ve read “Godel, Escher, Bach”, right? This is not a pipe. The hand that’s doing the drawing is in the drawing, no matter how realistically drawn it is. ;-)
Can I get your analysis of my “spouse cheating and what I know about it” example? I’ve understood your position on utility functions as stated in other branches; I’m curious as to how you would interpret my claim to have preferences over both states of reality and beliefs about reality in this fairly concrete and non-hypothetical case.
Can I get your analysis of my “spouse cheating and what I know about it” example? I’ve understood your position on utility functions as stated in other branches; I’m curious as to how you would interpret my claim to have preferences over both states of reality and beliefs about reality in this fairly concrete and non-hypothetical case.
My point is that your entire argument consists of pointing to the map and claiming it’s the territory. In the cases where reality and your belief conflict, you won’t know that’s the case. Your behavior will be exactly the same, either way, so the distinction is moot.
When you are trying to imagine, “my spouse is cheating and I think she isn’t”, you aren’t imagining that situation… you are actually imagining yourself perceiving that to be the case. That is, your map contains the idea of being deceived, and that this is an example of being deceived, and it is therefore bad.
None of that had anything to do with the reality over which you claim to be expressing a preference, because if it were the reality, you would not know you were being deceived.
This is just one neat little example of systemic bias in the systems we use to represent and reflect on preferences. They are designed to react to perceived circumstances, rather than to produce consistent reasoning about how things ought to be. So if you ever imagine that they are “about” reality, outside the relatively-narrow range of the here-and-now moment, you are on the path to error.
And just as errors accumulate in Newtonian physics as you approach the speed of light, so too do reasoning errors tend to accumulate as you turn your reasoning towards (abstract) self-reflection.
When you are trying to imagine, “my spouse is cheating and I think she isn’t”, you aren’t imagining that situation… you are actually imagining yourself perceiving that to be the case. That is, your map contains the idea of being deceived, and that this is an example of being deceived, and it is therefore bad.
Sure.
None of that had anything to do with the reality over which you claim to be expressing a preference, because if it were the reality, you would not know you were being deceived.
No. My imagination encompasses the fact that if it were the reality, I would not know I was being deceived. I know what my emotional state would be—it would be the same as it is now. That’s easy to get.
What it really comes down to is that saying that my preference ordering is “my spouse cheats and I find out” > “my spouse cheats and I believe my spouse never cheats” is equivalent to saying that if I am not sure (say 50% probability), I will seek out more information. I’m not sure how such a decision could be justified without considering that my preferences are over both the map and the territory.
ETA: Reading over what you’ve written in other branches, I’d like to point out a preference for not being deceived even if you will never realize it isn’t an error—it’s prima facie evidence that human preferences are over the territory as well as the map. That may not be the most useful way of thinking about it from a mindhacking perspective, but I don’t think it’s actually wrong.
I’d like to point out a preference for not being deceived even if you will never realize it isn’t an error—it’s prima facie evidence that human preferences are over the territory as well as the map.
That preference is not universal, which to me makes it absolutely part of the map. And it’s not just the fictional evidence of Cypher wanting to go back in the Matrix and forget, guys routinely pay women for various forms of fantasy fulfillment, willingly suspending disbelief in order to be deceived.
Not enough? How about the experimental philosophers who re-ran the virtual world thought experiment until they found that people’s decision about living in a fantasy world that they’d think was real, was heavily dependent upon whether they 1) had already been living in the fantasy, 2) whether their experience of life would significantly change, and 3) whether their friends and loved ones were also in the fantasy world.
If anything, those stats should be quite convincing that it’s philosophers and extreme rationalists who have a pathological fear of deception, rather than a inbuilt human preference for actually knowing the truth… and that most likely, if we have an inbuilt preference against deception, it’s probably aimed at obtaining social consensus rather than finding truth.
All that having been said, I will concede that there perhaps you could find some irreducible microkernel of “map” that actually corresponds to “territory”. I just don’t think it makes sense (on the understanding-people side) to worry about it. If you’re trying to understand what people want or how they’ll behave, the territory is absolutely the LAST place you should be looking. (Since the distinctions they’re using, and the meanings they attach to those distinctions, are 100% in the map.)
That preference is not universal, which to me makes it absolutely part of the map.
I don’t see how it supposed to follow from the fact that not everyone prefers not-being-decieved, that those who claim to prefer not-being-deceived must be wrong about their own preferences. Could you explain why you seem to think it does?
The claim others are defending here (as I understand it) is not that everyone’s preferences are really over the territory; merely that some people’s are. Pointing out that some people’s preferences aren’t about the territory isn’t a counterargument to that claim.
I don’t see how it supposed to follow from the fact that not everyone prefers not-being-decieved, that those who claim to prefer not-being-deceived must be wrong about their own preferences.
I’m saying that the preferences point to the map because your entire experience of reality is in the map—you can’t experience reality directly. The comments about people’s differences in not-being-deceived were just making the point that that preference is more about consensus reality than reality reaity. In truth, we all care about our model of reality, which we labeled reality and think is reality, but is actually not.
The comments about people’s differences in not-being-deceived were just making the point that that preference is more about consensus reality than reality reaity [sic].
I’m afraid I have no idea what this is supposed to mean. It seems to me like you’re just repeating your conclusion over and over again using different words, which unfortunately doesn’t constitute an argument. Maybe to you it seems like we’re doing the same thing, I don’t know.
Alternatively, maybe we’re still talking past each other for the reasons suggested here (which everyone seemed to agree with at the time.) In which case, I wonder why we’re still having this conversation at all, and apologise for my part in pointlessly extending it. ;)
Alternatively, maybe we’re still talking past each other for the reasons suggested here (which everyone seemed to agree with at the time.) In which case, I wonder why we’re still having this conversation at all, and apologise for my part in pointlessly extending it.
It’s probably because I replied to an unclosed subthread, causing an unintended resurrection. Also, at one point Vladimir Nesov did some resurrection too, and there have also been comments by Cyan and Saturn that kept things going.
Anyway, yes, as you said, we already agreed we are talking about different things, so let’s stop now. ;-)
If you agree that you are just talking about a different thing, and given that “utility” is a term understood to mean different thing from what you were talking about, kindly stop using that term for your separate concept to avoid unnecessary confusion and stop arguing about the sound of fallen tree.
1) Why would people differ so much? Even concrete preferences don’t get reversed, magical mutants don’t exist.
2) Even if you only care about your map, you still care about your map as a part of the territory, otherwise you make the next step and declare that you don’t care about state of your brain either, you only care about caring itself, at which point you disappear in a “puff!” of metaphysical confusion. It’s pretty much inevitable.
Even if you only care about your map, you still care about your map as a part of the territory,
...a “territory” that exists only in your brain, since you cannot directly perceive or operate upon the real territory.
otherwise you make the next step and declare that you don’t care about state of your brain either, you only care about caring itself, at which point you disappear in a “puff!” of metaphysical confusion.
...and note too, that all of this modeling is taking place in a brain. When you point to something and label it “reality”, you are pointing to a portion of your map. It is not, and cannot be the actual territory. It doesn’t matter how many times you try to say, “but no, I mean real reality”, because the only thing you have to represent that idea with is your brain.
It is still the idea of reality, a model of reality, that exists in your map.
Yes, the map is represented physically within the territory. The map “runs on” a platform made of territory, just as “you” run on a platform made of map. All I am saying is, “you” cannot access the territory directly, because your platform is made of map, just as the map’s platform is made of territory. Everything we perceive or think or imagine is therefore map… including the portion of the map we refer to as the territory.
We want our preferences to point to the territory, but they cannot do so in actual reality. We have the illusion that they can, because we have the same representative freedom as an artist drawing a picture of hands drawing each other—we can easily represent paradoxical and unreal things within the surface of the map.
(I remember reading at one point a tutorial in General Semantics that illustrated this point much better than I am doing, but sadly I cannot seem to find it at the moment.)
We want our preferences to point to the territory, but they cannot do so in actual reality.
There’s a distinction to be made between the fact that our knowledge about whether our preferences are satisfied is map-bound and the assertion that our preferences only take the map into account.
If you are operating under an assumption that nobody agrees with, you are wasting anyone’s time (assumption: map is about the map, territory be damned), as the argument never goes anywhere. As a compromise, compose your best argument as a top-level post (but only if you expect to convince at least someone).
That’s why I don’t understand you—I dropped this particular subthread for that very reason, but Cyan asked a second time for a reply. Otherwise, I’d have not said anything else in this particular subthread.
You could still write a meta-reply, taking that problem into account. The root of the disagreement can be stated in one line, and a succinct statement of at the moment unresolvable disagreement is resolution of an argument.
You could still write a meta-reply, taking that problem into account. The root of the disagreement can be stated in one line, and a succinct statement of at the moment unresolvable disagreement is resolution of an argument.
To be honest, I have not perceived anyone I’m speaking with to be treating this as an unresolvable conflict, besides you. Most of the other people have been saying things I perceive to mean, “I agree with such-and-such, but I am confused why you think that about this other case”, or “Ah, we are talking about different things in this area—how would you apply it to the thing I mean?”
You are the only one who appears to be simply stating dogma back at me, without seeking to understand where our maps do or do not overlap. (I’m not very quick on the OB URL citations, but ISTM that everything I’m saying about maps is consistent with EY’s models of reductionism, and with observable facts about how brains operate.)
You appear to have a pattern of responding to my descriptions of things (as I perceive them in my map, of course) as if they were attacks on your preferred prescriptions for how reality should be (in your map). It’s natural that this would lead to an impasse, since I am not actually disputing your opinions, and you are not actually objecting to my facts. Hence, we talk past each other.
(Btw, when I say “facts”, I mean “statements intended to be about actual conditions”, not “truths”. All models are false, but some are more useful than others.)
I’m just trying to be decisive in identifying the potential flaming patterns in the discussion. I could debate the specifics, but given my prior experience in debating stuff with you, and given the topics that could be debated in these last instances, I predict that the discussion won’t lead anywhere, and so I skip the debate and simply state my position, to avoid unnecessary text.
One way of stopping recurring thematic or person-driven flame wars (that kill Internet communities) is to require the sides to implement decent write-ups of their positions: even without reaching agreement, at some point there remains nothing to be said, and so the endless cycle of active mutual misunderstanding gets successfully broken.
I’m just trying to be decisive in identifying the potential flaming patterns in the discussion. I could debate the specifics, but given my prior experience in debating stuff with you, and given the topics that could be debated in these last instances, I predict that the discussion won’t lead anywhere, and so I skip the debate and simply state my position, to avoid unnecessary text.
I don’t understand how that’s supposed to work. If you don’t expect it to lead anywhere, why bother saying anything at all?
I’m registering the disagreement, and inviting you to sort the issue out for yourself, through reconsidering your position in response to apparent disagreement, or through engaging into a more constructive form of discussion.
I’m registering the disagreement, and inviting you to sort the issue out for yourself, through reconsidering your position in response to apparent disagreement, or through engaging into a more constructive form of discussion.
This appears to be a one-way street. If applied consistently, it would seem that your first step would be to reconsider your position in response to apparent disagreement… or that I should reply by registering my disagreement—which implicitly I’d have already done.
Or, better yet, you would begin (as other people usually do) by starting the “more constructive form of discussion”, i.e., raising specific objections or asking specific questions to determine where the differences in our maps lie.
Don’t take it lightly, it’s a well-vetted and well-understood position, extensively discussed and agreed upon. You should take such claims as strong evidence that you may have missed something crucial, that you need to go back and reread the standard texts.
Don’t take it lightly, it’s a well-vetted and well-understood position, extensively discussed and agreed upon.
It’s extensively discussed and agreed upon, that that is how we (for certain definitions of “we”) would like it to be, and it certainly has desirable properties for say, building Friendly AI, or any AI that doesn’t wirehead. And it is certainly a property of the human brain that it orients its preferences towards what it believes is the outside world—again, it has good consequences for preventing wireheading.
But that doesn’t make it actually true, just useful.
It’s also pretty well established as a tenet of e.g., General Semantics, that the “outside world” is unknowable, since all we can ever consciously perceive is our map. The whole point of discussing biases is that our maps are systematically biased—and this includes our preferences, which are being applied to our biased views of the world, rather than the actual world.
I am being descriptive here, not prescriptive. When we say we prefer a certain set of things to actually be true, we can only mean that we want the world to not dispute a certain map, because otherwise we are making the supernaturalist error of assuming that a thing could be true independent of the components that make it so.
To put it another way, if I say, “I prefer that the wings of this plane not fall off”, I am speaking about the map, since “wings” do not exist in the territory.
IOW, our statements about reality are about the intersection of some portion of “observable” reality and our particular mapping (division and labeling) of it. And it cannot be otherwise, since to even talk about it, we have to carve up and label the “reality” we are discussing.
IOW, our statements about reality are about the intersection of some portion of “observable” reality and our particular mapping (division and labeling) of it. And it cannot be otherwise, since to even talk about it, we have to carve up and label the “reality” we are discussing.
It’s funny that you talk of wordplay a few comments back, as it seems that you’re the one making a technically-correct-but-not-practically-meaningful argument here.
If I may attempt to explore your position: Suppose someone claims a preference for “blue skies”. The wirehead version of this that you endorse is “I prefer experiences that include the perception I label ‘blue sky’”. The “anti-wirehead” version you seem to be arguing against is “I prefer actual states of the world where the sky is actually blue”.
You seem to be saying that since the preference is really about the experience of blue skies, it makes no sense to talk about the sky actually being blue. Chasing after external definitions involving photons and atmospheric scattering is beside the point, because the actual preference wasn’t formed in terms of them.
This becomes another example of the general rule that it’s impossible to form preferences directly about reality, because “reality” is just another label on our subjective map.
As far as specifics go, I think the point you make is sound: Most (all?) of our preferences can’t just be about the territory, because they’re phrased in terms of things that themselves don’t exist in the territory, but at best simply point at the slice of experience labeled “the territory”.
That said, I think this perspective grossly downplays the practical importance of that label. It has very distinct subjective features connecting in special ways to other important concepts. For the non-solipsists among us, perhaps the most important role it plays is establishing a connection between our subjective reality and someone else’s. We have reason to believe that it mediates experiences we label as “physical interactions” in a manner causally unaffected by our state of mind alone.
When I say “I prefer the galaxy not to be tiled by paperclips”, I understand that, technically, the only building blocks I have for that preference are labeled experiences and concepts that aren’t themselves the “stuff” of their referents. In fact, I freely admit that I’m not exactly sure what constitutes “the galaxy”, but the preference I just expressed actually contains a massive number of implicit references to other concepts that I consider causally connected to it via my “external reality” label. What’s more, most people I communicate with can easily access a seemingly similar set of connections to their “external reality” label, assuming they don’t talk themselves out of it.
The territory concept plays a similar role to that of an opaque reference in a programming language. Its state may not be invariant, but its identity is. I don’t have to know any true facts concerning its actual structure for it to be meaningful and useful. Just as photons aren’t explicitly required to subjectively perceive a blue sky, the ontological status of my territory concept doesn’t really change its meaning or importance, which is acquired through its intimate connection to massive amounts of raw experience.
Claiming my preferences about the territory are really just about my map is true in the narrow technical sense that it’s impossible for me to refer directly to “reality”, but doing so completely glosses over the deep, implicit connections expressed by such preferences, most primarily the connection between myself and the things I label “other consciousnesses”. In contrast, the perception of these connections seems to come for free by “confusing” the invariant identity of my territory concept with the invariant “existence” of a real external world. The two notions are basically isomorphic, so where’s the value in the distinction?
Think of what difference is there between “referring directly” to the outside reality and “referring directly” to the brain. Not much, methinks. There is no homunculus whose hands are only so long to reach the brain, but not long enough to touch your nose.
You seem to be saying that since the preference is really about the experience of blue skies, it makes no sense to talk about the sky actually being blue.
That depends on whether you’re talking about “blue” in terms of human experience, or whether you’re talking about wavelengths of light. The former is clearly “map”, whereas discussing wavelengths of light at least might be considered “about” the territory in some sense.
However, if you are talking about preferences, I don’t think there’s any way for a preference to escape the map. We can define how the mapped preference relates to some mapped portion of the territory, but preferences (on human hardware at least) can only be “about” experience.
And that’s because all of our hypothetical preferences for states of the actual world are being modeled as experiences in order to compute a preference.
The entire context of this discussion (for me, anyway) has been about preferences. That doesn’t preclude the possibility of relating maps to territory, or maps to other people’s maps. I’m just saying that human beings have no way to model their preferences except by modeling experiences, in human-sensory terms.
The “anti-wirehead” version you seem to be arguing against is “I prefer actual states of the world where the sky is actually blue”.
But when this statement is uttered by a human, it is almost invariably a lie. Because inside, the person’s model is an experience of “blue sky”. The preference is about experiences, not the state of the world.
Even if you phrase this as, “I prefer the sky to be actually blue, even if I don’t know it”, it is still a lie, because now you are modeling an experience of the sky being blue, plus an experience of you not knowing it.
The two notions are basically isomorphic, so where’s the value in the distinction?
Well, it makes clear some of the limits of certain endeavors that are often discussed here. It dissolves confusions about the best ways to make people happy, and whether a world should be considered “real” or “virtual”, and whether it’s somehow “bad” to be virtual.
But the most important practical benefit is that it helps in understanding why sometimes the best thing for a person may be to update their preferences to match the constraints of reality, rather than trying in vain to make reality fit their preferences.
Consider the “blue sky” preference. The experience of a blue sky is likely associated with, and significantly colored by things like the person’s favorite color, the warmth of the sun or the cool breeze of that perfect day one summer when they were in love. For another person, it might be associated with the blinding heat of the desert and a sensation of thirst… and these two people can then end up arguing endlessly about whether a blue sky is obviously good or bad.
And both are utterly deluded to think this their preferences have anything to do with reality.
I am not saying that they disagree about how many angstroms the light of a blue sky is, or that the sky doesn’t really exist or anything like that. I’m saying their preference is (and can only be) about their maps, because the mere fact of a blue sky has no inherent “preferability”, without reference to some other purpose.
Even if we try to say in the abstract that it is good because it’s correlated with things about our planet that make life possible, we can only have that preference because we’re modeling an experience of “life” that we label “good”. (And perhaps a depressed teenager would disagree, using a differently-labeled experience of “life”!)
This does not mean preferences are invalid or meaningless. It doesn’t mean that we should only change our preferences and ignore reality. However, to the extent that our preferences produce negative experiences, it is saner to remove the negative portion of the preference.
Luckily, human beings are not limited to, or required to have, bidirectional preferences. Feeling pain at the absence of something is not required in order to experience pleasure at its presence, in other words. (Or vice versa.)
Awareness of this fact, combined with an awareness that it is really the experience we prefer (and mainly, the somatic markers we have attached to the experience) makes it plain that the logical thing to do is to remove the negative label, and leave any positive labels in place.
However, if we think that our internal labeling of experience has something to do with “reality”, then we are likely to engage in the confusion of thinking that removing a negative label of badness will somehow create or prolong badness in the territory.
And for that matter, we may be under the mistaken impression that changing the reality out there will change our experience of it… and that is often not the case. As the saying goes, if you want to make a human suffer, you can either not give them what they want, or else you can give it to them! Humans tend to create subgoals that get promoted to “what I want” status, without reference to what the original desired experience was.
For example, looking for blue skies...
When what you really want is to fall in love again.
Wow, −5! People here don’t seem to appreciate this sort of challenge to their conceptual framework.
I’m just saying that human beings have no way to model their preferences except by modeling experiences, in human-sensory terms.
I agree, but I wonder if I failed to communicate the distinction I was attempting to make. The human-sensory experience of being embedded in a concrete, indifferent reality is (drugs, fantasies, and dreams aside) basically constant. It’s a fundamental thread underlying our entire history of experience.
It’s this indifference to our mental state that makes it special. A preference expressed in terms of “reality” has subjective properties that it would otherwise lack. Maybe I want the sky to be blue so that other people will possess a similar experience of it that we can share. “Blueness” may still be a red herring, but my preference now demands some kind of invariant between minds that seemingly cannot be mediated except through a shared external reality. You might argue that I really just prefer shared experiences, but this ignores the implied consistency between such experiences and all other experiences involving the external reality, something I claim to value above and beyond any particular experience.
Even if you phrase this as, “I prefer the sky to be actually blue, even if I don’t know it”, it is still a lie, because now you are modeling an experience of the sky being blue, plus an experience of you not knowing it.
This is where the massive implicit context enters the scene. “Even if I don’t know it” is modeled after experience only in the degenerate sense that it’s modeled after experience of indifferent causality. A translation might look like “I prefer to experience a reality with the sorts of consequences I would predict from the sky being blue, even if I don’t consciously perceive blue skies”. That’s still an oversimplification, but it’s definitely more complex than just invoking a generic memory of “not having known something” and applying it to blue skies.
The two notions are basically isomorphic, so where’s the value in the distinction?
Well, it makes clear some of the limits of certain endeavors that are often discussed here. It dissolves confusions about the best ways to make people happy, and whether a world should be considered “real” or “virtual”, and whether it’s somehow “bad” to be virtual.
I don’t see how any of that is true. I can easily think of different concrete realizations of “real” and “virtual” that would interact differently with my experience of reality, thus provoking different labellings of “good” and “bad”. If your point is merely that “real” is technically underspecified, then I agree. But I don’t see how you can draw inferences from this underspecification.
For another person, it might be associated with the blinding heat of the desert and a sensation of thirst… and these two people can then end up arguing endlessly about whether a blue sky is obviously good or bad.
And both are utterly deluded to think this their preferences have anything to do with reality.
I’m going to have to turn your own argument against you here. To the extent that you have a concept of reality that is remotely consistent with your everyday experience, I claim that “in reality, blue skies are bad because they provoke suffering” is a preference stated in terms of an extremely similar reality-concept, plus a suffering-concept blended together from first-hand experience and compassion (itself also formed in terms of reality-as-connected-to-other-minds). For you to say it has “nothing to do with reality” is pure semantic hogwash. What definition of “reality” can you possibly be using to make this statement, except the one formed by your lifetime’s-worth of experience with indifferent causality? You seem to be denying the use of the term to relate your concept of reality to mine, despite their apparent similarity.
However, to the extent that our preferences produce negative experiences, it is saner to remove the negative portion of the preference.
This doesn’t make sense to me. Whether or not an experience is “negative” is a function of our preferences. If a preference “produces” negative experiences, then either they’re still better than the alternative (in which case it’s a reasonable preference, and it’s probably worthwhile to change your perception of the experience) or they’re not (in which case it’s not a true preference, just delusion).
Luckily, human beings are not limited to, or required to have, bidirectional preferences. Feeling pain at the absence of something is not required in order to experience pleasure at its presence, in other words. (Or vice versa.)
That’s a property of pain and pleasure, not preference. I may well decide not to feel pain due to preference X being thwarted, but I still prefer X, and I still prefer pleasure to the absence of pleasure.
Awareness of this fact, combined with an awareness that it is really the experience we prefer (and mainly, the somatic markers we have attached to the experience) makes it plain that the logical thing to do is to remove the negative label, and leave any positive labels in place.
This is where I think your oversimplification of “experience vs reality” produces invalid conclusions. Those labels don’t just apply to one experience or another, they apply to a massively complicated network of experience that I can’t even begin to hold in my mind at once. Given that, your logic doesn’t follow at all, because I really don’t know what I’m relabeling.
This relates to a general reservation I have with cavalier attitudes toward mind-hacks: I know full well that my preferences are complex, difficult to understand, and grossly underspecified in any conscious realization, so it’s not at all obvious to me that optimizing a simple preference concerning one particular scenario doesn’t carry loads of unintended consequences for the rest of them. I’ve had direct experience with my subconsciously directed behavior “making decisions for me” that I had conscious reasons to optimize against, only later to find out that my conscious understanding of the situation was flawed and incomplete. I think that ignoring the intuitive implications of an external reality leads to similar contradictions.
You seem to mostly be arguing against a strawman; as I said, I’m not saying reality doesn’t exist or that it’s not relevant to our experiences. What I’m saying is that the preferences are composed of map, and while there are connections between that map and external reality, we are essentially deluded to think our preferences refer to actual reality, and that this delusion leads us to believing that changing external reality will change our internal experience, when more often the reverse is more likely true. (That is, changing our internal experience will more likely result in our taking actions that will actually change external reality.)
Note, however that:
Whether or not an experience is “negative” is a function of our preferences.
Here you seem to be arguing my point. The experience is a function of preferences, the preferences are a product of, and point to, other experiences, in a self-sustaining loop that sometimes might as well not be connected to outside reality at all, for all that it has anything to do with what’s actually going on.
Lucky people live in a perpetual perception of good things happening, unlucky people the opposite, even when the same events are happening to both.
How can we say, then, that either person’s perceptions are “about” reality, if they are essentially unconditional? Clearly, something else is going on.
If we disagree at this point, I’d have to say it can only be because we disagree on what “about” means. When I say preferences are not “about” reality, it is in the same sense that Robin Hanson is always saying that politics is not “about” policy, etc.
Clearly, preferences are “about” reality in the same sense that politics are “about” policy. That is, reality is the subject of a preference, in the same way that a policy might be the subject of a political dispute.
However, in both cases, the point of the ostensible activity is not where it appears to be. In order for politics to function, people must sincerely believe that it is “about” policy, in precisely the same way as we must sincerely believe our preferences are “about” reality, in order to make them function—and for similar reasons.
But in neither case does either the sincerity or the necessity of the delusion change the fact that it’s nonetheless a delusion.
If we disagree at this point, I’d have to say it can only be because we disagree on what “about” means.
I don’t think I disagree with any of the above, except to dispute its full universality (which I’m not sure you’re even arguing). To attempt to rephrase your point: Our interactions with reality create experiences filtered through our particular way of characterizing such interactions. It’s these necessarily subjective characterizations (among other things), rather than the substance of the interaction itself, which generate our preferences. When reflecting on our preferences, we’re likely to look right past the interpretive layer we’ve introduced and attribute them to the external stimulus that produced the response, rather than the response itself.
Robin’s “X is not about Y” has the flavor of general, but not universal, rules. Would you extend your analogy to include this property?
Robin’s “X is not about Y” has the flavor of general, but not universal, rules. Would you extend your analogy to include this property?
Here’s an interesting question for you: why is it important that you consider this non-universal? What value does it provide you for me to concede an exception, or what difference will it make in your thinking if I say “yes” or “no”? I am most curious.
(Meanwhile, I agree with your summation as an accurate, if incomplete restatement of the bulk of my point.)
Because I’m trying to make sense of your position, but I don’t think I can with such a strict conclusion. I don’t see any fundamental reason why someone couldn’t form preferences more or less directly mediated by reality, it just seems that in practice, we don’t.
If you’re asking why I’m bringing up universality, it seemed clear that your claims about preferences were universal in scope until you brought up “X is not about Y”. “Must logically be” and “tends to be in practice” are pretty different types of statement.
I mean, you said some things that sound like answers, but they’re not answers to the questions I asked. Here they are again:
Why is it important that you consider this non-universal?
and
What value does it provide you for me to concede an exception, or what difference will it make in your thinking if I say “yes” or “no”?
Your almost-answer was that you don’t think you can “make sense” of my position with a strict conclusion. Why is that? What would it mean for there to be a strict conclusion? How, specifically, would that be a problem?
Why is it important that you consider this non-universal?
I didn’t answer this because it’s predicated on an assumption that has no origin in the conversation. I never claimed that it was “important” for me to consider this non-universal. As per being “liberal in what I accept” in the realm of communication, I tried to answer the nearest meaningful question I thought you might actually be asking. I thought the phrase “If you’re asking why I’m bringing up universality” made my confusion sufficiently clear.
If you really do mean to ask me why I think it’s important that I believe in some property of preference formation, then either I’ve said something fairly obvious to that end that I’m not remembering (or finding), or you’re asserting your own inferences as the basis of a question, instead of its substance. I try to give people the benefit of the doubt that I’ve misunderstood them in such cases, rather than just assume they’re speaking manipulatively.
What value does it provide you for me to concede an exception
No particular value in mind. I suppose the greatest value would be in you solidly refuting such exceptions in a way that made sense to me, as that would be a more surprising (therefore more informative) outcome. If you the concede the exception, I don’t gain any additional insight, so that’s of fairly neutral value.
what difference will it make in your thinking if I say “yes” or “no”?
Not really sure yet, especially in the “no” case (since in that case you may have reasons I haven’t yet thought of or understood). I suppose in the “yes” case I’d have greater confidence that I knew what you were talking about if I encountered similar concepts in your comments elsewhere. This discussion has had some difference on my thinking: I don’t think I understood the thrust of your point when I originally complained that your distinction lacked relevance.
Your almost-answer was that you don’t think you can “make sense” of my position with a strict conclusion. Why is that? What would it mean for there to be a strict conclusion?
By strict conclusion, I mean “preferences are modeled strictly in terms of the map: it is logically impossible to a hold preference expressed in terms of something other than that which is expressed in the map”. This seems very nearly true, but vulnerable to counterexamples when taken as a general principle or logical result of some other general principle. I’ll elaborate if you’d like, but I thought I’d clarify that you meant it that way. If you didn’t, theoretical or speculative counter-examples aren’t particularly relevant.
By strict conclusion, I mean “preferences are modeled strictly in terms of the map: it is logically impossible to a hold preference expressed in terms of something other than that which is expressed in the map”. This seems very nearly true, but vulnerable to counterexamples when taken as a general principle or logical result of some other general principle. I’ll elaborate if you’d like, but I thought I’d clarify that you meant it that way. If you didn’t, theoretical or speculative counter-examples aren’t particularly relevant.
I can imagine that, in principle, some other sort of mind than a human’s might be capable of being a counterexample, apart from, say, the trivial example of a thermostat, which shows a “preference” for reality being a certain way. An AI could presumably be built so that its preferences were based on properties of the world, rather than properties of its experience, or deduction from other properties based on experience. However, at some point that would need to be rooted in the goal system provided by its programmers… who presumably based it off of their own preferences.… ;-) (Nonetheless, if the AI didn’t have anything we’d label “experience”, then I’d have to agree that it has a preference about reality, rather than its experience of reality.)
I could also consider an argument that, say, hunger is about the state of one’s stomach, and that it therefore is “about” the territory, except that I’m not sure hunger qualifies as a preference, rather than an appetite or a drive. A person on a hunger strike or with anorexia still experiences hunger, yet prefers not to eat.
If you think you have other counterexamples, I’d like to hear them. I will be very surprised if they don’t involve some rather tortured reasoning and hypotheticals, though, or non-human minds. The only reason I even hedge my bets regarding humans is that (contrary to popular belief) I’m not under the mistaken impression that I have anything remotely approaching a complete theory of mind for human brains, versus a few crude maps that just happen to cover certain important chunks of “territory”. ;-)
I can imagine that, in principle, some other sort of mind than a human’s might be capable of being a counterexample, apart from, say, the trivial example of a thermostat, which shows a “preference” for reality being a certain way.
I don’t actually consider this a good counterexample. It can been trivially shown that the thermostat’s “preference” is not in terms of the “reality” of temperature: Just sabotage the sensor. The thermostat “prefers” its sensor reading to correspond to its set point. Wouldn’t you agree this is fairly analogous to plenty of human desires?
I could also consider an argument that, say, hunger is about the state of one’s stomach, and that it therefore is “about” the territory, except that I’m not sure hunger qualifies as a preference, rather than an appetite or a drive.
Agreed. The closest it seems you could come is to prefer satiation of said appetites, which is a subjective state.
If you think you have other counterexamples, I’d like to hear them. I will be very surprised if they don’t involve some rather tortured reasoning and hypotheticals, though, or non-human minds.
Actually, human minds are the primary source of my reservations. I don’t think my reasoning is particularly tortured, but it certainly seems incomplete. Like you, I really have no idea what a mind is.
That said, I do seem to have preferences that concern other minds. These don’t seem reducible to experiences of inter-personal behavior… they seem largely rooted in the empathic impulse, the “mirror neurons”. Of course, on its face, this is still just built from subjective experience, right? It’s the the experience of sympathetic response when modeling another mind. And there’s no question that this involves substituting my own experiences for theirs as part of the modeling process.
But when I reflect on a simple inter-personal preference like “I’d love for my friend to experience this”, I can’t see how it really reduces to pure experience, except as mediated by my concept of invariant reality. I don’t have a full anticipation of their reaction, and it doesn’t seem to be my experience of modeling their interaction that I’m after either.
Feel free to come up with a better explanation, but I find it difficult to deconstruct my desire to reproduce internally significant experiences in an external environment in a way that dismisses the role of “hard” reality. I can guess at the pre-reflective biological origin of this sort of preference, just like we can point at the biological origin of domesticated turkeys, but, just as turkeys can’t function without humans, I don’t know how it would function without some reasonable concept of a reality that implements things intrinsically inaccessible and indifferent to my own experience.
I chose to instantiate this particular example, but the general rule seems to be: The very fabric of what “another mind” means to me involves the concept of an objective but shared reality. The very fabric of what “another’s experiences” means to me involves the notion of an external system giving rise to external subjective experiences that bear some relation to my own.
You could claim my reasoning is tortured in that it resembles Russel’s paradox: One could talk about the set of all subjective preferences explicitly involving objective phenomena (i.e., not containing themselves). But it seems to me that I can in a sense relate to a very restricted class of objective preferences, those constructed from the vocabulary of my experience, reflected back into the world, and reinstantiated in the form of another mind.
Another simple example: Do you think a preference for honest communication is at all plausible? Doesn’t it involve something beyond “I hope the environment doesn’t trick me”?
That said, I do seem to have preferences that concern other minds. These don’t seem reducible to experiences of inter-personal behavior… they seem largely rooted in the empathic impulse, the “mirror neurons”. Of course, on its face, this is still just built from subjective experience, right? It’s the the experience of sympathetic response when modeling another mind. And there’s no question that this involves substituting my own experiences for theirs as part of the modeling process.
Right. And don’t forget the mind-projection machinery, that causes us to have, e.g. different inbuilt intuitions about things that are passively moved, move by themselves, or have faces that appear to express emotion. These are all inbuilt maps in humans.
But when I reflect on a simple inter-personal preference like “I’d love for my friend to experience this”, I can’t see how it really reduces to pure experience, except as mediated by my concept of invariant reality. I don’t have a full anticipation of their reaction, and it doesn’t seem to be my experience of modeling their interaction that I’m after either.
Most of us learn by experience that sharing positive experiences with others results in positive attention. That’s all that would be needed, but it’s also likely that humans have an evolved appetite to communicate and share positive experiences with their allies.
Another simple example: Do you think a preference for honest communication is at all plausible? Doesn’t it involve something beyond “I hope the environment doesn’t trick me”?
It just means you prefer one class of experiences to another, that you have come to associate with other experiences or actions coming before them, or co-incident with them.
The reason, btw, that I asked why it made a difference whether this is an absolute concept or a “mostly” concept, is that AFAICT, the idea that “some preferences are really about the territory” leads directly to “therefore, all of MY preferences are really about the territory”.
In contrast, thinking of all preferences being essentially delusional is a much better approach, especially if 99.999999999% of all human preferences are entirely about the map, if we presume that maybe there are some enlightened Zen masters or Beisutsukai out there who’ve successfully managed, against all odds, to win the epistemic lottery and have an actual “about the territory” preference.
Even if the probability of having such a preference were much higher, viewing it as still delusional with respect to “invariant reality” (as you call it) does not introduce any error. So the consequences of erring on the side of delusion are negligible, and there is a significant upside to being more able to notice when you’re looping, subgoal stomping, or just plain deluded.
That’s why it’s of little interest to me how many .9′s there are on the end of that %, or whether in fact it’s 100% - the difference is inconsequential for any practical purpose involving human beings. (Of course, if you’re doing FAI, you probably want to do some deeper thinking than this, since you want the AI to be just as deluded as humans are, in one sense, but not as deluded in another.)
The reason, btw, that I asked why it made a difference whether this is an absolute concept or a “mostly” concept, is that AFAICT, the idea that “some preferences are really about the territory” leads directly to “therefore, all of MY preferences are really about the territory”.
For the love of Bayes, NO. The people here are generally perfectly comfortable with the realization that much of their altruism, etc. is sincere signaling rather than actual altruism. (Same for me, before you ask.) So it’s not necessary to tell ourselves the falsehood that all of our preferences are only masked desires for certain states of mind.
As for your claim that the ratio of signaling to genuine preference is 1 minus epsilon, that’s a pretty strong claim, and it flies in the face of experience and certain well-supported causal models. For example, kin altruism is a widespread and powerful evolutionary adaptation; organisms with far less social signaling than humans are just hardwired to sacrifice at certain proportions for near relatives, because the genes that cause this flourish thereby. It is of course very useful for humans to signal even higher levels of care and devotion to our kin; but given two alleles such that
(X) makes a human want directly to help its kin to the right extent, plus a desire to signal to others and itself that it is a kin-helper, versus
(X’) makes a human only want to signal to others and itself that it is a kin-helper,
the first allele beats the second easily, because the second will cause searches for the cheapest ways to signal kin-helping, which ends up helping less than the optimal level for promoting those genes.
Thus we have a good deal of support for the hypothesis that our perceived preferences in some areas are a mix of signaling and genuine preferences, and not nearly 100% one or the other. Generally, those who make strong claims against such hypotheses should be expected to produce experimental evidence. Do you have any?
The people here are generally perfectly comfortable with the realization that much of their altruism, etc. is sincere signaling rather than actual altruism.
That’s nice, but not relevant, since I haven’t been talking about signaling.
Given that, I’m not going to go through the rest of your comment point by point, as it’s all about signaling and kin selection stuff that doesn’t in any way contest the idea that “preference is about experiences, not the reality being experienced”.
I don’t disagree with what you said, it’s just not in conflict with the main idea here. When I said that this is like Hanson’s “politics are not about policy”, I didn’t mean that it was therefore about signaling! (I said it was “not about” in the same way, not that it was about in the same way—i.e., that the mechanism of delusion was similar.)
The way human preferences work certainly supports signaling functions, and may be systematically biased by signaling drives, but that’s not the same thing as saying that preferences equal signaling, or that preferences are “about” signaling.
Well, this discussion might not be useful to either of us at this point, but I’ll give it one last go. My reason for bringing in talk of signaling is that throughout this conversation, it seems like one of the claims you have been making is that
The algorithm (more accurately, the collection of algorithms) that constitutes me makes its decisions based on a weighting of my current and extrapolated states of mind. To the extent that I perceive preferences about things that are distinct from my mental states (and especially when confronting thought-experiments in which my mental states will knowably diverge from the mental states I would ordinarily form given certain features of the world), I am deceiving myself.
Now, I brought up signaling because I and many others already accept a form of (A), in which we’ve evolved to deceive others and ourselves about our real priorities because such signalers appear to others to be better potential friends, lovers, etc. It looks perfectly meaningful to me to declare such preferences “illusory”, since in point of fact we find rationalizations for choosing not what we signaled we prefer, but rather the least costly available signs of these ‘preferences’.
However, kin altruism appears to be a clear case where not all action is signaling, where making decisions that are optimized to actually benefit my relatives confers an advantage in total fitness to my genes.
While my awareness and my decisions exist on separate tracks, my decisions seem to come out as they would for a certain preference relation, one of whose attributes is a concern for my relatives’ welfare. Less concern, of course, than I consciously think I have for them; but roughly the right amount of concern for Hamilton’s Rule of kin selection.
My understanding, then, is that I have both conscious and real preferences; the former are what I directly feel, but the latter determine parts of my action and are partially revealed by analysis of how I act. (One component of my real preferences is social, and even includes the preference to keep signaling my conscious preferences to myself and others when it doesn’t cost me too much; this at least gives my conscious preferences some role in my actions.) If my actions predictably come out in accordance with the choices of an actual preference relation, then the term “preference” has to be applied there if it’s applied anywhere.
There’s still the key functional sense in which my anticipation of future world-states (and not just my anticipation of future mind-states) enters into my real preferences; I feel an emotional response now about the possibility of my sister dying and me never knowing, because that is the form that evaluation of that imagined world takes. Furthermore, the reason I feel that emotional response in that situation is because it confers an advantage to have one’s real preferences more finely tuned to “model of the future world” than “model of the future mind”, because that leads to decisions that actually help when I need to help.
This is what I mean by having my real preferences sometimes care about the state of the future world (as modeled by my present mind) rather than just my future experience (ditto). Do you disagree on a functional level; and if so, in what situation do you predict a person would feel or act differently than I’d predict? If our disagreement is just about what sort of language is helpful or misleading when taking about the mind, then I’d be relieved.
The confusion that you have here is that kin altruism is only “about” your relatives from the outside of you. Within the map that you have, you have no such thing as “kin altruism”, any more than a thermostat’s map contains “temperature regulation”. You have features that execute to produce kin altruism, as a thermostat’s features produce temperature regulation. However, just as a thermostat simply tries to make its sensor match its setting, so too do your preferences simply try to keep your “sensors” within a desired range.
This is true regardless of the evolutionary, signaling, functional, or other assumed “purposes” of your preferences, because the reality in which those other concepts exist, is not contained within the system those preferences operate in. It is a self-applied mind projection fallacy to think otherwise, for reasons that have been done utterly to death in my interactions with Vladimir Nesov in this thread. If you follow that logic, you’ll see how preferences, aboutness, and “natural categories” can be completely reduced to illusions of the mind projection fallacy upon close examination.
Well, if this is just a disagreement over whether our typical uses of the word “about” are justified, then I’m satisfied with letting go of this thread; is that the case, or do you think there is a disagreement on our expectations for specific human thoughts and actions?
I suggest, by the way, that your novel backwards application of the Mind Projection Fallacy needs its own name so as not to get it confused with the usual one. (Eliezer’s MPF denotes the problem with exporting our mental/intentional concepts outside the sphere of human beings; you seem to be asserting that we imported the notion of preferences from the external world in the first place.)
you seem to be asserting that we imported the notion of preferences from the external world in the first place
No. I’m saying that the common ideas of “preference” and “about” are mind projection fallacies, in the original sense of the phrase (which Eliezer did not coin, btw, but which he does use correctly). Preference-ness and about-ness are qualities (like “sexiness”) that are attributed as intrinsic properties of the world, but to be properly specified must include the one doing the attribution.
IOW, for your preferences to be “about” the world, there must be someone who is making this attribution of aboutness, as the aboutness itself does not exist in the territory, any more than “sexiness” exists in the territory.
However, you cannot make this attribution, because the thing you think of as “the territory” is really only your model of the territory.
Well, if this is just a disagreement over whether our typical uses of the word “about” are justified, then I’m satisfied with letting go of this thread; is that the case, or do you think there is a disagreement on our expectations for specific human thoughts and actions?
This can be viewed as purely a Russellian argument about language levels, but the practical point I originally intended to make was that humans cannot actually make preferences about the actual territory because the only thing we can evaluate are our own experiences—which can be suspect. Inbuilt drives and biases are one source of experiences being suspect, but our own labeling of experiences is also suspect—labels are not only subject to random linkage, but are prone to spreading to related topics in time, space, or subject matter.
It is thus grossly delusional as a practical matter to assume that your preferences have anything to do with actual reality, as opposed to your emotionally-colored, recall-biased associations with imagined subsets of half-remembered experiences of events that occurred under entirely different conditions. (Plus, many preferences subtly lead to the recreation of circumstances that thwart the preference’s fulfillment—which calls into question precisely what “reality” that preference is about.)
Perhaps we could call our default thinking about such matters (i.e. preferences being about reality) “naive preferential realism”, by analogy to “naive moral realism”, as it is essentially the same error, applied to one’s own preferences rather than some absolute definition of good or evil.
This is pretty much what I meant by a semantic argument. If, as I’ve argued, my real preferences (as defined above) care about the projected future world (part of my map) and not just the projected future map (a sub-part of that map), then I see no difficulty with describing this by “I have preferences about the future territory”, as long as I remain aware that all the evaluation is happening within my map.
It is perhaps analogous to moral language in that when I talk about right and wrong, I keep in mind that these are patterns within my brain (analogous to those in other human brains) extrapolated from emotive desires, rather than objectively perceived entities. But with that understanding, right and wrong are still worth thinking about and discussing with others (although I need to be quite careful with my use of the terms when talking with a naive moral realist), since these are patterns that actually move me to act in certain ways, and to introspect in certain ways on my action and on the coherence of the patterns themselves.
In short, any theory of language levels or self-reference that ties you in Hofstadterian knots when discussing real, predictable human behavior (like the decision process for kin altruism) is problematic.
That said, I’m done with this thread. Thanks for an entertainingly slippery discussion!
ETA: To put it another way, learning about the Mind Projection Fallacy doesn’t mean you can never use the word “sexy” again; it just means that you should be aware of its context in the human mind, which will stop you from using it in certain novel but silly situations.
Consider the difference between a thermostat connected to a heater and a human maintaining the same temperature by looking at a thermometer and switching the heater on and off. Obviously there is a lot more going on inside the human’s brain, but I still don’t understand how the thermostat has any particular kind of connection to reality that the human lacks. The same applies whether the thermostat was built by humans with preferences or somehow formed without human design.
edit: I’m not trying to antagonize you, but I genuinely can’t tell whether you are trying to communicate something that I’m not understanding, or you’ve just read The Secret one too many times.
Obviously there is a lot more going on inside the human’s brain, but I still don’t understand how the thermostat has any particular kind of connection to reality that the human lacks.
The thermostat lacks the ability to reflect on itself, as well as the mind-projection machinery that deludes human beings into thinking that their preferences are “about” the reality they influence and are influenced by.
edit: I’m not trying to antagonize you, but I genuinely can’t tell whether you are trying to communicate something that I’m not understanding, or you’ve just read The Secret one too many times.
You’re definitely rounding to a cliche. The Secret folks think that our preferences create the universe, which is just as delusional as thinking our preferences are about the universe.
the trivial example of a thermostat, which shows a “preference” for reality being a certain way
Doesn’t it rather have a preference for its sensors showing a certain reading? (This doesn’t lead to thermostat wireheading because the thermostat’s action won’t make the sensor alter its mechanism.)
Really, it’s only systems that can model a scenario where its sensors say X but the situation is actually Y, that could possibly have preferences that go beyond the future readings of its sensors. If you assert that a thermostat can have preferences about the territory but a human can’t, then you are twisting language to an unhelpful degree.
Whether your preferences refer to your state, or to the rest of the world is indeed a wirehead-related issue. The problem with the idea that they refer to your state is that that idea tends to cause wirehead behaviour—surgery on your own brain to produce the desired state. So—it seems desirable to construct agents that believe that there is a real world, and that their preferences relate to it.
Whether your preferences refer to your state, or to the rest of the world is indeed a wirehead-related issue. The problem with the idea that they refer to your state is that that idea tends to cause wirehead behaviour—surgery on your own brain to produce the desired state. So—it seems desirable to construct agents that believe that there is a real world, and that their preferences relate to it.
I agree—that’s probably why humans appear to be constructed that way. The problem comes in when you expect the system to also be able to accurately reflect its preferences, as opposed to just executing them.
This does not preclude the possibility of creating systems that can; it’s just that they’re purely hypothetical.
To the greatest extent practical, I try to write here only about what I know about the practical effects of the hardware we actually run on today, if for no other reason than if I got into entirely-theoretical discussions I’d post WAY more than I already do. ;-)
Presumably, if you asked such an agent to reflect on its own purposes, it would claim that they related to the external world (unless it’s aim was to deceive you about its purposes for signalling reasons, of course).
For example, it might claim that its aim was to save the whales—rather than to feel good about saving the whales. It could do the latter by taking drugs or via hypnotherapy—and that is not how it actually acts.
Presumably, if you asked such an agent to reflect on its own purposes, it would claim that they related to the external world (unless it’s aim was to deceive you about its purposes for signalling reasons, of course).
Actually, if signaling was its true purpose, it would claim the same thing. And if it were hacked together by evolution to be convincing, it might even do so by genuinely believing that its reflections were accurate. ;-)
For example, it might claim that its aim was to save the whales—rather than to feel good about saving the whales. It could do the latter by taking drugs or via hypnotherapy—and that is not how it actually acts.
Indeed. But in the case of humans, note first that many people do in fact take drugs to feel good, and second, that we tend to dislike being deceived. When we try to imagine getting hypnotized into believing the whales are safe, we react as we would to being deceived, not as we would if we truly believed the whales were safe. It is this error in the map that gives us a degree of feed-forward consistency, in that it prevents us from certain classes of wireheading.
However, it’s also a source of other errors, because in the case of self-fulfilling beliefs, it leads to erroneous conclusions about our need for the belief. For example, if you think your fear of being fired is the only thing getting you to work at all, then you will be reluctant to give up that fear, even if it’s really the existence of the fear that is suppressing, say, the creativity or ambition that would replace the fear.
In each case, the error is the same: System 2 projection of the future implicitly relies on the current contents of System 1′s map, and does not take into account how that map would be different in the projected future.
(This is why, by the way, The Work’s fourth question is “who would you be without that thought?” The question is a trick to force System 1 to do a projection using the presupposition that the belief is already gone.)
It’s possible that the confusion here is (yet again) due to people using the word utility to mean different things. If PJ is using utility to refer to, e.g. an emotional or cognitive state, then he’s right that our utility cannot respond directly to the territory. But broader notions of utility, well-being, and preference are possible, and nothing PJ has said is especially relevant to whether they are coherent or not.
If PJ is using utility to refer to, e.g. an emotional or cognitive state, then he’s right that our utility cannot respond directly to the territory. But broader notions of utility, well-being, and preference are possible, and nothing PJ has said is especially relevant to whether they are coherent or not.
Right. I don’t dabble in discussing those broader notions, though, since they can’t be empirically grounded. How can you test a concept of utility that’s not grounded in human perception and emotion? What good can it ever do you if you can’t connect it back to actual living people?
I consider such discussions to be much more irrational than, say, talk of “The Secret”, which at least offers an empirical procedure that can be tested. ;-)
(In fairness, I do consider such discussions here on LW to be far less annoying than most discussions of the Secret and suchlike!)
These notions are about what it means for something to be good for “actual living people”. They’re difficult, if not impossible to “test” (about the best testing procedures we’ve come up with is thought experiments, which as discussed elsewhere are riddled with all sorts of problems). But it’s not like you can “test” the idea that positive emotions are good for you either.
I’m hard pressed, though, to conceive of a moral philosophy where improved health would not be considered “good for you”.
Preference utilitarianism applied to someone who thinks that it is only through suffering that life can achieve meaning.
To be clear, I don’t subscribe to such a view myself, but it’s conceivable. I agree with you that health is good for people. My point is just that this agreement owes more to shared intuition than conclusive empirical testing.
Preference utilitarianism applied to someone who thinks that it is only through suffering that life can achieve meaning.
Yes, but now we’re back to concrete feelings of actual people again. ;-)
To be clear, I don’t subscribe to such a view myself, but it’s conceivable. I agree with you that health is good for people. My point is just that this agreement owes more to shared intuition than conclusive empirical testing.
Right, which is one reason why, when we’re talking about this particular tiny (but important) domain (that at least partially overlaps with Eliezer’s notion of Fun Theory), conclusive empirical testing is a bit of a red herring, since the matter is subjective from the get-go. We can objectively predict certain classes of subjective events, but the subjectivity itself seems to be beyond that. At some point, you have to make an essentially arbitrary decision of what to value.
Part of the problem is the X is necessarily based in the map, not the territory. There will always be the chance that one will learn something that radically changes the map, so it seems like an explicit statement of f will have to involve all possible maps that one might have.
Necessarily? If I place value on e.g. “my friends actually liking and respecting me”, rather than just “the subjective sense that my friends like and respect me” then my utility function seems to be responding directly to the territory rather than the map. (It also means that I won’t ever really know my utility, but that’s true of lots of things.) Some people argue that things can’t affect one’s well-being unless one somehow experiences them, but that’s a contentious position.
Am I missing your intended meaning?
You’re right—I should have written “necessarily based in the map, not just the territory.”
My intended meaning has to do with fundamental shifts in one’s understanding of how reality works that make some previous apparent question of fact become a “wrong question” or category error or similar non-issue.
In practice, your definition of what “liking and respecting me” means—i.e., what evidence you expect to see in the world of that—is part of the map, not the territory.
Suppose, for example, that your friends really and truly like and respect you… but they have to beat you up and call you names, for some other reason. Does that match what you actually value? It’s out there in the territory, after all.
That is, is merely knowing that they “like and respect you” enough? Or is that phrase really just a shorthand in your map for a set of behaviors and non-behaviors that you actually value?
Note that if you argue that, “if they really liked and respected me, they wouldn’t do that”, then you are now back to talking about your map of what that phrase means, as opposed to what someone else’s map is.
System 2 thinking is very tricky this way—it’s prone to manipulating symbols as if they were the things they’re merely pointing at, as though the map were the territory… when the only things that exist in its perceptual sphere are the labels on the map.
Most of the time, when we think we’re talking about the territory, we’re talking about the shapes on the map, but words aren’t even the shapes on the map!
We have only access to our current map to tell us about the territory, yes. But we have strong intuitions about how we would act if we could explicitly choose that our future map permanently diverge from our current map (which we currently see as the territory). If we (again by our current map) believe that this divergence would conform less to the territory (as opposed to a new map created by learning information), many of us would oppose that change even against pretty high stakes.
I mean, if Omega told me that I had to choose between
(A) my sister on Mars being well but cut off from all contact with me, or
(B) my sister being killed but a nonsentient chatbot impersonating her to me in happy weekly chats,
and that in either case my memory of this choice would be wiped when I made it, I would choose (A) without hesitation.
I understand that calling our current map “the territory” looks like a categorical error, but rejecting conchis’ point entirely is the wrong response. There’s a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.
Of course, because the immediate pain of the thought of choosing B would outweigh the longer-term lesser pain of the thought of losing contact with your sister.
This has nothing to do with whether the events actually occur, and everything to do with your mapping of the experience of the conditions, as you imagine them for purposes of making a decision.
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences
It so happens that there is some correspondence between this (real) process and the way we would prefer to think we establish and evaluate our preferences. Specifically, both models will generate similar results, most of the time. It’s just that the reasons we end up with for the responses are quite different.
But calling that latter concept “territory” is still a category error, because what you are using to evaluate it is still your perception of how you would experience the change.
We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have “rational” preferences it is because they will ultimately lead to some desired emotion or sensation.
However, our brains are constructed in such a way so as to allow us to plausibly overlook and deny this fact, so that we can be honestly “sincere” in our altruism… specifically by claiming that our responses are “really” about things outside ourselves.
For example, your choice of “A” allows you to self-signal altruism, even if your sister would actually prefer death to being imprisoned on Mars for the rest of her life! Your choice isn’t about making her life better, it’s about you feeling better for the brief moment that you’re aware you did something.
(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)
I affirm this, but it does not follow that:
Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.
And a thermostat’s map is also “entangled” with the territory, but as loqi pointed out, what it really prefers is only that its input sensor match its temperature setting!
I am not saying there are no isomorphisms between the shape of our preferences and the shape of reality, I am saying that assuming this isomorphism means the preferences are therefore “about” the territory is mind projection.
If you look at a thermostat, you can project that it was made by an optimizing process that “wanted” it to do certain things by responding to the territory, and that thus, the thermostat’s map is “about” the territory. And in the same way, you can look at a human and project that it was made by an optimizing process (evolution) that “wanted” it to do certain thing by responding to the territory.
However, the “aboutness” of the thermostat does not reside in the thermostat; it resides in the maker of the thermostat, if it can be said to exist at all! (In fact, this “aboutness” cannot exist, because it is not a material entity; it’s a mental entity—the idea of aboutness.)
So despite the existence of inputs and outputs, both the human and the thermostat do their “preference” calculations inside the closed box of their respective models of the world.
It just so happens that humans’ model of the world also includes a Mind Projection device, that causes humans to see intention and purpose everywhere they look. And when they look through this lens at themselves, they imagine that their preferences are about the territory… which then keeps them from noticing various kinds of erroneous reasoning and subgoal stomps.
For that matter, it keeps them from noticing things like the idea that if you practice being a pessimist, nothing good can last for you, because you’ve trained yourself to find bad things about anything. (And vice versa for optimists.)
Ostensibly, optimism and pessimism are “about” the outside world, but in fact, they’re simply mechanical, homeostatic processes very much like a thermostat.
I am not a solipsist nor do I believe people “create your own reality”, with respect to the actual territory. What I’m saying is that people are deluded about the degree of isomorphism between their preferences and reality, because they confuse the map with the territory. And even with maximal isomorphism between preference and reality, they are still living in the closed box of their model.
It is reasonable to assume that existence actually exists, but all we can actually reason about is our experience of it, “inside the box”.
And what if he did ask?
Then, as I said, he cares about something closer to the reality.
The major point I’ve been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations. And not asking the question is a perfect example of the sort of decision error this can produce!
However, asking the question doesn’t magically make the preference about the territory either; in order to prefer the future include his sister’s best interests, he must first have an experience of the sister and a reason to wish well of her. But it’s still better than not asking, which is basically wireheading.
The irony I find in this discussion is that people seem to think I’m in favor of wireheading because I point out that we’re all doing it, all the time. When in fact, the usefulness of being aware that it’s all wireheading, is that it makes you better at noticing when you’re doing it less-usefully.
The fact that he hadn’t asked his sister, or about his sister’s actual well-being instantly jumped off the screen at me, because it was (to me) obvious wireheading.
So, you could say that I’m biased by my belief to notice wireheading more, but that’s an advantage for a rationalist, not a disadvantage.
Is human knowledge also not just in the map, but exclusively of the map? If not, what’s the difference?
Any knowledge about the actual territory can in principle be reduced to mechanical form without the presence of a human being in the system.
To put it another way, a preference is not a procedure, process, or product. The very use of the word “preference” is a mind projection—mechanical systems do not have “preferences”—they just have behavior.
The only reason we even think we have preferences in the first place (let alone that they’re about the territory!) is because we have inbuilt mind projection. The very idea of having preferences is hardwired into the model we use for thinking about other animals and people.
You never answered my question.
You said, “if not, what’s the difference”, and I gave you the difference. i..e, we can have “knowledge” of the territory.
So, knowledge exists in the structure of map and is about the territory, while preference can’t be implemented in natural artifacts. Preference is a magical property of subjective experience, and it is over maps, or about subjective experience, but not, for example, about the brain. Saying that preference exists in the structure of map or that it is about the territory is a confusion, that you call “mind projection” Does that summarize your position? What are the specific errors in this account?
No, “preference” is an illusory magical property projected by brains onto reality, which contains only behaviors.
Our brains infer “preferences” as a way of modeling expected behaviors of other agents: humans, animals, and anything else we perceive as having agency (e.g. gods, spirits, monsters). When a thing has a behavior, our brains conclude that the thing “prefers” to have either the behavior or the outcome of the behavior, in a particular circumstance. In other words, “preference” is a label attached to a clump of behavior-tendency observations and predictions in the brain—not a statement about the nature of the thing being observed.
Thus, presuming that these “preferences” actually exist in the territory is supernaturalism, i.e., acting as though basic mental entities exist.
My original point had more to do with the types of delusion that occur when we reason on the basis of preferences actually existing, rather than the idea simply being a projection of our own minds. However, the above will do for a start, as I believe my other conclusions can be easily reached from this point.
Do you think someone is advocating the position that goodness of properties of the territory is an inherent property of territory (that sounds like a kind of moral realism)? This looks like the lack of distinction between 1-place and 2-place words. You could analogize preference (and knowledge) as a relation between the mind and the (possible states of the) territory, that is neither a property of the mind alone, nor of the territory alone, but a property of them being involved in a certain interaction.
No, I assume that everybody who’s been seriously participating has at least got that part straight.
Now you’re getting close to what I’m saying, but on the wrong logical level. What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
If I assert that my preferences are “about” the real world, I am making a category error because my preferences are relationships between portions of my map, some of which I have labeled as representing the territory.
The fact that there is a limited isomorphism between that portion of my map, and the actual territory, does not make my preferences “about” the territory, unless you represent that idea in another map.
That is, I can represent the idea that “your” preferences are about the territory in my map… in that I can posit a relationship between the part of my map referring to “you”, and the part of my map referring to “the territory”. But that “aboutness” relationship is only contained in my map; it doesn’t exist in reality either.
That’s why it’s always a mind projection fallacy to assert that preferences are “about” territory: one cannot assert it of one’s own preferences, because that implies the territory is inside the map. And if one asserts it of another person’s preferences, then that one is projecting their own map onto the territory.
I initially only picked on the specific case of self-applied projection, because understanding that case can be very practically useful for mind hacking. In particular, it helps to dissolve certain irrational fears that changing one’s preferences will necessarily result in undesirable futures. (That is, these fears are worrying that the gnomes and fairies will be destroyed by the truth, when in fact they were never there to start with.)
How’s that? You can write Newton’s law of universal gravitation describing the orbit of the Earth around the Sun on a piece of paper located on the surface of a table standing in a house on the surface of the Earth. Where does this analogy break from your point of view?
″...but, you can’t fold up the territory and put it in your glove compartment”
The “aboutness” relationship between the written version of Newton’s law and the actual instances of it is something that lives in the map in your head.
IOW, the aboutness is not on the piece of paper. Nor does it exist in some supernatural link between the piece of paper and the objects acting in accordance with the expressed law.
Located on the planet Earth.
And this helps your position how?
Your head describes how your head rotates around the Sun.
No, your head is rotating around the Sun, and it contains a description relating the ideas of “head” and “Sun”. You are confusing head 1 (the real head) with head 2 (the “head” pictured inside head 1), as well as Sun 1 (the real Sun) and Sun 2 (the “Sun” pictured inside head 1).
No, I’m not confusing them. They are different things. Yet the model simulates the real thing, which means the following (instead of magical aboutness): By examining the model it’s possible to discover new properties of its real counterpart, that were not apparent when the model was being constructed, and that can’t be observed directly (or it’s just harder to do), yet can be computed from the model.
Indeed. Although more precisely, examining the model merely suggests or predicts these “new” (rather, previously undiscovered, unnoticed, or unobservable) properties.
That is what I mean by isomorphism between model and territory. The common usage of “about”, however, projects an intention onto this isomorphism—a link that can only exist in the mind of the observer, not the similarity of shapes between one physical process and another.
Since agent’s possible actions are one of the things in the territory captured by the model, it’s possible to use the model to select an action leading to a preferable outcome, and to perform thus selected action, determining the territory to conform with the plan. The correspondence between the preferred state of the world in the mind and the real world is ensured by this mechanism for turning plans into actuality. Pathologies aside, or course.
I don’t disagree with anything you’ve just said, but it does nothing to support the idea of an isomorphism inherently meaning that one thing is “about” another.
If I come across a near-spherical rock that resembles the moon, does this make the rock “about” the moon? If I find another rock that is shaped the same, does that mean it is about the moon? The first rock? Something else entirely?
The :”aboutness” of a thing can’t be in the thing, and that applies equally to thermostats and humans.
The (external) aboutness of a thermostat’s actions don’t reside in the thermostat’s map, and humans are deluded when they project that the (external) aboutness of their own actions actually resides within the same map they’re using to decide those actions. It is merely a sometimes-useful (but often harmful) fiction.
Taboo “aboutness” already. However unfathomably confused the philosophic and folk usage of this word is doesn’t interest me much. What I mean by this word I described in these comments, and this usage seems reasonably close to the usual one, which justifies highjacking the word for the semi-technical meaning rather than inventing a new one. This is also the way meaning/aboutness is developed in formal theories of semantics.
So, you are saying that you have no argument with my position, because you have not been using either “about” or “preference” with their common usage?
If that is the case, why couldn’t you simply say that, instead of continued argument and posturing about your narrower definition of the words? ISTM you could have pointed that out days ago and saved us all a lot of time.
This is also not the first time where I have been reducing the common usage of a word (e.g. “should”) and then had you argue that I was wrong, based on your own personal meaning of the word.
Since I have no way of knowing in advance all of the words you have chosen to redefine in your specialized vocabulary, would it be too much to ask if you point out which words you are treating as specialized when you argue that my objection to (or reduction of) the common meaning of the word is incorrect, because it does not apply to your already-reduced personal version of the word?
Then, I could simply nod, and perhaps ask for your reduction in the case where I do not have a good one already, and we would not need to have an extended argument where we are using utterly incompatible definitions for such words as “about”, “preference”, and “should.”
Actually, the point is that most of the other usages of these words are meaningless confusion, and the argument is that this particular semi-technical sense is what the word actually means, when you get the nonsense out of it. It’s not how it’s used, but it’s the only meaningful thing that fits the idea.
Since you don’t just describe the usage of the word, but argue for the confusion behind it, we have a disagreement. Presenting a clear definition is the easy part. Showing that ten volumes of the Encyclopedia of Astrology is utter nonsense is harder, and arguing with each point made in its chapters is a wrong approach. It should be debunked on meta-level, with an argument that doesn’t require the object-level details, but that requires the understanding of the general shape of the confusion.
Yes, but ones which most people do not understand to be confusion, and the only reason I started this discussion in the first place was because I was trying to clear up one point in that confusion.
I am arguing against the confusion, not for the confusion. So, as far as I can tell, there should be no disagreement.
In practice, however, you have been making arguments that sound like you are still confusing map and territory in your own thinking, despite seeming to agree with my reasoning on the surface. You are consistently treating “about” as a 2-way relationship, when to be minimally cohesive, it requires 3 entities: the 2 entities that have an isomorphism, and the third entity whose map ascribes some significance to this isomorphism.
You’ve consistently omitted the presence of the third entity, making it sound as though you do not believe it to be required, and thereby committing the mind projection fallacy.
So you are saying that my definition with which you’ve just agreed is unreasonable. Pick something tangible.
(Also, please stop using “mind projection fallacy”, you are misapplying the term.)
How is that, precisely? My understanding is that it is mind projection when you mistakenly believe a property of an object to be intrinsic, rather than merely attributed.
I am pointing out that “aboutness” (whose definition I never agreed on, because you handwaved away the subject by saying it is I who should define it), is not an intrinsic property of isomorphic relationships.
Rather, it is a property being attributed to that relationship, a label that is being expressed in some map.
That sounds like a textbook case of the mind projection fallacy, i.e. “the error of projecting your own mind’s properties into the external world.”
(Prediction: your next reply will still not address this point, nor clarify your definition of “about”, but simply handwave again why it is that I am doing something else wrong. Anything but actually admitting that you have been using a mind-projecting definition of “about” since the beginning of this conversation, right up until the point where you ducked the question by asking me to taboo it, rather than defend the imprecise definition you’ve been using, or clear up any of the other handwaving you’ve been using to beg the question. I base this prediction on the rapid increase in non-responsive replies that, instead of defending the weak points of your position, represent attempts to find new ways to attack me and/or my position. A rationalist should be able to attack their own weak points, let alone being able to defend them, without resorting to distraction, subject-changing, and playing to the gallery.)
There are natural categories, like “tigers”, that don’t require much of a mind to define. It’s not mind projection fallacy to say that something is a tiger.
P.S. I’m correcting self-censoring threshold, so expect silence where before I’d say something for the fifth time.
Is that actually an argument? ’cause it sounds like a random sentence injected into the conversation, perhaps as an invitation for me to waste time tearing “natural categories” to shreds, while leaving you still able to deny that your statement actually relates in any substantial way to your point… thereby once again relieving you of any need to actually defend your position.
That is, are you actually claiming aboutness to be a natural category? Or just trying to get me to treat your argument as if you were doing so?
I already did and do expect it; see my “prediction” in the parent to your comment. I predicted that you would remain silent on any substantive issues, and avoid admitting anywhere where you were mistaken or incorrect. (I notice, for example, that you went back and deleted the comment where you said I was using “mind projection fallacy” incorrectly, rather than admit your attack was in error.)
And, as predicted, you avoided directly addressing the actual point of contention, instead choosing to enter a new piece of handwaving to imply that I am doing something else wrong.
That is, you appear to now be implying that I am using an overbroad definition of the MPF, without actually saying that I am doing it, or that your statement is in any way connected to your own position. This is a nice double bind, since either way I interpret the statement, you can retreat… and throw in more irrelevancies.
I don’t know if “troll” is a natural category, but you’re sure getting close to where I’d mind-project your behavior as matching that of one. ;-)
For the record, I thought it obvious that my argument above implied that I claim aboutness to be a natural category (although I’m not perfectly sure it’s a sound argument). I deleted my comment because I deemed it low-quality, before knowing you responded to it.
It’s not.
First, the only way it can be one is if “natural category” has the reductionist meaning of “a category based on distinctions that humans are biased towards using as discriminators”, rather than “a category that ‘naturally’ exists in the territory”. (Categories are abstractions, not physical entities, after all.)
And second, even if you do use the reductionist meaning of “natural category”, then this does not in any way undermine the conclusion that “aboutness” is mind projection when you omit the entity mapping that aboutness from the description.
In other words, this argument appears to result in only one of two possibilities: either “aboutness” is not a natural category per the reductionist definition, and thus inherently a mind projection when the attribution source is omitted, or “aboutness” is a natural category per the reductionist definition… in which case the attribution source has to be a human brain (i.e., in another map).
Finally, if we entirely reject the reductionist definition of “natural category”, then “natural category” is itself an instance of the mind projection fallacy, since the description omits any definition of for whom the category is “natural”.
In short, QED: the argument is not sound. (I just didn’t want to bother typing all this if you were going to retreat to a claim this was never your argument.)
Indeed. If this didn’t work then there wouldn’t be any practical point in modeling physics!
To the (unknowable*) extent that the portion of my map labelled “territory” is an accurate reflection of the relevant portion of the territory, do I get to say that my preferences are “about” the territory (implicitly including disclaimers like “as mediated by the map”)?
* due at the very least to Matrix/simulation scenarios
You can say it all you want, it just won’t make it true. ;-) Your preference is “about” your experience, just as the thermostat’s heating and cooling preferences are “about” the temperature of its sensor, relative to its setting.
For there to be an “about”, there has to be another observer, projecting a relationship of intention onto the two things. It’s a self-applied mind projection—a “strange loop” in your model—to assert that you can make such statements about your own preferences, like a drawing of Escher wherein Escher is pictured, making the drawing. The whole thing only makes sense within the surface of the paper.
(Heck, it’s probably a similar strange loop to make statements about one’s self in general, but this probably doesn’t lead to the same kind of confusion and behavioral problems that result from making assertions about one’s preferences.… No, wait, actually, yes it does! Self-applied nominalizations, like “I’m bad at math” are an excellent example. Huh. I keep learning interesting new things in this discussion.)
That’s one way of writing. Another is to edit what you intend to post before you click ‘comment’.
I feel your frustration, but throwing the word “magical” in there is just picking a fight, IMO. Anyway, I too would like to see P.J. Eby summarize his position in this format.
I have a certain technical notion of magic in mind. This particular comment wasn’t about frustration (some of the others were), I’m trying out something different of which I might write a post later.
Be charitable in your interpretation, and remember the Least Convenient Possible World principle. I was presuming that the setup was such that being alive on Mars wouldn’t be a ‘fate worse than death’ for her; if it were, I’d choose differently. If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.
I would say rather that my reaction is my evaluation of an imagined future world. The essence of many decision algorithms is to model possible futures and compare them to some criteria. In this case, I have complicated unconscious affective criteria for imagined futures (which dovetail well with my affective criteria for states of affairs I directly experience), and my affective reaction generally determines my actions.
To the extent this is true (as in the sense of my previous sentence), it is a tautology. I understand what you’re arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals. I am not asserting this; I believe that our affective algorithms do often operate under more selfish and basic criteria, and that they fixate on the most salient possibilities instead of weighing probabilities properly, among other things.
However, these affective algorithms do appear to respond more strongly to certain facets of “how I expect the world to be” than to facets of “how I expect to think the world is” when the two conflict (with an added penalty for the expectation of being deceived), and I don’t find that problematic on any level.
As I said, it’s still going to be about your experience during the moments until your memory is erased.
I took that as a given, actually. ;-) What I’m really arguing against is the naive self-applied mind projection fallacy that causes people to see themselves as decision-making agents—i.e., beings with “souls”, if you will. Asserting that your preferences are “about” the territory is the same sort of error as saying that the thermostat “wants” it to be a certain temperature. The “wanting” is not in the thermostat, it’s in the thermostat’s maker.
Of course, it makes for convenient language to say it wants, but we should not confuse this with thinking the thermostat can really “want” anything but for its input and setting to match. And the same goes for humans.
(This is not a mere fine point of tautological philosophy; human preferences in general suffer from high degrees of subgoal stomp, chaotic loops, and other undesirable consequences arising as a direct result of this erroneous projection. Understanding the actual nature of preferences makes it easier to dissolve these confusions.)
I wish I could upvote this two or three times. Thank you.
What features of that comment made it communicate something new to you? What was it that got communicated?
The comment restated a claim that a certain relationship is desirable as a claim that given that it’s desirable, there is a process that establishes it to be true. It’s interesting how this restatement could pierce inferential distance: is preference less trustworthy than a fact, and so demonstrating the conversion of preference into a fact strengthens the case?
Given the length of the thread I branched from, it looks like you and P.J. Eby ended up talking past each other to some extent, and I think that you both failed to distinguish explicitly between the current map (which is what you calculate the territory to be) and a hypothetical future map.
P.J. Eby was (correctly) insisting that your utility function is only in contact with your current map, not the territory directly. You were (correctly) insisting that your utility function cares about (what it calculates to be) the future territory, and not just the future map.
Is that a fair statement of the key points?
Utility function is no more “in contact” with your current map than the actual truth of 2+2=4 is in contact with display of a calculator that displays the statement. Utility function may care about past territory (and even counterfactual territory) as well as future territory, with map being its part. Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn’t be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we’re talking about practical purposes, this is a relevant point.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I’m generally more repulsed by the ones in which my map says it’s fine where it’s not than by the opposite.
This something of a nitpick, but this isn’t strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
Oh, good point. I should have restricted it to “any calculation you personally do”, in which case I believe it holds.
At which point it becomes trivial: any calculation that is done on your map is done using your map, just Markovity of computation...
A related point is that you can create tools that make decisions themselves, in situations only of possibility of which you are aware.
Right. It’s trivial, but relevant when discussing in what sense our decision algorithms refer to territory versus map.
I can’t parse this. What do you mean?
If you install an alarm system that uses a video camera to recognize movement and calls the police if it’s armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we’ve resorted to arguing about the semantics of “map” versus “territory”, as I don’t see a scenario where we’d predict or decide differently from each other on account of this disagreement. As such, I’m willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
It appears to lead nowhere: your comments are clear, while his are all smoke and mirrors, in many many words.
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system’s controller—an implementation of values that are not our own.
If there are any “preferences about reality” in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker’s values.
By analogy, if our preferences are the implementation of any values, they are the “values” of natural selection, not our own.
If now you say that natural selection doesn’t have any preferences or values, then we are left with no preferences anywhere—merely isomorphism between control systems and their environments. Saying this isomorphism is “about” something is saying that a mental entity (the “about” relationship) exists in the real world, i.e., supernaturalism.
In short, what I’m saying is that anybody who argues human preferences are “about” reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of “preference”, and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are “about” reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its “preferences” still “about” reality in that case?
Both. You are now trying to explain away the rainbow, by insisting that it consists of atoms, which can’t in themselves possess the properties of a rainbow.
So an alarm system has preferences? That is not most people’s understanding of the word “preference”, which requires a degree of agency that most rationalists wouldn’t attribute to an alarm system.
Nonetheless, let us say an alarm system has preferences. You didn’t answer any of my follow-on questions for that case.
As for explaining away the rainbow, you seem to have me confused with an anti-reductionist. See Explaining vs. Explaining Away, in particular:
At this point, I am attempting to show that the very concept of a “preference” existing in the first place is something projected onto the world by an inbuilt bias in human perception. Reality does not have preferences, it has behaviors.
This is not erasing the rainbow from the world, it’s attempting to erase the projection of a mind-modeling variable (“preference”) from the world, in much the same way as Eliezer broke down the idea of “possible” actions in one of his series.
So, if you are claiming that preference actually exists, please give your definition of a preference, such that alarm systems and humans both have them.
A good reply, if only you approached the discussion this constructively more often.
Note that probability is also in the mind, but yet your see all the facts through it, and you can’t ever revoke it, each mind is locked in its subjectively objective character. What do you think of that?
I think that those things have already been very well explained by Eliezer—so much so that I assumed that you (and the others participating in this discussion) would have already internalized them to the same degree as I have, such that asserting “preferences” to be “about” things would be a blatantly obvious instance of the mind projection fallacy.
That’s why, early on, I tended to just speak as though it was bloody obvious, and why I haven’t been painstakingly breaking it all out piece by piece, and why I’ve been baffled by the argument, confusion, and downvoting from people for whom this sort of basic reductionism ought to be a bloody simple matter.
Oh, and finally, I think that you still haven’t given your definition of “preference”, such that humans and alarm systems both have it, so that we can then discuss how it can then be “about” something… and whether that “aboutness” exists in the thing having the preference, or merely in your mental model of the thing.
That in reply to a comment full of links to Eliezer’s articles. You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument. I wasn’t even talking about preference.
I know. That’s the problem. See this comment and this one, where I asked for your definition of preference, which you still haven’t given.
That’s because you also “didn’t answer my comment, but wrote some text that doesn’t help me in our argument.” I was attempting to redirect you to answering the question which you’ve now ducked twice in a row.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation. Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation. I’m currently trying to understand your position in terms of my position, not to explain to you my position.
We reached a point in the discussion where it appears the only way we could disagree is if we had a different definition of “preference”. Since I believe I’ve made my definition quite clear, I wanted to know what yours is.
It might not help you, but it would certainly help me to understand your position, if you are not using the common definition of preference.
I asked you first, and you responded with (AFAICT) a non-answer. You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
So, I actually answered your question (i.e. “what do you think?”), even though you still haven’t answered mine.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
I chose the articles in the comment above because they were in surface-match with what you are talking about, as a potential point on establishing understanding. I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
And I answered that I agree with them, and that I considered it foundational material to what I’m talking about.
Indeed, which is why I’d now like to have the answer to my question, please. What definition of “preferences” are you using, such that an alarm system, thermostat, and human all have them? (Since this is not the common, non-metaphorical usage of “preference”.)
Preference is order on the lotteries of possible worlds (ideally established by expected utility), usually with agent a part of the world. Computations about this structure are normally performed by a mind inside the mind. The agent tries to find actions that determine the world to be as high as possible on the preference order, given the knowledge about it. Now, does it really help?
Yes, as it makes clear that what you’re talking about is a useful reduction of “preference”, unrelated to the common, “felt” meaning of “preference”. That alleviates the need to further discuss that portion of the reduction.
The next step of reduction would be to unpack your phrase “determine the world”… because that’s where you’re begging the question that the agent is determining the world, rather than determining the thing it models as “the world”.
So far, I have seen no-one explain how an agent can go beyond its own model of the world, except as perceived by another agent modeling the relationship between that agent and the world. It is simply repeatedly asserted (as you have effectively just done) as an obvious fact.
But if it is an obvious fact, it should be reducible, as “preference” is reducible, should it not?
Hmm… Okay, this should’ve been easier if the possibility of this agreement was apparent to you. This thread is thereby merged here.
I’d been following this topic and getting frustrated with my inability to put my opinion on the whole preferences-about-the-territory thing into words, and I thought that orthonomal’s comment accomplished it very nicely. I don’t think I understand your other question.
Utility is about the territory in the same sense that the map is about the territory; the map tells us the way the territory is, utility tells us the way we want the territory to be. Us non-wireheaders want an accurate map because it’s the territory we care about.
Supposing utility is not about the territory but about the map, we get people who want nothing more than to sabotage their own mapmaking capabilities. If the territory is not what we care about, maintaining the correspondence of map to territory would be a pointless waste of effort. Wireheading would look like an unambiguously good idea, not just to some people but to everyone.
Conchis’ example of wanting his friends to really like and respect him is correct. He may have failed to explicitly point out that he also has an unrelated preference for not being beaten up. He’s also in the unfortunate position of valuing something he can’t know about without using long chains of messy inductive inferences. But his values are still about the territory, and he wants his map to accurately reflect the territory because it’s the territory he cares about.
I am only saying that the entire stack of concepts you have just mentioned exists only in your map.
Permit me to translate: supposing utility is not about the (portion of map labeled) territory but about the (portion of map labeled) map, we get people who want nothing more than to sabotage their own mapmaking capabilities.
Does that make it any clearer what I’m saying?
This is a “does the tree make a sound” argument, and I’m on the, “no it doesn’t” side, due to using a definition of “sound” that means “the representation of audio waves within a human nervous system”. You are on the “of course it makes a sound” side, because your definition of sound is “pressure waves in the air.”
Make sense?
As far as I can tell, you’re saying that there is no territory, or that the territory is irrelevant. In other words, solipsism. You’ve overcome the naive map/territory confusion, but only to wind up with a more sophisticated form of confusion.
This isn’t a “does the tree make a sound” argument. It’s more like a “dude… how do we even really know reality is really real” argument. Rationality is entirely pointless if all we’re doing is manipulating completely arbitrary map-symbols. But in that case, why not leave us poor, deluded believers in reality to define the words “map”, “territory”, and “utility” the way we have always done?
No, general semantics. There’s a difference.
Can you point out the difference?
Even though “this is not a pipe”, the form of a depiction of a pipe is nevertheless highly constrained by the physical properties of actual pipes. Do you deny that? If not, how do you explain it?
I’ve been trying to be on the “it depends on your definition and my definition sits within the realm of acceptable definitions” side. Unfortunately, whether this is what you intend or not, most of your comments come across as though you’re on the “it depends on the definition, and my (PJ’s) defintion is right and yours is wrong” side, which is what seems to be getting people’s backs up.
This confusion is dissolved in the post Disputing Definitions.
Which confusion? I didn’t think I was confused. Now I’m confused about whether I’m confused. ;)
You mentioned this confusion as possibly playing a role in you and Eby talking past each other, the ambiguous use of the word “utility”.
OK, cool. Now, given that we’ve already identified that, what does Disputing Definitions tell us that we don’t already know?
There’s two things to say in response to this: first, I can define “liking and respecting me” as “experiencing analogous brain states to mine when I like and respect someone else”. That’s in the territory (modulo some assumptions about the cognitive unity of humankind): I could verify it in principle, although not in practice.
The second thing is that even if we grant that the example was poor, the point was still valid. For example, one might prefer that one’s spouse never cheat to one’s spouse cheating but never being aware of that fact. (ETA: but maybe you weren’t arguing against the point, only the example.)
But what if they experience that state, and still, say, beat you up and treat you like jerks, because that’s what their map says you should do when you feel that way?
This isn’t about the example being poor, it’s about people thinking things in the map actually exist in the territory. Everything you perceive is mediated by your maps, even if only in the minimal sense of being reduced to human sensory-pattern recognition symbols first, let alone all the judgments about the symbols that we add on top.
How about the case where you absolutely believe the spouse is cheating, but they really aren’t?
This is certainly a better example, in that it’s easier to show that it’s not reality that you value, but the part of your map that you label “reality”. If you really truly believe the spouse is cheating, then you will feel exactly the same as if they really are.
IOW, when you say that you value something “in the territory”, all you are ever really talking about is the part of your map that you label “the territory”, whether that portion of the map actually corresponds to the territory or not.
This is not some sort of hypothetical word-argument, btw. (I have no use for them, which is why I mostly avoid the Omega discussions.) This is a practical point for minimizing one’s suffering and unwanted automatic responses to events in the world. To the extent you believe that your map is the territory, you will suffer when it is out-of-sync.
It’s still possible to prefer this state of affairs to one where they are beating me because they are contemptuous of me. Remember, we’re talking about a function from some set X to the real numbers, and we’re trying to figure out what sorts of things are members of X. In general, people do have preferences about the way things actually are.
But my spouse won’t, and I have preferences about that fact. All other things being equal, my preference ordering is “my spouse never cheats and I believe my spouse never cheats” > “my spouse cheats and I find out” > “my spouse cheats and I believe my spouse never cheats” > “my spouse never cheats but I believe she does”. If a utility function exists that captures this preference, it will be a function that takes both reality and my map as arguments.
Right, which is where this veers off into “hypothetical word-arguments” for me, because the entire point I’m making is that all your preferences are still about the map, no matter how many times you point to a region of the map marked, “the way things actually are”, and distinguish it from another part of the map labeled, “my map”.
You’ve read “Godel, Escher, Bach”, right? This is not a pipe. The hand that’s doing the drawing is in the drawing, no matter how realistically drawn it is. ;-)
Can I get your analysis of my “spouse cheating and what I know about it” example? I’ve understood your position on utility functions as stated in other branches; I’m curious as to how you would interpret my claim to have preferences over both states of reality and beliefs about reality in this fairly concrete and non-hypothetical case.
My point is that your entire argument consists of pointing to the map and claiming it’s the territory. In the cases where reality and your belief conflict, you won’t know that’s the case. Your behavior will be exactly the same, either way, so the distinction is moot.
When you are trying to imagine, “my spouse is cheating and I think she isn’t”, you aren’t imagining that situation… you are actually imagining yourself perceiving that to be the case. That is, your map contains the idea of being deceived, and that this is an example of being deceived, and it is therefore bad.
None of that had anything to do with the reality over which you claim to be expressing a preference, because if it were the reality, you would not know you were being deceived.
This is just one neat little example of systemic bias in the systems we use to represent and reflect on preferences. They are designed to react to perceived circumstances, rather than to produce consistent reasoning about how things ought to be. So if you ever imagine that they are “about” reality, outside the relatively-narrow range of the here-and-now moment, you are on the path to error.
And just as errors accumulate in Newtonian physics as you approach the speed of light, so too do reasoning errors tend to accumulate as you turn your reasoning towards (abstract) self-reflection.
Sure.
No. My imagination encompasses the fact that if it were the reality, I would not know I was being deceived. I know what my emotional state would be—it would be the same as it is now. That’s easy to get.
What it really comes down to is that saying that my preference ordering is “my spouse cheats and I find out” > “my spouse cheats and I believe my spouse never cheats” is equivalent to saying that if I am not sure (say 50% probability), I will seek out more information. I’m not sure how such a decision could be justified without considering that my preferences are over both the map and the territory.
ETA: Reading over what you’ve written in other branches, I’d like to point out a preference for not being deceived even if you will never realize it isn’t an error—it’s prima facie evidence that human preferences are over the territory as well as the map. That may not be the most useful way of thinking about it from a mindhacking perspective, but I don’t think it’s actually wrong.
That preference is not universal, which to me makes it absolutely part of the map. And it’s not just the fictional evidence of Cypher wanting to go back in the Matrix and forget, guys routinely pay women for various forms of fantasy fulfillment, willingly suspending disbelief in order to be deceived.
Not enough? How about the experimental philosophers who re-ran the virtual world thought experiment until they found that people’s decision about living in a fantasy world that they’d think was real, was heavily dependent upon whether they 1) had already been living in the fantasy, 2) whether their experience of life would significantly change, and 3) whether their friends and loved ones were also in the fantasy world.
If anything, those stats should be quite convincing that it’s philosophers and extreme rationalists who have a pathological fear of deception, rather than a inbuilt human preference for actually knowing the truth… and that most likely, if we have an inbuilt preference against deception, it’s probably aimed at obtaining social consensus rather than finding truth.
All that having been said, I will concede that there perhaps you could find some irreducible microkernel of “map” that actually corresponds to “territory”. I just don’t think it makes sense (on the understanding-people side) to worry about it. If you’re trying to understand what people want or how they’ll behave, the territory is absolutely the LAST place you should be looking. (Since the distinctions they’re using, and the meanings they attach to those distinctions, are 100% in the map.)
I don’t see how it supposed to follow from the fact that not everyone prefers not-being-decieved, that those who claim to prefer not-being-deceived must be wrong about their own preferences. Could you explain why you seem to think it does?
The claim others are defending here (as I understand it) is not that everyone’s preferences are really over the territory; merely that some people’s are. Pointing out that some people’s preferences aren’t about the territory isn’t a counterargument to that claim.
I’m saying that the preferences point to the map because your entire experience of reality is in the map—you can’t experience reality directly. The comments about people’s differences in not-being-deceived were just making the point that that preference is more about consensus reality than reality reaity. In truth, we all care about our model of reality, which we labeled reality and think is reality, but is actually not.
I’m afraid I have no idea what this is supposed to mean. It seems to me like you’re just repeating your conclusion over and over again using different words, which unfortunately doesn’t constitute an argument. Maybe to you it seems like we’re doing the same thing, I don’t know.
Alternatively, maybe we’re still talking past each other for the reasons suggested here (which everyone seemed to agree with at the time.) In which case, I wonder why we’re still having this conversation at all, and apologise for my part in pointlessly extending it. ;)
It’s probably because I replied to an unclosed subthread, causing an unintended resurrection. Also, at one point Vladimir Nesov did some resurrection too, and there have also been comments by Cyan and Saturn that kept things going.
Anyway, yes, as you said, we already agreed we are talking about different things, so let’s stop now. ;-)
If you agree that you are just talking about a different thing, and given that “utility” is a term understood to mean different thing from what you were talking about, kindly stop using that term for your separate concept to avoid unnecessary confusion and stop arguing about the sound of fallen tree.
Yay!
1) Why would people differ so much? Even concrete preferences don’t get reversed, magical mutants don’t exist.
2) Even if you only care about your map, you still care about your map as a part of the territory, otherwise you make the next step and declare that you don’t care about state of your brain either, you only care about caring itself, at which point you disappear in a “puff!” of metaphysical confusion. It’s pretty much inevitable.
...a “territory” that exists only in your brain, since you cannot directly perceive or operate upon the real territory.
...and note too, that all of this modeling is taking place in a brain. When you point to something and label it “reality”, you are pointing to a portion of your map. It is not, and cannot be the actual territory. It doesn’t matter how many times you try to say, “but no, I mean real reality”, because the only thing you have to represent that idea with is your brain.
It is still the idea of reality, a model of reality, that exists in your map.
Yes, the map is represented physically within the territory. The map “runs on” a platform made of territory, just as “you” run on a platform made of map. All I am saying is, “you” cannot access the territory directly, because your platform is made of map, just as the map’s platform is made of territory. Everything we perceive or think or imagine is therefore map… including the portion of the map we refer to as the territory.
We want our preferences to point to the territory, but they cannot do so in actual reality. We have the illusion that they can, because we have the same representative freedom as an artist drawing a picture of hands drawing each other—we can easily represent paradoxical and unreal things within the surface of the map.
(I remember reading at one point a tutorial in General Semantics that illustrated this point much better than I am doing, but sadly I cannot seem to find it at the moment.)
There’s a distinction to be made between the fact that our knowledge about whether our preferences are satisfied is map-bound and the assertion that our preferences only take the map into account.
You’ve just ignored Cyan’s counterexample, and presented a few of your own that support your point of view.
I answered it here, actually.
Write up your argument, make a top post, refer to it if it’s convincing. But guerilla arguing is evil: many words and low signal-to-noise.
I don’t understand you.
If you are operating under an assumption that nobody agrees with, you are wasting anyone’s time (assumption: map is about the map, territory be damned), as the argument never goes anywhere. As a compromise, compose your best argument as a top-level post (but only if you expect to convince at least someone).
That’s why I don’t understand you—I dropped this particular subthread for that very reason, but Cyan asked a second time for a reply. Otherwise, I’d have not said anything else in this particular subthread.
You could still write a meta-reply, taking that problem into account. The root of the disagreement can be stated in one line, and a succinct statement of at the moment unresolvable disagreement is resolution of an argument.
To be honest, I have not perceived anyone I’m speaking with to be treating this as an unresolvable conflict, besides you. Most of the other people have been saying things I perceive to mean, “I agree with such-and-such, but I am confused why you think that about this other case”, or “Ah, we are talking about different things in this area—how would you apply it to the thing I mean?”
You are the only one who appears to be simply stating dogma back at me, without seeking to understand where our maps do or do not overlap. (I’m not very quick on the OB URL citations, but ISTM that everything I’m saying about maps is consistent with EY’s models of reductionism, and with observable facts about how brains operate.)
You appear to have a pattern of responding to my descriptions of things (as I perceive them in my map, of course) as if they were attacks on your preferred prescriptions for how reality should be (in your map). It’s natural that this would lead to an impasse, since I am not actually disputing your opinions, and you are not actually objecting to my facts. Hence, we talk past each other.
(Btw, when I say “facts”, I mean “statements intended to be about actual conditions”, not “truths”. All models are false, but some are more useful than others.)
I’m just trying to be decisive in identifying the potential flaming patterns in the discussion. I could debate the specifics, but given my prior experience in debating stuff with you, and given the topics that could be debated in these last instances, I predict that the discussion won’t lead anywhere, and so I skip the debate and simply state my position, to avoid unnecessary text.
One way of stopping recurring thematic or person-driven flame wars (that kill Internet communities) is to require the sides to implement decent write-ups of their positions: even without reaching agreement, at some point there remains nothing to be said, and so the endless cycle of active mutual misunderstanding gets successfully broken.
I don’t understand how that’s supposed to work. If you don’t expect it to lead anywhere, why bother saying anything at all?
I’m registering the disagreement, and inviting you to sort the issue out for yourself, through reconsidering your position in response to apparent disagreement, or through engaging into a more constructive form of discussion.
This appears to be a one-way street. If applied consistently, it would seem that your first step would be to reconsider your position in response to apparent disagreement… or that I should reply by registering my disagreement—which implicitly I’d have already done.
Or, better yet, you would begin (as other people usually do) by starting the “more constructive form of discussion”, i.e., raising specific objections or asking specific questions to determine where the differences in our maps lie.
Preferences are computed by the map, but they are NOT about the map.
Oh, right. It says so right here… on the map! ;-)
Don’t take it lightly, it’s a well-vetted and well-understood position, extensively discussed and agreed upon. You should take such claims as strong evidence that you may have missed something crucial, that you need to go back and reread the standard texts.
It’s extensively discussed and agreed upon, that that is how we (for certain definitions of “we”) would like it to be, and it certainly has desirable properties for say, building Friendly AI, or any AI that doesn’t wirehead. And it is certainly a property of the human brain that it orients its preferences towards what it believes is the outside world—again, it has good consequences for preventing wireheading.
But that doesn’t make it actually true, just useful.
It’s also pretty well established as a tenet of e.g., General Semantics, that the “outside world” is unknowable, since all we can ever consciously perceive is our map. The whole point of discussing biases is that our maps are systematically biased—and this includes our preferences, which are being applied to our biased views of the world, rather than the actual world.
I am being descriptive here, not prescriptive. When we say we prefer a certain set of things to actually be true, we can only mean that we want the world to not dispute a certain map, because otherwise we are making the supernaturalist error of assuming that a thing could be true independent of the components that make it so.
To put it another way, if I say, “I prefer that the wings of this plane not fall off”, I am speaking about the map, since “wings” do not exist in the territory.
IOW, our statements about reality are about the intersection of some portion of “observable” reality and our particular mapping (division and labeling) of it. And it cannot be otherwise, since to even talk about it, we have to carve up and label the “reality” we are discussing.
It’s funny that you talk of wordplay a few comments back, as it seems that you’re the one making a technically-correct-but-not-practically-meaningful argument here.
If I may attempt to explore your position: Suppose someone claims a preference for “blue skies”. The wirehead version of this that you endorse is “I prefer experiences that include the perception I label ‘blue sky’”. The “anti-wirehead” version you seem to be arguing against is “I prefer actual states of the world where the sky is actually blue”.
You seem to be saying that since the preference is really about the experience of blue skies, it makes no sense to talk about the sky actually being blue. Chasing after external definitions involving photons and atmospheric scattering is beside the point, because the actual preference wasn’t formed in terms of them.
This becomes another example of the general rule that it’s impossible to form preferences directly about reality, because “reality” is just another label on our subjective map.
As far as specifics go, I think the point you make is sound: Most (all?) of our preferences can’t just be about the territory, because they’re phrased in terms of things that themselves don’t exist in the territory, but at best simply point at the slice of experience labeled “the territory”.
That said, I think this perspective grossly downplays the practical importance of that label. It has very distinct subjective features connecting in special ways to other important concepts. For the non-solipsists among us, perhaps the most important role it plays is establishing a connection between our subjective reality and someone else’s. We have reason to believe that it mediates experiences we label as “physical interactions” in a manner causally unaffected by our state of mind alone.
When I say “I prefer the galaxy not to be tiled by paperclips”, I understand that, technically, the only building blocks I have for that preference are labeled experiences and concepts that aren’t themselves the “stuff” of their referents. In fact, I freely admit that I’m not exactly sure what constitutes “the galaxy”, but the preference I just expressed actually contains a massive number of implicit references to other concepts that I consider causally connected to it via my “external reality” label. What’s more, most people I communicate with can easily access a seemingly similar set of connections to their “external reality” label, assuming they don’t talk themselves out of it.
The territory concept plays a similar role to that of an opaque reference in a programming language. Its state may not be invariant, but its identity is. I don’t have to know any true facts concerning its actual structure for it to be meaningful and useful. Just as photons aren’t explicitly required to subjectively perceive a blue sky, the ontological status of my territory concept doesn’t really change its meaning or importance, which is acquired through its intimate connection to massive amounts of raw experience.
Claiming my preferences about the territory are really just about my map is true in the narrow technical sense that it’s impossible for me to refer directly to “reality”, but doing so completely glosses over the deep, implicit connections expressed by such preferences, most primarily the connection between myself and the things I label “other consciousnesses”. In contrast, the perception of these connections seems to come for free by “confusing” the invariant identity of my territory concept with the invariant “existence” of a real external world. The two notions are basically isomorphic, so where’s the value in the distinction?
Think of what difference is there between “referring directly” to the outside reality and “referring directly” to the brain. Not much, methinks. There is no homunculus whose hands are only so long to reach the brain, but not long enough to touch your nose.
Agreed, as the brain is a physical object. Referring “directly” to subjective experiences is a different story though.
That depends on whether you’re talking about “blue” in terms of human experience, or whether you’re talking about wavelengths of light. The former is clearly “map”, whereas discussing wavelengths of light at least might be considered “about” the territory in some sense.
However, if you are talking about preferences, I don’t think there’s any way for a preference to escape the map. We can define how the mapped preference relates to some mapped portion of the territory, but preferences (on human hardware at least) can only be “about” experience.
And that’s because all of our hypothetical preferences for states of the actual world are being modeled as experiences in order to compute a preference.
The entire context of this discussion (for me, anyway) has been about preferences. That doesn’t preclude the possibility of relating maps to territory, or maps to other people’s maps. I’m just saying that human beings have no way to model their preferences except by modeling experiences, in human-sensory terms.
But when this statement is uttered by a human, it is almost invariably a lie. Because inside, the person’s model is an experience of “blue sky”. The preference is about experiences, not the state of the world.
Even if you phrase this as, “I prefer the sky to be actually blue, even if I don’t know it”, it is still a lie, because now you are modeling an experience of the sky being blue, plus an experience of you not knowing it.
Well, it makes clear some of the limits of certain endeavors that are often discussed here. It dissolves confusions about the best ways to make people happy, and whether a world should be considered “real” or “virtual”, and whether it’s somehow “bad” to be virtual.
But the most important practical benefit is that it helps in understanding why sometimes the best thing for a person may be to update their preferences to match the constraints of reality, rather than trying in vain to make reality fit their preferences.
Consider the “blue sky” preference. The experience of a blue sky is likely associated with, and significantly colored by things like the person’s favorite color, the warmth of the sun or the cool breeze of that perfect day one summer when they were in love. For another person, it might be associated with the blinding heat of the desert and a sensation of thirst… and these two people can then end up arguing endlessly about whether a blue sky is obviously good or bad.
And both are utterly deluded to think this their preferences have anything to do with reality.
I am not saying that they disagree about how many angstroms the light of a blue sky is, or that the sky doesn’t really exist or anything like that. I’m saying their preference is (and can only be) about their maps, because the mere fact of a blue sky has no inherent “preferability”, without reference to some other purpose.
Even if we try to say in the abstract that it is good because it’s correlated with things about our planet that make life possible, we can only have that preference because we’re modeling an experience of “life” that we label “good”. (And perhaps a depressed teenager would disagree, using a differently-labeled experience of “life”!)
This does not mean preferences are invalid or meaningless. It doesn’t mean that we should only change our preferences and ignore reality. However, to the extent that our preferences produce negative experiences, it is saner to remove the negative portion of the preference.
Luckily, human beings are not limited to, or required to have, bidirectional preferences. Feeling pain at the absence of something is not required in order to experience pleasure at its presence, in other words. (Or vice versa.)
Awareness of this fact, combined with an awareness that it is really the experience we prefer (and mainly, the somatic markers we have attached to the experience) makes it plain that the logical thing to do is to remove the negative label, and leave any positive labels in place.
However, if we think that our internal labeling of experience has something to do with “reality”, then we are likely to engage in the confusion of thinking that removing a negative label of badness will somehow create or prolong badness in the territory.
And for that matter, we may be under the mistaken impression that changing the reality out there will change our experience of it… and that is often not the case. As the saying goes, if you want to make a human suffer, you can either not give them what they want, or else you can give it to them! Humans tend to create subgoals that get promoted to “what I want” status, without reference to what the original desired experience was.
For example, looking for blue skies...
When what you really want is to fall in love again.
Wow, −5! People here don’t seem to appreciate this sort of challenge to their conceptual framework.
I agree, but I wonder if I failed to communicate the distinction I was attempting to make. The human-sensory experience of being embedded in a concrete, indifferent reality is (drugs, fantasies, and dreams aside) basically constant. It’s a fundamental thread underlying our entire history of experience.
It’s this indifference to our mental state that makes it special. A preference expressed in terms of “reality” has subjective properties that it would otherwise lack. Maybe I want the sky to be blue so that other people will possess a similar experience of it that we can share. “Blueness” may still be a red herring, but my preference now demands some kind of invariant between minds that seemingly cannot be mediated except through a shared external reality. You might argue that I really just prefer shared experiences, but this ignores the implied consistency between such experiences and all other experiences involving the external reality, something I claim to value above and beyond any particular experience.
This is where the massive implicit context enters the scene. “Even if I don’t know it” is modeled after experience only in the degenerate sense that it’s modeled after experience of indifferent causality. A translation might look like “I prefer to experience a reality with the sorts of consequences I would predict from the sky being blue, even if I don’t consciously perceive blue skies”. That’s still an oversimplification, but it’s definitely more complex than just invoking a generic memory of “not having known something” and applying it to blue skies.
I don’t see how any of that is true. I can easily think of different concrete realizations of “real” and “virtual” that would interact differently with my experience of reality, thus provoking different labellings of “good” and “bad”. If your point is merely that “real” is technically underspecified, then I agree. But I don’t see how you can draw inferences from this underspecification.
I’m going to have to turn your own argument against you here. To the extent that you have a concept of reality that is remotely consistent with your everyday experience, I claim that “in reality, blue skies are bad because they provoke suffering” is a preference stated in terms of an extremely similar reality-concept, plus a suffering-concept blended together from first-hand experience and compassion (itself also formed in terms of reality-as-connected-to-other-minds). For you to say it has “nothing to do with reality” is pure semantic hogwash. What definition of “reality” can you possibly be using to make this statement, except the one formed by your lifetime’s-worth of experience with indifferent causality? You seem to be denying the use of the term to relate your concept of reality to mine, despite their apparent similarity.
This doesn’t make sense to me. Whether or not an experience is “negative” is a function of our preferences. If a preference “produces” negative experiences, then either they’re still better than the alternative (in which case it’s a reasonable preference, and it’s probably worthwhile to change your perception of the experience) or they’re not (in which case it’s not a true preference, just delusion).
That’s a property of pain and pleasure, not preference. I may well decide not to feel pain due to preference X being thwarted, but I still prefer X, and I still prefer pleasure to the absence of pleasure.
This is where I think your oversimplification of “experience vs reality” produces invalid conclusions. Those labels don’t just apply to one experience or another, they apply to a massively complicated network of experience that I can’t even begin to hold in my mind at once. Given that, your logic doesn’t follow at all, because I really don’t know what I’m relabeling.
This relates to a general reservation I have with cavalier attitudes toward mind-hacks: I know full well that my preferences are complex, difficult to understand, and grossly underspecified in any conscious realization, so it’s not at all obvious to me that optimizing a simple preference concerning one particular scenario doesn’t carry loads of unintended consequences for the rest of them. I’ve had direct experience with my subconsciously directed behavior “making decisions for me” that I had conscious reasons to optimize against, only later to find out that my conscious understanding of the situation was flawed and incomplete. I think that ignoring the intuitive implications of an external reality leads to similar contradictions.
You seem to mostly be arguing against a strawman; as I said, I’m not saying reality doesn’t exist or that it’s not relevant to our experiences. What I’m saying is that the preferences are composed of map, and while there are connections between that map and external reality, we are essentially deluded to think our preferences refer to actual reality, and that this delusion leads us to believing that changing external reality will change our internal experience, when more often the reverse is more likely true. (That is, changing our internal experience will more likely result in our taking actions that will actually change external reality.)
Note, however that:
Here you seem to be arguing my point. The experience is a function of preferences, the preferences are a product of, and point to, other experiences, in a self-sustaining loop that sometimes might as well not be connected to outside reality at all, for all that it has anything to do with what’s actually going on.
Lucky people live in a perpetual perception of good things happening, unlucky people the opposite, even when the same events are happening to both.
How can we say, then, that either person’s perceptions are “about” reality, if they are essentially unconditional? Clearly, something else is going on.
If we disagree at this point, I’d have to say it can only be because we disagree on what “about” means. When I say preferences are not “about” reality, it is in the same sense that Robin Hanson is always saying that politics is not “about” policy, etc.
Clearly, preferences are “about” reality in the same sense that politics are “about” policy. That is, reality is the subject of a preference, in the same way that a policy might be the subject of a political dispute.
However, in both cases, the point of the ostensible activity is not where it appears to be. In order for politics to function, people must sincerely believe that it is “about” policy, in precisely the same way as we must sincerely believe our preferences are “about” reality, in order to make them function—and for similar reasons.
But in neither case does either the sincerity or the necessity of the delusion change the fact that it’s nonetheless a delusion.
I don’t think I disagree with any of the above, except to dispute its full universality (which I’m not sure you’re even arguing). To attempt to rephrase your point: Our interactions with reality create experiences filtered through our particular way of characterizing such interactions. It’s these necessarily subjective characterizations (among other things), rather than the substance of the interaction itself, which generate our preferences. When reflecting on our preferences, we’re likely to look right past the interpretive layer we’ve introduced and attribute them to the external stimulus that produced the response, rather than the response itself.
Robin’s “X is not about Y” has the flavor of general, but not universal, rules. Would you extend your analogy to include this property?
Here’s an interesting question for you: why is it important that you consider this non-universal? What value does it provide you for me to concede an exception, or what difference will it make in your thinking if I say “yes” or “no”? I am most curious.
(Meanwhile, I agree with your summation as an accurate, if incomplete restatement of the bulk of my point.)
Because I’m trying to make sense of your position, but I don’t think I can with such a strict conclusion. I don’t see any fundamental reason why someone couldn’t form preferences more or less directly mediated by reality, it just seems that in practice, we don’t.
If you’re asking why I’m bringing up universality, it seemed clear that your claims about preferences were universal in scope until you brought up “X is not about Y”. “Must logically be” and “tends to be in practice” are pretty different types of statement.
You didn’t answer my questions.
I mean, you said some things that sound like answers, but they’re not answers to the questions I asked. Here they are again:
and
Your almost-answer was that you don’t think you can “make sense” of my position with a strict conclusion. Why is that? What would it mean for there to be a strict conclusion? How, specifically, would that be a problem?
I didn’t answer this because it’s predicated on an assumption that has no origin in the conversation. I never claimed that it was “important” for me to consider this non-universal. As per being “liberal in what I accept” in the realm of communication, I tried to answer the nearest meaningful question I thought you might actually be asking. I thought the phrase “If you’re asking why I’m bringing up universality” made my confusion sufficiently clear.
If you really do mean to ask me why I think it’s important that I believe in some property of preference formation, then either I’ve said something fairly obvious to that end that I’m not remembering (or finding), or you’re asserting your own inferences as the basis of a question, instead of its substance. I try to give people the benefit of the doubt that I’ve misunderstood them in such cases, rather than just assume they’re speaking manipulatively.
No particular value in mind. I suppose the greatest value would be in you solidly refuting such exceptions in a way that made sense to me, as that would be a more surprising (therefore more informative) outcome. If you the concede the exception, I don’t gain any additional insight, so that’s of fairly neutral value.
Not really sure yet, especially in the “no” case (since in that case you may have reasons I haven’t yet thought of or understood). I suppose in the “yes” case I’d have greater confidence that I knew what you were talking about if I encountered similar concepts in your comments elsewhere. This discussion has had some difference on my thinking: I don’t think I understood the thrust of your point when I originally complained that your distinction lacked relevance.
By strict conclusion, I mean “preferences are modeled strictly in terms of the map: it is logically impossible to a hold preference expressed in terms of something other than that which is expressed in the map”. This seems very nearly true, but vulnerable to counterexamples when taken as a general principle or logical result of some other general principle. I’ll elaborate if you’d like, but I thought I’d clarify that you meant it that way. If you didn’t, theoretical or speculative counter-examples aren’t particularly relevant.
I can imagine that, in principle, some other sort of mind than a human’s might be capable of being a counterexample, apart from, say, the trivial example of a thermostat, which shows a “preference” for reality being a certain way. An AI could presumably be built so that its preferences were based on properties of the world, rather than properties of its experience, or deduction from other properties based on experience. However, at some point that would need to be rooted in the goal system provided by its programmers… who presumably based it off of their own preferences.… ;-) (Nonetheless, if the AI didn’t have anything we’d label “experience”, then I’d have to agree that it has a preference about reality, rather than its experience of reality.)
I could also consider an argument that, say, hunger is about the state of one’s stomach, and that it therefore is “about” the territory, except that I’m not sure hunger qualifies as a preference, rather than an appetite or a drive. A person on a hunger strike or with anorexia still experiences hunger, yet prefers not to eat.
If you think you have other counterexamples, I’d like to hear them. I will be very surprised if they don’t involve some rather tortured reasoning and hypotheticals, though, or non-human minds. The only reason I even hedge my bets regarding humans is that (contrary to popular belief) I’m not under the mistaken impression that I have anything remotely approaching a complete theory of mind for human brains, versus a few crude maps that just happen to cover certain important chunks of “territory”. ;-)
I don’t actually consider this a good counterexample. It can been trivially shown that the thermostat’s “preference” is not in terms of the “reality” of temperature: Just sabotage the sensor. The thermostat “prefers” its sensor reading to correspond to its set point. Wouldn’t you agree this is fairly analogous to plenty of human desires?
Agreed. The closest it seems you could come is to prefer satiation of said appetites, which is a subjective state.
Actually, human minds are the primary source of my reservations. I don’t think my reasoning is particularly tortured, but it certainly seems incomplete. Like you, I really have no idea what a mind is.
That said, I do seem to have preferences that concern other minds. These don’t seem reducible to experiences of inter-personal behavior… they seem largely rooted in the empathic impulse, the “mirror neurons”. Of course, on its face, this is still just built from subjective experience, right? It’s the the experience of sympathetic response when modeling another mind. And there’s no question that this involves substituting my own experiences for theirs as part of the modeling process.
But when I reflect on a simple inter-personal preference like “I’d love for my friend to experience this”, I can’t see how it really reduces to pure experience, except as mediated by my concept of invariant reality. I don’t have a full anticipation of their reaction, and it doesn’t seem to be my experience of modeling their interaction that I’m after either.
Feel free to come up with a better explanation, but I find it difficult to deconstruct my desire to reproduce internally significant experiences in an external environment in a way that dismisses the role of “hard” reality. I can guess at the pre-reflective biological origin of this sort of preference, just like we can point at the biological origin of domesticated turkeys, but, just as turkeys can’t function without humans, I don’t know how it would function without some reasonable concept of a reality that implements things intrinsically inaccessible and indifferent to my own experience.
I chose to instantiate this particular example, but the general rule seems to be: The very fabric of what “another mind” means to me involves the concept of an objective but shared reality. The very fabric of what “another’s experiences” means to me involves the notion of an external system giving rise to external subjective experiences that bear some relation to my own.
You could claim my reasoning is tortured in that it resembles Russel’s paradox: One could talk about the set of all subjective preferences explicitly involving objective phenomena (i.e., not containing themselves). But it seems to me that I can in a sense relate to a very restricted class of objective preferences, those constructed from the vocabulary of my experience, reflected back into the world, and reinstantiated in the form of another mind.
Another simple example: Do you think a preference for honest communication is at all plausible? Doesn’t it involve something beyond “I hope the environment doesn’t trick me”?
Right. And don’t forget the mind-projection machinery, that causes us to have, e.g. different inbuilt intuitions about things that are passively moved, move by themselves, or have faces that appear to express emotion. These are all inbuilt maps in humans.
Most of us learn by experience that sharing positive experiences with others results in positive attention. That’s all that would be needed, but it’s also likely that humans have an evolved appetite to communicate and share positive experiences with their allies.
It just means you prefer one class of experiences to another, that you have come to associate with other experiences or actions coming before them, or co-incident with them.
The reason, btw, that I asked why it made a difference whether this is an absolute concept or a “mostly” concept, is that AFAICT, the idea that “some preferences are really about the territory” leads directly to “therefore, all of MY preferences are really about the territory”.
In contrast, thinking of all preferences being essentially delusional is a much better approach, especially if 99.999999999% of all human preferences are entirely about the map, if we presume that maybe there are some enlightened Zen masters or Beisutsukai out there who’ve successfully managed, against all odds, to win the epistemic lottery and have an actual “about the territory” preference.
Even if the probability of having such a preference were much higher, viewing it as still delusional with respect to “invariant reality” (as you call it) does not introduce any error. So the consequences of erring on the side of delusion are negligible, and there is a significant upside to being more able to notice when you’re looping, subgoal stomping, or just plain deluded.
That’s why it’s of little interest to me how many .9′s there are on the end of that %, or whether in fact it’s 100% - the difference is inconsequential for any practical purpose involving human beings. (Of course, if you’re doing FAI, you probably want to do some deeper thinking than this, since you want the AI to be just as deluded as humans are, in one sense, but not as deluded in another.)
For the love of Bayes, NO. The people here are generally perfectly comfortable with the realization that much of their altruism, etc. is sincere signaling rather than actual altruism. (Same for me, before you ask.) So it’s not necessary to tell ourselves the falsehood that all of our preferences are only masked desires for certain states of mind.
As for your claim that the ratio of signaling to genuine preference is 1 minus epsilon, that’s a pretty strong claim, and it flies in the face of experience and certain well-supported causal models. For example, kin altruism is a widespread and powerful evolutionary adaptation; organisms with far less social signaling than humans are just hardwired to sacrifice at certain proportions for near relatives, because the genes that cause this flourish thereby. It is of course very useful for humans to signal even higher levels of care and devotion to our kin; but given two alleles such that
(X) makes a human want directly to help its kin to the right extent, plus a desire to signal to others and itself that it is a kin-helper, versus
(X’) makes a human only want to signal to others and itself that it is a kin-helper,
the first allele beats the second easily, because the second will cause searches for the cheapest ways to signal kin-helping, which ends up helping less than the optimal level for promoting those genes.
Thus we have a good deal of support for the hypothesis that our perceived preferences in some areas are a mix of signaling and genuine preferences, and not nearly 100% one or the other. Generally, those who make strong claims against such hypotheses should be expected to produce experimental evidence. Do you have any?
That’s nice, but not relevant, since I haven’t been talking about signaling.
Given that, I’m not going to go through the rest of your comment point by point, as it’s all about signaling and kin selection stuff that doesn’t in any way contest the idea that “preference is about experiences, not the reality being experienced”.
I don’t disagree with what you said, it’s just not in conflict with the main idea here. When I said that this is like Hanson’s “politics are not about policy”, I didn’t mean that it was therefore about signaling! (I said it was “not about” in the same way, not that it was about in the same way—i.e., that the mechanism of delusion was similar.)
The way human preferences work certainly supports signaling functions, and may be systematically biased by signaling drives, but that’s not the same thing as saying that preferences equal signaling, or that preferences are “about” signaling.
Well, this discussion might not be useful to either of us at this point, but I’ll give it one last go. My reason for bringing in talk of signaling is that throughout this conversation, it seems like one of the claims you have been making is that
The algorithm (more accurately, the collection of algorithms) that constitutes me makes its decisions based on a weighting of my current and extrapolated states of mind. To the extent that I perceive preferences about things that are distinct from my mental states (and especially when confronting thought-experiments in which my mental states will knowably diverge from the mental states I would ordinarily form given certain features of the world), I am deceiving myself.
Now, I brought up signaling because I and many others already accept a form of (A), in which we’ve evolved to deceive others and ourselves about our real priorities because such signalers appear to others to be better potential friends, lovers, etc. It looks perfectly meaningful to me to declare such preferences “illusory”, since in point of fact we find rationalizations for choosing not what we signaled we prefer, but rather the least costly available signs of these ‘preferences’.
However, kin altruism appears to be a clear case where not all action is signaling, where making decisions that are optimized to actually benefit my relatives confers an advantage in total fitness to my genes.
While my awareness and my decisions exist on separate tracks, my decisions seem to come out as they would for a certain preference relation, one of whose attributes is a concern for my relatives’ welfare. Less concern, of course, than I consciously think I have for them; but roughly the right amount of concern for Hamilton’s Rule of kin selection.
My understanding, then, is that I have both conscious and real preferences; the former are what I directly feel, but the latter determine parts of my action and are partially revealed by analysis of how I act. (One component of my real preferences is social, and even includes the preference to keep signaling my conscious preferences to myself and others when it doesn’t cost me too much; this at least gives my conscious preferences some role in my actions.) If my actions predictably come out in accordance with the choices of an actual preference relation, then the term “preference” has to be applied there if it’s applied anywhere.
There’s still the key functional sense in which my anticipation of future world-states (and not just my anticipation of future mind-states) enters into my real preferences; I feel an emotional response now about the possibility of my sister dying and me never knowing, because that is the form that evaluation of that imagined world takes. Furthermore, the reason I feel that emotional response in that situation is because it confers an advantage to have one’s real preferences more finely tuned to “model of the future world” than “model of the future mind”, because that leads to decisions that actually help when I need to help.
This is what I mean by having my real preferences sometimes care about the state of the future world (as modeled by my present mind) rather than just my future experience (ditto). Do you disagree on a functional level; and if so, in what situation do you predict a person would feel or act differently than I’d predict? If our disagreement is just about what sort of language is helpful or misleading when taking about the mind, then I’d be relieved.
The confusion that you have here is that kin altruism is only “about” your relatives from the outside of you. Within the map that you have, you have no such thing as “kin altruism”, any more than a thermostat’s map contains “temperature regulation”. You have features that execute to produce kin altruism, as a thermostat’s features produce temperature regulation. However, just as a thermostat simply tries to make its sensor match its setting, so too do your preferences simply try to keep your “sensors” within a desired range.
This is true regardless of the evolutionary, signaling, functional, or other assumed “purposes” of your preferences, because the reality in which those other concepts exist, is not contained within the system those preferences operate in. It is a self-applied mind projection fallacy to think otherwise, for reasons that have been done utterly to death in my interactions with Vladimir Nesov in this thread. If you follow that logic, you’ll see how preferences, aboutness, and “natural categories” can be completely reduced to illusions of the mind projection fallacy upon close examination.
Well, if this is just a disagreement over whether our typical uses of the word “about” are justified, then I’m satisfied with letting go of this thread; is that the case, or do you think there is a disagreement on our expectations for specific human thoughts and actions?
I suggest, by the way, that your novel backwards application of the Mind Projection Fallacy needs its own name so as not to get it confused with the usual one. (Eliezer’s MPF denotes the problem with exporting our mental/intentional concepts outside the sphere of human beings; you seem to be asserting that we imported the notion of preferences from the external world in the first place.)
No. I’m saying that the common ideas of “preference” and “about” are mind projection fallacies, in the original sense of the phrase (which Eliezer did not coin, btw, but which he does use correctly). Preference-ness and about-ness are qualities (like “sexiness”) that are attributed as intrinsic properties of the world, but to be properly specified must include the one doing the attribution.
IOW, for your preferences to be “about” the world, there must be someone who is making this attribution of aboutness, as the aboutness itself does not exist in the territory, any more than “sexiness” exists in the territory.
However, you cannot make this attribution, because the thing you think of as “the territory” is really only your model of the territory.
This can be viewed as purely a Russellian argument about language levels, but the practical point I originally intended to make was that humans cannot actually make preferences about the actual territory because the only thing we can evaluate are our own experiences—which can be suspect. Inbuilt drives and biases are one source of experiences being suspect, but our own labeling of experiences is also suspect—labels are not only subject to random linkage, but are prone to spreading to related topics in time, space, or subject matter.
It is thus grossly delusional as a practical matter to assume that your preferences have anything to do with actual reality, as opposed to your emotionally-colored, recall-biased associations with imagined subsets of half-remembered experiences of events that occurred under entirely different conditions. (Plus, many preferences subtly lead to the recreation of circumstances that thwart the preference’s fulfillment—which calls into question precisely what “reality” that preference is about.)
Perhaps we could call our default thinking about such matters (i.e. preferences being about reality) “naive preferential realism”, by analogy to “naive moral realism”, as it is essentially the same error, applied to one’s own preferences rather than some absolute definition of good or evil.
This is pretty much what I meant by a semantic argument. If, as I’ve argued, my real preferences (as defined above) care about the projected future world (part of my map) and not just the projected future map (a sub-part of that map), then I see no difficulty with describing this by “I have preferences about the future territory”, as long as I remain aware that all the evaluation is happening within my map.
It is perhaps analogous to moral language in that when I talk about right and wrong, I keep in mind that these are patterns within my brain (analogous to those in other human brains) extrapolated from emotive desires, rather than objectively perceived entities. But with that understanding, right and wrong are still worth thinking about and discussing with others (although I need to be quite careful with my use of the terms when talking with a naive moral realist), since these are patterns that actually move me to act in certain ways, and to introspect in certain ways on my action and on the coherence of the patterns themselves.
In short, any theory of language levels or self-reference that ties you in Hofstadterian knots when discussing real, predictable human behavior (like the decision process for kin altruism) is problematic.
That said, I’m done with this thread. Thanks for an entertainingly slippery discussion!
ETA: To put it another way, learning about the Mind Projection Fallacy doesn’t mean you can never use the word “sexy” again; it just means that you should be aware of its context in the human mind, which will stop you from using it in certain novel but silly situations.
Consider the difference between a thermostat connected to a heater and a human maintaining the same temperature by looking at a thermometer and switching the heater on and off. Obviously there is a lot more going on inside the human’s brain, but I still don’t understand how the thermostat has any particular kind of connection to reality that the human lacks. The same applies whether the thermostat was built by humans with preferences or somehow formed without human design.
edit: I’m not trying to antagonize you, but I genuinely can’t tell whether you are trying to communicate something that I’m not understanding, or you’ve just read The Secret one too many times.
The thermostat lacks the ability to reflect on itself, as well as the mind-projection machinery that deludes human beings into thinking that their preferences are “about” the reality they influence and are influenced by.
You’re definitely rounding to a cliche. The Secret folks think that our preferences create the universe, which is just as delusional as thinking our preferences are about the universe.
You don’t understand how something can be about something else, but declare it meaningless.
Doesn’t it rather have a preference for its sensors showing a certain reading? (This doesn’t lead to thermostat wireheading because the thermostat’s action won’t make the sensor alter its mechanism.)
Really, it’s only systems that can model a scenario where its sensors say X but the situation is actually Y, that could possibly have preferences that go beyond the future readings of its sensors. If you assert that a thermostat can have preferences about the territory but a human can’t, then you are twisting language to an unhelpful degree.
Whether your preferences refer to your state, or to the rest of the world is indeed a wirehead-related issue. The problem with the idea that they refer to your state is that that idea tends to cause wirehead behaviour—surgery on your own brain to produce the desired state. So—it seems desirable to construct agents that believe that there is a real world, and that their preferences relate to it.
I agree—that’s probably why humans appear to be constructed that way. The problem comes in when you expect the system to also be able to accurately reflect its preferences, as opposed to just executing them.
This does not preclude the possibility of creating systems that can; it’s just that they’re purely hypothetical.
To the greatest extent practical, I try to write here only about what I know about the practical effects of the hardware we actually run on today, if for no other reason than if I got into entirely-theoretical discussions I’d post WAY more than I already do. ;-)
Presumably, if you asked such an agent to reflect on its own purposes, it would claim that they related to the external world (unless it’s aim was to deceive you about its purposes for signalling reasons, of course).
For example, it might claim that its aim was to save the whales—rather than to feel good about saving the whales. It could do the latter by taking drugs or via hypnotherapy—and that is not how it actually acts.
Actually, if signaling was its true purpose, it would claim the same thing. And if it were hacked together by evolution to be convincing, it might even do so by genuinely believing that its reflections were accurate. ;-)
Indeed. But in the case of humans, note first that many people do in fact take drugs to feel good, and second, that we tend to dislike being deceived. When we try to imagine getting hypnotized into believing the whales are safe, we react as we would to being deceived, not as we would if we truly believed the whales were safe. It is this error in the map that gives us a degree of feed-forward consistency, in that it prevents us from certain classes of wireheading.
However, it’s also a source of other errors, because in the case of self-fulfilling beliefs, it leads to erroneous conclusions about our need for the belief. For example, if you think your fear of being fired is the only thing getting you to work at all, then you will be reluctant to give up that fear, even if it’s really the existence of the fear that is suppressing, say, the creativity or ambition that would replace the fear.
In each case, the error is the same: System 2 projection of the future implicitly relies on the current contents of System 1′s map, and does not take into account how that map would be different in the projected future.
(This is why, by the way, The Work’s fourth question is “who would you be without that thought?” The question is a trick to force System 1 to do a projection using the presupposition that the belief is already gone.)
It’s possible that the confusion here is (yet again) due to people using the word utility to mean different things. If PJ is using utility to refer to, e.g. an emotional or cognitive state, then he’s right that our utility cannot respond directly to the territory. But broader notions of utility, well-being, and preference are possible, and nothing PJ has said is especially relevant to whether they are coherent or not.
Right. I don’t dabble in discussing those broader notions, though, since they can’t be empirically grounded. How can you test a concept of utility that’s not grounded in human perception and emotion? What good can it ever do you if you can’t connect it back to actual living people?
I consider such discussions to be much more irrational than, say, talk of “The Secret”, which at least offers an empirical procedure that can be tested. ;-)
(In fairness, I do consider such discussions here on LW to be far less annoying than most discussions of the Secret and suchlike!)
These notions are about what it means for something to be good for “actual living people”. They’re difficult, if not impossible to “test” (about the best testing procedures we’ve come up with is thought experiments, which as discussed elsewhere are riddled with all sorts of problems). But it’s not like you can “test” the idea that positive emotions are good for you either.
I thought this was well established scientifically, if by “good for you”, you mean health, persistence, or success in general. (see e.g. Seligman)
The argument is precisely about what “good for you” means, so this would be assuming the conclusion that needs to be established.
Ow. That makes my head hurt. (See, that’s why I try not to get into these discussions!)
(I’m hard pressed, though, to conceive of a moral philosophy where improved health would not be considered “good for you”.)
Preference utilitarianism applied to someone who thinks that it is only through suffering that life can achieve meaning.
To be clear, I don’t subscribe to such a view myself, but it’s conceivable. I agree with you that health is good for people. My point is just that this agreement owes more to shared intuition than conclusive empirical testing.
Yes, but now we’re back to concrete feelings of actual people again. ;-)
Right, which is one reason why, when we’re talking about this particular tiny (but important) domain (that at least partially overlaps with Eliezer’s notion of Fun Theory), conclusive empirical testing is a bit of a red herring, since the matter is subjective from the get-go. We can objectively predict certain classes of subjective events, but the subjectivity itself seems to be beyond that. At some point, you have to make an essentially arbitrary decision of what to value.
Preference utilitarianism applied to someone who thinks that it is only through suffering that life can achieve meaning.
(To be clear, I don’t subscribe to this view; but it is conceivable.)
Ah, right. Good call.