It seems, perhaps, that your main point is that usefulness can come apart from correspondence:
Great. So the rest of the argument follows: the existence of social constructs means that there is more to usefulness than correspondence to reality.
The argument against your point is that scientifically inaccurate maps can have, other, compensatory, kinds of usefulness. You haven’t refuted that.
I don’t believe that Zack disagreed with this? Indeed, Zack mentions several examples where the two come apart:
There is an important difference between “not including mountains on a map because it’s a political map that doesn’t show any mountains” and “not including Mt. Everest on a geographic map, because my sister died trying to climb Everest and seeing it on the map would make me feel sad.”
There is an important difference between “identifying this pill as not being ‘poison’ allows me to focus my uncertainty about what I’ll observe after administering the pill to a human (even if most possible minds have never seen a ‘human’ and would never waste cycles imagining administering the pill to one)” and “identifying this pill as not being ‘poison’, because if I publicly called it ‘poison’, then the manufacturer of the pill might sue me.”
These are both examples where “useful” is importantly different from “corresponds to reality”.
He’s disagreeing with someone over something. In think my point is the same as Scott’s, and he seems to be responding to Scott.
Edit:
If you read back, I’m responding to the point that:
”...in order for your map to be useful in the service of your values, it needs to reflect the statistical structure of things in the territory—which depends on the territory, not your values.”
That’s a pretty clear rejection of useful-but-not-corresponding even if there are examples of useful-but-not-corresponding further down.
These are both examples where “useful” is importantly different from “corresponds to reality”.
Yes, but they are examples with negative connotations.
If you read back, I’m responding to the point that: ”...in order for your map to be useful in the service of your values, it needs to reflect the statistical structure of things in the territory—which depends on the territory, not your values.”
That’s fair.
Yes, but they are examples with negative connotations.
I also agree with the negative connotations. There’s something special, worth defending, about epistemics focusing only on reflecting the territory, screening off other considerations as much as possible.
There’s something special, worth defending, about epistemics focusing only on reflecting the territory, screening off other considerations as much as possible.
That’s quite a vague claim. Are you saying that realistic epistemology is special in some sense that it should be applied to everything, or that everything should be reduced to it?
I’m saying that epistemics focused on usefulness-to-predicting is broadly useful in a way that epistemics optimized in other ways is not. It is more trustworthy in that the extent to which it’s optimized for some people at the expense of other people must be very limited. (Of course it will still be more useful to some people than others, but the Schelling-point-nature means that we tend to take it as the gold standard against which other things are judged as “manipulative”.)
Another defense of this Schelling point is that as we depart from it, it becomes increasingly difficult to objectively judge whether we are benefiting or hurting as a result. We get a web of contagious lies spreading through our epistemology.
I’m not saying this is a Schelling fence which has held firm through the ages, by any means; indeed, it is rarely held firm. But, speaking very roughly and broadly, this is a fight between “scientists” and “politicians” (or, as Benquo has put it, between engineers and diplomats).
I’m saying that epistemics focused on usefulness-to-predicting is broadly useful in a way that epistemics optimized in other ways is not
That’s still not very clear. As opposed to other epistemics being useless, or as opposed to other epistemics having specialized usefulness?
It is more trustworthy in that the extent to which it’s optimized for some people at the expense of other people must be very limited.
Why assume it’s necessarily conflictual and zero sum? For one thing, there’s a lot of social constructs and unscientific semantics out there.
the gold standard against which other things are judged as “manipulative”.)
Why assume anything unscientific is manipulative?
We get a web of contagious lies spreading through our epistemology.
If you are going to use a contagion metaphor, why not use an immune system metaphor? Which would be a metaphor for critical thinking.
Another defense of this Schelling point is that as we depart from it
We were never there!
ETA: I don’t buy that a unscientific concept is necessarily a lie, but even so, if lies are contagious, and no process deletes them, then we should already be in a sea of lies.
But, speaking very roughly and broadly, this is a fight between “scientists” and “politicians” (or, as Benquo has put it, between engineers and diplomats).
Why? Science and politics do not have to fight over the same territory.
>I’m saying that epistemics focused on usefulness-to-predicting is broadly useful in a way that epistemics optimized in other ways is not
That’s still not very clear. As opposed to other epistemics being useless, or as opposed to other epistemics having specialized usefulness?
What I meant by “broadly useful” is, having usefulness in many situations and for many people, rather than having usefulness in one specific situation or for one specific person.
For example, it’s often more useful to have friends who optimize their epistemics mostly based on usefulness-for-predicting, because those beliefs are more likely to be useful to you as well, rather than just them.
In contrast, if you have friends who optimize their beliefs based on a lot of other things, then you will have to do more work to figure out whether those beliefs are useful to you as well. Simply put, their beliefs will be less trustworthy.
Scaling up from “friends” to “society”, this effect gets much more pronounced, so that in the public sphere we really have to ask who benefits from claims/beliefs, and uncontaminated beliefs are much more valuable (so truly unbiased science and journalism are quite valuable as a social good).
Similarly, we can go to the smaller scale of one person communicating with themselves over time. If you optimize your beliefs based on a lot of things other than usefulness-for-predicting, the usefulness of your beliefs will have a tendency to be very situation-specific, so your may have to rethink things a lot more when situations change, compared with someone who left their beliefs unclouded.
Why assume it’s necessarily conflictual and zero sum? For one thing, there’s a lot of social constructs and unscientific semantics out there.
Because when it is not, then beliefs optimized for predictive value only are optimal. If several agents have sufficiently similar goals such that their only focus is on achieving common goals, then the most predictively accurate beliefs are also going to be the highest utility.
For example, if there is a high social incentive in a community to believe in some specific deity, it could be because there is low trust that people without that belief would act cooperatively. This in turn is because people are assumed to have selfish (IE non-shared) goals. Belief in the deity aligns goals because the deity is said to punish selfish behavior. So, given the belief, everyone can act cooperatively.
Why assume anything unscientific is manipulative?
I’ll grant you one caveat: self-fulfilling prophecies. In situations where those are possible, there are several equally predictively accurate beliefs with different utilities, and we should choose the “best” according to our full preferences.
It’s a pretty large concession, since it includes all sorts of traditions and norms.
Aside from that, though, optimizing for something else that predictive value is very probably manipulative for the reason I stated above: if you’re optimizing for something else, it suggests you’re not working in a team with shared goals, since assuming shared goals, the best collective beliefs are the most predictive.
ETA: I don’t buy that a unscientific concept is necessarily a lie, but even so, if lies are contagious, and no process deletes them, then we should already be in a sea of lies.
I think this part is just a misunderstanding. The post I linked to argues that lies are contagious not in the sense that they spread, but rather, in the sense that in order to justify one lie, you often have to make more lies, so that the lie spreads throughout your web of beliefs. Ultimately, under scrutiny, you would have to lie (eg to yourself) about epistemology itself, since you would need to justify where you got these beliefs from (so for example, Christian scholars will tend to disagree with Bayesians about what constitutes justification for a belief).
Why? Science and politics do not have to fight over the same territory.
I think this has to do with our other disagreement, so I’ll just say that in an ordinary conversation (which I think normally has some mix between “engineer culture” and “diplomat culture”), I personally think there is a lot of overlap in the territory those two modes might be concerned with.
What I meant by “broadly useful” is, having usefulness in many situations and for many people, rather than having usefulness in one specific situation or for one specific person.
That still didn’t tell me whether specialised purposes are non existent , ineffective, or morally wrong.
For example, it’s often more useful to have friends who optimize their epistemics mostly based on usefulness-for-predicting, because those beliefs are more likely to be useful to you as well, rather than just them.
So..ineffective?
What you are saying would be true if people chose friends and projects at random. And if you can only use one toolkit for everything. Neither assumption is realistic. People gather over common interests, and common interests lead to specialised vocabulary. That’s as true of rationalism as anything else.
In contrast, if you have friends who optimize their beliefs based on a lot of other things, then you will have to do more work to figure out whether those beliefs are useful to you as well.
Assuming friends are as randomly distributed as strangers.
Scaling up from “friends” to “society”, this effect gets much more pronounced, so that in the public sphere we really have to ask who benefits from claims/beliefs, and uncontaminated beliefs are much more valuable (so truly unbiased science and journalism are quite valuable as a social good).
Yes, but it’s been that way forever. It’s not like something recently happened to kick us out if the garden if Eden, and it’s not like we never developed any ways of coping.
Similarly, we can go to the smaller scale of one person communicating with themselves over time. If you optimize your beliefs based on a lot of things other than usefulness-for-predicting, the usefulness of your beliefs will have a tendency to be very situation-specific, so your may have to rethink things a lot more when situations change, compared with someone who left their beliefs unclouded.
And if you use generic concepts for everything you lose the advantages of specialised ones.
Why assume it’s necessarily conflictual and zero sum? For one thing, there’s a lot of social constructs and unscientific semantics out there.
Because when it is not, then beliefs optimized for predictive value only are optimal. If several agents have sufficiently similar goals such that their only focus is on achieving common goals, then the most predictively accurate beliefs are also going to be the highest utility.
Assuming that everything is prediction. If several agents have sufficiently similar goals such that their only focus is on achieving common goals,the most optimal concepts will be ones that are specialised for achieving the goal.
For examplee, in cookery school, you will be taught the scientific untruth that tomatoes are vegetables. The manipulates them into into putting them into savoury dishes instead of deserts. This is more efficient than discovering by trial and error what to do with them.
For example, if there is a high social incentive in a community to believe in some specific deity, it could be because there is low trust that people without that belief would act cooperatively. This in turn is because people are assumed to have selfish (IE non-shared) goals. Belief in the deity aligns goals because the deity is said to punish selfish behavior. So, given the belief, everyone can act cooperatively.
There isn’t just one kind of unscientific concept. Shared myths can iron out differences in goals, as in your example, or they can optimise the achievement of shared goals, as in mine.
I’ll grant you one caveat: self-fulfilling prophecies. In situations where those are possible, there are several equally predictively accurate beliefs with different utilities, and we should choose the “best” according to our full preferences.
Assuming, wrongly, that everything is prediction.
Aside from that, though, optimizing for something else that predictive value is very probably manipulative for the reason I stated above:
So...evil?
Low level manipulation is ubiquitous. You need to argue for “manipulative in an egregiously bad way” separately
if you’re optimizing for something else, it suggests you’re not working in a team with shared goals, since assuming shared goals, the best collective beliefs are the most predictive.
What you are saying would be true if people chose friends and projects at random. And if you can only use one toolkit for everything. Neither assumption is realistic. People gather over common interests, and common interests lead to specialised vocabulary. That’s as true of rationalism as anything else.
>In contrast, if you have friends who optimize their beliefs based on a lot of other things, then you will have to do more work to figure out whether those beliefs are useful to you as well.
Assuming friends are as randomly distributed as strangers.
I agree that in practice, people choose friends who share memes (in particular, these “optimized for reasons other than pure accuracy” memes) -- both in that they will select friends on the basis of shared memes, and in that other ways of selecting friends will often result in selecting those who share memes.
But remember my point about agents with fully shared goals. Then, memes optimized to predict what they mutually care about will be optimal for them to use.
So if your friends are using concepts which are optimized for other things, then either (1) you’ve got differing goals and you now would do well to sort out which of their concepts have been gerrymandered, (2) they’ve inherited gerrymandered concepts from someone else with different goals, or (3) your friends and you are all cooperating to gerrymander someone else’s concepts (or, (4), someone is making a mistake somewhere and gerrymandering concepts unnecessarily).
I’m not saying that any of these are fundamentally ineffective, untenable, or even morally reprehensible (though I do think of 1-3 as a bit morally reprehensible, it’s not really the position I want to defend here). I’m just saying there’s something special about avoiding these things, whenever possible, which has good reason to be attractive to a math/science/rationalist flavored person—because if you care deeply about clear thinking, and don’t want the overhead of optimizing your memes for political ends (or de-optimizing memes from friends from those ends), this is the way to do it. So for that sort of person, fighting against gerrymandered concepts is a very reasonable policy decision, and those who have made that choice will find allies with each other. They will naturally prefer to have their own discussions in their own places.
I do, of course, think that the LessWrong community should be and to an extent is such a place.
>>Why assume it’s necessarily conflictual and zero sum?
>Because when it is not, then beliefs optimized for predictive value only are optimal. If several agents have sufficiently similar goals such that their only focus is on achieving common goals, then the most predictively accurate beliefs are also going to be the highest utility.
Assuming that everything is prediction. If several agents have sufficiently similar goals such that their only focus is on achieving common goals,the most optimal concepts will be ones that are specialised for achieving the goal.
For examplee, in cookery school, you will be taught the scientific untruth that tomatoes are vegetables. The manipulates them into into putting them into savoury dishes instead of deserts. This is more efficient than discovering by trial and error what to do with them.
This point was dealt with in the OP. This is why Zack refers to optimizing for prediction of things we care about. Zack is ruling in things like classifying tomatoes as vegetables for culinary purposes, and fruits for biological purposes.1 A cook cares about whether something goes well with savory dishes, whereas a biologist cares about properties relating to the functioning and development of an organism, and its evolutionary relationships with other organisms. So each will use concepts optimized for predicting those things.
So why sanction this sort of goal-dependence, while leaving other sorts of goal-dependence unsanctioned? Can’t I apply the same arguments I made previously, about this creating a lot of friction when people with different goals try to use each other’s concepts?
I think it does create a lot of friction, but the cost of not doing this is simply too high. To live in this universe, humans have to focus on predicting things which are useful to them. Our intellect is not so vast that we can predict things in a completely unbiased way and still have the capacity to, say, cook a meal.
Furthermore, although this does create some friction between agents with different goals, what it doesn’t do (which conceptual gerrymandering does do) is cloud your judgement when you are doing your best to figure things out on your own. By definition, your concepts are optimized to help you predict things you care about, ie, think as clearly as possible. Whereas if your concepts are optimized for other goals, then you must be sacrificing some of your ability to predict things you care about, in order to achieve other things. Yes, it might be worth it, but it must be recognized as a sacrifice. And it’s natural for some people to be unwilling to make that sort of sacrifice.
I imagine that, perhaps, you aren’t fully internalizing this cost because you are imagining using gerrymandered concepts in conversation while internally thinking in clear concepts. But I see the argument as about how to think, not how to talk (although both are important). If you use a gerrymandered concept, you may have no understanding of the non-gerrymandered versions; or you may have some understanding, but in any case not the fluency to think in them. Otherwise you’d risk not achieving your purpose, like a Christian who shows too much fluency in the atheist ontology, thus losing credibility as a Christian. (If they think in the atheist ontology and only speak in the Christian one, that just makes them a liar, which is a different matter.)
There isn’t just one kind of unscientific concept. Shared myths can iron out differences in goals, as in your example, or they can optimise the achievement of shared goals, as in mine.
To summarize, I continue to assume a somewhat adversarial scenario (not necessarily zero sum!) because I see Zack as (correctly) ruling in mere optimization of concepts to predict the things we care about, but ruling out other forms of optimization of concepts to be useful. I believe that this rules in all the non-adversarial examples which you would point at, leaving only the cases where something adversarial is going on.
Low level manipulation is ubiquitous. You need to argue for “manipulative in an egregiously bad way” separately
I’m arguing that Zack’s definition is a very good Schelling fence to put up.
One of Zack’s recurring arguments is that appeal to consequences is an invalid argument when considering where to draw conceptual boundaries. “We can’t define Vargaths as anyone who supports Varg, because the President would be a Vargath by that definition, which she would find offensive; and we don’t want to offend the president!” would be, by Zack’s lights, transparent conceptual gerrymandering and an invalid argument.
Zack’s argument is not itself conceptual gerrymandering because this argument is being made on epistemic grounds, IE, pointing out that accepting “appeals to consequences” arguments reduces your ability to predict things you care about.
My argument in support of Zack’s argument appeals to consequences, but does so in service of the normative question of whether a community of truth-seekers should adopt norms against appeals to consequences. Being a normative question, this is precisely where appeals to consequences are valid and desired.
I think you should think of the validity/invalidity of appeals to consequences as the main thing at stake in this argument, in so far as you are wondering what it’s all about (ie trying to ask me exactly what kind of claim I’m making). Fighting against ubiquitous low-level manipulation would be nice, but there isn’t really a proposal on the table for accomplishing that.
1: For the record, I believe the classical “did you know tomatoes aren’t vegetables, they’re fruits?” is essentially an urban legend with no basis in scientific classification. Vegetable is essentially a culinary term. If you want to speak in biology terms, then yes, it’s also a fruit, but that’s not mutually exclusive with it being a vegetable. But in any case, it’s clear that there can be terminological conflicts like this, even if “vegetable” isn’t one of them; and “tomato” is a familiar example, even if it’s spurious. So we can carry on using it as an example for the sake of argument.
I do, of course, think that the LessWrong community should be and to an extent is such a place.
Something about this has been bugging me and I maybe finally have a grasp on it.
It’s perhaps somewhat entangled with this older Benquo comment elsewhere in this thread. I’m not sure if you endorse this phrasing but your prior paragraph seems similar:
Discourse about how to speak the truth efficiently, on a site literally called “Less Wrong,” shouldn’t have to explicitly disclaim that it’s meant as advice within that context every time, even if it’s often helpful to examine what that means and when and how it is useful to prioritize over other desiderata.
Since a couple-years-ago, I’ve updated “yes, LessWrong should be a fundamentally truthseeking place, optimizing for that at the expense of other things.” (this was indeed an update for me, since I came here for the Impact and vague-appreciation-of-truthseeking, and only later updated that yes, Epistemics are one of the most important cause areas)
But, one of the most important things I want to get out of LessWrong is a clear map of how the rest of the world works, and how to interface with it.
So when I read the conclusion here...
Similarly, the primary thing when you take a word in your lips is your intention to reflect the territory, whatever the means. Whenever you categorize, label, name, define, or draw boundaries, you must cut through to the correct answer in the same movement. If you think only of categorizing, labeling, naming, defining, or drawing boundaries, you will not be able actually to reflect the territory.
...I feel like my epistemics are kind of under attack.
I feel like this statement is motte-and-bailey-ing between “These are the rules of thinking rationally, for forming accurate beliefs, and for communicating about that in particular contexts” and “these are the rules of communicating, in whatever circumstances you might find yourself.”
And it’s actually a pretty big epistemic deal for a site called LessWrong to not draw that distinction. “How coordination works, even with non-rationalists”, is as really big deal, not a special edge case, and I want to be maintaining an accurate map of it the entire time that I’m building out my theory of rationality.
Sort of relatedly, or on the flipside of the coin:
In these threads, I’ve seen a lot of concern with using language “consequentially”, rather than rooted in pure epistemics and map-territory correspondence.
And those arguments have always seemed weird to me. Because… what could you possibly be grounding this all out in, other than consequences? It seems useful to have a concept of “appeals to consequence” being logically invalid. But in terms of what norms to have on as public forum, the key issue on a public forum is that appeals to shortsighted consequences are bad, for the same reason shortsighted consequentialism is often bad.
If you don’t call the president a Vargath (despite them obviously supporting Varg), because they’d be offended, it seems fairly straightforward to argue that this has bad consequences. You just have to model it out more steps.
I would agree with the claim “if you’re constantly checking ‘hey, in this particular instance, maybe it’s net positive to lie?’ you end lying all the time, and end up in a world where people can’t trust each other”, so it’s worth treating appeals to consequences as forbidden as part of a Rule Consequentialism framework. But, why not just say that?
Because… what could you possibly be grounding this all out in, other than consequences?
In my mind, it stands as an open problem whether you can “usually” expect an intelligent system to remain “agent-like in design” under powerful self-modification. By “agent-like in design” I mean having subcomponents which transparently contribute to the overall agentiness, such as true beliefs, coherent goal systems, etc.
The argument in favor is: it becomes really difficult to self-optimize as your own mind-design becomes less modular. At some point you’re just a massive policy with each part fine-tuned to best shape the future (a future which you had some model of at some point in the past); at some point you have to lose general-purpose learning. Therefore, agents with complicated environments and long time horizons will stay modular.
The argument against is: it just isn’t very probable that the nice clean design is the most optimal. Even if there’s only a small incentive to do weird screwy things with your head (ie a small chance you encounter Newcomblike problems where Omega cares about aspects of your ritual of cognition, rather than just output), the agent will follow that incentive where it leads. Plus, general self-optimization can lead to weird, non-modular designs. Why shouldn’t it?
So, in my mind, it stands as an open problem whether purely consequentialist arguments tend to favor a separate epistemic module “in the long term”.
Therefore, I don’t think we can always ground pure epistemic talk in consequences. At least, not without further work.
However, I do think it’s a coherent flag to rally around, and I do think it’s an important goal in the short term, and I think it’s particularly important for a large number of agents trying to coordinate, and it’s also possible that it’s something approaching a terminal goal for humans (ie, curiosity wants to be satisfied by truth).
So I do want to defend pure epistemics as its own goal which doesn’t continuously answer to broad consequentialism. I perceive some reactions to Zack’s post as isolated demands for rigor, invoking the entire justificatory chain to consequentialism when it would not be similarly invoked for a post about, say, p-values.
(A post about p-values vs bayesian hypothesis testing might give rise do discussions of consequences, but not questions of whether the whole argument about bayes vs p-values makes sense because isn’t epistemics ultimately consequentialist anyway or similar.)
I would agree with the claim “if you’re constantly checking ‘hey, in this particular instance, maybe it’s net positive to lie?’ you end lying all the time, and end up in a world where people can’t trust each other”, so it’s worth treating appeals to consequences as forbidden as part of a Rule Consequentialism framework. But, why not just say that?
I would respond:
Partly for the same reason that a post on Bayes’ Law vs p-values wouldn’t usually bother to say that; it’s at least one meta level up from the chief concerns. Granted, unlike a hypothetical post about p-values, Zack’s post was about the appeal-to-consequences argument from its inception, since it responds to an inappropriate appeal to consequences. However, Zack’s primary argument is on the object level, pointing out that how you define words is of epistemic import, and therefore cannot be chosen freely without making epistemic compromises.
TAG and perhaps other critics of this post are not conceding that much; so, the point you make doesn’t seem like it’s sufficient to address the meta-level questions which are being raised.
I would concede that there is, perhaps, something funny about the way I’ve been responding to the discussion—I have a sense that I might be doing some motte/bailey thing around (motte:) this is an isolated demand for rigor, and we should be able to talk about pure epistemics as a goal without explicitly qualifying everything with “if you’re after pure epistemics”; vs (bailey:) we should pursue pure epistemics. In writing comments here, I’ve attempted to carefully argue the two separately. However, I perceive TAG as not having received these as separate arguments. And it is quite possible I’ve blurred the lines at times. They are pretty relevant to each other.
(I say all of this largely agreeing with the thrust of what the post and your (Abram’s) comments are pointing at, but feeling like something about the exact reasoning is off. And it feeling consistently off has been part of why I’ve taken awhile to come around to the reasoning)
So if your friends are using concepts which are optimized for other things, then either (1) you’ve got differing goals and you now would do well to sort out which of their concepts have been gerrymandered, (2) they’ve inherited gerrymandered concepts from someone else with different goals, or (3) your friends and you are all cooperating to gerrymander someone else’s concepts (or, (4), someone is making a mistake somewhere and gerrymandering concepts unnecessarily).
So? That’s a very particular set of problems. If you try to solve them by banning all unscientific concepts, then you lose all the usefulness they have in other contexts.
I’m just saying there’s something special about avoiding these things, whenever possible,
Wherever possible, or wherever beneficial? Does it make the world a better place to keep pointing out that tomatoes are fruit?
because if you care deeply about clear thinking, and don’t want the overhead of optimizing your memes for political ends (or de-optimizing memes from friends from those ends), this is the way to do it.
You personally can do what you like. If you don’t assume that everyone has to have the same solution, then there is no need for conflict.
If you use a gerrymandered concept, you may have no understanding of the non-gerrymandered versions; or you may have some understanding, but in any case not the fluency to think in them.
I’m not following you any more. Of course unscientific concepts can go wrong—anything can. But if you’re not saying everyone should use scientific conceotts all the time, what are you saying?
I see Zack as (correctly) ruling in mere optimization of concepts to predict the things we care about, but ruling out other forms of optimization of concepts to be useful.
I think that is Zacks argument, and that it s fallacious. Because we do things other than predict.
Low level manipulation is ubiquitous. You need to argue for “manipulative in an egregiously bad way” separately
I’m arguing that Zack’s definition is a very good Schelling fence to put up
You are arguing that it is remotely possible to eliminate all manipulation???
One of Zack’s recurring arguments is that appeal to consequences is an invalid argument when considering where to draw conceptual boundaries
Obtaining good consequences is a very good reason to do a lot of things.
So if your friends are using concepts which are optimized for other things, then either (1) you’ve got differing goals and you now would do well to sort out which of their concepts have been gerrymandered, (2) they’ve inherited gerrymandered concepts from someone else with different goals, or (3) your friends and you are all cooperating to gerrymander someone else’s concepts (or, (4), someone is making a mistake somewhere and gerrymandering concepts unnecessarily).
So? That’s a very particular set of problems. If you try to solve them by banning all unscientific concepts, then you lose all the usefulness they have in other contexts.
It seems like part of our persistent disagreement is:
I see this as one of very few pathways, and by far the dominant pathway, by which beliefs can be beneficial in a different way from useful-for-prediction
You see this as one of many many pathways, and very much a corner case
I frankly admit that I think you’re just wrong about this, and you seem quite mistaken in many of the other pathways you point out. The argument you quoted above was supposed to help establish my perspective, by showing that there would be no reason to use gerrymandered concepts unless there was some manipulation going on. Yet you casually brush this off as a very particular set of problems.
I’m just saying there’s something special about avoiding these things, whenever possible,
Wherever possible, or wherever beneficial? Does it make the world a better place to keep pointing out that tomatoes are fruit?
As a general policy, I think that yes, frequently pointing out subtler inaccuracies in language helps practice specificity and gradually refines concepts. For example, if you keep pointing out that tomatoes are fruit, you might eventually be corrected by someone pointing out that “vegetable” is a culinary distinction rather than a biological one, and so there is no reason to object to the classification of a tomato as a vegetable. This could help you develop philosophically, by providing a vivid example of how we use multiple overlapping classification systems rather than one; and further, that scientific-sounding classification criteria don’t always take precedence (IE culinary knowledge is just as valid as biology knowledge).
If you use a gerrymandered concept, you may have no understanding of the non-gerrymandered versions; or you may have some understanding, but in any case not the fluency to think in them.
I’m not following you any more. Of course unscientific concepts can go wrong—anything can. But if you’re not saying everyone should use scientific conceotts all the time, what are you saying?
In what you quoted, I was trying to point out the distinction between speaking a certain way vs thinking a certain way. My overall conversational strategy was to try to separate out the question of whether you should speak a specific way from the question of whether you should think a specific way. This was because I had hoped that we could more easily reach agreement about the “thinking” side of the question.
More specifically, I was pointing out that if we restrict our attention to how to think, then (I claim) the cost of using concepts for non-epistemic reasons is very high, because you usually cannot also be fluent in the more epistemically robust concepts, without the non-epistemic concepts losing a significant amount of power. I gave an example of a Christian who understands the atheist worldview in too much detail.
I see Zack as (correctly) ruling in mere optimization of concepts to predict the things we care about, but ruling out other forms of optimization of concepts to be useful.
I think that is Zacks argument, and that it s fallacious. Because we do things other than predict.
I need some kind of map of the pathways you think are important here.
I 100% agree that we do things other than predict. Specifically, we act. However, the effectiveness of action seems to be very dependent on the accuracy of predictions. We either (a) come up with good plans by virtue of having good models of the world, or (b) learn how to take effective actions “directly” by interacting with the world and responding to feedback. Both of these rely on good epistemics (because learning to act “directly” still relies on our understanding of the world to interpret the feedback—ie the same reason ML people sometimes say that reinforcement learning is essentially learning a classifier).
That view—that by far the primary way in which concepts influence the world is via the motor output channels, which primarily rely on good predictions—is the foundation of my view that most of the benefits of concepts optimized for things other than prediction must be manipulation.
Low level manipulation is ubiquitous. You need to argue for “manipulative in an egregiously bad way” separately
I’m arguing that Zack’s definition is a very good Schelling fence to put up
You are arguing that it is remotely possible to eliminate all manipulation???
Suppose we’re starting a new country, and we are making the decision to outlaw theft. Someone comes to you and says “it isn’t remotely possible to eliminate all theft!!!” … you aren’t going to be very concerned with their argument, right? The point of laws is not to entirely eliminate a behavior (although it would be nice). The point is to help make the behavior uncommon enough that the workings of society are not too badly impacted.
In Zack’s case, he isn’t even suggesting criminal punishment be applied to violations. It’s more like someone just saying “stealing is bad”. So the reply “you’re saying that we can eliminate all theft???” seems even less relevant.
One of Zack’s recurring arguments is that appeal to consequences is an invalid argument when considering where to draw conceptual boundaries
Obtaining good consequences is a very good reason to do a lot of things.
Again, I’m going to need some kind of map of how you see the consequences flowing, because I think the main pathway for those “good consequences” you’re seeing is manipulation.
I frankly admit that I think you’re just wrong about this, and you seem quite mistaken in many of the other pathways you point out
I dont think you have shown that.
Wherever possible, or wherever beneficial? Does it make the world a better place to keep pointing out that tomatoes are fruit?
As a general policy, I think that yes, frequently pointing out subtler inaccuracies in language helps practice specificity and gradually refines concepts. Everything else is Manipulation, and Manipulation is always bad”.
I agree that gaining a meta level undertanding of jargons and the assumptions behind them is useful. I don’t agree that, once you have such an understanding, it reduces to, “everything is or should be passive reflection of statistical regularities in pre-existing reality.
In what you quoted, I was trying to point out the distinction between speaking a certain way vs thinking a certain way. My overall conversational strategy was to try to separate out the question of whether you should speak a specific way from the question of whether you should think a specific way. This was because I had hoped that we could more easily reach agreement about the “thinking” side of the question.
Arguing against whom? I dont believe that ones thinking should be constrained by some narrow set of interests. I have never said it should. On the contrary, I have been arguing against the narrowness of “everything is or should be passive reflection of statistical regularities in pre existing reality”.
More specifically, I was pointing out that if we restrict our attention to how to think, then (I claim) the cost of using concepts for non-epistemic reasons is very high, because you usually cannot also be fluent in the more epistemically robust concepts, without the non-epistemic concepts losing a significant amount of power.
That is yet another surreptitious appeal to the unproven assumption that passive reflection is the only game in town. The agument can easilly be inverted: assuming that what we are doing is constructing a better world or ourselves, then we would be hampered by only using concepts that are”epistemic” in the sense of being restricted to restricted to labelling what is already there.
Of course, construction isnt the only game in town either.
I need some kind of map of the pathways you think are important here.
what has been offered already are the ideas of:-
self-fulffilling prophecies, AKA blueprints AKA social constructs
co-ordination.
fuctionality. Treating a tomato as a vegeable tells you wha to do with it for culinary puposes.
What hasn’t been offered is any reason to think those things don’t exist, or aren’t important, or aren’t useful. My 1) and 2) are Zack’s b) and d). Zack dismissed b) and d) without argument.
We either (a) come up with good plans by virtue of having good models of the world,
Of course, you can’t come up with a plan for making the world better that consists of nothing but a passive model of the world, however accurate it might be.
You seem to be confusing necessity and sufficiency.
That view—that by far the primary way in which concepts influence the world is via the motor output channels, which primarily rely on good predictions—is the foundation of my view that most of the benefits of concepts optimized for things other than prediction must be manipulation
There’s nothing anyone can say to you that would change the automatic and unconscious operation of your motor channels.
In Zack’s case, he isn’t even suggesting criminal punishment be applied to violations. It’s more like someone just saying “stealing is bad”. So the reply “you’re saying that we can eliminate all theft???” seems even less relevant.
You are arguing that not wanting to eliminate all manipulation is compatible with believing all manipulation to be bad. That falls short of showing that all manipulation is bad. (We’re holding a debate. So, you’re trying to change my mind, and I yours..isn’t that manipulation?)
I think the main pathway for those “good consequences” you’re seeing is manipulation.
I don’t think you have shown that either. And it wouldn’t matter unless All Manipulation is Bad.
You haven’t refuted the counterexamples to everything-that-isnt-reflection-is-manipulation, and you havent shown that all manipulation is bad, either.
I feel like you’re taking my attempts to explain my position and requiring that each one be a rigorous defense. Sometimes we just have to spend some time trying to understand each other before we can bring the knives out or whatever, yeah? Sorry if I’m guilty of the same thing—I tried to unpack some more details after my flat statement that I thought you were wrong, but it probably came off as just being argumentative.
>>>If you use a gerrymandered concept, you may have no understanding of the non-gerrymandered versions; or you may have some understanding, but in any case not the fluency to think in them.
>>I’m not following you any more. Of course unscientific concepts can go wrong—anything can. But if you’re not saying everyone should use scientific conceotts all the time, what are you saying?
>In what you quoted, I was trying to point out the distinction between speaking a certain way vs thinking a certain way. My overall conversational strategy was to try to separate out the question of whether you should speak a specific way from the question of whether you should think a specific way. This was because I had hoped that we could more easily reach agreement about the “thinking” side of the question.
Arguing against whom? I dont believe that ones thinking should be constrained by some narrow set of interests. I have never said it should. On the contrary, I have been arguing against the narrowness of “everything is or should be passive reflection of statistical regularities in pre existing reality”.
(Sorry, I just don’t get how this is relevant to the quote you’re apparently responding to; I didn’t use the words ‘arguing against’ there, and was describing my conversational goal, rather than arguing something. So I’m going to try to make some more clarifying remarks which may not answer your question:)
You ask “if you’re not saying everyone should use scientific concepts all the time, what are you saying?”
I have attempted to separately argue the following:
Much of the time, using “unscientific concepts” is a mistake. In particular, by trying to separate thinking vs speaking, I was trying to point out that even in cases where it’s plausible that you are better off speaking in epistemically unhygenic ways, it’s not plausible that you’re better off thinking in those ways: there’s a high cost to pay in not understanding the world. (Note the weak “much of the time” qualifier here—I endorse this point and think it’s important to the discussion, but I’m endorsing a rather weak statement, on purpose.)
Most of the time, using “unscientific concepts” is useful only for manipulative purposes. My argument here is based on the idea that agents with shared goals will communicate in a way which shares as much information as possible (in the bits communicated—IE, modulo communication costs, redundancy built into the language to ensure communication over noisy channels, etc). Therefore, behavior contrary to this must be either uncooperative or simply sub-optimal. This doesn’t mean it’s irrational (a consequentialist might manipulate others), but I presume that you would be less happy to argue in favor of unscientific concepts if you conceded that they were almost always manipulative. Your response to this was to call my argument a “very special case”. I do not concede this; I think it is a very general case. (I do not currently understand why you called it a very special case.)
Very nearly all of the time, it makes sense to separate out pure epistemic quality and consider it as a coherent goal, talk about how to achieve it, etc. (Not pursue it singlemindedly, but distinguish it as a comprehensible thing.) In particular, it makes sense to have this discussion about nearly any statement. I perceive you as having a large disagreement with me about this, thinking that it makes a lot less sense for some statements, EG those about marriage and money.
Some of the time, it makes sense to have a social norm against appeals to consequences (as an argument for changing epistemic stances), in order to safeguard ‘scientific’ thought-processes against distortion. In particular, I think it makes sense on lesswrong. This is not a claim that all conceptual gerrymandering can be eliminated, but rather, that we should make the attempt (at least in specific arenas of discourse).
what has been offered already are the ideas of:-
1) self-fulffilling prophecies, AKA blueprints AKA social constructs
2) co-ordination.
3) fuctionality. Treating a tomato as a vegeable tells you wha to do with it for culinary puposes.
What hasn’t been offered is any reason to think those things don’t exist, or aren’t important, or aren’t useful. My 1) and 2) are Zack’s b) and d). Zack dismissed b) and d) without argument.
I fully conceded #1 earlier in our discussion—I have no qualms with this pathway, and I think it’s important. I don’t think it entails accepting less-accurate beliefs (a self-fulfilling prophecy is, after all, true!), but I do think it entails valid appeals-to-consequences for what might otherwise seem like purely epistemic questions. Furthermore I think this is relatively common.
I fully concede #3, and also perceive Zack as explicitly doing so, as part of his central argument.
I am not trying to defend a norm against #1 or #3, nor am I defending a concept of “pure epistemics” which regards #1 or #3 as impurities, in my own points 1-4 earlier. I think “pure epistemics” without your #1 would be very limited, because it becomes ill-defined in the presence of self-fulfilling prophecies or other predictions which are relevant to their own outcomes. I think “pure epistemics” without your #3 is very nearly useless, due to a lack of focus on useful questions. Both of these things are coherent things to talk about, but not very useful to agents, and therefore less descriptively apt for discussing and understanding agents, nor as normatively apt for a community of agents.
As for #2, I think some of this is covered by #1. Everything else, I claim is manipulative, like EG promising a good afterlife if you help build a pyramid in the middle of the desert. Manipulation works, but I continue to presume it’s not what you’re defending when you defend ‘unscientific concepts’.
So I suppose either (a) we can agree on all of that, and don’t have any remaining disagreement, or (b) our main disagreement is with #2, and we should focus on my argument that epistemic impurities are going to be manipulative, or (c) your 1-3 don’t cover all the bases you think are important, and we should talk about what other channels make unscientific concepts useful. (Or perhaps some mix of a-c.)
I feel like you’re taking my attempts to explain my position and requiring that each one be a rigorous defense.
If someone has made a position clear, they need to move onto defending it at some stage, or else it’s all just opinion.
You clearly think that some concepts lack objectivity .. that’s been explained a great length with equations and diagrams...and you think that the very existence of scientific objectivity is in danger. But between these two claims there are any number of intermediate steps that have not been explained or defended.
Much of the time, using “unscientific concepts” is a mistake
I don’t see why. It’s not a mistake to use special purpose or value laden concepts appropriately. So how can it be usually be a mistake to use them? Are you saying that they are usually used inappropriately?
Most of the time, using “unscientific concepts” is useful only for manipulative purposes. My argument here is based on the idea that agents with shared goals will communicate in a way which shares as much information as possible (in the bits communicated—IE, modulo communication costs, redundancy built into the language to ensure communication over noisy channels, etc).
No. If they have shared goals, they will already have a lot of shared information ( ie. small inferential distance) and they will already use a special purpose jargon.
Special interest groups always have special language. Objective, scientific language is what scientists use, and not that many people are scientists, so it is not the default.
In any case, how is that evidence of manipulation?
but I presume that you would be less happy to argue in favor of unscientific concepts if you conceded that they were almost always manipulative.
I don’t concede that they are always manipulative, in an objectionable sense. We are at the stage where you need to clarify that.
Your response to this was to call my argument a “very special case”. I do not concede this; I think it is a very general case. (I do not currently understand why you called it a very special case).
How common is manipulation? If you set the bar on what constitutes manipulation very low, then it is very common, even including this discussion. But if it is very common, how can it be very bad? If you think that all gerrymandered concepts are “manipulative” in the sense of micro manipulations, where’s the problem?
I think this a central weakness of your case: you need to choose one of “manipulation common”, and “manipulation bad”.
Very nearly all of the time, it makes sense to separate out pure epistemic quality and consider it as a coherent goal, talk about how to achieve it, etc.
Why? And for whom?
Some of the time, it makes sense to have a social norm against appeals to consequences (as an argument for changing epistemic stances), in order to safeguard ‘scientific’ thought-processes against distortion
Well, if it’s only some of the time, you can achieve that by saying that scientists are special people who do have an obligation to be as objective as possible , but no obligation to be consequentialist. But that’s not novel.
As for #2, I think some of this is covered by #1. Everything else, I claim is manipulative, like EG promising a good afterlife if you help build a pyramid in the middle of the desert
That seems like a weakman to me. What about cases where coordination is of benefit to the people doing the coordinating...like obeying traffic laws? A speed limit is a gerrymandered concept.
It seems, perhaps, that your main point is that usefulness can come apart from correspondence:
I don’t believe that Zack disagreed with this? Indeed, Zack mentions several examples where the two come apart:
These are both examples where “useful” is importantly different from “corresponds to reality”.
He’s disagreeing with someone over something. In think my point is the same as Scott’s, and he seems to be responding to Scott.
Edit:
If you read back, I’m responding to the point that: ”...in order for your map to be useful in the service of your values, it needs to reflect the statistical structure of things in the territory—which depends on the territory, not your values.”
That’s a pretty clear rejection of useful-but-not-corresponding even if there are examples of useful-but-not-corresponding further down.
Yes, but they are examples with negative connotations.
That’s fair.
I also agree with the negative connotations. There’s something special, worth defending, about epistemics focusing only on reflecting the territory, screening off other considerations as much as possible.
That’s quite a vague claim. Are you saying that realistic epistemology is special in some sense that it should be applied to everything, or that everything should be reduced to it?
I’m saying that epistemics focused on usefulness-to-predicting is broadly useful in a way that epistemics optimized in other ways is not. It is more trustworthy in that the extent to which it’s optimized for some people at the expense of other people must be very limited. (Of course it will still be more useful to some people than others, but the Schelling-point-nature means that we tend to take it as the gold standard against which other things are judged as “manipulative”.)
Another defense of this Schelling point is that as we depart from it, it becomes increasingly difficult to objectively judge whether we are benefiting or hurting as a result. We get a web of contagious lies spreading through our epistemology.
I’m not saying this is a Schelling fence which has held firm through the ages, by any means; indeed, it is rarely held firm. But, speaking very roughly and broadly, this is a fight between “scientists” and “politicians” (or, as Benquo has put it, between engineers and diplomats).
That’s still not very clear. As opposed to other epistemics being useless, or as opposed to other epistemics having specialized usefulness?
Why assume it’s necessarily conflictual and zero sum? For one thing, there’s a lot of social constructs and unscientific semantics out there.
Why assume anything unscientific is manipulative?
If you are going to use a contagion metaphor, why not use an immune system metaphor? Which would be a metaphor for critical thinking.
We were never there!
ETA: I don’t buy that a unscientific concept is necessarily a lie, but even so, if lies are contagious, and no process deletes them, then we should already be in a sea of lies.
Why? Science and politics do not have to fight over the same territory.
What I meant by “broadly useful” is, having usefulness in many situations and for many people, rather than having usefulness in one specific situation or for one specific person.
For example, it’s often more useful to have friends who optimize their epistemics mostly based on usefulness-for-predicting, because those beliefs are more likely to be useful to you as well, rather than just them.
In contrast, if you have friends who optimize their beliefs based on a lot of other things, then you will have to do more work to figure out whether those beliefs are useful to you as well. Simply put, their beliefs will be less trustworthy.
Scaling up from “friends” to “society”, this effect gets much more pronounced, so that in the public sphere we really have to ask who benefits from claims/beliefs, and uncontaminated beliefs are much more valuable (so truly unbiased science and journalism are quite valuable as a social good).
Similarly, we can go to the smaller scale of one person communicating with themselves over time. If you optimize your beliefs based on a lot of things other than usefulness-for-predicting, the usefulness of your beliefs will have a tendency to be very situation-specific, so your may have to rethink things a lot more when situations change, compared with someone who left their beliefs unclouded.
Because when it is not, then beliefs optimized for predictive value only are optimal. If several agents have sufficiently similar goals such that their only focus is on achieving common goals, then the most predictively accurate beliefs are also going to be the highest utility.
For example, if there is a high social incentive in a community to believe in some specific deity, it could be because there is low trust that people without that belief would act cooperatively. This in turn is because people are assumed to have selfish (IE non-shared) goals. Belief in the deity aligns goals because the deity is said to punish selfish behavior. So, given the belief, everyone can act cooperatively.
I’ll grant you one caveat: self-fulfilling prophecies. In situations where those are possible, there are several equally predictively accurate beliefs with different utilities, and we should choose the “best” according to our full preferences.
It’s a pretty large concession, since it includes all sorts of traditions and norms.
Aside from that, though, optimizing for something else that predictive value is very probably manipulative for the reason I stated above: if you’re optimizing for something else, it suggests you’re not working in a team with shared goals, since assuming shared goals, the best collective beliefs are the most predictive.
I think this part is just a misunderstanding. The post I linked to argues that lies are contagious not in the sense that they spread, but rather, in the sense that in order to justify one lie, you often have to make more lies, so that the lie spreads throughout your web of beliefs. Ultimately, under scrutiny, you would have to lie (eg to yourself) about epistemology itself, since you would need to justify where you got these beliefs from (so for example, Christian scholars will tend to disagree with Bayesians about what constitutes justification for a belief).
I think this has to do with our other disagreement, so I’ll just say that in an ordinary conversation (which I think normally has some mix between “engineer culture” and “diplomat culture”), I personally think there is a lot of overlap in the territory those two modes might be concerned with.
That still didn’t tell me whether specialised purposes are non existent , ineffective, or morally wrong.
So..ineffective?
What you are saying would be true if people chose friends and projects at random. And if you can only use one toolkit for everything. Neither assumption is realistic. People gather over common interests, and common interests lead to specialised vocabulary. That’s as true of rationalism as anything else.
Assuming friends are as randomly distributed as strangers.
Yes, but it’s been that way forever. It’s not like something recently happened to kick us out if the garden if Eden, and it’s not like we never developed any ways of coping.
And if you use generic concepts for everything you lose the advantages of specialised ones.
Why assume it’s necessarily conflictual and zero sum? For one thing, there’s a lot of social constructs and unscientific semantics out there.
Assuming that everything is prediction. If several agents have sufficiently similar goals such that their only focus is on achieving common goals,the most optimal concepts will be ones that are specialised for achieving the goal.
For examplee, in cookery school, you will be taught the scientific untruth that tomatoes are vegetables. The manipulates them into into putting them into savoury dishes instead of deserts. This is more efficient than discovering by trial and error what to do with them.
There isn’t just one kind of unscientific concept. Shared myths can iron out differences in goals, as in your example, or they can optimise the achievement of shared goals, as in mine.
Assuming, wrongly, that everything is prediction.
So...evil?
Low level manipulation is ubiquitous. You need to argue for “manipulative in an egregiously bad way” separately
No, see above.
I agree that in practice, people choose friends who share memes (in particular, these “optimized for reasons other than pure accuracy” memes) -- both in that they will select friends on the basis of shared memes, and in that other ways of selecting friends will often result in selecting those who share memes.
But remember my point about agents with fully shared goals. Then, memes optimized to predict what they mutually care about will be optimal for them to use.
So if your friends are using concepts which are optimized for other things, then either (1) you’ve got differing goals and you now would do well to sort out which of their concepts have been gerrymandered, (2) they’ve inherited gerrymandered concepts from someone else with different goals, or (3) your friends and you are all cooperating to gerrymander someone else’s concepts (or, (4), someone is making a mistake somewhere and gerrymandering concepts unnecessarily).
I’m not saying that any of these are fundamentally ineffective, untenable, or even morally reprehensible (though I do think of 1-3 as a bit morally reprehensible, it’s not really the position I want to defend here). I’m just saying there’s something special about avoiding these things, whenever possible, which has good reason to be attractive to a math/science/rationalist flavored person—because if you care deeply about clear thinking, and don’t want the overhead of optimizing your memes for political ends (or de-optimizing memes from friends from those ends), this is the way to do it. So for that sort of person, fighting against gerrymandered concepts is a very reasonable policy decision, and those who have made that choice will find allies with each other. They will naturally prefer to have their own discussions in their own places.
I do, of course, think that the LessWrong community should be and to an extent is such a place.
This point was dealt with in the OP. This is why Zack refers to optimizing for prediction of things we care about. Zack is ruling in things like classifying tomatoes as vegetables for culinary purposes, and fruits for biological purposes.1 A cook cares about whether something goes well with savory dishes, whereas a biologist cares about properties relating to the functioning and development of an organism, and its evolutionary relationships with other organisms. So each will use concepts optimized for predicting those things.
So why sanction this sort of goal-dependence, while leaving other sorts of goal-dependence unsanctioned? Can’t I apply the same arguments I made previously, about this creating a lot of friction when people with different goals try to use each other’s concepts?
I think it does create a lot of friction, but the cost of not doing this is simply too high. To live in this universe, humans have to focus on predicting things which are useful to them. Our intellect is not so vast that we can predict things in a completely unbiased way and still have the capacity to, say, cook a meal.
Furthermore, although this does create some friction between agents with different goals, what it doesn’t do (which conceptual gerrymandering does do) is cloud your judgement when you are doing your best to figure things out on your own. By definition, your concepts are optimized to help you predict things you care about, ie, think as clearly as possible. Whereas if your concepts are optimized for other goals, then you must be sacrificing some of your ability to predict things you care about, in order to achieve other things. Yes, it might be worth it, but it must be recognized as a sacrifice. And it’s natural for some people to be unwilling to make that sort of sacrifice.
I imagine that, perhaps, you aren’t fully internalizing this cost because you are imagining using gerrymandered concepts in conversation while internally thinking in clear concepts. But I see the argument as about how to think, not how to talk (although both are important). If you use a gerrymandered concept, you may have no understanding of the non-gerrymandered versions; or you may have some understanding, but in any case not the fluency to think in them. Otherwise you’d risk not achieving your purpose, like a Christian who shows too much fluency in the atheist ontology, thus losing credibility as a Christian. (If they think in the atheist ontology and only speak in the Christian one, that just makes them a liar, which is a different matter.)
To summarize, I continue to assume a somewhat adversarial scenario (not necessarily zero sum!) because I see Zack as (correctly) ruling in mere optimization of concepts to predict the things we care about, but ruling out other forms of optimization of concepts to be useful. I believe that this rules in all the non-adversarial examples which you would point at, leaving only the cases where something adversarial is going on.
I’m arguing that Zack’s definition is a very good Schelling fence to put up.
One of Zack’s recurring arguments is that appeal to consequences is an invalid argument when considering where to draw conceptual boundaries. “We can’t define Vargaths as anyone who supports Varg, because the President would be a Vargath by that definition, which she would find offensive; and we don’t want to offend the president!” would be, by Zack’s lights, transparent conceptual gerrymandering and an invalid argument.
Zack’s argument is not itself conceptual gerrymandering because this argument is being made on epistemic grounds, IE, pointing out that accepting “appeals to consequences” arguments reduces your ability to predict things you care about.
My argument in support of Zack’s argument appeals to consequences, but does so in service of the normative question of whether a community of truth-seekers should adopt norms against appeals to consequences. Being a normative question, this is precisely where appeals to consequences are valid and desired.
I think you should think of the validity/invalidity of appeals to consequences as the main thing at stake in this argument, in so far as you are wondering what it’s all about (ie trying to ask me exactly what kind of claim I’m making). Fighting against ubiquitous low-level manipulation would be nice, but there isn’t really a proposal on the table for accomplishing that.
1: For the record, I believe the classical “did you know tomatoes aren’t vegetables, they’re fruits?” is essentially an urban legend with no basis in scientific classification. Vegetable is essentially a culinary term. If you want to speak in biology terms, then yes, it’s also a fruit, but that’s not mutually exclusive with it being a vegetable. But in any case, it’s clear that there can be terminological conflicts like this, even if “vegetable” isn’t one of them; and “tomato” is a familiar example, even if it’s spurious. So we can carry on using it as an example for the sake of argument.
Something about this has been bugging me and I maybe finally have a grasp on it.
It’s perhaps somewhat entangled with this older Benquo comment elsewhere in this thread. I’m not sure if you endorse this phrasing but your prior paragraph seems similar:
Since a couple-years-ago, I’ve updated “yes, LessWrong should be a fundamentally truthseeking place, optimizing for that at the expense of other things.” (this was indeed an update for me, since I came here for the Impact and vague-appreciation-of-truthseeking, and only later updated that yes, Epistemics are one of the most important cause areas)
But, one of the most important things I want to get out of LessWrong is a clear map of how the rest of the world works, and how to interface with it.
So when I read the conclusion here...
...I feel like my epistemics are kind of under attack.
I feel like this statement is motte-and-bailey-ing between “These are the rules of thinking rationally, for forming accurate beliefs, and for communicating about that in particular contexts” and “these are the rules of communicating, in whatever circumstances you might find yourself.”
And it’s actually a pretty big epistemic deal for a site called LessWrong to not draw that distinction. “How coordination works, even with non-rationalists”, is as really big deal, not a special edge case, and I want to be maintaining an accurate map of it the entire time that I’m building out my theory of rationality.
Sort of relatedly, or on the flipside of the coin:
In these threads, I’ve seen a lot of concern with using language “consequentially”, rather than rooted in pure epistemics and map-territory correspondence.
And those arguments have always seemed weird to me. Because… what could you possibly be grounding this all out in, other than consequences? It seems useful to have a concept of “appeals to consequence” being logically invalid. But in terms of what norms to have on as public forum, the key issue on a public forum is that appeals to shortsighted consequences are bad, for the same reason shortsighted consequentialism is often bad.
If you don’t call the president a Vargath (despite them obviously supporting Varg), because they’d be offended, it seems fairly straightforward to argue that this has bad consequences. You just have to model it out more steps.
I would agree with the claim “if you’re constantly checking ‘hey, in this particular instance, maybe it’s net positive to lie?’ you end lying all the time, and end up in a world where people can’t trust each other”, so it’s worth treating appeals to consequences as forbidden as part of a Rule Consequentialism framework. But, why not just say that?
In my mind, it stands as an open problem whether you can “usually” expect an intelligent system to remain “agent-like in design” under powerful self-modification. By “agent-like in design” I mean having subcomponents which transparently contribute to the overall agentiness, such as true beliefs, coherent goal systems, etc.
The argument in favor is: it becomes really difficult to self-optimize as your own mind-design becomes less modular. At some point you’re just a massive policy with each part fine-tuned to best shape the future (a future which you had some model of at some point in the past); at some point you have to lose general-purpose learning. Therefore, agents with complicated environments and long time horizons will stay modular.
The argument against is: it just isn’t very probable that the nice clean design is the most optimal. Even if there’s only a small incentive to do weird screwy things with your head (ie a small chance you encounter Newcomblike problems where Omega cares about aspects of your ritual of cognition, rather than just output), the agent will follow that incentive where it leads. Plus, general self-optimization can lead to weird, non-modular designs. Why shouldn’t it?
So, in my mind, it stands as an open problem whether purely consequentialist arguments tend to favor a separate epistemic module “in the long term”.
Therefore, I don’t think we can always ground pure epistemic talk in consequences. At least, not without further work.
However, I do think it’s a coherent flag to rally around, and I do think it’s an important goal in the short term, and I think it’s particularly important for a large number of agents trying to coordinate, and it’s also possible that it’s something approaching a terminal goal for humans (ie, curiosity wants to be satisfied by truth).
So I do want to defend pure epistemics as its own goal which doesn’t continuously answer to broad consequentialism. I perceive some reactions to Zack’s post as isolated demands for rigor, invoking the entire justificatory chain to consequentialism when it would not be similarly invoked for a post about, say, p-values.
(A post about p-values vs bayesian hypothesis testing might give rise do discussions of consequences, but not questions of whether the whole argument about bayes vs p-values makes sense because isn’t epistemics ultimately consequentialist anyway or similar.)
I would respond:
Partly for the same reason that a post on Bayes’ Law vs p-values wouldn’t usually bother to say that; it’s at least one meta level up from the chief concerns. Granted, unlike a hypothetical post about p-values, Zack’s post was about the appeal-to-consequences argument from its inception, since it responds to an inappropriate appeal to consequences. However, Zack’s primary argument is on the object level, pointing out that how you define words is of epistemic import, and therefore cannot be chosen freely without making epistemic compromises.
TAG and perhaps other critics of this post are not conceding that much; so, the point you make doesn’t seem like it’s sufficient to address the meta-level questions which are being raised.
I would concede that there is, perhaps, something funny about the way I’ve been responding to the discussion—I have a sense that I might be doing some motte/bailey thing around (motte:) this is an isolated demand for rigor, and we should be able to talk about pure epistemics as a goal without explicitly qualifying everything with “if you’re after pure epistemics”; vs (bailey:) we should pursue pure epistemics. In writing comments here, I’ve attempted to carefully argue the two separately. However, I perceive TAG as not having received these as separate arguments. And it is quite possible I’ve blurred the lines at times. They are pretty relevant to each other.
(I say all of this largely agreeing with the thrust of what the post and your (Abram’s) comments are pointing at, but feeling like something about the exact reasoning is off. And it feeling consistently off has been part of why I’ve taken awhile to come around to the reasoning)
So? That’s a very particular set of problems. If you try to solve them by banning all unscientific concepts, then you lose all the usefulness they have in other contexts.
Wherever possible, or wherever beneficial? Does it make the world a better place to keep pointing out that tomatoes are fruit?
You personally can do what you like. If you don’t assume that everyone has to have the same solution, then there is no need for conflict.
I’m not following you any more. Of course unscientific concepts can go wrong—anything can. But if you’re not saying everyone should use scientific conceotts all the time, what are you saying?
I think that is Zacks argument, and that it s fallacious. Because we do things other than predict.
You are arguing that it is remotely possible to eliminate all manipulation???
Obtaining good consequences is a very good reason to do a lot of things.
It seems like part of our persistent disagreement is:
I see this as one of very few pathways, and by far the dominant pathway, by which beliefs can be beneficial in a different way from useful-for-prediction
You see this as one of many many pathways, and very much a corner case
I frankly admit that I think you’re just wrong about this, and you seem quite mistaken in many of the other pathways you point out. The argument you quoted above was supposed to help establish my perspective, by showing that there would be no reason to use gerrymandered concepts unless there was some manipulation going on. Yet you casually brush this off as a very particular set of problems.
As a general policy, I think that yes, frequently pointing out subtler inaccuracies in language helps practice specificity and gradually refines concepts. For example, if you keep pointing out that tomatoes are fruit, you might eventually be corrected by someone pointing out that “vegetable” is a culinary distinction rather than a biological one, and so there is no reason to object to the classification of a tomato as a vegetable. This could help you develop philosophically, by providing a vivid example of how we use multiple overlapping classification systems rather than one; and further, that scientific-sounding classification criteria don’t always take precedence (IE culinary knowledge is just as valid as biology knowledge).
In what you quoted, I was trying to point out the distinction between speaking a certain way vs thinking a certain way. My overall conversational strategy was to try to separate out the question of whether you should speak a specific way from the question of whether you should think a specific way. This was because I had hoped that we could more easily reach agreement about the “thinking” side of the question.
More specifically, I was pointing out that if we restrict our attention to how to think, then (I claim) the cost of using concepts for non-epistemic reasons is very high, because you usually cannot also be fluent in the more epistemically robust concepts, without the non-epistemic concepts losing a significant amount of power. I gave an example of a Christian who understands the atheist worldview in too much detail.
I need some kind of map of the pathways you think are important here.
I 100% agree that we do things other than predict. Specifically, we act. However, the effectiveness of action seems to be very dependent on the accuracy of predictions. We either (a) come up with good plans by virtue of having good models of the world, or (b) learn how to take effective actions “directly” by interacting with the world and responding to feedback. Both of these rely on good epistemics (because learning to act “directly” still relies on our understanding of the world to interpret the feedback—ie the same reason ML people sometimes say that reinforcement learning is essentially learning a classifier).
That view—that by far the primary way in which concepts influence the world is via the motor output channels, which primarily rely on good predictions—is the foundation of my view that most of the benefits of concepts optimized for things other than prediction must be manipulation.
Suppose we’re starting a new country, and we are making the decision to outlaw theft. Someone comes to you and says “it isn’t remotely possible to eliminate all theft!!!” … you aren’t going to be very concerned with their argument, right? The point of laws is not to entirely eliminate a behavior (although it would be nice). The point is to help make the behavior uncommon enough that the workings of society are not too badly impacted.
In Zack’s case, he isn’t even suggesting criminal punishment be applied to violations. It’s more like someone just saying “stealing is bad”. So the reply “you’re saying that we can eliminate all theft???” seems even less relevant.
Again, I’m going to need some kind of map of how you see the consequences flowing, because I think the main pathway for those “good consequences” you’re seeing is manipulation.
I dont think you have shown that.
I agree that gaining a meta level undertanding of jargons and the assumptions behind them is useful. I don’t agree that, once you have such an understanding, it reduces to, “everything is or should be passive reflection of statistical regularities in pre-existing reality.
Arguing against whom? I dont believe that ones thinking should be constrained by some narrow set of interests. I have never said it should. On the contrary, I have been arguing against the narrowness of “everything is or should be passive reflection of statistical regularities in pre existing reality”.
That is yet another surreptitious appeal to the unproven assumption that passive reflection is the only game in town. The agument can easilly be inverted: assuming that what we are doing is constructing a better world or ourselves, then we would be hampered by only using concepts that are”epistemic” in the sense of being restricted to restricted to labelling what is already there.
Of course, construction isnt the only game in town either.
what has been offered already are the ideas of:-
self-fulffilling prophecies, AKA blueprints AKA social constructs
co-ordination.
fuctionality. Treating a tomato as a vegeable tells you wha to do with it for culinary puposes.
What hasn’t been offered is any reason to think those things don’t exist, or aren’t important, or aren’t useful. My 1) and 2) are Zack’s b) and d). Zack dismissed b) and d) without argument.
Of course, you can’t come up with a plan for making the world better that consists of nothing but a passive model of the world, however accurate it might be.
You seem to be confusing necessity and sufficiency.
There’s nothing anyone can say to you that would change the automatic and unconscious operation of your motor channels.
You are arguing that not wanting to eliminate all manipulation is compatible with believing all manipulation to be bad. That falls short of showing that all manipulation is bad. (We’re holding a debate. So, you’re trying to change my mind, and I yours..isn’t that manipulation?)
I don’t think you have shown that either. And it wouldn’t matter unless All Manipulation is Bad.
You haven’t refuted the counterexamples to everything-that-isnt-reflection-is-manipulation, and you havent shown that all manipulation is bad, either.
I feel like you’re taking my attempts to explain my position and requiring that each one be a rigorous defense. Sometimes we just have to spend some time trying to understand each other before we can bring the knives out or whatever, yeah? Sorry if I’m guilty of the same thing—I tried to unpack some more details after my flat statement that I thought you were wrong, but it probably came off as just being argumentative.
(Sorry, I just don’t get how this is relevant to the quote you’re apparently responding to; I didn’t use the words ‘arguing against’ there, and was describing my conversational goal, rather than arguing something. So I’m going to try to make some more clarifying remarks which may not answer your question:)
You ask “if you’re not saying everyone should use scientific concepts all the time, what are you saying?”
I have attempted to separately argue the following:
Much of the time, using “unscientific concepts” is a mistake. In particular, by trying to separate thinking vs speaking, I was trying to point out that even in cases where it’s plausible that you are better off speaking in epistemically unhygenic ways, it’s not plausible that you’re better off thinking in those ways: there’s a high cost to pay in not understanding the world. (Note the weak “much of the time” qualifier here—I endorse this point and think it’s important to the discussion, but I’m endorsing a rather weak statement, on purpose.)
Most of the time, using “unscientific concepts” is useful only for manipulative purposes. My argument here is based on the idea that agents with shared goals will communicate in a way which shares as much information as possible (in the bits communicated—IE, modulo communication costs, redundancy built into the language to ensure communication over noisy channels, etc). Therefore, behavior contrary to this must be either uncooperative or simply sub-optimal. This doesn’t mean it’s irrational (a consequentialist might manipulate others), but I presume that you would be less happy to argue in favor of unscientific concepts if you conceded that they were almost always manipulative. Your response to this was to call my argument a “very special case”. I do not concede this; I think it is a very general case. (I do not currently understand why you called it a very special case.)
Very nearly all of the time, it makes sense to separate out pure epistemic quality and consider it as a coherent goal, talk about how to achieve it, etc. (Not pursue it singlemindedly, but distinguish it as a comprehensible thing.) In particular, it makes sense to have this discussion about nearly any statement. I perceive you as having a large disagreement with me about this, thinking that it makes a lot less sense for some statements, EG those about marriage and money.
Some of the time, it makes sense to have a social norm against appeals to consequences (as an argument for changing epistemic stances), in order to safeguard ‘scientific’ thought-processes against distortion. In particular, I think it makes sense on lesswrong. This is not a claim that all conceptual gerrymandering can be eliminated, but rather, that we should make the attempt (at least in specific arenas of discourse).
I fully conceded #1 earlier in our discussion—I have no qualms with this pathway, and I think it’s important. I don’t think it entails accepting less-accurate beliefs (a self-fulfilling prophecy is, after all, true!), but I do think it entails valid appeals-to-consequences for what might otherwise seem like purely epistemic questions. Furthermore I think this is relatively common.
I fully concede #3, and also perceive Zack as explicitly doing so, as part of his central argument.
I am not trying to defend a norm against #1 or #3, nor am I defending a concept of “pure epistemics” which regards #1 or #3 as impurities, in my own points 1-4 earlier. I think “pure epistemics” without your #1 would be very limited, because it becomes ill-defined in the presence of self-fulfilling prophecies or other predictions which are relevant to their own outcomes. I think “pure epistemics” without your #3 is very nearly useless, due to a lack of focus on useful questions. Both of these things are coherent things to talk about, but not very useful to agents, and therefore less descriptively apt for discussing and understanding agents, nor as normatively apt for a community of agents.
As for #2, I think some of this is covered by #1. Everything else, I claim is manipulative, like EG promising a good afterlife if you help build a pyramid in the middle of the desert. Manipulation works, but I continue to presume it’s not what you’re defending when you defend ‘unscientific concepts’.
So I suppose either (a) we can agree on all of that, and don’t have any remaining disagreement, or (b) our main disagreement is with #2, and we should focus on my argument that epistemic impurities are going to be manipulative, or (c) your 1-3 don’t cover all the bases you think are important, and we should talk about what other channels make unscientific concepts useful. (Or perhaps some mix of a-c.)
If someone has made a position clear, they need to move onto defending it at some stage, or else it’s all just opinion.
You clearly think that some concepts lack objectivity .. that’s been explained a great length with equations and diagrams...and you think that the very existence of scientific objectivity is in danger. But between these two claims there are any number of intermediate steps that have not been explained or defended.
I don’t see why. It’s not a mistake to use special purpose or value laden concepts appropriately. So how can it be usually be a mistake to use them? Are you saying that they are usually used inappropriately?
No. If they have shared goals, they will already have a lot of shared information ( ie. small inferential distance) and they will already use a special purpose jargon. Special interest groups always have special language. Objective, scientific language is what scientists use, and not that many people are scientists, so it is not the default.
In any case, how is that evidence of manipulation?
I don’t concede that they are always manipulative, in an objectionable sense. We are at the stage where you need to clarify that.
How common is manipulation? If you set the bar on what constitutes manipulation very low, then it is very common, even including this discussion. But if it is very common, how can it be very bad? If you think that all gerrymandered concepts are “manipulative” in the sense of micro manipulations, where’s the problem?
I think this a central weakness of your case: you need to choose one of “manipulation common”, and “manipulation bad”.
Why? And for whom?
Well, if it’s only some of the time, you can achieve that by saying that scientists are special people who do have an obligation to be as objective as possible , but no obligation to be consequentialist. But that’s not novel.
That seems like a weakman to me. What about cases where coordination is of benefit to the people doing the coordinating...like obeying traffic laws? A speed limit is a gerrymandered concept.