An action does not naturally “have” an expected value, it is assigned an expected value by a combination of decision theory, prior, and utility function, so we can’t describe their disagreement as “about what action has the highest expected value”.
Fair enough. Let me try again: “They still disagree about what action is most likely to fulfill the agents desires when the agent is faced with Newcomb’s dilemma.” Or something like that.
But how do they (or you) know that they actually do disagree? According to their Taboo transcript, they do not disagree.
According to their Taboo transcript, they don’t disagree over the solutions of Newcomb’s problem recommended by EDT and CDT. But they might still disagree about whether EDT or CDT is most likely to fulfill the agent’s desires when faced with Newcomb’s problem.
It seems that there must be an alternative way to detect substantive disagreement, other than by asking people to Taboo?
Yes. Ask about anticipations.
If people actually disagree, but through the process of Tabooing conclude that they do not disagree (like in the above example), that should count as a lose for Tabooing, right?
That didn’t happen in this example. They do not, in fact, disagree over the solutions to Newcomb’s problem recommended by EDT and CDT. If they disagree, it’s about something else, like who is the tallest living person on Earth or which action is most likely to fulfill an agent’s desires when faced with Newcomb’s dilemma.
Of course Tabooing can go wrong, but it’s a useful tool. So is testing for differences of anticipation, though that can also go wrong.
In the case of “morality”, why do you trust the process of Tabooing so much that you do not give this possibility much credence?
No, I think it’s quite plausible that Tabooing can be done wrong when talking about morality. In fact, it may be more likely to go wrong there than anywhere else. But it’s also better to Taboo than to simply not use such a test for surface-level confusion. It’s also another option to not Taboo and instead propose that we try to decode the cognitive algorithms involved in order to get a clearer picture of our intuitive notion of moral terms than we can get using introspection and intuition.
“They still disagree about what action is most likely to fulfill the agents desires when the agent is faced with Newcomb’s dilemma.”
This introduces even more assumptions into the picture. Why fulfillment of desires or specifically agent desires is relevant? Why is “most likely” in there? You are trying to make things precise at the expense of accuracy, that’s the big taboo failure mode, increasingly obscure lost purposes.
I’m just providing an example. It’s not my story. I invite you or Wei Dai to say what it is the two speakers disagree about even after they agree about the conclusions of CDT and EDT for Newcomb’s problem. If all you can say is that they disagree about what they ‘should’ do, or what it would be ‘rational’ to do, then we’ll have to talk about things at that level of understanding, but that will be tricky.
If all you can say is that they disagree about what they ‘should’ do, or what it would be ‘rational’ to do, then we’ll have to talk about things at that level of understanding, but that will be tricky.
What other levels of understanding do we have? The question needs to be addressed on its own terms. Very tricky. There are ways of making this better, platonism extended to everything seems to help a lot, for example. Toy models of epistemic and decision-theoretic primitives also clarify things, training intuition.
We’re making progress on what it means for brains to value things, for example. Or we can talk in an ends-relational sense, and specify ends. Or we can keep things even more vague but then we can’t say much at all about ‘ought’ or ‘rational’.
The problem is that it doesn’t look any better than figuring out what CDT or EDT recommend. What the brain recommends is not automatically relevant to the question of what should be done.
The problem is that it doesn’t look any better than figuring out what CDT or EDT recommend. What the brain recommends is not automatically relevant to the question of what should be done.
If by ‘should’ in this sense you mean the ‘intended’ meaning of ‘should’ that we don’t have access too, then I agree.
Note: Wei Dai and I chatted for a while, and this resulted in three new clarifying paragraphs at the end of the is-ought section of my post ’Pluralistic Moral Reductionism.
Even given your disclaimer, I suspect we still disagree on the merits of Taboo as it apply to metaethics. Have you tried having others who are metaethically confused play Taboo in real life, and if so, did it help?
People like Eliezer and Drescher, von Neumann and Savage, have been able to make clear progress in understanding the nature of rationality, and the methods they used did not involve much (if any) neuroscience. On “morality” we don’t have such past successes to guide us, but your focus on neuroscience still seems misguided according to my intuitions.
Have you tried having others who are metaethically confused play Taboo in real life, and if so, did it help?
Yes. The most common result is that people come to realize they don’t know what they mean by ‘morally good’, unless they are theists.
People like Eliezer and Drescher, von Neumann and Savage, have been able to make clear progress in understanding the nature of rationality, and the methods they used did not involve much (if any) neuroscience. On “morality” we don’t have such past successes to guide us, but your focus on neuroscience still seems misguided according to my intuitions.
If it looks like I’m focusing on neuroscience, I think that’s an accident of looking at work I’ve produced in a 4-month period rather than over a longer period (that hasn’t occurred yet). I don’t think neuroscience is as central to metaethics or rationality as my recent output might suggest. Humans with meat-brains are strange agents who will make up a tiny minority of rational and moral agents in the history of intelligent agents in our light-cone (unless we bring an end to intelligent agents in our light-cone).
Yes. The most common result is that people come to realize they don’t know what they mean by ‘morally good’, unless they are theists.
Huh, I think that would have been good to mention in one of your posts. (Unless you did and I failed to notice it.)
It occurs to me that with a bit of tweaking to Austere Metaethics (which I’ll call Interim Metaethics), we can help everyone realize that they don’t know what they mean by “morally good”.
For example:
Deontologist: Should we build a deontic seed AI?
Interim Metaethicist: What do you mean by “should X”?
Deontologist: “X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z.”
Interim Metaethicist: Are you sure? If that’s really what you mean, then when a consequentialist says “should X” he probably means “X maximizes expected utility according to decision theory Y and utility function Z”. In which case the two of you do not actually disagree. But you do disagree with him, right?
Deontologist: Good point. I guess I don’t really mean that by “should”. I’m confused.
(Doesn’t that seem like an improvement over Austere Metaethics?)
I guess one difference between us is that I don’t see anything particularly ‘wrong’ with using stipulative definitions as long as you’re aware that they don’t match the intended meaning (that we don’t have access to yet), whereas you like to characterize stipulative definitions as ‘wrong’ when they don’t match the intended meaning.
But perhaps I should add one post before my empathic metaethics post which stresses that the stipulative definitions of ‘austere metaethics’ don’t match the intended meaning—and we can make this point by using all the standard thought experiments that deontologists and utilitarians and virtue ethicists and contractarian theorists use against each other.
After the above conversation, wouldn’t the deontologist want to figure out what he actually means by “should” and what its properties are? Why would he want to continue to use the stipulated definition that he knows he doesn’t actually mean? I mean I can imagine something like:
Deontologist: I guess I don’t really mean that by “should”, but I need to publish a few more papers for tenure, so please just help me figure out whether we should build an deontic seed AI under that stipulated definition of “should”, so I can finish my paper and submit it to the Journal of Machine Deontology.
But even in this case it would make more sense for him to avoid “stipulative definition” and instead say
Deontologist: Ok, by “should” I actually mean a concept that I can’t define at this point. But I guess it has something to do with deontic logic, and it would be useful to explore the properties of deontic logic in more detail. So, can you please help me figure out whether building a deontic seed AI is obligatory (by deontic logic) if we assume axiomatic imperatives Y and Z?
This way, he clarifies to himself and others that “”X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z.” is not what he means by “should X”, but instead a guess about the nature of morality (a concept that we can’t yet precisely define).
Perhaps you’d answer that a stipulated meaning is just that, a guess about the nature of something. But as you know, words have connotations, and I think the connotation of “guess” is more appropriate here than “meaning”.
Perhaps you’d answer that a stipulated meaning is just that, a guess about the nature of something. But as you know, words have connotations, and I think the connotation of “guess” is more appropriate here than “meaning”.
The problem is that we have to act in the world now. We can’t wait around for metaethics and decision theory to be solved. Thus, science books have glossaries in the back full of highly useful operationalized and stiuplated definitions for hundreds of terms, whether or not they match the intended meanings (that we don’t have access to) of those terms for person A, or the intended meanings of those terms for person B, or the intended meanings for those terms for person C.
I think this glossary business is a familiar enough practice that calling that thing a glossary of ‘meanings’ instead of a glossary of ‘guesses at meanings’ is fine. Maybe ‘meaning’ doesn’t have the connotations for me that it has for you.
Science needs doing, laws need to be written and enforced, narrow AIs need to be programmed, best practices in medicine need to be written, agents need to act… all before metaethics and decision theory are solved. In a great many cases, we need to have meaning_stipulated before we can figure out meaning_intended.
Sigh… Maybe I should just put a sticky note on my monitor that says
REMEMBER: You probably don’t actually disagree with Luke, because whenever he says “X means Z by Y”, he might just mean “X stipulated Y to mean Z”, which in turn is just another way of saying “X guesses that the nature of Y is Z”.
We humans have different intuitions about the meanings of terms and the nature of meaning itself, and thus we’re all speaking slightly different languages. We always need to translate between our languages, which is where Taboo and testing for anticipations come in handy.
I’m using the concept of meaning from linguistics, which seems fair to me. In linguistics, stipulated meaning is most definitely a kind of meaning (and not merely a kind of guessing at meaning), for it is often “what is expressed by the writer or speaker, and what is conveyed to the reader or listener, provided that they talk about the same thing.”
Whatever the case, this language looks confusing/misleading enough to avoid. It conflates the actual search for intended meaning with all those irrelevant stipulations, and assigns misleading connotations to the words referring to these things. In Eliezer’s sequences, the term was “fake utility function”. The presence of “fake” in the term is important, it reminds of incorrectness of the view.
So far, you’ve managed to confuse me and Wei with this terminology alone, probably many others as well.
So far, you’ve managed to confuse me and Wei with this terminology alone, probably many others as well.
Perhaps, though I’ve gotten comments from others that it was highly clarifying for them. Maybe they’re more used to the meaning of ‘meaning’ from linguistics.
But one must not fall into the trap of thinking that a definition you’ve stipulated (aloud or in your head) for ‘ought’ must match up to your intuitive concept of ‘ought’. In fact, I suspect it never does, which is why the conceptual analysis of ‘ought’ language can go in circles for thousands of years, and why any stipulated meaning of ‘ought’ is a fake utility function. To see clearly to our intuitive concept of ought, we’ll have to try empathic metaethics (see below).
It’s not clear from this paragraph whether “intuitive concept” refers to the oafish tools in human brain (which have the same problems as stipulated definitions, including irrelevance) or the intended meaning that those tools seek. Conceptual analysis, as I understand, is concerned with analysis of the imperfect intuitive tools, so it’s also unclear in what capacity you mention conceptual analysis here.
(I do think this and other changes will probably make new readers less confused.)
Roger has an intuitive concept of ‘morally good’, the intended meaning of which he doesn’t fully have access to (but it could be discovered by something like CEV). Roger is confused enough to think that his intuitive concept of ‘morally good’ is ‘that which produces the greatest pleasure for the greatest number’.
The conceptual analyst comes along and says: “Suppose that an advanced team of neuroscientists and computer scientists could hook everyone’s brains up to a machine that gave each of them maximal, beyond-orgasmic pleasure for the rest of their abnormally long lives. Then they will blast each person and their pleasure machine into deep space at near light-speed so that each person could never be interfered with. Would this be morally good?”
ROGER: Huh. I guess that’s not quite what I mean by ‘morally good’. I think what I mean by ‘morally good’ is ‘that which produces the greatest subjective satisfaction of wants in the greatest number’.
CONCEPTUAL ANALYST: Okay, then. Suppose that an advanced team of neuroscientists and computer scientists could hook everyone’s brains up to ‘The Matrix’ and made them believe and feel that all their wants were being satisfied, for the rest of their abnormally long lives. Then they will blast each person and their pleasure machine into deep space at near light-speed so that each person could never be interfered with. Would this be morally good?
ROGER: No, I guess that’s not what I mean, either. What I really mean is...
And around and around we go, for centuries.
The problem with trying to access our intended meaning for ‘morally good’ by this intuitive process is that it brings into play, as you say, all the ‘oafish tools’ in the human brain. And philosophers have historically not paid much attention to the science of how intuitions work.
Roger is confused enough to think that his intuitive concept of ‘morally good’ is ‘that which produces the greatest pleasure for the greatest number’.
That intuition says the same thing as “pleasure-maximization”, or that intended meaning can be captured as “pleasure-maximization”? Even if intuition is saying exactly “pleasure-maximization”, it’s not necessarily the intended meaning, and so it’s unclear why one would try to replicate the intuitive tool, rather than search for a characterization of the intended meaning that is better than the intuitive tool. This is the distinction I was complaining about.
(This is an isolated point unrelated to the rest of your comment.)
Understood. I think I’m trying to figure out if there’s a better way to talk about this ‘intended meaning’ (that we don’t yet have access to) than to say ‘intended meaning’ or ‘intuitive meaning’. But maybe I’ll just have to say ‘intended meaning (that we don’t yet have access to)’.
New paragraph version:
But one must not fall into the trap of thinking that a definition you’ve stipulated (aloud or in your head) for ‘ought’ must match up to your intended meaning of ‘ought’ (to which you don’t have introspective access). In fact, I suspect it never does, which is why the conceptual analysis of ‘ought’ language can go in circles for centuries, and why any stipulated meaning of ‘ought’ is a fake utility function. To see clearly to our intuitive concept of ought, we’ll have to try empathic metaethics (see below).
I’ve been veryclearmanytimes that ‘austere metaethics’ is for clearing up certain types of confusions, but won’t do anything to solve FAI, which is why we need ‘empathic metaethics’.
I was discussing that particular comment, not rehashing the intention behind ‘austere metaethics’.
More specifically, you made a statement “We can’t wait around for metaethics and decision theory to be solved.” It’s not clear to me what purpose is being served by what alternative action to “waiting around for metaethics to be solved”. It looks like you were responding to Wei’s invitation to justify the use of word “meaning” instead of “guess”, but it’s not clear how your response relates to that question.
Like I said over here, I’m using the concept of ‘meaning’ from linguistics. I’m hoping that fewer people are confused by my use of ‘meaning’ as employed in the field that studies meaning than if I had used ‘meaning’ in a more narrow and less standard way, like Wei Dai’s. Perhaps I’m wrong about that, but I’m not sure.
My comment above about how “we have to act in the world now” gives one reason why, I suspect, the linguist’s sense of ‘meaning’ includes stipulated meaning, and why stipulated meaning is so common.
In any case, I think you and Wei Dai have helped me think about how to be more clear to more people by adding such clarifications as this.
In those paragraphs, you add intuition as an alternative to stipulated meaning. But this is not what we are talking about, we are talking about some unknown, but normative meaning that can’t be presently stipulated, and is referred partly through intuition in a way that is more accurate than any currently available stipulation. What intuition tells is as irrelevant as what the various stipulations tell, what matters is the thing that the imperfect intuition refers. This idea doesn’t require a notion of automated stipulation (“empathic” discussion).
“some unknown, but normative meaning that can’t be presently stipulated” is what I meant by “intuitive meaning” in this case.
automated stipulation (“empathic” discussion)
I’ve never thought of ‘empathic’ discussion as ‘automated stipulation’. What do you mean by that?
Even our stipulated definitions are only promissory notes for meaning. Luckily, stipulated definitions can be quite useful for achieving our goals. Figuring out what we ‘really want’, or what we ‘rationally ought to do’ when faced with Newcomb’s problem, would also be useful. Such terms are carry even more vague promissory notes for meaning than stipulated definitions, and yet they are worth pursuing.
Treat intuition as just another stipulated definition, that happens to be expressed as a pattern of mind activity, as opposed to a sequence of words. The intuition itself doesn’t define the thing it refers to, it can be slightly wrong, or very wrong. The same goes for words. Both intuition and various words we might find are tools for referring to some abstract structure (intended meaning), that is not accurately captured by any of these tools. The purpose of intuition, and of words, is in capturing this structure accurately, accessing its properties. We can develop better understanding by inventing new words, training new intuitions, etc.
None of these tools hold a privileged position with respect to the target structure, some of them just happen to more carefully refer to it. At the beginning of any investigation, we would typically only have intuitions, which specify the problem that needs solving. They are inaccurate fuzzy lumps of confusion, too. At the same time, any early attempt at finding better tools will be unsuccessful, explicit definitions will fail to capture the intended meaning, even as intuition doesn’t capture it precisely. Attempts at guiding intuition to better precision can likewise make it a less accurate tool for accessing the original meaning. On the other hand, when the topic is well-understood, we might find an explicit definition that is much better than the original intuition. We might train new intuitions that reflect the new explicit definition, and are much better tools than the original intuition.
And as far as I can tell, you don’t agree. You express agreement too much, like your stipulated-meaning thought experiments, this is one of the problems. But I’d probably need a significantly more clear presentation of what feels wrong to make progress on our disagreement.
Fair enough. Let me try again: “They still disagree about what action is most likely to fulfill the agents desires when the agent is faced with Newcomb’s dilemma.” Or something like that.
According to their Taboo transcript, they don’t disagree over the solutions of Newcomb’s problem recommended by EDT and CDT. But they might still disagree about whether EDT or CDT is most likely to fulfill the agent’s desires when faced with Newcomb’s problem.
Yes. Ask about anticipations.
That didn’t happen in this example. They do not, in fact, disagree over the solutions to Newcomb’s problem recommended by EDT and CDT. If they disagree, it’s about something else, like who is the tallest living person on Earth or which action is most likely to fulfill an agent’s desires when faced with Newcomb’s dilemma.
Of course Tabooing can go wrong, but it’s a useful tool. So is testing for differences of anticipation, though that can also go wrong.
No, I think it’s quite plausible that Tabooing can be done wrong when talking about morality. In fact, it may be more likely to go wrong there than anywhere else. But it’s also better to Taboo than to simply not use such a test for surface-level confusion. It’s also another option to not Taboo and instead propose that we try to decode the cognitive algorithms involved in order to get a clearer picture of our intuitive notion of moral terms than we can get using introspection and intuition.
This introduces even more assumptions into the picture. Why fulfillment of desires or specifically agent desires is relevant? Why is “most likely” in there? You are trying to make things precise at the expense of accuracy, that’s the big taboo failure mode, increasingly obscure lost purposes.
I’m just providing an example. It’s not my story. I invite you or Wei Dai to say what it is the two speakers disagree about even after they agree about the conclusions of CDT and EDT for Newcomb’s problem. If all you can say is that they disagree about what they ‘should’ do, or what it would be ‘rational’ to do, then we’ll have to talk about things at that level of understanding, but that will be tricky.
What other levels of understanding do we have? The question needs to be addressed on its own terms. Very tricky. There are ways of making this better, platonism extended to everything seems to help a lot, for example. Toy models of epistemic and decision-theoretic primitives also clarify things, training intuition.
We’re making progress on what it means for brains to value things, for example. Or we can talk in an ends-relational sense, and specify ends. Or we can keep things even more vague but then we can’t say much at all about ‘ought’ or ‘rational’.
The problem is that it doesn’t look any better than figuring out what CDT or EDT recommend. What the brain recommends is not automatically relevant to the question of what should be done.
If by ‘should’ in this sense you mean the ‘intended’ meaning of ‘should’ that we don’t have access too, then I agree.
Note: Wei Dai and I chatted for a while, and this resulted in three new clarifying paragraphs at the end of the is-ought section of my post ’Pluralistic Moral Reductionism.
Some remaining issues:
Even given your disclaimer, I suspect we still disagree on the merits of Taboo as it apply to metaethics. Have you tried having others who are metaethically confused play Taboo in real life, and if so, did it help?
People like Eliezer and Drescher, von Neumann and Savage, have been able to make clear progress in understanding the nature of rationality, and the methods they used did not involve much (if any) neuroscience. On “morality” we don’t have such past successes to guide us, but your focus on neuroscience still seems misguided according to my intuitions.
Yes. The most common result is that people come to realize they don’t know what they mean by ‘morally good’, unless they are theists.
If it looks like I’m focusing on neuroscience, I think that’s an accident of looking at work I’ve produced in a 4-month period rather than over a longer period (that hasn’t occurred yet). I don’t think neuroscience is as central to metaethics or rationality as my recent output might suggest. Humans with meat-brains are strange agents who will make up a tiny minority of rational and moral agents in the history of intelligent agents in our light-cone (unless we bring an end to intelligent agents in our light-cone).
Huh, I think that would have been good to mention in one of your posts. (Unless you did and I failed to notice it.)
It occurs to me that with a bit of tweaking to Austere Metaethics (which I’ll call Interim Metaethics), we can help everyone realize that they don’t know what they mean by “morally good”.
For example:
Deontologist: Should we build a deontic seed AI?
Interim Metaethicist: What do you mean by “should X”?
Deontologist: “X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z.”
Interim Metaethicist: Are you sure? If that’s really what you mean, then when a consequentialist says “should X” he probably means “X maximizes expected utility according to decision theory Y and utility function Z”. In which case the two of you do not actually disagree. But you do disagree with him, right?
Deontologist: Good point. I guess I don’t really mean that by “should”. I’m confused.
(Doesn’t that seem like an improvement over Austere Metaethics?)
I guess one difference between us is that I don’t see anything particularly ‘wrong’ with using stipulative definitions as long as you’re aware that they don’t match the intended meaning (that we don’t have access to yet), whereas you like to characterize stipulative definitions as ‘wrong’ when they don’t match the intended meaning.
But perhaps I should add one post before my empathic metaethics post which stresses that the stipulative definitions of ‘austere metaethics’ don’t match the intended meaning—and we can make this point by using all the standard thought experiments that deontologists and utilitarians and virtue ethicists and contractarian theorists use against each other.
After the above conversation, wouldn’t the deontologist want to figure out what he actually means by “should” and what its properties are? Why would he want to continue to use the stipulated definition that he knows he doesn’t actually mean? I mean I can imagine something like:
Deontologist: I guess I don’t really mean that by “should”, but I need to publish a few more papers for tenure, so please just help me figure out whether we should build an deontic seed AI under that stipulated definition of “should”, so I can finish my paper and submit it to the Journal of Machine Deontology.
But even in this case it would make more sense for him to avoid “stipulative definition” and instead say
Deontologist: Ok, by “should” I actually mean a concept that I can’t define at this point. But I guess it has something to do with deontic logic, and it would be useful to explore the properties of deontic logic in more detail. So, can you please help me figure out whether building a deontic seed AI is obligatory (by deontic logic) if we assume axiomatic imperatives Y and Z?
This way, he clarifies to himself and others that “”X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z.” is not what he means by “should X”, but instead a guess about the nature of morality (a concept that we can’t yet precisely define).
Perhaps you’d answer that a stipulated meaning is just that, a guess about the nature of something. But as you know, words have connotations, and I think the connotation of “guess” is more appropriate here than “meaning”.
The problem is that we have to act in the world now. We can’t wait around for metaethics and decision theory to be solved. Thus, science books have glossaries in the back full of highly useful operationalized and stiuplated definitions for hundreds of terms, whether or not they match the intended meanings (that we don’t have access to) of those terms for person A, or the intended meanings of those terms for person B, or the intended meanings for those terms for person C.
I think this glossary business is a familiar enough practice that calling that thing a glossary of ‘meanings’ instead of a glossary of ‘guesses at meanings’ is fine. Maybe ‘meaning’ doesn’t have the connotations for me that it has for you.
Science needs doing, laws need to be written and enforced, narrow AIs need to be programmed, best practices in medicine need to be written, agents need to act… all before metaethics and decision theory are solved. In a great many cases, we need to have meaning_stipulated before we can figure out meaning_intended.
Sigh… Maybe I should just put a sticky note on my monitor that says
REMEMBER: You probably don’t actually disagree with Luke, because whenever he says “X means Z by Y”, he might just mean “X stipulated Y to mean Z”, which in turn is just another way of saying “X guesses that the nature of Y is Z”.
That might work.
We humans have different intuitions about the meanings of terms and the nature of meaning itself, and thus we’re all speaking slightly different languages. We always need to translate between our languages, which is where Taboo and testing for anticipations come in handy.
I’m using the concept of meaning from linguistics, which seems fair to me. In linguistics, stipulated meaning is most definitely a kind of meaning (and not merely a kind of guessing at meaning), for it is often “what is expressed by the writer or speaker, and what is conveyed to the reader or listener, provided that they talk about the same thing.”
Whatever the case, this language looks confusing/misleading enough to avoid. It conflates the actual search for intended meaning with all those irrelevant stipulations, and assigns misleading connotations to the words referring to these things. In Eliezer’s sequences, the term was “fake utility function”. The presence of “fake” in the term is important, it reminds of incorrectness of the view.
So far, you’ve managed to confuse me and Wei with this terminology alone, probably many others as well.
Perhaps, though I’ve gotten comments from others that it was highly clarifying for them. Maybe they’re more used to the meaning of ‘meaning’ from linguistics.
Does this new paragraph at the end of this section in PMR help?
It’s not clear from this paragraph whether “intuitive concept” refers to the oafish tools in human brain (which have the same problems as stipulated definitions, including irrelevance) or the intended meaning that those tools seek. Conceptual analysis, as I understand, is concerned with analysis of the imperfect intuitive tools, so it’s also unclear in what capacity you mention conceptual analysis here.
(I do think this and other changes will probably make new readers less confused.)
Here’s the way I’m thinking about it.
Roger has an intuitive concept of ‘morally good’, the intended meaning of which he doesn’t fully have access to (but it could be discovered by something like CEV). Roger is confused enough to think that his intuitive concept of ‘morally good’ is ‘that which produces the greatest pleasure for the greatest number’.
The conceptual analyst comes along and says: “Suppose that an advanced team of neuroscientists and computer scientists could hook everyone’s brains up to a machine that gave each of them maximal, beyond-orgasmic pleasure for the rest of their abnormally long lives. Then they will blast each person and their pleasure machine into deep space at near light-speed so that each person could never be interfered with. Would this be morally good?”
ROGER: Huh. I guess that’s not quite what I mean by ‘morally good’. I think what I mean by ‘morally good’ is ‘that which produces the greatest subjective satisfaction of wants in the greatest number’.
CONCEPTUAL ANALYST: Okay, then. Suppose that an advanced team of neuroscientists and computer scientists could hook everyone’s brains up to ‘The Matrix’ and made them believe and feel that all their wants were being satisfied, for the rest of their abnormally long lives. Then they will blast each person and their pleasure machine into deep space at near light-speed so that each person could never be interfered with. Would this be morally good?
ROGER: No, I guess that’s not what I mean, either. What I really mean is...
And around and around we go, for centuries.
The problem with trying to access our intended meaning for ‘morally good’ by this intuitive process is that it brings into play, as you say, all the ‘oafish tools’ in the human brain. And philosophers have historically not paid much attention to the science of how intuitions work.
Does that make sense?
That intuition says the same thing as “pleasure-maximization”, or that intended meaning can be captured as “pleasure-maximization”? Even if intuition is saying exactly “pleasure-maximization”, it’s not necessarily the intended meaning, and so it’s unclear why one would try to replicate the intuitive tool, rather than search for a characterization of the intended meaning that is better than the intuitive tool. This is the distinction I was complaining about.
(This is an isolated point unrelated to the rest of your comment.)
Understood. I think I’m trying to figure out if there’s a better way to talk about this ‘intended meaning’ (that we don’t yet have access to) than to say ‘intended meaning’ or ‘intuitive meaning’. But maybe I’ll just have to say ‘intended meaning (that we don’t yet have access to)’.
New paragraph version:
You think this applies to figuring out decision theory for FAI? If not, how is that relevant in this context?
Vladimir,
I’ve been very clear many times that ‘austere metaethics’ is for clearing up certain types of confusions, but won’t do anything to solve FAI, which is why we need ‘empathic metaethics’.
I was discussing that particular comment, not rehashing the intention behind ‘austere metaethics’.
More specifically, you made a statement “We can’t wait around for metaethics and decision theory to be solved.” It’s not clear to me what purpose is being served by what alternative action to “waiting around for metaethics to be solved”. It looks like you were responding to Wei’s invitation to justify the use of word “meaning” instead of “guess”, but it’s not clear how your response relates to that question.
Like I said over here, I’m using the concept of ‘meaning’ from linguistics. I’m hoping that fewer people are confused by my use of ‘meaning’ as employed in the field that studies meaning than if I had used ‘meaning’ in a more narrow and less standard way, like Wei Dai’s. Perhaps I’m wrong about that, but I’m not sure.
My comment above about how “we have to act in the world now” gives one reason why, I suspect, the linguist’s sense of ‘meaning’ includes stipulated meaning, and why stipulated meaning is so common.
In any case, I think you and Wei Dai have helped me think about how to be more clear to more people by adding such clarifications as this.
(This is similar to my reaction expressed here.)
In those paragraphs, you add intuition as an alternative to stipulated meaning. But this is not what we are talking about, we are talking about some unknown, but normative meaning that can’t be presently stipulated, and is referred partly through intuition in a way that is more accurate than any currently available stipulation. What intuition tells is as irrelevant as what the various stipulations tell, what matters is the thing that the imperfect intuition refers. This idea doesn’t require a notion of automated stipulation (“empathic” discussion).
“some unknown, but normative meaning that can’t be presently stipulated” is what I meant by “intuitive meaning” in this case.
I’ve never thought of ‘empathic’ discussion as ‘automated stipulation’. What do you mean by that?
Even our stipulated definitions are only promissory notes for meaning. Luckily, stipulated definitions can be quite useful for achieving our goals. Figuring out what we ‘really want’, or what we ‘rationally ought to do’ when faced with Newcomb’s problem, would also be useful. Such terms are carry even more vague promissory notes for meaning than stipulated definitions, and yet they are worth pursuing.
My understanding of this topic is as follows.
Treat intuition as just another stipulated definition, that happens to be expressed as a pattern of mind activity, as opposed to a sequence of words. The intuition itself doesn’t define the thing it refers to, it can be slightly wrong, or very wrong. The same goes for words. Both intuition and various words we might find are tools for referring to some abstract structure (intended meaning), that is not accurately captured by any of these tools. The purpose of intuition, and of words, is in capturing this structure accurately, accessing its properties. We can develop better understanding by inventing new words, training new intuitions, etc.
None of these tools hold a privileged position with respect to the target structure, some of them just happen to more carefully refer to it. At the beginning of any investigation, we would typically only have intuitions, which specify the problem that needs solving. They are inaccurate fuzzy lumps of confusion, too. At the same time, any early attempt at finding better tools will be unsuccessful, explicit definitions will fail to capture the intended meaning, even as intuition doesn’t capture it precisely. Attempts at guiding intuition to better precision can likewise make it a less accurate tool for accessing the original meaning. On the other hand, when the topic is well-understood, we might find an explicit definition that is much better than the original intuition. We might train new intuitions that reflect the new explicit definition, and are much better tools than the original intuition.
As far as I can tell, I agree with all of this.
And as far as I can tell, you don’t agree. You express agreement too much, like your stipulated-meaning thought experiments, this is one of the problems. But I’d probably need a significantly more clear presentation of what feels wrong to make progress on our disagreement.
I look forward to it.
I’m not sure what you mean by “you agree too much”, though. Like I said, as far as I can tell I agree with everything in this comment of yours.