I hadn’t come across the von Neumann-Morgenstern utility theorem before reading this post, thanks for drawing it to my attention.
Looking at Moral Philosophy through the lens of agents working with utility/value functions is an interesting exercise; it’s something I’m still working on. In the long run, I think some deep thinking needs to be done about what we end up selecting as terminal values, and how we incorporate them into a utility function. (I hope that isn’t stating something that is blindingly obvious.)
I guess where you might be headed is into Meta-Ethics. As I understand it, meta-ethics includes debates on moral relativism that are closely related to the existence terminal/intrinsic values. Moral relativism asserts that all values are subjective (i.e., only the beliefs of individuals), rather than objective (i.e., universally true). So no practice or activity is inherently right or wrong, it is just the perception of people that makes it so. As you might imagine, this can be used as a defense of violent cultural practices (it could even be used in defense of baby-eating).
I tend to agree with the position of moral relativism; unfortunate though it may be, I’m not convinced there are things that are objectively valuable. I’m of the belief that if there are no agents to value something, then that something has effectively no value. That holds for people and their values too. That said, we do exist, and I think subjective values count for something.
Humanity has come to some degree of consensus over what should be valued. Probably largely as a result of evolution and social conditioning. So from here, I think it mightn’t be wasted effort to explore the selection of different intrinsic values.
Luke Muehlhauser has called morality an engineering problem. While Sam Harris has described morality as a landscape, i.e., the surface is our terminal value we are trying to maximize (Harris picked the well-being of conscious creatures) and the societal practices as the variables. Though I don’t know that well-being is the best terminal value, I like the idea of treating morality as an optimization problem. I think this is a reasonable way to view ethics. Without objective values, it might just be a matter of testing different sets of terminal subjective values, until we find the optimum (an hopefully don’t get trapped in a local maximum).
Nevertheless, I think it’s interesting to suppose that something is objectively valuable. It doesn’t seem like a stretch to me to say that the knowledge of what is objectively valuable, would, itself, be objectively valuable. And that the search for that knowledge would probably also be objectively valuable. After all that, it would be somewhat ironic if it turned out the universal objective values don’t include the survival of life on Earth.
I’m not convinced there are things that are objectively valuable. I’m of the belief that if there are no agents to value something, then that something has effectively no value.
The second sentence doesn’t follow from the first. If rational agents converge on their values, that is objective enough. Analogy: one can accept that mathematical truth is objective (mathematicians will converge) without being a Platonists (mathematical truths have an existence separate from humans)
Without objective values, it might just be a matter of testing different sets of terminal subjective values, until we find the optimum (an hopefully don’t get trapped in a local maximum).
I fin d that hard to follow. If the test i rationally justifiable, and leads to the uniform results, how is that not objective?
You seem to be using “objective” (having a truth value independent of individual humans) to mean what I would mean by “real” (having existence independent of humans).
First of all, thanks for the comment. You have really motivated me to read and think about this more—starting with getting clearer on the meanings of “objective”, “subjective”, and “intrinsic”. I apologise for any confusion caused by my incorrect use of terminology. I guess that is why Eliezer likes to taboo words. I hope you don’t mind me persisting in trying to explain my view and using those “taboo” words.
Since I was talking about meta-ethical moral relativism, I hope that it was sufficiently clear that I was referring to moral values. What I meant by “objective values” was “objectively true moral values” or “objectively true intrinsic values”.
The second sentence doesn’t follow from the first.
The second sentence was an explanation of the first: not logically derived from the first sentence, but a part of the argument. I’ll try to construct my arguments more linearly in future.
If I had to rephrase that passage I’d say:
If there are no agents to value something, intrinsically or extrinsically, then there is also nothing to act on those values. In the absence of agents to act, values are effectively meaningless. Therefore, I’m not convinced that there is objective truth in intrinsic or moral values.
However, the lack of meaningful values in the absence of agents hints at agents themselves being valuable. If value can only have meaning in the presence of an agent, then that agent probably has, at the very least, extrinsic/instrumental value. Even a paperclip maximiser would probably consider itself to have instrumental value, right?
If rational agents converge on their values, that is objective enough.
I think there is a difference between it being objectively true that, in certain circumstances, the values of rational agents converge, and it being objectively true that those values are moral. A rational agent can do really “bad” things if the beliefs and intrinsic values on which it is acting are “bad”. Why else would anyone be scared of AI?
Analogy: one can accept that mathematical truth is objective (mathematicians will converge) without being a Platonists (mathematical truths have an existence separate from humans)
I accept the possibility of objective truth values. I’m not convinced that it is objectively true that the convergence of subjectively true moral values indicates objectively true moral values. As far as values go, moral values don’t seem to be as amenable to rigorous proofs as formal mathematical theorems. We could say that intrinsic values seem to be analogous to mathematical axioms.
I fin d that hard to follow. If the test i rationally justifiable, and leads to the uniform results, how is that not objective?
I’ll have a go at clarifying that passage with the right(?) terminology:
Without the objective truth of intrinsic values, it might just be a matter of testing different sets of assumed intrinsic values until we find an “optimal” or acceptable convergent outcome.
Morality might be somewhat like an NP-hard optimisation problem. It might be objectively true that we get a certain result from a test. It’s more difficult to say that it is objectively true that we have solved a complex optimisation problem.
You seem to be using “objective” (having a truth value independent of individual humans) to mean what I would mean by “real” (having existence independent of humans).
Thanks for informing me that my use of the term “objective” was confused/confusing. I’ll keep trying to improve the clarity of my communication and understanding of the terminology.
First of all, thanks for the comment. You have really motivated me to read and think about this more
That’s what I like to hear!
If there are no agents to value something, intrinsically or extrinsically, then there is also nothing to act on those values. In the absence of agents to act, values are effectively meaningless. Therefore, I’m not convinced that there is objective truth in intrinsic or moral values.
But there is no need for morality in the absence of agents. When agents are there, values will be there, when agents are not there, the absence of values doesn’t matter.
I think there is a difference between it being objectively true that, in certain circumstances, the values of rational agents converge, and it being objectively true that those values are moral. A rational agent can do really “bad” things if the beliefs and intrinsic values on which it is acting are “bad”. Why else would anyone be scared of AI?
I don’t require their values to converge, I require them to accept the truths of certain claims. This happens in real
life. People say “I don’t like X, but I respect your right to do it”. The first part says X is a disvalue, the second is an override coming from rationality.
I’m assuming a lot of background in this post that you don’t seem to have. Have you read the sequences, specifically the metaethics stuff?
Moral philosophy on LW is decades (at the usual philosophical pace) ahead of what you would learn elsewhere and a lot of the stuff you mentioned is considered solved or obsolete.
Moral philosophy on LW is decades (at the usual philosophical pace) ahead of what you would learn elsewhere
Really? That’s kind of scary if true. Moral philosophy on LW doesn’t strike me as especially well developed (particularly compared to other rationality related subjects LW covers).
Moral philosophy is not well developed on LW, but I think it’s further than it is elsewhere, and when I look at the pace of developments in philosophy, it looks like it will take decades for everyone else to catch up. Maybe I’m underestimating the quality of mainstream philosophy, though.
All I know is that people who are interested in moral philosophy who haven’t been exposed to LW are a lot more confused than those on LW. And that those on LW are more confused than they think they are (hence the OP).
Moral philosophy is not well developed on LW, but I think it’s further than it is elsewhere
What do you think represents the best moral philosophy that LW has to offer?
And that those on LW are more confused than they think they are (hence the OP).
Just a few months ago you seemed to be saying that we didn’t need to study moral philosophy, but just try to maximize “awesomeness”, which “You already know that you know how to compute”. I find it confusing that this post doesn’t mention that one at all. Have you changed your mind since then, if so why? Or are you clarifying your position, or something else?
What do you think represents the best moral philosophy that LW has to offer?
The metaethics sequence sinks most of the standard confusions, though it doesn’t offer actual conclusions or procedures.
Complexity of value. Value being human specific. morality as optimization target. etc.
Maybe it’s just the epistemic quality around here though. LWers talking about morality are able to go much further without getting derailed than the best I’ve seen elsewhere, even if there weren’t much good work on moral philosophy on LW.
Just a few months ago you seemed to be saying that we didn’t need to study moral philosophy, but just try to maximize “awesomeness”, which “You already know that you know how to compute”. I find it confusing that this post doesn’t mention that one at all. Have you changed your mind since then, if so why? Or are you clarifying your position, or something else?
Right. This is a good question.
For actually making decisions, use Awesomeness or something as your moral proxy, because it more or less just works. For those of us who want to go deeper and understand the theory of morality declaratively, the OP applies; we basically don’t have any good theory. They are two sides of the same coin; the situation in moral philosophy is like the situation in physics a few hundred (progress subjective) years ago, and we need to recognize this before trying to build the house on sand, so to speak. So we are better off just using our current buggy procedural morality.
I could have made the connection clearer I suppose.
This post is actually a sort of precurser to some new and useful (I hope) work on the subject that I’ve written up but haven’t gotten around to polishing and posting. I have maybe 5 posts worth of morality related stuff in the works, and then I’m getting out of this godforsaken dungeon.
For actually making decisions, use Awesomeness or something as your moral proxy, because it more or less just works.
Given that we don’t have a good explicit theory of what morality really is, how do you know (and how could you confidently claim in that earlier post) that Awesomeness is a good moral proxy?
So we are better off just using our current buggy procedural morality.
I think I understand what you’re saying now, thanks for the clarification. However, my current buggy procedural morality is not “maximize awesomeness” but more like an instinctive version of Bostrom and Ord’s moral parliament.
You don’t tend to find much detailed academic discussion regarding metaethical philosophy on the blogosphere at all.
Disclaimers: strictly comparing it to other subjects which I consider similar from an outside view, and supported only by personal experience and observation.
Have you read the sequences, specifically the metaethics stuff?
I have, and I found it unclear and inconclusive. A number of people have offered to explain it , and they all ended up bowing out unable to do so
Moral philosophy on LW is decades (at the usual philosophical pace) ahead of what you would learn elsewhere and a lot of the stuff you mentioned is considered solved or obsolete.
Sorry, I have only read selections of the sequences, and not many of the posts on metaethics. Though as far as I’ve gotten, I’m not convinced that the sequences really solve, or make obsolete, many of the deeper problems or moral philosophy.
The original post, and this one, seems to be running into the “is-ought” gap and moral relativism. Being unable to separate terminal values from biases is due to there being no truly objective terminal values. Despite Eliezer’s objections, this is a fundamental problem for determining what terminal values or utility function we should use—a task you and I are both interested in undertaking.
I think this community vastly over-estimates its grip on meta-ethical concepts like moral realism or moral anti-realism. (E.g. the hopelessly confused discussion in this thread). I don’t think the meta-ethics sequence resolves these sorts of basic issues.
I’m still coming to terms with the philosophical definitions of different positions and their implications, and the Stanford Encyclopedia of Philosophy seems like a more rounded account of the different view points than the meta-ethics sequences. I think I might be better off first spending my time continuing to read the SEP and trying to make my own decisions, and then reading the meta-ethics sequences with that understanding of the philosophical background.
By the way, I can see your point that objections to moral anti-realism in this community may be somewhat motivated by the possibility that friendly AI becomes unprovable. As I understand it, any action can be “rational” if the value/utility function is arbitrary.
There is a lot of diversity of opinions in philosophers and that may be true as a whole of the discipline, there is some good stuff to be found there. I’d recommend staying here for the most part rather than wading through philosophy elsewhere, though.
Also, many moral philosophers may have very different moral sentiments from you and that maybe that makes them seem like idiots more than they actually are. Different moral sentiments as to whether consequentialism rather than just within consequentialism among other things.
I hadn’t come across the von Neumann-Morgenstern utility theorem before reading this post, thanks for drawing it to my attention.
Looking at Moral Philosophy through the lens of agents working with utility/value functions is an interesting exercise; it’s something I’m still working on. In the long run, I think some deep thinking needs to be done about what we end up selecting as terminal values, and how we incorporate them into a utility function. (I hope that isn’t stating something that is blindingly obvious.)
I guess where you might be headed is into Meta-Ethics. As I understand it, meta-ethics includes debates on moral relativism that are closely related to the existence terminal/intrinsic values. Moral relativism asserts that all values are subjective (i.e., only the beliefs of individuals), rather than objective (i.e., universally true). So no practice or activity is inherently right or wrong, it is just the perception of people that makes it so. As you might imagine, this can be used as a defense of violent cultural practices (it could even be used in defense of baby-eating).
I tend to agree with the position of moral relativism; unfortunate though it may be, I’m not convinced there are things that are objectively valuable. I’m of the belief that if there are no agents to value something, then that something has effectively no value. That holds for people and their values too. That said, we do exist, and I think subjective values count for something.
Humanity has come to some degree of consensus over what should be valued. Probably largely as a result of evolution and social conditioning. So from here, I think it mightn’t be wasted effort to explore the selection of different intrinsic values.
Luke Muehlhauser has called morality an engineering problem. While Sam Harris has described morality as a landscape, i.e., the surface is our terminal value we are trying to maximize (Harris picked the well-being of conscious creatures) and the societal practices as the variables. Though I don’t know that well-being is the best terminal value, I like the idea of treating morality as an optimization problem. I think this is a reasonable way to view ethics. Without objective values, it might just be a matter of testing different sets of terminal subjective values, until we find the optimum (an hopefully don’t get trapped in a local maximum).
Nevertheless, I think it’s interesting to suppose that something is objectively valuable. It doesn’t seem like a stretch to me to say that the knowledge of what is objectively valuable, would, itself, be objectively valuable. And that the search for that knowledge would probably also be objectively valuable. After all that, it would be somewhat ironic if it turned out the universal objective values don’t include the survival of life on Earth.
The second sentence doesn’t follow from the first. If rational agents converge on their values, that is objective enough. Analogy: one can accept that mathematical truth is objective (mathematicians will converge) without being a Platonists (mathematical truths have an existence separate from humans)
I fin d that hard to follow. If the test i rationally justifiable, and leads to the uniform results, how is that not objective? You seem to be using “objective” (having a truth value independent of individual humans) to mean what I would mean by “real” (having existence independent of humans).
First of all, thanks for the comment. You have really motivated me to read and think about this more—starting with getting clearer on the meanings of “objective”, “subjective”, and “intrinsic”. I apologise for any confusion caused by my incorrect use of terminology. I guess that is why Eliezer likes to taboo words. I hope you don’t mind me persisting in trying to explain my view and using those “taboo” words.
Since I was talking about meta-ethical moral relativism, I hope that it was sufficiently clear that I was referring to moral values. What I meant by “objective values” was “objectively true moral values” or “objectively true intrinsic values”.
The second sentence was an explanation of the first: not logically derived from the first sentence, but a part of the argument. I’ll try to construct my arguments more linearly in future.
If I had to rephrase that passage I’d say:
If there are no agents to value something, intrinsically or extrinsically, then there is also nothing to act on those values. In the absence of agents to act, values are effectively meaningless. Therefore, I’m not convinced that there is objective truth in intrinsic or moral values.
However, the lack of meaningful values in the absence of agents hints at agents themselves being valuable. If value can only have meaning in the presence of an agent, then that agent probably has, at the very least, extrinsic/instrumental value. Even a paperclip maximiser would probably consider itself to have instrumental value, right?
I think there is a difference between it being objectively true that, in certain circumstances, the values of rational agents converge, and it being objectively true that those values are moral. A rational agent can do really “bad” things if the beliefs and intrinsic values on which it is acting are “bad”. Why else would anyone be scared of AI?
I accept the possibility of objective truth values. I’m not convinced that it is objectively true that the convergence of subjectively true moral values indicates objectively true moral values. As far as values go, moral values don’t seem to be as amenable to rigorous proofs as formal mathematical theorems. We could say that intrinsic values seem to be analogous to mathematical axioms.
I’ll have a go at clarifying that passage with the right(?) terminology:
Without the objective truth of intrinsic values, it might just be a matter of testing different sets of assumed intrinsic values until we find an “optimal” or acceptable convergent outcome.
Morality might be somewhat like an NP-hard optimisation problem. It might be objectively true that we get a certain result from a test. It’s more difficult to say that it is objectively true that we have solved a complex optimisation problem.
Thanks for informing me that my use of the term “objective” was confused/confusing. I’ll keep trying to improve the clarity of my communication and understanding of the terminology.
That’s what I like to hear!
But there is no need for morality in the absence of agents. When agents are there, values will be there, when agents are not there, the absence of values doesn’t matter.
I don’t require their values to converge, I require them to accept the truths of certain claims. This happens in real life. People say “I don’t like X, but I respect your right to do it”. The first part says X is a disvalue, the second is an override coming from rationality.
I’m assuming a lot of background in this post that you don’t seem to have. Have you read the sequences, specifically the metaethics stuff?
Moral philosophy on LW is decades (at the usual philosophical pace) ahead of what you would learn elsewhere and a lot of the stuff you mentioned is considered solved or obsolete.
Really? That’s kind of scary if true. Moral philosophy on LW doesn’t strike me as especially well developed (particularly compared to other rationality related subjects LW covers).
I don’t believe anyone’s really taken the metaethics sequence out for a test drive to see if it solves any nontrivial problems in moral philosophy.
Its worse than that. No-one even knows what the theory laid out is. EY says different things in different places.
If I recall correctly it struck me as an ok introduction to metaethics but it stopped before it got to the hard (ie. interesting) stuff.
Moral philosophy is not well developed on LW, but I think it’s further than it is elsewhere, and when I look at the pace of developments in philosophy, it looks like it will take decades for everyone else to catch up. Maybe I’m underestimating the quality of mainstream philosophy, though.
All I know is that people who are interested in moral philosophy who haven’t been exposed to LW are a lot more confused than those on LW. And that those on LW are more confused than they think they are (hence the OP).
What do you think represents the best moral philosophy that LW has to offer?
Just a few months ago you seemed to be saying that we didn’t need to study moral philosophy, but just try to maximize “awesomeness”, which “You already know that you know how to compute”. I find it confusing that this post doesn’t mention that one at all. Have you changed your mind since then, if so why? Or are you clarifying your position, or something else?
The metaethics sequence sinks most of the standard confusions, though it doesn’t offer actual conclusions or procedures.
Complexity of value. Value being human specific. morality as optimization target. etc.
Maybe it’s just the epistemic quality around here though. LWers talking about morality are able to go much further without getting derailed than the best I’ve seen elsewhere, even if there weren’t much good work on moral philosophy on LW.
Right. This is a good question.
For actually making decisions, use Awesomeness or something as your moral proxy, because it more or less just works. For those of us who want to go deeper and understand the theory of morality declaratively, the OP applies; we basically don’t have any good theory. They are two sides of the same coin; the situation in moral philosophy is like the situation in physics a few hundred (progress subjective) years ago, and we need to recognize this before trying to build the house on sand, so to speak. So we are better off just using our current buggy procedural morality.
I could have made the connection clearer I suppose.
This post is actually a sort of precurser to some new and useful (I hope) work on the subject that I’ve written up but haven’t gotten around to polishing and posting. I have maybe 5 posts worth of morality related stuff in the works, and then I’m getting out of this godforsaken dungeon.
Given that we don’t have a good explicit theory of what morality really is, how do you know (and how could you confidently claim in that earlier post) that Awesomeness is a good moral proxy?
I think I understand what you’re saying now, thanks for the clarification. However, my current buggy procedural morality is not “maximize awesomeness” but more like an instinctive version of Bostrom and Ord’s moral parliament.
It seems to fit with intuition. How exactly my intuitions are supposed to imply actual morality is an open question.
Could you nominate some confusions that are unsunk amongst professional philosophers (vis a vis your “decades ahead” claim).
You don’t tend to find much detailed academic discussion regarding metaethical philosophy on the blogosphere at all.
Disclaimers: strictly comparing it to other subjects which I consider similar from an outside view, and supported only by personal experience and observation.
I have, and I found it unclear and inconclusive. A number of people have offered to explain it , and they all ended up bowing out unable to do so
I find no evidence for that claim.
Sorry, I have only read selections of the sequences, and not many of the posts on metaethics. Though as far as I’ve gotten, I’m not convinced that the sequences really solve, or make obsolete, many of the deeper problems or moral philosophy.
The original post, and this one, seems to be running into the “is-ought” gap and moral relativism. Being unable to separate terminal values from biases is due to there being no truly objective terminal values. Despite Eliezer’s objections, this is a fundamental problem for determining what terminal values or utility function we should use—a task you and I are both interested in undertaking.
I think this community vastly over-estimates its grip on meta-ethical concepts like moral realism or moral anti-realism. (E.g. the hopelessly confused discussion in this thread). I don’t think the meta-ethics sequence resolves these sorts of basic issues.
I’m still coming to terms with the philosophical definitions of different positions and their implications, and the Stanford Encyclopedia of Philosophy seems like a more rounded account of the different view points than the meta-ethics sequences. I think I might be better off first spending my time continuing to read the SEP and trying to make my own decisions, and then reading the meta-ethics sequences with that understanding of the philosophical background.
By the way, I can see your point that objections to moral anti-realism in this community may be somewhat motivated by the possibility that friendly AI becomes unprovable. As I understand it, any action can be “rational” if the value/utility function is arbitrary.
There is a lot of diversity of opinions in philosophers and that may be true as a whole of the discipline, there is some good stuff to be found there. I’d recommend staying here for the most part rather than wading through philosophy elsewhere, though.
Also, many moral philosophers may have very different moral sentiments from you and that maybe that makes them seem like idiots more than they actually are. Different moral sentiments as to whether consequentialism rather than just within consequentialism among other things.