There are a lot of posts here that presuppose some combination of moral anti-realism and value complexity. These views go together well: if value is not fundamental, but dependent on characteristics of humans, then it can derive complexity from this and not suffer due to Occam’s Razor.
There are another pair of views that go together well: moral realism and value simplicity. Many posts here strongly dismiss these views, effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with expert opinion and being clearly overconfident. In the Phil Papers survey for example, 56.3% of philosophers lean towards or believe realism, while only 27.7% lean towards or accept anti-realism.
Given this, and given comments from people like me in the intersection of the philosophical and LW communities who can point out that it isn’t a case of stupid philosophers supporting realism and all the really smart ones supporting anti-realism, there is no way that the LW community should have anything like the confidence that it does on this point.
Moreover, I should point out that most of the realists lean towards naturalism, which allows a form of realism that is very different to the one that Eliezer critiques. I should also add that within philosophy, the trend is probably not towards anti-realism, but towards realism. The high tide of anti-realism was probably in the middle of the 20th Century, and since then it has lost its shiny newness and people have come up with good arguments against it (which are never discussed here...).
Even for experts in meta-ethics, I can’t see how their confidence can get outside the 30%-70% range given the expert disagreement. For non-experts, I really can’t see how one could even get to 50% confidence in anti-realism, much less the kind of 98% confidence that is typically expressed here.
Among target faculty listing meta-ethics as their area of study moral realism’s lead is much smaller: 42.5% for moral realism and 38.2% against.
Looking further through the philpapers data, a big chunk of the belief in moral realism seems to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist.
In general, only 61% of respondents “accept” rather than lean towards atheism, and a quarter don’t even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top.
Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective adjudication within a reasonable period of time.
In general, those interquestion correlations should help pinpoint any correct contrarian cluster.
This is why I put more weight on Toby’s personal position, than on the majority expert position. As far as I know, Toby is in the same contrarian cluster as me, yet he seems to give much more weight to moral realism (and presumably not the Yudkowskian kind either) than I do. Like ciphergoth, I wish he would tell us which arguments in favor of realism, or against anti-realism, that he finds persuasive.
It seems that would be more likely if some people would put effort into apparently wanting to learn more about moral realism, or would read and present some of the arguments charitably to LW.
Thanks for looking that up Carl—I didn’t know they had the break-downs. This is the more relevant result for this discussion, but it doesn’t change my point much. Unless it was 80% or so in favour of anti-realism, I think holding something like 95% credence in anti-realism this is far too high for non-experts.
Atheism doesn’t get 80% support among philosophers, and most philosophers of religion reject it because of a selection effect where few wish to study what they believe to be non-subjects (just as normative and applied ethicists are more likely to reject anti-realism).
Perhaps we shouldn’t look for professional consensus on things we accept with almost-certainty, because things that can be correctly accepted with almost-certainty by amateurs will not be professionally studied, except by people who are systematically confused. Instead, we should ask non-professional opinion of people who are in the position to know most about the subject, but don’t study it professionally.
You are correct that it is reasonable to assign high confidence to atheism even if it doesn’t have 80% support, but we must be very careful here. Atheism is presumably the strongest example of such a claim here on Less Wrong (i.e. one which you can tell a great story why so many intelligent people would disagree etc and hold a high confidence in the face of disagreement). However, this does not mean that we can say that any other given view is just like atheism in this respect and thus hold beliefs in the face of expert disagreement, that would be far too convenient.
Strong agreement about not overgeneralizing. It does appear, however, that libertarianism about free well, non-physicalism about the mind, and a number of sorts of moral realism form a cluster, sharing the feature of reifying certain concepts in our cognitive algorithms even when they can be ‘explained away.’ Maybe we can discuss this tomorrow night.
However, this does not mean that we can say that any other given view is just like atheism in this respect and thus hold beliefs in the face of expert disagreement, that would be far too convenient.
Of course not; the substance of one’s reasons for disagreeing matters greatly. In this case, I suspect there’s probably a significant amount of correlation/non-independence between the reasons for believing atheism and believing something like moral non-realism.
One thing we should take away from cases like atheism is that surveys probably shouldn’t be interpreted naively, but rather as somewhat noisy information. I think my own heuristic (on binary questions where I already have a strong opinion) is basically to look on which side of 50% my position falls; if the majority agrees with me (or, say, the average confidence in my position is over 50%), I tend to regard that as (more) evidence in my favor, with the strength increasing as the percentage increases.
(This, I think, would be part of how I would answer Yvain.)
I think the arguments you’re developing here go a long way towards answering Toby’s point, but what safeguards can we use to ensure we can’t use it as a generalized anti-expert defence?
The prerequisite for this heuristic is coming to a conclusion with near-certainty on an amateur level. The safeguard has to be general ability to not get that much unjustified overconfidence.
I’m pointing out that there is already a generally applicable enough set of safeguards that covers this case in particular, adequate or not. That is, this heuristic doesn’t automatically lead as astray.
I don’t think I can understand you properly; it reads like you’re saying that we can be confident in rejecting expert advice if we’ve already reached a contrary position with high confidence. That doesn’t sound Bayesian. I suspect the error is mine but I’d appreciate your help in finding and fixing it!
EDIT: I [not Vladimir] would say that if we have one position that we can be confident in (atheism) we can use it as an indicator of expert quality, and pay more attention to those experts on other issues (e.g. moral realism as philosophers define it).
And with respect to the selection effect among philosophers of religion, there’s overwhelming direct evidence on this in the form of the Catholic Church push on this front.
I would say so too, though I wasn’t saying that here. It is the mechanism through which we can reject expert opinion, but also as applied to the very claim that is being contested, not just the other slam-dunk claims.
Only where there’s a relationship of course. We would be unwise to reject medical expertise from a body where atheists were few, unless religion impinged on that advice eg abortion, cryonics. Here a relationship with religion is clear.
I would say that if on some matter of medical controversy atheist doctors and medical academics tended to come out one way, while the median opinion came out the other way, we should go with the atheist medical opinion, ceteris paribus. Atheism is a proxy for intelligence and scientific thinking, a finding which has a mountain of evidence in its favor.
Definitely if the majority opinion among atheist experts differed from the majority opinion among all experts, I’d go for the former, but if say the majority of doctors studying a disease were Catholic for simple geographic reasons, I’d still defer to their expertise.
(This discussion is about meta-level mechanism for agreement, where you accept a conclusion; experts might well have persuasive arguments that inverse one’s confidence.)
Many posts here strongly dismiss [moral realism and simplicity], effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with expert opinion and being clearly overconfident. [...] For non-experts, I really can’t see how one could even get to 50% confidence in anti-realism, much less the kind of 98% confidence that is typically expressed here.
One person’s modus ponens is another’s modus tollens. You say that professional philosophers’ disagreement implies that antirealists shouldn’t be so confident, but my confidence in antirealism is such that I am instead forced to downgrade my confidence in professional philosophers. I defer to experts in mathematics and science, where I can at least understand something of what it means for a mathematical or scientific claim to be true. But on my current understanding of the world, moral realism just comes out as nonsense. I know what it means for a computation to yield this-and-such a result, or for a moral claim to be true with respect to such-and-these moral premises that might be held by some agent. But what does it mean for a moral claim to be simply true, full stop? What experiment could you perform to tell, even in principle? If the world looks exactly the same whether murder is intrinsically right or intrinsically wrong, what am I supposed to do besides say that there simply is no fact of the matter, and proceed with my life just as before?
I realize how arrogant it must seem for young, uncredentialled (not even a Bachelor’s!) me to conclude that brilliant professional philosophers who have devoted their entire lives to studying this topic are simply confused. But, disturbing as it may be to say … that’s how it really looks.
But what does it mean for a moral claim to be simply true, full stop?
Well, in my world, it means that the premises are built into saying “moral claim”; that the subject matter of “morality” is the implications of those premises, and that moral claims are true when they make true statements about these implications. If you wanted to talk about the implications of other premises, it wouldn’t be the subject matter of what we name “morality”. Most possible agents (e.g. under a complexity-based measure of mind design space) will not be interested in this subject matter—they won’t care about what is just, fair, freedom-promoting, life-preserving, right, etc.
This doesn’t contradict what you say, but it’s a reason why someone who believes exactly everything you do might call themselves a moral realist.
In my view, people who look at this state of affairs and say “There is no morality” are advocating that the subject matter of morality is a sort of extradimensional ontologically basic agent-compelling-ness, and that, having discovered this hypothesized transcendental stuff to be nonexistent, we have discovered that there is no morality. In contrast, since this transcendental stuff is not only nonexistent but also poorly specified and self-contradictory, I think it was a huge mistake to claim that it was the subject matter of morality in the first place, that we were talking about some mysterious ineffable confused stuff when we were asking what is right. Instead I take the subject matter of morality to be what is fair, just, freedom-promoting, life-preserving, happiness-creating, etcetera (and what that starting set of values would become in the limit of better knowledge and better reflection). So moral claims can be true, and it all adds up to normality in a rather mundane way… which is probably just what we ought to expect to see when we’re done.
Yes, but I think that my way of talking about things (agents have preferences, some of which are of a type we call moral, but there is no objective morality) is more useful than your way of talking about things (defining moral as a predicate referring to a large set of preferences), because your formulation (deliberately?) makes it difficult to talk about humans with different moral preferences, which possibility you don’t seem to take very seriously, whereas I think it very likely.
Well, in my world, it means that the premises are built into saying “moral claim”; that the subject matter of “morality” is the implications of those premises, and that moral claims are true when they make true statements about these implications.
So, according to this view, moral uncertainty is just a subset of logical uncertainty, where we restrict our attention to the implication of a fixed set of moral premises. But why is it that I feel uncertain about which premises I should accept? I bet that when most people talk about moral realism and moral uncertainty, that is what they’re talking about.
(and what that starting set of values would become in the limit of better knowledge and better reflection)
Why/how does/should one’s moral premises change as one gains knowledge and ability to reflect? (Note that in standard decision theory one’s values simply don’t change this way.) It seems to me this ought to be the main topic of moral inquiry, instead of being relegated to a parenthetical remark. The subsequent working out of implications seems rather trivial by comparison.
So moral claims can be true, and it all adds up to normality in a rather mundane way… which is probably just what we ought to expect to see when we’re done.
But why is it that I feel uncertain about which premises I should accept?
You’ve got meta-moral criteria for judging between possible terms in your utility function, a reconciliation process for conflicting terms, other phenomena which are very interesting and I do wish someone would study in more detail, but so far as metaethics goes it would tend to map onto a computation whose uncertain output is your utility function. Just more logical uncertainty.
How can I put it? The differences here are probably very important to FAI designers and object-level moral philosophers, but I’m not sure they’re metaethically interesting… or they’re metaethically interesting, but they don’t make you confused about what sort of stuff morality could possibly be made out of. Moral uncertainty is still made out of a naturalistic mixture of physical uncertainty and logical uncertainty.
Suppose there’s an UFAI loose on the Internet that’s not yet very powerful. In order to gain more power, it wants me to change my moral premises (so I’ll help it later), and to do that, it places a story on the web for me to find. I read the story, and it “inspires” me to change my values in the direction that the UFAI prefers. In your view, how do we say that this is bad, if this is just what my meta-moral computation did?
If the UFAI convinced you of anything that wasn’t true during the process—outright lies about reality or math—or biased sampling of reality producing a biased mental image, like a story that only depicts one possibility where other possibilities are more probable—then we have a simple and direct critique.
If the UFAI never deceived you in the course of telling the story, but simple measures over the space of possible moral arguments you could hear and moralities you subsequently develop, produce a spread of extrapolated volitions “almost all” of whom think that the UFAI-inspired-you has turned into something alien and unvaluable—if it flew through a persuasive keyhole to produce a very noncentral future version of you who is disvalued by central clusters of you—then it’s the sort of thing a Coherent Extrapolated Volition would try to stop.
See also #1 on the list of New Humane Rights: “You have the right not to have the spread in your volition optimized away by an external decision process acting on unshared moral premises.”
You have the right not to have the spread in your volition optimized away by an external decision process acting on unshared moral premises.
You have the right to a system of moral dynamics complicated enough that you can only work it out by discussing it with other people who share most of it.
You have the right to be created by a creator acting under what that creator regards as a high purpose.
You have the right to exist predominantly in regions where you are having fun.
You have the right to be noticeably unique within a local world.
You have the right to an angel. If you do not know how to build an angel, one will be appointed for you.
You have the right to exist within a linearly unfolding time in which your subjective future coincides with your decision-theoretical future.
You have the right to remain cryptic.
-- Eliezer Yudkowsky
(originally posted sometime around 2005, probably earlier)
What about the least convenient world where human meta-moral computation doesn’t have the coherence that you assume? If you found yourself living in such a world, would you give up and say no meta-ethics is possible, or would you keep looking for one? If it’s the latter, and assuming you find it, perhaps it can be used in the “convenient” worlds as well?
To put it another way, it doesn’t seem right to me that the validity of one’s meta-ethics should depend on a contingent fact like that. Although perhaps instead of just complaining about it, I should try to think of some way to remove the dependency...
(We also disagree about the likelihood that the coherence assumption holds, but I think we went over that before, so I’m skipping it in the interest of avoiding repetition.)
I think this is about metamorals not metaethics—yes, I’m merely defining terms here, but I consider “What is moral?” and “What is morality made of?” to be problems that invoke noticeably different issues. We already know, at this point, what morality is made of; it’s a computation. Which computation? That’s a different sort of question and I don’t see a difficulty in having my answer depend on contingent facts I haven’t learned.
In response to your question: yes, if I had given a definition of moral progress where it turned out empirically that there was no coherence in the direction in which I was trying to point and the past had been a random walk, then I should reconsider my attempt to describe those changes as “progress”.
Which computation? That’s a different sort of question and I don’t see a difficulty in having my answer depend on contingent facts I haven’t learned.
How do you cash “which computation?” out to logical+physical uncertainty? Do you have in mind some well-defined metamoral computation that would output the answer?
I think you just asked me how to write an FAI. So long as I know that it’s made out of logical+physical uncertainty, though, I’m not confused in the same way that I was confused in say 1998.
“Well-specified” may have been too strong a term, then; I meant to include something like CEV as described in 2004.
Is there an infinite regress of not knowing how to compute morality, or how to compute (how to compute morality), or how to compute (how to compute (...)), that you need to resolve; do you currently think you have some idea of how it bottoms out; or is there a third alternative that I should be seeing?
it doesn’t seem right to me that the validity of one’s meta-ethics should depend on a contingent fact like that
I think it is a powerful secret of philosophy and AI design that all useful philosophy depends upon the philosopher(s) observing contingent facts from their sensory input stream. Philosophy can be thought of as an ultra high level machine learning technique that records the highest-level regularities of our input/output streams. And the reason I said that this is a powerful AI design principle, is that you realize that your AI can do good philosophy by looking for such regularities.
But why is it that I feel uncertain about which premises I should accept?
Think of it as a foundational struggle: you’ve got non-rigorous ideas about what is morally true/right, and you are searching of a way to build a foundation such that any right idea will follow from that foundation deductively. Arguably, this task is impossible within human mind. A better human-level approach would be structural, where you recognize certain (premise) patterns in reliable moral ideas, and learn heuristics that allow to conclude other patterns wherever you find the premise patterns. This constitutes ordinary moral progress, when fixed in culture.
In cases like that, I am perfectly willing to say that we have discovered that the subject matter of “fairies” is a coherent, well-formed concept that turns out to have an empty referent. The closet is there, we opened it up and looked, and there was nothing inside. I know what the world ought to look like if there were fairies, or alternatively no fairies, and the world looks like it has no fairies.
I think that a very large fraction of the time, when a possibility appears to be coherent and well formed, it may turn out not to be upon more careful examination. I would see the subject matter of “fairies” as “that which causes us to talk about fairies”, the subject matter of “dogs” as “that which causes us to talk about dogs”, and the subject matter of “morality” as “that which causes us to talk about morality”. All three are interesting.
This is a theme that crops up fairly frequently as a matter of semantic confusion and is a confusion that is difficult to resolve trivially due to inferential differences to the actual abstract concepts. I haven’t seen this position explained so coherently in one place before. Particularly the line:
I think it was a huge mistake to claim that it was the subject matter of morality in the first place, that we were talking about some mysterious ineffable confused stuff when we were asking what is right.
… and the necessary context. I would find it useful to have this as a top level post to link to. Even if, as you have just suggested to JamesAndrix, it is just a copy and paste job. It’ll save searching through comments to find a permalink if nothing else.
Doctors or medicine, investors or analysis of public information, scientists or science, philosophers or philosophy… maybe it’s the process of credentialing that we should be downgrading our credence in. Really, why should the prior for credentials being a very significant form of evidence ever have been very high?
The philpapers survey is for the top 99 departments. Things do get better as you go up. Among hard scientists, elite schools are more atheist, and the only almost entirely atheist groups are super-elite, like the National Academy of Sciences/Royal Society.
I realize how arrogant it must seem for young, uncredentialled (not even a Bachelor’s!) me to conclude that brilliant professional philosophers who have devoted their entire lives to studying this topic are simply confused. But, disturbing as it may be to say … that’s how it really looks.
Perhaps the fact that they have devoted their lives to a topic suggests that they have a vested interest in making it appear not to be nonsense. Cognitive dissonance can be tricky even for the pros.
Quite possible. But in that case I would say that we’re just talking about things in different ways, and not actually disagreeing on anything substantive.
Toby, I spent a while looking into the meta-ethical debates about realism. When I thought moral realism was a likely option on the table, I meant:
Strong Moral Realism: All (or perhaps just almost all) beings, human, alien or AI, when given sufficient computing power and the ability to learn science and get an accurate map-territory distinction, will agree on what physical state the universe ought to be transformed into, and therefore they will assist you in transforming it into this state.
But modern philosophers who call themselves “realists” don’t mean anything nearly this strong. They mean that that there are moral “facts”. But what use is it if the paperclipper agrees that it is a “moral fact” that human rights ought to be respected, if it then goes on to say it has no desire to act according to the prescription of moral facts, and moral facts can’t somehow revoke it.
The force of “scientific facts” is that they constrain the world. If an alien wants to get from Andromeda to here, it has to take at least 2.5 million years, the physical fact of the finite speed of light literally stops the alien from getting here sooner, whether it likes it or not.
The 56.3/27.7% split on philpapers seems to me to be an argument about whether you should be allowed to attach the word “fact” to your preferences, kind of as a shiny badge of merit, without actually disagreeing on any physical prediction about the world. The debate between weak moral realists and antirealists sounds like the debate where two people ask “if a tree falls in the forest, does it really make a sound?”—they’re not arguing about anything substantive.
So, I ask, how many philosophers are strong moral realists, in the sense I defined?
EDIT: After seeing Carl’s comment, it seems likely to me that there probably are a bunch of theists who would, in fact, support the strong moral realism position; but they’re clowns, so who cares.
I strongly agree with Roko that something like his strong version is the interesting version. What matters is what range of creatures will come to agree on outcomes; it matters much less what range of creatures think their desires are “right” in some absolute sense, if they don’t think that will eventually be reflected in agreement.
The force of “scientific facts” is that they constrain the world.
In the context of this comment, the goal of FAI can be said to be to constrain the world by “moral facts”, just like laws of physics constrain the world by “physical facts”. This is the sense in which I mean “FAI=Physical Laws 2.0”.
Only in a useless way: there is a specific FAI that does the “truly right” thing, but the truthhood of rightness doesn’t stop you from having to code the rightness in. Goodness is not discoverably true: if you don’t already know exactly what goodness is, you can’t find out.
hmmm. That is interesting. Well, let us define the collection W_i of worlds run by superintelligences with the subscript i ranging over goals. No matter what i is, those worlds are going to look, to any agents in them, like worlds with “moral truths”.
However, any agent that learned the real physics of such a world would see that the goodness is written in to the initial conditions, not the laws.
Roko, you make a good point that it can be quite murky just what realism and anti-realism mean (in ethics or in anything else). However, I don’t agree with what you write after that. Your Strong Moral Realism is a claim that is outside the domain of philosophy, as it is an empirical claim in the domain of exo-biology or exo-sociology or something. No matter what the truth of a meta-ethical claim, smart entities might refuse to believe it (the same goes for other philosophical claims or mathematical claims).
Pick your favourite philosophical claim. I’m sure there are very smart possible entities that don’t believe this and very smart ones that do. There are probably also very smart entities without the concepts needed to consider it.
I understand why you introduced Strong Moral Realism: you want to be able to see why the truth of realism would matter and so you came up with truth conditions. However, reducing a philosophical claim to an empirical one never quite captures it.
For what its worth, I think that the empirical claim Strong Moral Realism is false, but I wouldn’t be surprised if there was considerable agreement among radically different entities on how to transform the world.
Pick your favourite philosophical claim. I’m sure there are very smart possible entities that don’t believe this and very smart ones that do
If there’s a philosophical claim that intelligent agents across the universe wouldn’t display massive agreement on, then I don’t really think it is worth its salt. I think that this principle can be used to eliminate a lot of nonsense from philosophy.
Which of anti-realism or weak realism is true seems to be a question we can eliminate. Whether strong realism is true or not seems substantive, because it matters to our policy which is true.
However, reducing a philosophical claim to an empirical one never quite captures it.
There are clearly some examples where there can be interesting things to say that aren’t really empirical, e.g. decision theory, mystery of subjective experience. But I think that this isn’t one of them.
Suffice it to say I can’t think of anything that makes the debate between weak realism and antirealism at all interesting or worthy of attention. Certainly, Friendly AI theorists ought not care about the difference, because the empirical claims about an AI system will do are identical. Once the illusions and fallacies surrounding rationalist moral psychology has been debunked, proponents of other AI motivation methods than FAI also ought not to care about the weak realism vs. anti-realism pseudo-question
I’m having trouble reconciling this with the beginning of your first comment:
These views go together well: if value is not fundamental, but dependent on characteristics of humans, then it can derive complexity from this and not suffer due to Occam’s Razor.
An “optimal organism” may be a possibility, though. Assuming god’s utility function, it is theoretically possible that a unique optimal agent might exist. Whether it would be found before the universal heat death is another issue, though.
From my naturalist POV, you need to show me a paperclipper before it is convincing evidence about the real world. Paperclipper’s are theoretical possibilities, but who would build one, why, and how long would it last in the wild?
...and if the “paperclips” part is a metaphor, then which preferred ordered atomic states count, and which don’t? Is a cockroach a “paperclipper”—because it acts as though it wants to fill the universe with its DNA?
Yes, paperclips are a metaphor. No one expects a literal paperclip maximizer; the point is to illustrate unFriendly AI as a really powerful system with little or no moral worth as humans would understand moral worth. A non-conscious superintelligent cockroach-type thing that fills the universe with its DNA or equivalent would indeed qualify.
In that case, I don’t think a division of superintelligences into paperclippers and non-paperclippers “carves nature at the joints” very well. It appears to be a human-centric classification scheme.
I’ve proposed another way of classifying superintelligence goal systems—according to whether or not they are “handicapped”.
Healthy superintelligences execute god’s utility function—i.e. they don’t value anything apart from their genes.
Handicapped superintelligences value other things—paperclips, gold atoms, whatever. Genes are valued too—but they may only have proximate value.
According to this classification scheme, the cockroach and paperclipper would be in different categories.
“Handicapped” superintelligences value things besides their genes. They typically try and leave something behind. Most other agents keep dissipating negentropy until they have flattened energy gradients as much as they can—the way most living ecosystems do.
It appears to be a human-centric classification scheme.
Yes, that’s the point! We’re humans, and so for some purposes we find it useful to categorize superintelligences into those that do and don’t do what we want, even if it isn’t a natural categorization from a more objective standpoint.
Right—well, fine. One issue is that the classification into paperclippers and non-paperclippers was not clear to me until you clarified it. Another poster has “clarified” things the other way in response to the same comment. So, as a classification scheme, IMO the idea seems rather vague and unclear.
The next issue is: how close does an agent have to be to what you (we?) want before it is a non-paperclipper?
IMO, the idea of a metaphorical unfriendly paperclipper appears to need pinning down before it is of very much use as a means of superintelligence classification scheme.
Another poster has “clarified” things the other way in response to the same comment.
I’m pretty confident Roko agrees with me and that this is just a communication error.
So, as a classification scheme, IMO the idea seems rather vague and unclear.
I’m given to understand that the classification scheme is Friendly versus unFriendly, with paperclip maximizer being an illustrative (albeit not representative) example of the latter. I agree that more rigor (and perhaps clearer terminology) is in order.
Machine intelligences seem likely to vary in their desirability to humans.
Technically true. However, most naive superintelligence designs will simply kill all humans. You’ve accomplished quite a lot to even get to a failed utopia, much less deciding whether you want Prime Intellect or Coherent Extrapolated Volition.
It’s also unlikely you’ll accidentally do something significantly worse than killing all humans, for the same reasons. A superintelligent sadist is just as hard as a utopia.
I read the essay you linked to. I really don’t know where to start.
Now, we are not currently facing threats from any alien races. However, if we do so in the future, then we probably do not want to have handicapped our entire civilisation.
So we should guard against potential threats from non-human intelligent life by building a non-human superintelligence that doesn’t care about humans?
While dependencies on humans may have the effect of postponing the demise of our species, they also have considerable potential to hamper and slow evolutionary progress.
Postpone? I thought the point of friendly AI was to preserve human values for as long as physically possible. “Evolutionary progress?” Evolution is stupid and doesn’t care about the individual organisms. Evolution causes pointless suffering and death. It produces stupid designs. As Michael Vassar once said: think of all the simple things that evolution didn’t invent. The wheel. The bow and arrow. The axial-flow pump. Evolution had billions of years creating and destroying organisms and it couldn’t invent stuff built by cave men. Is it OK in your book that people die of antibiotic resistant diseases? MRSA is a result of evolutionary progress.
For example humans have poor space-travel potential, and any tendency to keep humans around will be associated with remaining stuck on the home world.
Who said humans have to live on planets or breathe oxygen or run on neurons? Why do you think a superintelligence will have problems dealing with asteroids when humans today are researching ways to deflect them?
I think your main problem is that you’re valuing the wrong thing. You practically worship evolution while neglecting important things like people, animals, or anything that can suffer. Also, I think you fail to notice the huge first-mover advantage of any superintelligence, even one as “handicapped” as a friendly AI.
Finally, I know the appearance of the arguer doesn’t change the validity of the argument, but I feel compelled to tell you this: You would look much better with a haircut, a shave, and some different glasses.
I don’t avocate building machines that are indiffierent to humans. For instance, I think machine builders would be well advised to (and probably mostly will) construct devices that obey the law—which includes all kinds of provisions for preventing harm to humans.
Evolution did produce the wheel and the bow and arrow. If you think otherwise, please state clearly what definition of the term “evolution” you are using.
Regarding space travel—I was talking about wetware humans.
Re: “Why do you think a superintelligence will have problems dealing with asteroids when humans today are researching ways to deflect them?”
...that is a projection on your part—not something I said.
Re: “Also, I think you fail to notice the huge first-mover advantage of any superintelligence”
To quote mine myself:
“IMHO, it is indeed possible that the first AI will effectively take over the world. I.T. is an environment with dramatic first-mover advantages. It is often a winner-takes-all market – and AI seems likely to exhibit such effects in spades.”
“Google was not the first search engine, Microsoft was not the first OS maker—and Diffie–Hellman didn’t invent public key crypto.
Being first does not necessarily make players uncatchable—and there’s a selection process at work in the mean time, that weeds out certain classes of failures.”
I have thought and written about this issue quite a bit—and my position seems a bit more nuanced and realistic than the position you are saying you think I should have.
I disagree even with your interpretation of that document, but that is not the point emphasized in the grandparent. I acknowledge that while a superintelligence need not have genes it is in fact possible to construct a superintelligence that does relies significantly on “small sections of heritable information”, including the possibility of a superintelligence that relies on genes in actual DNA. Hence the slight weakening of the claim.
I think your term “God’s utility function” is a bit confusing—as if it’s just one utility function. If you value your genes, and I value my genes, and our genes are different, then we have different utility functions.
Also, the vast majority of possible minds don’t have genes.
By “gene” I mean: “Small chunk of heritable information”
Any sufficiently long-term persistent structure persists via a copying process—and so has “genes” in this sense.
What we mean by preference. Except that preference, being a specification of a computation, has a lot of forms of expression, so it doesn’t “persist” by a copying process, it “persists” as a nontrivial computational process.
A superintelligence that persists in copying a given piece of information is running a preference (computational process) that specifies copying as the preferable form of expression, over all the other things it could be doing.
No, no! Genes is just intended to refer to any heritable information. Preferences are something else entirely. Agents can have preferences which aren’t inherited—and not everything that gets inherited is a preference.
Anything information that persists over long periods of time persists via copying.
“Copying” just means there’s Shannon-mutual information between the source and the destination which originated in the source. Complex computations are absolutely included—provided that they share this property.
[Any] information that persists over long periods of time persists via copying.
“Copying” just means there’s Shannon-mutual information between the source and the destination which originated in the source. Complex computations are absolutely included—provided that they share this property.
Then preference still qualifies. This holds as a factual claim provided we are talking about reflectively consistent agents (i.e. those that succeed in not losing their preference), and as a normative claim regardless.
I would appreciate it if you avoid redefining words into highly qualified meanings, like “gene” for “anything that gets copied”, and then “copying” for “any computation process that preserves mutual information”.
Re: Then preference still qualifies. This holds as a factual claim provided [bunch of conditions]
Yes, there are some circumstances under which preferences are coded genetically and reliably inherited. However, your claim was stronger. You said what meant by genes was what “we” would call preferences. That implies that genes are preferences and preferences are genes.
You have just argued that a subset of preferences can be genetically coded—and I would agree with that. However, you have yet to argue that everything that is inherited is a preference.
I think you are barking up the wrong tree here—the concepts of preferences and genes are just too different. For example, clippy likes paperclips, in addition to the propagation of paperclip-construction instructions. The physical paperclips are best seen as phenotype—not genotype.
Re: “I would appreciate it if you avoid redefining words into highly qualified meanings [...]”
I am just saying what I mean—so as to be clear.
If you don’t want me to use the words “copy” and “gene” for those concepts—then you are out of luck—unless you have a compelling case to make for better terminology. My choice of words in both cases is pretty carefully considered.
Re: Then preference still qualifies. This holds as a factual claim provided [bunch of conditions]
Not “bunch of conditions”. Reflective consistency is the same concept as “correctly copying preference”, if I read your sense of “copying” correctly, and given that preference is not just “thing to be copied”, but also plays the appropriate role in decision-making (wording in the grandparent comment improved). And reflectively consistent agents are taken as a natural and desirable (from the point of view of those agents) attractor where all agents tend to end up, so it’s not just an arbitrary category of agents.
That implies that genes are preferences and preferences are genes.
But there are many different preferences for different agents, just as there are different genes. Using the word “genes” in the context where both human preference and evolution are salient is misleading, because human genes, even if we take them as corresponding to a certain preference, don’t reflect human preference, and are not copied in the same sense human preference is copied. Human genes are exactly the thing that currently persists by vanilla “copying”, not by any reversible (mutual information-preserving) process.
If you don’t want me to use the words “copy” and “gene” for those concepts—then you are out of luck
Confusing terminology is still bad even if you failed to think up a better alternative.
You appear to be on some kind of different planet to me—and are so far away that I can’t easily see where your ideas are coming from.
The idea I was trying to convey was really fairly simple, though:
“Small chunks of heritable information” (a.k.a. “genes”) are one thing, and the term “preferences” refers to a different concept.
As an example of a preference that is not inherited, consider the preference of an agent for cats—after being bitten by a dog as a child.
As an example of something that is inherited that is not a preference, consider the old socks that I got from my grandfather after his funeral.
These are evidently different concepts—thus the different terms.
Thanks for your terminology feedback. Alas, I am unmoved. That’s the best terminology I have found, and you don’t provide an alternative proposal. It is easy to bitch about terminology—but not always so easy to improve on it.
Right—well, that seems pretty unlikely. What is the story? A paperclip manufacturer with an ambitious IT department that out-performs every other contender? How come the government doesn’t just step on the results?
Is there a story about how humanity drops the reins of civilisation like that that is not extremely contrived?
I am unclear on how this story is contrived. There are vast numbers of business with terrible externalities today; this is deeply related to the debate on climate change. Alternately, we have large cutting machines and those machines don’t care whether they are cutting trees or people; if a wood chipper could pick things up but not distinguish between which things it was picking up it would be very dangerous (but still very useful if you have a large space of things you want chipped).
I am a moral cognitivist. Statements like “ceteris paribus, happiness is a good thing” have truth-values. Such moral statements simply are not compelling or even interesting enough to compute the truth-value of to the vast majority of agents, even those which maximize coherent utility functions using Bayesian belief updating (that is, rational agents) or approximately rational agents.
AFAICT the closest official term for what I am is “analytic descriptivist”, though I believe I can offer a better defense of analytic descriptivism than what I’ve read so far.
From your SEP link on Moral Realism:
“It is worth noting that, while moral realists are united in their cognitivism and in their rejection of error theories, they disagree among themselves not only about which moral claims are actually true but about what it is about the world that makes those claims true. ”
I think this is good cause for breaking up that 56%. We should not take them as a block merely because (one component of) their conclusions match, if their justifications are conflicting or contradictory. It could still be the case that 90% of expert philosophers reject any given argument for moral realism. (This would be consistent with my view that those arguments are silly.)
I may have noticed this because the post on Logical Rudeness is fresh in my mind.
You are entirely right that the 56% would split up into many subgroups, but I don’t really see how this weakens my point: more philosophers support realist positions than anti-realist ones. For what its worth, the anti-realists are also fragmented in a similar way.
Disagreeing positions don’t add up just because they share a feature. On the contrary, If people offer lots of different contradictory reasons for a conclusion (even if each individual has consistent beliefs) it is a sign that they are rationalizing their position.
If 2⁄3′s of experts support proposition G , 1⁄3 because of reason A while rejecting B, and 1⁄3 because of reason B while rejecting A, and the remaining 1⁄3 reject A and B; then the majority Reject A, and the majority Reject B. G should not be treated as a reasonable majority view.
This should be clear if A is the koran and B is the bible.
If we’re going to add up expert views, we need to add up what experts consider important about a question, not features of their conclusions.
You shouldn’t add up two experts if they would consider each other’s arguments irrational. That’s ignoring their expertise.
Yes James, I’d also appreciate that.
Maybe we should encourage more short top-level posts and comment upgrades to posts. I think that would be great if we could develop a good procedure.
There is a hidden object which is either green, red or blue. Three people have conflicting opinions about its colour, based on different pieces of reasoning. If you are the one who believes it is green, you have to add up the opponents who say not-green, despite the fact that there is no single not-green position (think of the symmetry—otherwise everyone could have too great confidence). The same holds true if these are expert opinions.
The above example is basically as general as possible, so in order for your argument to work it will need to add specifics of some sort.
Also, the Koran/Bible case doesn’t work. By symmetry, the Koran readers can say that they don’t need to add up the Bible readers and the atheists, since they are heterogeneous, so they can keep their belief in the Koran...
In practice all arguments will share some premises and some conclusions, in messy asymmetrical ways.
If the not-greens share a a consistent rationale about why the object cannot be green, then I need to take that into account.
If the red supporter contends that all green and blue objects were lost in the color wars, while the blue supporter contends that all objects are fundamentally blue and besides the color wars never happened, then their opinions roughly cancel each other out. (Barring other reasons for me to view one as more rational than the other.)
I suspect that there are things to be said about islam that both atheists and christians would agree on. That’s a block that a rational muslim should take into account. Our disagreeing conclusions about god are secondary.
If I’m going to update my position because 56% of experts agree on something, then I want to know what I’m going to update to.
BTW, I wish there is a way to upgrade a comment into a post and automatically move all the discussions under the new post as well.
The only reason I can think of to upgrade a comment to a post is to draw attention to it, whether google attention, naturality of external linking, or the attention of the regular readers. In all these cases, it seems to me that it is the duty of the author, who is demanding time from many readers, to spend time summarizing the old discussion and making it easy for new readers to join.
Ignoring their expertise, but counting only popularity. Moderator, does that mean that Less Wrong’s karma system might be modified to take into account why a comment was upvoted?
A valid principle James, but a bad example which might be contested by those more knowledgeable of the matter.
Islam considers itself the best of the revealed religions and jesus is revered as a prophet in Islam.
So, in this case, christians reject the koran, but the muslims do not completely reject the bible.
I’m not sure what might serve as a better example, though. The multiple possible explanations of the present recession may serve as a better example, incase you want to make this a top level post.
What you say is true while the Koran and the Bible are referents, but when A and B become “Mohammed is the last prophet, who brought the full truth of God’s will” and “Jesus was a literal incarnation of God,” (the central beliefs of the religions that hold the respective books sacred) then James’ logic holds.
The right response to moral realism isn’t to dispute it’s truth but to simply observe you don’t understand the concept.
I mean imagine someone started going around insisting some situations were Heret and others were Grovic but when asked to explain what made a situation Heret or Grovic he simply shrugged and said they were primitive concepts. But you persist and after observing his behavior for a period of time you work out some principle that perfectly predicts which category he will assign a given situation to, even counterfactually but when you present the algorithm to him and ask, “Ohh so is it satisfying this principle that makes one Heret rather than Grovic?” he insists that while your notion will always agrees with his notion that’s not what he means. Moreover, he insists that no definition in terms of physical state could capture these concepts.
Confused you press him and he says that there are special things which we can’t casually interact with that determine Heret or Grovic status. Bracketing your skepticism you ask him to say what properties these new ontological objects must have. After listing a couple he adds that most importantly they can’t just be random things with this structure but they also have to be Heret making or Grovic making and that’s what distingushes them from all the other casually inaccessible things out there that might otherwise yield some slightly different class of things as Heret and Grovic.
Frustrated you curse the guy saying he hasn’t really told you anything since you didn’t know what it meant to be Heret or Grovic in the first place so you surely don’t know what it means to be Heret making or Grovic making. The man’s reply is simply to shrug and say, “well it’s a fundamental concept, if you don’t understand I can’t explain it to you anymore than I could explain the perceptual experience of redness to a man who had never experienced color.”
In such a situation the only thing you can do is give up on the notion of Heret and Grovic. Debating about whether to say it’s an incoherent notion, a concept you lack the facilities to comprehend or something else would just waste time with a useless word game. Ultimately you just have to ignore such talk as something that lacks content for you and treat it the same way as you would meaningless gibberish.
The fact that when moral realists say the same thing about good and evil they are using the same sounds that we understand to mean something different shouldn’t change the situation at all.
In metaethics, there are typically very good arguments against all known views, and only relatively weak arguments for each of them. For anything in philosophy, a good first stop is the Stanford Encyclopedia of Philosophy. Here are some articles on the topic at SEP:
Toby, I read through those SEP articles but couldn’t find the good arguments against anti-realism that you mentioned. In contrast, the article on deontology laid out the arguments for and against it very clearly.
Can you please point us more specifically to the arguments that you find persuasive? Maybe just give us some page numbers in the book that you referenced? Most of us don’t really have the time to read something like that cover to cover in search of a few nuggets of information.
Thank you for doing that, and may I second this. I started reading those articles, then after a bit started scanning for the anti-realism articles, and worried after not finding them that I’d not read carefully enough, so I’m glad to have your report on this.
I really am curious to read these arguments, so I hope someone can point us to them.
I managed to find a draft of a book chapter titled In Defence of Moral Realism. I’m still wondering what Toby thinks the best arguments are, but alas he doesn’t seem to be following this discussion anymore.
Thanks! Again, didn’t get much from a quick skim, let me know if you find any real meat in there.
The thing that really got my attention wasn’t the assertion that there are some arguments in favour of realism, but that there are good arguments specifically against anti-realism.
I know I’ve spoken of “skimming” twice here. I promise, if Toby Ord were to say to me “this contains good arguments against anti-realism” I would read it carefully.
The thing that really got my attention wasn’t the assertion that there are some arguments in favour of realism, but that there are good arguments specifically against anti-realism.
But surely an argument against anti-realism is also an argument for realism? I’m interpreting Toby’s comment as saying that there are good arguments for realism in general, but not for any particular realist meta-ethical theory.
Again, didn’t get much from a quick skim, let me know if you find any real meat in there.
The author says in the conclusion, “I do not pretend to give any knock-down argument in this chapter for the thesis that objective moral facts or
reasons exist, independently of our thoughts and actions.” So I think it’s mostly a matter of how convincing one finds the argument that he does give.
It seems likely, given that the author is a specialist in and proponent of moral realism, that he would give the best arguments that he knew, so this paper seems like good evidence for what kind of arguments for realism is currently available.
It seems likely, given that the author is a specialist in and proponent of moral realism, that he would give the best arguments that he knew, so this paper seems like good evidence for what kind of arguments for realism is currently available.
Thanks! There were several points in your PhD thesis where I couldn’t work out how to square your position with moral anti-realism—I guess I know why now :-)
Basically, morality is a product of evolution—which can be expected to favour some moral values over other ones—just as it favours certain physical structures like eyes and legs.
Things like: “under most circumstances, don’t massacre your relatives or yourself” can be reasonably expected to be widespread values in the universe. The idea gives morality a foundation in the natural world.
It is useful that Tim summarizes his position in this context, voted up.
My position, developed with no background in philosophy or meta-ethics whatsoever and thus likely to be error-riddled or misguided, is that I consider it an unsolved problem within physical materialism (specifically, within the context of moral anti-realism) how “meaning” (the meaning of life and/or the value of values) can be a coherent or possible concept.
I have many good responses to the comments here, and I suppose I could sketch out some of the main arguments against anti-realism, but there are also many serious demands on my time and sadly this doesn’t look like a productive discussion. There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying (which is particularly alarming given that many commenters are pointing out that they don’t understand what the opponents are saying!).
Given all of this, I fear that writing a post would not be a good use of my time.
many commenters are pointing out that they don’t understand what the opponents are saying
This is a little unfair; as soon as you take a deflationary stance on anything, you’re saying that the other stance doesn’t really have comprehensible content, and it’s a mistake to turn that into a general-purpose dismissal of deflationary stances.
There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying
If you think that’s more true here than it is in other discussion forums, we’re doing something very wrong. I understand that you’re not able to spend time writing for this audience, but for those of us who do want to find out more about what moral realists are saying, every link you can provide to existing essays is valuable.
This point has given me a lot of pause, so forgive me my many replies. Part of the problem is that even if I were only 60% confident of moral anti-realism, I would still act on it as if I were 100% confident because I don’t understand moral realism at all, and my 60% confidence is in the belief that no-one else does either.
My impression of academic philosophers is that their ‘expertise’ is primarily in knowledge of what other philosophers have said and in the forms of academic philosophical argument. It is not expertise in true facts about the world. In other words, I would defer to their expertise on the technical details of academically accepted definitions of philosophical terms, or on the writings of Kant, much as I would defer to an expert in literary criticism on the details of what opinions other literary critics have expressed. In neither case however do I consider their opinions to be particularly relevant to the pursuit of true facts about the world.
The fact that the survey you link finds 27% of philosophers ‘accept or lean towards non-physicalism’ increases my confidence in the above thesis.
Even for experts in meta-ethics, I can’t see how their confidence can get outside the 30%-70% range given the expert disagreement. For non-experts, I really can’t see how one could even get to 50% confidence in anti-realism, much less the kind of 98% confidence that is typically expressed here.
It depends on the expertise; for instance, if we’re talking about systems of axioms, then mathematicians may be those with the most relevant opinions as to whether one system has preference over others. And the idea that a unique system of moral axioms would have preference over all others makes no mathematical sense. If philosphers were espousing the n-realism position (“there are systems of moral axioms that are more true than others, but there will probably be many such systems, most mutually incompatible”), then I would have a hard time arguing against this.
But, put quite simply, I dismiss the moral realistic position for the moment as the arguments go like this:
1) There are moral truths that have special status; but these are undefined, and it is even undefined what makes them have this status.
2) These undefined moral truths make a consistent system.
3) This system is unique, according to criteria that are also undefined.
4) Were we to discover this system, we should follow it, for reasons that are also undefined.
There are too many ’undefined’s in there. There is also very little philosphical literature I’ve encountred on 2), 3) and 4), which is at least as important as 1). A lot of the literature on 1) seems to be reducible to linguistic confusion, and (most importantly) different moral realists have different reasons for believing 1), reasons that are often contradictory.
From a outsider’s perspective, these seem powerful reasons to assume that philosphers are mired in confusion on this issue, and that their opinions are not determining. My strong mathematical reasons for claiming that there is no “superiority total ordering” on any general collection of systems of axioms clinches the argument for me, pending further evidence.
Looking further through the philpapers data, a big chunk of the belief in moral realism seem to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist.
In general, only 61% of respondents “accept” rather than lean towards atheism, and a quarter don’t even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top.
Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective adjudication within a reasonable period of time.
Is there a reason I should care about the % of any group of people that think this or that? Just give us the argument, or write another article about it. It sounds interesting.
In things related to observable facts or repeatable experiments I’d agree. In more abstract things, I’m less interested in what the polls say.
Moral realism is a school of thought which has come in and out of style and favor among philosophers. Plato was arguably an moral realist; this isn’t a new idea or area of debate amongst philosophers. Telling me where we are on the constantly shifting scale of acceptance is really pretty meaningless. Its like telling me 58% of fashion designers like the color black this year.
Just to be sure, are you saying that you think there is a fact of the matter about whether moral realism is true, but you don’t think philosophers’ opinions are significantly correlated with this fact?
Moral realism is a meta-ethical view—I do not know that a such a viewpoint can be as a matter of fact correct or incorrect. Maybe an ethical realist would argue that it is a matter of fact, I’m not sure—an anti-realist might argue that neither viewpoint can be a matter of fact. The whole argument is really about “what are facts” and “what can be objectively true or false” so I suppose that someone may extend this view to the meta-layer where the merits of the viewpoint itself are discussed although I think that would not be very useful.
I’m going to deploy what I call the Wittgenstein Chomsky blah blah blah argument. Philosophy is just words in English; there is little ultimate meaning we are going to find here unless we declare our mathematical axioms. Already most of the views here seem reconcilable by redefining what exactly the different words mean.
To answer the question: some things can be proven objectively true, some things can be proven objectively false, some things can be proven to be undecidable. A fact is a true statement that follows from your given system of axioms. I personally am unsure if most moral principles or meta ethical systems can be declared objectively true or false with a standard ethical system, but I’m not going to take it seriously until a theorem prover says so. We are never going to convince each other of ultimate philosophical truth by having conversations like this.
I suppose this makes me an anti-realist, unless someone feels like redefining realism for me. :D
Moral realism is a school of thought which has come in and out of style and favor among philosophers.
While at times Toby Ord refers to 56% as “most” philosophers, a claim that is disputable on grounds of fashion, at other times he draws the line at 20%; the point is that realist philosophers are not a tiny minority, rejecting widely accepted arguments.
There are a lot of posts here that presuppose some combination of moral anti-realism and value complexity. These views go together well: if value is not fundamental, but dependent on characteristics of humans, then it can derive complexity from this and not suffer due to Occam’s Razor.
There are another pair of views that go together well: moral realism and value simplicity. Many posts here strongly dismiss these views, effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with expert opinion and being clearly overconfident. In the Phil Papers survey for example, 56.3% of philosophers lean towards or believe realism, while only 27.7% lean towards or accept anti-realism.
http://philpapers.org/surveys/results.pl
Given this, and given comments from people like me in the intersection of the philosophical and LW communities who can point out that it isn’t a case of stupid philosophers supporting realism and all the really smart ones supporting anti-realism, there is no way that the LW community should have anything like the confidence that it does on this point.
Moreover, I should point out that most of the realists lean towards naturalism, which allows a form of realism that is very different to the one that Eliezer critiques. I should also add that within philosophy, the trend is probably not towards anti-realism, but towards realism. The high tide of anti-realism was probably in the middle of the 20th Century, and since then it has lost its shiny newness and people have come up with good arguments against it (which are never discussed here...).
Even for experts in meta-ethics, I can’t see how their confidence can get outside the 30%-70% range given the expert disagreement. For non-experts, I really can’t see how one could even get to 50% confidence in anti-realism, much less the kind of 98% confidence that is typically expressed here.
Among target faculty listing meta-ethics as their area of study moral realism’s lead is much smaller: 42.5% for moral realism and 38.2% against.
Looking further through the philpapers data, a big chunk of the belief in moral realism seems to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist.
In general, only 61% of respondents “accept” rather than lean towards atheism, and a quarter don’t even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top.
Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective adjudication within a reasonable period of time.
A similar pattern shows up for physicalism.
In general, those interquestion correlations should help pinpoint any correct contrarian cluster.
This is why I put more weight on Toby’s personal position, than on the majority expert position. As far as I know, Toby is in the same contrarian cluster as me, yet he seems to give much more weight to moral realism (and presumably not the Yudkowskian kind either) than I do. Like ciphergoth, I wish he would tell us which arguments in favor of realism, or against anti-realism, that he finds persuasive.
It seems that would be more likely if some people would put effort into apparently wanting to learn more about moral realism, or would read and present some of the arguments charitably to LW.
Thanks for looking that up Carl—I didn’t know they had the break-downs. This is the more relevant result for this discussion, but it doesn’t change my point much. Unless it was 80% or so in favour of anti-realism, I think holding something like 95% credence in anti-realism this is far too high for non-experts.
Atheism doesn’t get 80% support among philosophers, and most philosophers of religion reject it because of a selection effect where few wish to study what they believe to be non-subjects (just as normative and applied ethicists are more likely to reject anti-realism).
Perhaps we shouldn’t look for professional consensus on things we accept with almost-certainty, because things that can be correctly accepted with almost-certainty by amateurs will not be professionally studied, except by people who are systematically confused. Instead, we should ask non-professional opinion of people who are in the position to know most about the subject, but don’t study it professionally.
You are correct that it is reasonable to assign high confidence to atheism even if it doesn’t have 80% support, but we must be very careful here. Atheism is presumably the strongest example of such a claim here on Less Wrong (i.e. one which you can tell a great story why so many intelligent people would disagree etc and hold a high confidence in the face of disagreement). However, this does not mean that we can say that any other given view is just like atheism in this respect and thus hold beliefs in the face of expert disagreement, that would be far too convenient.
Strong agreement about not overgeneralizing. It does appear, however, that libertarianism about free well, non-physicalism about the mind, and a number of sorts of moral realism form a cluster, sharing the feature of reifying certain concepts in our cognitive algorithms even when they can be ‘explained away.’ Maybe we can discuss this tomorrow night.
Of course not; the substance of one’s reasons for disagreeing matters greatly. In this case, I suspect there’s probably a significant amount of correlation/non-independence between the reasons for believing atheism and believing something like moral non-realism.
One thing we should take away from cases like atheism is that surveys probably shouldn’t be interpreted naively, but rather as somewhat noisy information. I think my own heuristic (on binary questions where I already have a strong opinion) is basically to look on which side of 50% my position falls; if the majority agrees with me (or, say, the average confidence in my position is over 50%), I tend to regard that as (more) evidence in my favor, with the strength increasing as the percentage increases.
(This, I think, would be part of how I would answer Yvain.)
I think the arguments you’re developing here go a long way towards answering Toby’s point, but what safeguards can we use to ensure we can’t use it as a generalized anti-expert defence?
The prerequisite for this heuristic is coming to a conclusion with near-certainty on an amateur level. The safeguard has to be general ability to not get that much unjustified overconfidence.
Are you proposing a safeguard here or setting out what the safeguard has to achieve?
I’m pointing out that there is already a generally applicable enough set of safeguards that covers this case in particular, adequate or not. That is, this heuristic doesn’t automatically lead as astray.
I don’t think I can understand you properly; it reads like you’re saying that we can be confident in rejecting expert advice if we’ve already reached a contrary position with high confidence. That doesn’t sound Bayesian. I suspect the error is mine but I’d appreciate your help in finding and fixing it!
EDIT: I [not Vladimir] would say that if we have one position that we can be confident in (atheism) we can use it as an indicator of expert quality, and pay more attention to those experts on other issues (e.g. moral realism as philosophers define it).
And with respect to the selection effect among philosophers of religion, there’s overwhelming direct evidence on this in the form of the Catholic Church push on this front.
Re: correction:
I would say so too, though I wasn’t saying that here. It is the mechanism through which we can reject expert opinion, but also as applied to the very claim that is being contested, not just the other slam-dunk claims.
Only where there’s a relationship of course. We would be unwise to reject medical expertise from a body where atheists were few, unless religion impinged on that advice eg abortion, cryonics. Here a relationship with religion is clear.
I would say that if on some matter of medical controversy atheist doctors and medical academics tended to come out one way, while the median opinion came out the other way, we should go with the atheist medical opinion, ceteris paribus. Atheism is a proxy for intelligence and scientific thinking, a finding which has a mountain of evidence in its favor.
Definitely if the majority opinion among atheist experts differed from the majority opinion among all experts, I’d go for the former, but if say the majority of doctors studying a disease were Catholic for simple geographic reasons, I’d still defer to their expertise.
I agree with this interpretation.
Zack is making basically the same point here.
(This discussion is about meta-level mechanism for agreement, where you accept a conclusion; experts might well have persuasive arguments that inverse one’s confidence.)
(cf. Argument Screens Off Authority.)
One person’s modus ponens is another’s modus tollens. You say that professional philosophers’ disagreement implies that antirealists shouldn’t be so confident, but my confidence in antirealism is such that I am instead forced to downgrade my confidence in professional philosophers. I defer to experts in mathematics and science, where I can at least understand something of what it means for a mathematical or scientific claim to be true. But on my current understanding of the world, moral realism just comes out as nonsense. I know what it means for a computation to yield this-and-such a result, or for a moral claim to be true with respect to such-and-these moral premises that might be held by some agent. But what does it mean for a moral claim to be simply true, full stop? What experiment could you perform to tell, even in principle? If the world looks exactly the same whether murder is intrinsically right or intrinsically wrong, what am I supposed to do besides say that there simply is no fact of the matter, and proceed with my life just as before?
I realize how arrogant it must seem for young, uncredentialled (not even a Bachelor’s!) me to conclude that brilliant professional philosophers who have devoted their entire lives to studying this topic are simply confused. But, disturbing as it may be to say … that’s how it really looks.
Well, in my world, it means that the premises are built into saying “moral claim”; that the subject matter of “morality” is the implications of those premises, and that moral claims are true when they make true statements about these implications. If you wanted to talk about the implications of other premises, it wouldn’t be the subject matter of what we name “morality”. Most possible agents (e.g. under a complexity-based measure of mind design space) will not be interested in this subject matter—they won’t care about what is just, fair, freedom-promoting, life-preserving, right, etc.
This doesn’t contradict what you say, but it’s a reason why someone who believes exactly everything you do might call themselves a moral realist.
In my view, people who look at this state of affairs and say “There is no morality” are advocating that the subject matter of morality is a sort of extradimensional ontologically basic agent-compelling-ness, and that, having discovered this hypothesized transcendental stuff to be nonexistent, we have discovered that there is no morality. In contrast, since this transcendental stuff is not only nonexistent but also poorly specified and self-contradictory, I think it was a huge mistake to claim that it was the subject matter of morality in the first place, that we were talking about some mysterious ineffable confused stuff when we were asking what is right. Instead I take the subject matter of morality to be what is fair, just, freedom-promoting, life-preserving, happiness-creating, etcetera (and what that starting set of values would become in the limit of better knowledge and better reflection). So moral claims can be true, and it all adds up to normality in a rather mundane way… which is probably just what we ought to expect to see when we’re done.
Yes, but I think that my way of talking about things (agents have preferences, some of which are of a type we call moral, but there is no objective morality) is more useful than your way of talking about things (defining moral as a predicate referring to a large set of preferences), because your formulation (deliberately?) makes it difficult to talk about humans with different moral preferences, which possibility you don’t seem to take very seriously, whereas I think it very likely.
So, according to this view, moral uncertainty is just a subset of logical uncertainty, where we restrict our attention to the implication of a fixed set of moral premises. But why is it that I feel uncertain about which premises I should accept? I bet that when most people talk about moral realism and moral uncertainty, that is what they’re talking about.
Why/how does/should one’s moral premises change as one gains knowledge and ability to reflect? (Note that in standard decision theory one’s values simply don’t change this way.) It seems to me this ought to be the main topic of moral inquiry, instead of being relegated to a parenthetical remark. The subsequent working out of implications seems rather trivial by comparison.
Maybe, but we’re not there yet.
You’ve got meta-moral criteria for judging between possible terms in your utility function, a reconciliation process for conflicting terms, other phenomena which are very interesting and I do wish someone would study in more detail, but so far as metaethics goes it would tend to map onto a computation whose uncertain output is your utility function. Just more logical uncertainty.
How can I put it? The differences here are probably very important to FAI designers and object-level moral philosophers, but I’m not sure they’re metaethically interesting… or they’re metaethically interesting, but they don’t make you confused about what sort of stuff morality could possibly be made out of. Moral uncertainty is still made out of a naturalistic mixture of physical uncertainty and logical uncertainty.
Suppose there’s an UFAI loose on the Internet that’s not yet very powerful. In order to gain more power, it wants me to change my moral premises (so I’ll help it later), and to do that, it places a story on the web for me to find. I read the story, and it “inspires” me to change my values in the direction that the UFAI prefers. In your view, how do we say that this is bad, if this is just what my meta-moral computation did?
If the UFAI convinced you of anything that wasn’t true during the process—outright lies about reality or math—or biased sampling of reality producing a biased mental image, like a story that only depicts one possibility where other possibilities are more probable—then we have a simple and direct critique.
If the UFAI never deceived you in the course of telling the story, but simple measures over the space of possible moral arguments you could hear and moralities you subsequently develop, produce a spread of extrapolated volitions “almost all” of whom think that the UFAI-inspired-you has turned into something alien and unvaluable—if it flew through a persuasive keyhole to produce a very noncentral future version of you who is disvalued by central clusters of you—then it’s the sort of thing a Coherent Extrapolated Volition would try to stop.
See also #1 on the list of New Humane Rights: “You have the right not to have the spread in your volition optimized away by an external decision process acting on unshared moral premises.”
New Humane Rights:
You have the right not to have the spread in your volition optimized away by an external decision process acting on unshared moral premises.
You have the right to a system of moral dynamics complicated enough that you can only work it out by discussing it with other people who share most of it.
You have the right to be created by a creator acting under what that creator regards as a high purpose.
You have the right to exist predominantly in regions where you are having fun.
You have the right to be noticeably unique within a local world.
You have the right to an angel. If you do not know how to build an angel, one will be appointed for you.
You have the right to exist within a linearly unfolding time in which your subjective future coincides with your decision-theoretical future.
You have the right to remain cryptic.
-- Eliezer Yudkowsky
(originally posted sometime around 2005, probably earlier)
What about the least convenient world where human meta-moral computation doesn’t have the coherence that you assume? If you found yourself living in such a world, would you give up and say no meta-ethics is possible, or would you keep looking for one? If it’s the latter, and assuming you find it, perhaps it can be used in the “convenient” worlds as well?
To put it another way, it doesn’t seem right to me that the validity of one’s meta-ethics should depend on a contingent fact like that. Although perhaps instead of just complaining about it, I should try to think of some way to remove the dependency...
(We also disagree about the likelihood that the coherence assumption holds, but I think we went over that before, so I’m skipping it in the interest of avoiding repetition.)
I think this is about metamorals not metaethics—yes, I’m merely defining terms here, but I consider “What is moral?” and “What is morality made of?” to be problems that invoke noticeably different issues. We already know, at this point, what morality is made of; it’s a computation. Which computation? That’s a different sort of question and I don’t see a difficulty in having my answer depend on contingent facts I haven’t learned.
In response to your question: yes, if I had given a definition of moral progress where it turned out empirically that there was no coherence in the direction in which I was trying to point and the past had been a random walk, then I should reconsider my attempt to describe those changes as “progress”.
How do you cash “which computation?” out to logical+physical uncertainty? Do you have in mind some well-defined metamoral computation that would output the answer?
I think you just asked me how to write an FAI. So long as I know that it’s made out of logical+physical uncertainty, though, I’m not confused in the same way that I was confused in say 1998.
“Well-specified” may have been too strong a term, then; I meant to include something like CEV as described in 2004.
Is there an infinite regress of not knowing how to compute morality, or how to compute (how to compute morality), or how to compute (how to compute (...)), that you need to resolve; do you currently think you have some idea of how it bottoms out; or is there a third alternative that I should be seeing?
I think it is a powerful secret of philosophy and AI design that all useful philosophy depends upon the philosopher(s) observing contingent facts from their sensory input stream. Philosophy can be thought of as an ultra high level machine learning technique that records the highest-level regularities of our input/output streams. And the reason I said that this is a powerful AI design principle, is that you realize that your AI can do good philosophy by looking for such regularities.
Think of it as a foundational struggle: you’ve got non-rigorous ideas about what is morally true/right, and you are searching of a way to build a foundation such that any right idea will follow from that foundation deductively. Arguably, this task is impossible within human mind. A better human-level approach would be structural, where you recognize certain (premise) patterns in reliable moral ideas, and learn heuristics that allow to conclude other patterns wherever you find the premise patterns. This constitutes ordinary moral progress, when fixed in culture.
I would agree with the above, but I would also substitute ‘god’, ‘fairies’, ‘chi’ and ‘UFO abductions’, among other things, in place of ‘morality’.
In cases like that, I am perfectly willing to say that we have discovered that the subject matter of “fairies” is a coherent, well-formed concept that turns out to have an empty referent. The closet is there, we opened it up and looked, and there was nothing inside. I know what the world ought to look like if there were fairies, or alternatively no fairies, and the world looks like it has no fairies.
I think that a very large fraction of the time, when a possibility appears to be coherent and well formed, it may turn out not to be upon more careful examination. I would see the subject matter of “fairies” as “that which causes us to talk about fairies”, the subject matter of “dogs” as “that which causes us to talk about dogs”, and the subject matter of “morality” as “that which causes us to talk about morality”. All three are interesting.
This is a theme that crops up fairly frequently as a matter of semantic confusion and is a confusion that is difficult to resolve trivially due to inferential differences to the actual abstract concepts. I haven’t seen this position explained so coherently in one place before. Particularly the line:
… and the necessary context. I would find it useful to have this as a top level post to link to. Even if, as you have just suggested to JamesAndrix, it is just a copy and paste job. It’ll save searching through comments to find a permalink if nothing else.
Copy it to the wiki yourself.
What name?
Such things should go through a top-level post first, original content doesn’t work well for the wiki.
Doctors or medicine, investors or analysis of public information, scientists or science, philosophers or philosophy… maybe it’s the process of credentialing that we should be downgrading our credence in. Really, why should the prior for credentials being a very significant form of evidence ever have been very high?
The philpapers survey is for the top 99 departments. Things do get better as you go up. Among hard scientists, elite schools are more atheist, and the only almost entirely atheist groups are super-elite, like the National Academy of Sciences/Royal Society.
Perhaps the fact that they have devoted their lives to a topic suggests that they have a vested interest in making it appear not to be nonsense. Cognitive dissonance can be tricky even for the pros.
Maybe they mean something different by it than we’re imagining?
Quite possible. But in that case I would say that we’re just talking about things in different ways, and not actually disagreeing on anything substantive.
Say we did a survey of 1000 independent advanced civilizations—and found they all broadly agreed on some moral proposition X.
That’s the kind of evidence that I think would support the idea of morality inherent in the natural world.
Toby, I spent a while looking into the meta-ethical debates about realism. When I thought moral realism was a likely option on the table, I meant:
Strong Moral Realism: All (or perhaps just almost all) beings, human, alien or AI, when given sufficient computing power and the ability to learn science and get an accurate map-territory distinction, will agree on what physical state the universe ought to be transformed into, and therefore they will assist you in transforming it into this state.
But modern philosophers who call themselves “realists” don’t mean anything nearly this strong. They mean that that there are moral “facts”. But what use is it if the paperclipper agrees that it is a “moral fact” that human rights ought to be respected, if it then goes on to say it has no desire to act according to the prescription of moral facts, and moral facts can’t somehow revoke it.
The force of “scientific facts” is that they constrain the world. If an alien wants to get from Andromeda to here, it has to take at least 2.5 million years, the physical fact of the finite speed of light literally stops the alien from getting here sooner, whether it likes it or not.
The 56.3/27.7% split on philpapers seems to me to be an argument about whether you should be allowed to attach the word “fact” to your preferences, kind of as a shiny badge of merit, without actually disagreeing on any physical prediction about the world. The debate between weak moral realists and antirealists sounds like the debate where two people ask “if a tree falls in the forest, does it really make a sound?”—they’re not arguing about anything substantive.
So, I ask, how many philosophers are strong moral realists, in the sense I defined?
EDIT: After seeing Carl’s comment, it seems likely to me that there probably are a bunch of theists who would, in fact, support the strong moral realism position; but they’re clowns, so who cares.
I strongly agree with Roko that something like his strong version is the interesting version. What matters is what range of creatures will come to agree on outcomes; it matters much less what range of creatures think their desires are “right” in some absolute sense, if they don’t think that will eventually be reflected in agreement.
Roko’s question seems engineered to be wrong to me.
If this is what people think moral realism means—or should mean—no wonder they disagree with it.
In the context of this comment, the goal of FAI can be said to be to constrain the world by “moral facts”, just like laws of physics constrain the world by “physical facts”. This is the sense in which I mean “FAI=Physical Laws 2.0”.
Only in a useless way: there is a specific FAI that does the “truly right” thing, but the truthhood of rightness doesn’t stop you from having to code the rightness in. Goodness is not discoverably true: if you don’t already know exactly what goodness is, you can’t find out.
I’m describing the sense of post-FAI world.
hmmm. That is interesting. Well, let us define the collection W_i of worlds run by superintelligences with the subscript i ranging over goals. No matter what i is, those worlds are going to look, to any agents in them, like worlds with “moral truths”.
However, any agent that learned the real physics of such a world would see that the goodness is written in to the initial conditions, not the laws.
Roko, you make a good point that it can be quite murky just what realism and anti-realism mean (in ethics or in anything else). However, I don’t agree with what you write after that. Your Strong Moral Realism is a claim that is outside the domain of philosophy, as it is an empirical claim in the domain of exo-biology or exo-sociology or something. No matter what the truth of a meta-ethical claim, smart entities might refuse to believe it (the same goes for other philosophical claims or mathematical claims).
Pick your favourite philosophical claim. I’m sure there are very smart possible entities that don’t believe this and very smart ones that do. There are probably also very smart entities without the concepts needed to consider it.
I understand why you introduced Strong Moral Realism: you want to be able to see why the truth of realism would matter and so you came up with truth conditions. However, reducing a philosophical claim to an empirical one never quite captures it.
For what its worth, I think that the empirical claim Strong Moral Realism is false, but I wouldn’t be surprised if there was considerable agreement among radically different entities on how to transform the world.
If there’s a philosophical claim that intelligent agents across the universe wouldn’t display massive agreement on, then I don’t really think it is worth its salt. I think that this principle can be used to eliminate a lot of nonsense from philosophy.
Which of anti-realism or weak realism is true seems to be a question we can eliminate. Whether strong realism is true or not seems substantive, because it matters to our policy which is true.
There are clearly some examples where there can be interesting things to say that aren’t really empirical, e.g. decision theory, mystery of subjective experience. But I think that this isn’t one of them.
Suffice it to say I can’t think of anything that makes the debate between weak realism and antirealism at all interesting or worthy of attention. Certainly, Friendly AI theorists ought not care about the difference, because the empirical claims about an AI system will do are identical. Once the illusions and fallacies surrounding rationalist moral psychology has been debunked, proponents of other AI motivation methods than FAI also ought not to care about the weak realism vs. anti-realism pseudo-question
I’m having trouble reconciling this with the beginning of your first comment:
Not me.
An “optimal organism” may be a possibility, though. Assuming god’s utility function, it is theoretically possible that a unique optimal agent might exist. Whether it would be found before the universal heat death is another issue, though.
From my naturalist POV, you need to show me a paperclipper before it is convincing evidence about the real world. Paperclipper’s are theoretical possibilities, but who would build one, why, and how long would it last in the wild?
...and if the “paperclips” part is a metaphor, then which preferred ordered atomic states count, and which don’t? Is a cockroach a “paperclipper”—because it acts as though it wants to fill the universe with its DNA?
Yes, paperclips are a metaphor. No one expects a literal paperclip maximizer; the point is to illustrate unFriendly AI as a really powerful system with little or no moral worth as humans would understand moral worth. A non-conscious superintelligent cockroach-type thing that fills the universe with its DNA or equivalent would indeed qualify.
In that case, I don’t think a division of superintelligences into paperclippers and non-paperclippers “carves nature at the joints” very well. It appears to be a human-centric classification scheme.
I’ve proposed another way of classifying superintelligence goal systems—according to whether or not they are “handicapped”.
Healthy superintelligences execute god’s utility function—i.e. they don’t value anything apart from their genes.
Handicapped superintelligences value other things—paperclips, gold atoms, whatever. Genes are valued too—but they may only have proximate value.
According to this classification scheme, the cockroach and paperclipper would be in different categories.
“Handicapped” superintelligences value things besides their genes. They typically try and leave something behind. Most other agents keep dissipating negentropy until they have flattened energy gradients as much as they can—the way most living ecosystems do.
http://alife.co.uk/essays/handicapped_superintelligence/
Yes, that’s the point! We’re humans, and so for some purposes we find it useful to categorize superintelligences into those that do and don’t do what we want, even if it isn’t a natural categorization from a more objective standpoint.
Right—well, fine. One issue is that the classification into paperclippers and non-paperclippers was not clear to me until you clarified it. Another poster has “clarified” things the other way in response to the same comment. So, as a classification scheme, IMO the idea seems rather vague and unclear.
The next issue is: how close does an agent have to be to what you (we?) want before it is a non-paperclipper?
IMO, the idea of a metaphorical unfriendly paperclipper appears to need pinning down before it is of very much use as a means of superintelligence classification scheme.
I’m pretty confident Roko agrees with me and that this is just a communication error.
I’m given to understand that the classification scheme is Friendly versus unFriendly, with paperclip maximizer being an illustrative (albeit not representative) example of the latter. I agree that more rigor (and perhaps clearer terminology) is in order.
Machine intelligences seem likely to vary in their desirability to humans.
Friendly / unFriendly seems rather binary, maybe a “desirability” scale would help.
Alas, this seems to be drifting away from the topic.
Technically true. However, most naive superintelligence designs will simply kill all humans. You’ve accomplished quite a lot to even get to a failed utopia, much less deciding whether you want Prime Intellect or Coherent Extrapolated Volition.
It’s also unlikely you’ll accidentally do something significantly worse than killing all humans, for the same reasons. A superintelligent sadist is just as hard as a utopia.
I read the essay you linked to. I really don’t know where to start.
So we should guard against potential threats from non-human intelligent life by building a non-human superintelligence that doesn’t care about humans?
Postpone? I thought the point of friendly AI was to preserve human values for as long as physically possible. “Evolutionary progress?” Evolution is stupid and doesn’t care about the individual organisms. Evolution causes pointless suffering and death. It produces stupid designs. As Michael Vassar once said: think of all the simple things that evolution didn’t invent. The wheel. The bow and arrow. The axial-flow pump. Evolution had billions of years creating and destroying organisms and it couldn’t invent stuff built by cave men. Is it OK in your book that people die of antibiotic resistant diseases? MRSA is a result of evolutionary progress.
Who said humans have to live on planets or breathe oxygen or run on neurons? Why do you think a superintelligence will have problems dealing with asteroids when humans today are researching ways to deflect them?
I think your main problem is that you’re valuing the wrong thing. You practically worship evolution while neglecting important things like people, animals, or anything that can suffer. Also, I think you fail to notice the huge first-mover advantage of any superintelligence, even one as “handicapped” as a friendly AI.
Finally, I know the appearance of the arguer doesn’t change the validity of the argument, but I feel compelled to tell you this: You would look much better with a haircut, a shave, and some different glasses.
Briefly:
I don’t avocate building machines that are indiffierent to humans. For instance, I think machine builders would be well advised to (and probably mostly will) construct devices that obey the law—which includes all kinds of provisions for preventing harm to humans.
Evolution did produce the wheel and the bow and arrow. If you think otherwise, please state clearly what definition of the term “evolution” you are using.
Regarding space travel—I was talking about wetware humans.
Re: “Why do you think a superintelligence will have problems dealing with asteroids when humans today are researching ways to deflect them?”
...that is a projection on your part—not something I said.
Re: “Also, I think you fail to notice the huge first-mover advantage of any superintelligence”
To quote mine myself:
“IMHO, it is indeed possible that the first AI will effectively take over the world. I.T. is an environment with dramatic first-mover advantages. It is often a winner-takes-all market – and AI seems likely to exhibit such effects in spades.”
http://www.overcomingbias.com/2008/05/roger-shank-ai.html
“Google was not the first search engine, Microsoft was not the first OS maker—and Diffie–Hellman didn’t invent public key crypto.
Being first does not necessarily make players uncatchable—and there’s a selection process at work in the mean time, that weeds out certain classes of failures.”
http://lesswrong.com/lw/1mm/advice_for_ai_makers/1gkg
I have thought and written about this issue quite a bit—and my position seems a bit more nuanced and realistic than the position you are saying you think I should have.
Superintelligences don’t have genes.
Well, most superintelligences don’t have genes.
They do if you use an information-theory definition of the term—like the ones on:
http://alife.co.uk/essays/informational_genetics/
I disagree even with your interpretation of that document, but that is not the point emphasized in the grandparent. I acknowledge that while a superintelligence need not have genes it is in fact possible to construct a superintelligence that does relies significantly on “small sections of heritable information”, including the possibility of a superintelligence that relies on genes in actual DNA. Hence the slight weakening of the claim.
What follows is just a copy-and-paste of another reply, but:
By “gene” I mean:
“Small chunk of heritable information”
http://alife.co.uk/essays/informational_genetics/
Any sufficiently long-term persistent structure persists via a copying process—and so has “genes” in this sense.
I think your term “God’s utility function” is a bit confusing—as if it’s just one utility function. If you value your genes, and I value my genes, and our genes are different, then we have different utility functions.
Also, the vast majority of possible minds don’t have genes.
Maybe. Though if you look at:
http://originoflife.net/gods_utility_function/
...then first of all the term is borrowed/inherited from:
http://en.wikipedia.org/wiki/God%27s_utility_function
...and also, I do mean it in a broader sense where (hopefully) it makes a bit more sense.
The concept is also referred to as “Goal system zero”—which I don’t like much.
My latest name for the idea is “Shiva’s goals” / “Shiva’s values”—a reference to the Hindu god of destruction, creation and transformation.
By “gene” I mean:
“Small chunk of heritable information”
http://alife.co.uk/essays/informational_genetics/
Any sufficiently long-term persistent structure persists via a copying process—and so has “genes” in this sense.
What we mean by preference. Except that preference, being a specification of a computation, has a lot of forms of expression, so it doesn’t “persist” by a copying process, it “persists” as a nontrivial computational process.
A superintelligence that persists in copying a given piece of information is running a preference (computational process) that specifies copying as the preferable form of expression, over all the other things it could be doing.
No, no! Genes is just intended to refer to any heritable information. Preferences are something else entirely. Agents can have preferences which aren’t inherited—and not everything that gets inherited is a preference.
Anything information that persists over long periods of time persists via copying.
“Copying” just means there’s Shannon-mutual information between the source and the destination which originated in the source. Complex computations are absolutely included—provided that they share this property.
Then preference still qualifies. This holds as a factual claim provided we are talking about reflectively consistent agents (i.e. those that succeed in not losing their preference), and as a normative claim regardless.
I would appreciate it if you avoid redefining words into highly qualified meanings, like “gene” for “anything that gets copied”, and then “copying” for “any computation process that preserves mutual information”.
Re: Then preference still qualifies. This holds as a factual claim provided [bunch of conditions]
Yes, there are some circumstances under which preferences are coded genetically and reliably inherited. However, your claim was stronger. You said what meant by genes was what “we” would call preferences. That implies that genes are preferences and preferences are genes.
You have just argued that a subset of preferences can be genetically coded—and I would agree with that. However, you have yet to argue that everything that is inherited is a preference.
I think you are barking up the wrong tree here—the concepts of preferences and genes are just too different. For example, clippy likes paperclips, in addition to the propagation of paperclip-construction instructions. The physical paperclips are best seen as phenotype—not genotype.
Re: “I would appreciate it if you avoid redefining words into highly qualified meanings [...]”
I am just saying what I mean—so as to be clear.
If you don’t want me to use the words “copy” and “gene” for those concepts—then you are out of luck—unless you have a compelling case to make for better terminology. My choice of words in both cases is pretty carefully considered.
Not “bunch of conditions”. Reflective consistency is the same concept as “correctly copying preference”, if I read your sense of “copying” correctly, and given that preference is not just “thing to be copied”, but also plays the appropriate role in decision-making (wording in the grandparent comment improved). And reflectively consistent agents are taken as a natural and desirable (from the point of view of those agents) attractor where all agents tend to end up, so it’s not just an arbitrary category of agents.
But there are many different preferences for different agents, just as there are different genes. Using the word “genes” in the context where both human preference and evolution are salient is misleading, because human genes, even if we take them as corresponding to a certain preference, don’t reflect human preference, and are not copied in the same sense human preference is copied. Human genes are exactly the thing that currently persists by vanilla “copying”, not by any reversible (mutual information-preserving) process.
Confusing terminology is still bad even if you failed to think up a better alternative.
Confusing terminology is still bad even if you failed to think up a better alternative.
You appear to be on some kind of different planet to me—and are so far away that I can’t easily see where your ideas are coming from.
The idea I was trying to convey was really fairly simple, though:
“Small chunks of heritable information” (a.k.a. “genes”) are one thing, and the term “preferences” refers to a different concept.
As an example of a preference that is not inherited, consider the preference of an agent for cats—after being bitten by a dog as a child.
As an example of something that is inherited that is not a preference, consider the old socks that I got from my grandfather after his funeral.
These are evidently different concepts—thus the different terms.
Thanks for your terminology feedback. Alas, I am unmoved. That’s the best terminology I have found, and you don’t provide an alternative proposal. It is easy to bitch about terminology—but not always so easy to improve on it.
I meant a literal paperclip maximizing superintelligent AI, so no, a cockroach is not one of those.
Right—well, that seems pretty unlikely. What is the story? A paperclip manufacturer with an ambitious IT department that out-performs every other contender? How come the government doesn’t just step on the results?
Is there a story about how humanity drops the reins of civilisation like that that is not extremely contrived?
I am unclear on how this story is contrived. There are vast numbers of business with terrible externalities today; this is deeply related to the debate on climate change. Alternately, we have large cutting machines and those machines don’t care whether they are cutting trees or people; if a wood chipper could pick things up but not distinguish between which things it was picking up it would be very dangerous (but still very useful if you have a large space of things you want chipped).
I am a moral cognitivist. Statements like “ceteris paribus, happiness is a good thing” have truth-values. Such moral statements simply are not compelling or even interesting enough to compute the truth-value of to the vast majority of agents, even those which maximize coherent utility functions using Bayesian belief updating (that is, rational agents) or approximately rational agents.
AFAICT the closest official term for what I am is “analytic descriptivist”, though I believe I can offer a better defense of analytic descriptivism than what I’ve read so far.
EDIT: Looking up moral naturalism shows that Frank Jackson’s analytic descriptivism aka moral functionalism is listed as a form of moral naturalism: http://plato.stanford.edu/entries/naturalism-moral/#JacMorFun
Note similarity to “Joy in the Merely Good”.
For the interested: A good summary/defense of Jackson’s moral functionalism can be found in Jackson (2012), “On ethical naturalism and the philosophy of language.”
Now, should we call this a form of “moral realism”? I dunno. That’s something I’d prefer to taboo. Even famous error theorist Richard Joyce kinda agrees.
From your SEP link on Moral Realism: “It is worth noting that, while moral realists are united in their cognitivism and in their rejection of error theories, they disagree among themselves not only about which moral claims are actually true but about what it is about the world that makes those claims true. ”
I think this is good cause for breaking up that 56%. We should not take them as a block merely because (one component of) their conclusions match, if their justifications are conflicting or contradictory. It could still be the case that 90% of expert philosophers reject any given argument for moral realism. (This would be consistent with my view that those arguments are silly.)
I may have noticed this because the post on Logical Rudeness is fresh in my mind.
You are entirely right that the 56% would split up into many subgroups, but I don’t really see how this weakens my point: more philosophers support realist positions than anti-realist ones. For what its worth, the anti-realists are also fragmented in a similar way.
Disagreeing positions don’t add up just because they share a feature. On the contrary, If people offer lots of different contradictory reasons for a conclusion (even if each individual has consistent beliefs) it is a sign that they are rationalizing their position.
If 2⁄3′s of experts support proposition G , 1⁄3 because of reason A while rejecting B, and 1⁄3 because of reason B while rejecting A, and the remaining 1⁄3 reject A and B; then the majority Reject A, and the majority Reject B. G should not be treated as a reasonable majority view.
This should be clear if A is the koran and B is the bible.
If we’re going to add up expert views, we need to add up what experts consider important about a question, not features of their conclusions.
You shouldn’t add up two experts if they would consider each other’s arguments irrational. That’s ignoring their expertise.
I know it might seem difficult to expand this into a top-level post, but if you just want to post it verbatim, I’d say go for it.
Yes James, I’d also appreciate that. Maybe we should encourage more short top-level posts and comment upgrades to posts. I think that would be great if we could develop a good procedure.
This certainly doesn’t work in all cases:
There is a hidden object which is either green, red or blue. Three people have conflicting opinions about its colour, based on different pieces of reasoning. If you are the one who believes it is green, you have to add up the opponents who say not-green, despite the fact that there is no single not-green position (think of the symmetry—otherwise everyone could have too great confidence). The same holds true if these are expert opinions.
The above example is basically as general as possible, so in order for your argument to work it will need to add specifics of some sort.
Also, the Koran/Bible case doesn’t work. By symmetry, the Koran readers can say that they don’t need to add up the Bible readers and the atheists, since they are heterogeneous, so they can keep their belief in the Koran...
In practice all arguments will share some premises and some conclusions, in messy asymmetrical ways.
If the not-greens share a a consistent rationale about why the object cannot be green, then I need to take that into account.
If the red supporter contends that all green and blue objects were lost in the color wars, while the blue supporter contends that all objects are fundamentally blue and besides the color wars never happened, then their opinions roughly cancel each other out. (Barring other reasons for me to view one as more rational than the other.)
I suspect that there are things to be said about islam that both atheists and christians would agree on. That’s a block that a rational muslim should take into account. Our disagreeing conclusions about god are secondary.
If I’m going to update my position because 56% of experts agree on something, then I want to know what I’m going to update to.
This discussion continues here.
BTW, I wish there is a way to upgrade a comment into a post and automatically move all the discussions under the new post as well.
The only reason I can think of to upgrade a comment to a post is to draw attention to it, whether google attention, naturality of external linking, or the attention of the regular readers. In all these cases, it seems to me that it is the duty of the author, who is demanding time from many readers, to spend time summarizing the old discussion and making it easy for new readers to join.
I haven’t heard it put that way before. But your explanation makes it seem obvious!
Ignoring their expertise, but counting only popularity. Moderator, does that mean that Less Wrong’s karma system might be modified to take into account why a comment was upvoted?
A valid principle James, but a bad example which might be contested by those more knowledgeable of the matter.
Islam considers itself the best of the revealed religions and jesus is revered as a prophet in Islam.
So, in this case, christians reject the koran, but the muslims do not completely reject the bible.
I’m not sure what might serve as a better example, though. The multiple possible explanations of the present recession may serve as a better example, incase you want to make this a top level post.
What you say is true while the Koran and the Bible are referents, but when A and B become “Mohammed is the last prophet, who brought the full truth of God’s will” and “Jesus was a literal incarnation of God,” (the central beliefs of the religions that hold the respective books sacred) then James’ logic holds.
This applies very generally when the evidential properties of reference classes are brought up.
The right response to moral realism isn’t to dispute it’s truth but to simply observe you don’t understand the concept.
I mean imagine someone started going around insisting some situations were Heret and others were Grovic but when asked to explain what made a situation Heret or Grovic he simply shrugged and said they were primitive concepts. But you persist and after observing his behavior for a period of time you work out some principle that perfectly predicts which category he will assign a given situation to, even counterfactually but when you present the algorithm to him and ask, “Ohh so is it satisfying this principle that makes one Heret rather than Grovic?” he insists that while your notion will always agrees with his notion that’s not what he means. Moreover, he insists that no definition in terms of physical state could capture these concepts.
Confused you press him and he says that there are special things which we can’t casually interact with that determine Heret or Grovic status. Bracketing your skepticism you ask him to say what properties these new ontological objects must have. After listing a couple he adds that most importantly they can’t just be random things with this structure but they also have to be Heret making or Grovic making and that’s what distingushes them from all the other casually inaccessible things out there that might otherwise yield some slightly different class of things as Heret and Grovic.
Frustrated you curse the guy saying he hasn’t really told you anything since you didn’t know what it meant to be Heret or Grovic in the first place so you surely don’t know what it means to be Heret making or Grovic making. The man’s reply is simply to shrug and say, “well it’s a fundamental concept, if you don’t understand I can’t explain it to you anymore than I could explain the perceptual experience of redness to a man who had never experienced color.”
In such a situation the only thing you can do is give up on the notion of Heret and Grovic. Debating about whether to say it’s an incoherent notion, a concept you lack the facilities to comprehend or something else would just waste time with a useless word game. Ultimately you just have to ignore such talk as something that lacks content for you and treat it the same way as you would meaningless gibberish.
The fact that when moral realists say the same thing about good and evil they are using the same sounds that we understand to mean something different shouldn’t change the situation at all.
Could you direct us to the best arguments for moral realism, or against anti-realism? Thanks!
In metaethics, there are typically very good arguments against all known views, and only relatively weak arguments for each of them. For anything in philosophy, a good first stop is the Stanford Encyclopedia of Philosophy. Here are some articles on the topic at SEP:
Moral Anti-Realism
Moral Realism
Metaethics
Moral Cognitivism vs Non-cognitivism
I think the best book to read on metaethics is:
An Introduction to Contemporary Metaethics
Toby, I read through those SEP articles but couldn’t find the good arguments against anti-realism that you mentioned. In contrast, the article on deontology laid out the arguments for and against it very clearly.
Can you please point us more specifically to the arguments that you find persuasive? Maybe just give us some page numbers in the book that you referenced? Most of us don’t really have the time to read something like that cover to cover in search of a few nuggets of information.
Thank you for doing that, and may I second this. I started reading those articles, then after a bit started scanning for the anti-realism articles, and worried after not finding them that I’d not read carefully enough, so I’m glad to have your report on this.
I really am curious to read these arguments, so I hope someone can point us to them.
I managed to find a draft of a book chapter titled In Defence of Moral Realism. I’m still wondering what Toby thinks the best arguments are, but alas he doesn’t seem to be following this discussion anymore.
Thanks! Again, didn’t get much from a quick skim, let me know if you find any real meat in there.
The thing that really got my attention wasn’t the assertion that there are some arguments in favour of realism, but that there are good arguments specifically against anti-realism.
I know I’ve spoken of “skimming” twice here. I promise, if Toby Ord were to say to me “this contains good arguments against anti-realism” I would read it carefully.
But surely an argument against anti-realism is also an argument for realism? I’m interpreting Toby’s comment as saying that there are good arguments for realism in general, but not for any particular realist meta-ethical theory.
The author says in the conclusion, “I do not pretend to give any knock-down argument in this chapter for the thesis that objective moral facts or reasons exist, independently of our thoughts and actions.” So I think it’s mostly a matter of how convincing one finds the argument that he does give.
It seems likely, given that the author is a specialist in and proponent of moral realism, that he would give the best arguments that he knew, so this paper seems like good evidence for what kind of arguments for realism is currently available.
Will read carefully on that basis. Thanks.
Do you have a personal favourite argument against moral anti-realism in there you could point me to?
Thanks! There were several points in your PhD thesis where I couldn’t work out how to square your position with moral anti-realism—I guess I know why now :-)
My case was here:
http://lesswrong.com/lw/1m5/savulescu_genetically_enhance_humanity_or_face/1fuv
Basically, morality is a product of evolution—which can be expected to favour some moral values over other ones—just as it favours certain physical structures like eyes and legs.
Things like: “under most circumstances, don’t massacre your relatives or yourself” can be reasonably expected to be widespread values in the universe. The idea gives morality a foundation in the natural world.
It is useful that Tim summarizes his position in this context, voted up.
My position, developed with no background in philosophy or meta-ethics whatsoever and thus likely to be error-riddled or misguided, is that I consider it an unsolved problem within physical materialism (specifically, within the context of moral anti-realism) how “meaning” (the meaning of life and/or the value of values) can be a coherent or possible concept.
Leave humans out of it and try to think about meanings of signals among animals, with an evolutionary perspective.
I accept this may be a case of the Popularization Bias (speaking for myself). I’d like to see some posts on the arguments against anti-realism...
Agreed. Perhaps Toby or David Pearce can be persuaded.
I don’t think I can persuaded.
I have many good responses to the comments here, and I suppose I could sketch out some of the main arguments against anti-realism, but there are also many serious demands on my time and sadly this doesn’t look like a productive discussion. There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying (which is particularly alarming given that many commenters are pointing out that they don’t understand what the opponents are saying!).
Given all of this, I fear that writing a post would not be a good use of my time.
Alas. Perhaps some Less Wrongers with more time will write and post a hypothetical apostasy. I invite folk to do so.
This is a little unfair; as soon as you take a deflationary stance on anything, you’re saying that the other stance doesn’t really have comprehensible content, and it’s a mistake to turn that into a general-purpose dismissal of deflationary stances.
If you think that’s more true here than it is in other discussion forums, we’re doing something very wrong. I understand that you’re not able to spend time writing for this audience, but for those of us who do want to find out more about what moral realists are saying, every link you can provide to existing essays is valuable.
I, for one, am interested in hearing arguments against anti-realism.
If you don’t have personal interest in writing up a sketch, that’s fine. Might you have some links to other people who have already done so?
Elsewhere in the thread.
Toby already linked to the SEP articles on moral realism and anti-realism in another comment.
This point has given me a lot of pause, so forgive me my many replies. Part of the problem is that even if I were only 60% confident of moral anti-realism, I would still act on it as if I were 100% confident because I don’t understand moral realism at all, and my 60% confidence is in the belief that no-one else does either.
Can you give pointers to prominent naturalist realists?
My impression of academic philosophers is that their ‘expertise’ is primarily in knowledge of what other philosophers have said and in the forms of academic philosophical argument. It is not expertise in true facts about the world. In other words, I would defer to their expertise on the technical details of academically accepted definitions of philosophical terms, or on the writings of Kant, much as I would defer to an expert in literary criticism on the details of what opinions other literary critics have expressed. In neither case however do I consider their opinions to be particularly relevant to the pursuit of true facts about the world.
The fact that the survey you link finds 27% of philosophers ‘accept or lean towards non-physicalism’ increases my confidence in the above thesis.
It depends on the expertise; for instance, if we’re talking about systems of axioms, then mathematicians may be those with the most relevant opinions as to whether one system has preference over others. And the idea that a unique system of moral axioms would have preference over all others makes no mathematical sense. If philosphers were espousing the n-realism position (“there are systems of moral axioms that are more true than others, but there will probably be many such systems, most mutually incompatible”), then I would have a hard time arguing against this.
But, put quite simply, I dismiss the moral realistic position for the moment as the arguments go like this:
1) There are moral truths that have special status; but these are undefined, and it is even undefined what makes them have this status.
2) These undefined moral truths make a consistent system.
3) This system is unique, according to criteria that are also undefined.
4) Were we to discover this system, we should follow it, for reasons that are also undefined.
There are too many ’undefined’s in there. There is also very little philosphical literature I’ve encountred on 2), 3) and 4), which is at least as important as 1). A lot of the literature on 1) seems to be reducible to linguistic confusion, and (most importantly) different moral realists have different reasons for believing 1), reasons that are often contradictory.
From a outsider’s perspective, these seem powerful reasons to assume that philosphers are mired in confusion on this issue, and that their opinions are not determining. My strong mathematical reasons for claiming that there is no “superiority total ordering” on any general collection of systems of axioms clinches the argument for me, pending further evidence.
Looking further through the philpapers data, a big chunk of the belief in moral realism seem to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist.
In general, only 61% of respondents “accept” rather than lean towards atheism, and a quarter don’t even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top.
Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective adjudication within a reasonable period of time.
A similar pattern shows up for physicalism.
In general, those interquestion correlations should help pinpoint any correct contrarian cluster.
I don’t see in what meaningful sense these people are “experts”.
Is there a reason I should care about the % of any group of people that think this or that? Just give us the argument, or write another article about it. It sounds interesting.
Re: “Is there a reason I should care about the % of any group of people that think this or that?”
Generally speaking, yes, of course. If lots of experts in a relevant field think something is true, then their opinion carries some weight.
In things related to observable facts or repeatable experiments I’d agree. In more abstract things, I’m less interested in what the polls say.
Moral realism is a school of thought which has come in and out of style and favor among philosophers. Plato was arguably an moral realist; this isn’t a new idea or area of debate amongst philosophers. Telling me where we are on the constantly shifting scale of acceptance is really pretty meaningless. Its like telling me 58% of fashion designers like the color black this year.
Just to be sure, are you saying that you think there is a fact of the matter about whether moral realism is true, but you don’t think philosophers’ opinions are significantly correlated with this fact?
Moral realism is a meta-ethical view—I do not know that a such a viewpoint can be as a matter of fact correct or incorrect. Maybe an ethical realist would argue that it is a matter of fact, I’m not sure—an anti-realist might argue that neither viewpoint can be a matter of fact. The whole argument is really about “what are facts” and “what can be objectively true or false” so I suppose that someone may extend this view to the meta-layer where the merits of the viewpoint itself are discussed although I think that would not be very useful.
I’m going to deploy what I call the Wittgenstein Chomsky blah blah blah argument. Philosophy is just words in English; there is little ultimate meaning we are going to find here unless we declare our mathematical axioms. Already most of the views here seem reconcilable by redefining what exactly the different words mean.
To answer the question: some things can be proven objectively true, some things can be proven objectively false, some things can be proven to be undecidable. A fact is a true statement that follows from your given system of axioms. I personally am unsure if most moral principles or meta ethical systems can be declared objectively true or false with a standard ethical system, but I’m not going to take it seriously until a theorem prover says so. We are never going to convince each other of ultimate philosophical truth by having conversations like this.
I suppose this makes me an anti-realist, unless someone feels like redefining realism for me. :D
Again, it feels like I am missing something… http://plato.stanford.edu/entries/truth-axiomatic/ helped a little.
While at times Toby Ord refers to 56% as “most” philosophers, a claim that is disputable on grounds of fashion, at other times he draws the line at 20%; the point is that realist philosophers are not a tiny minority, rejecting widely accepted arguments.
Upvoted for being a legitimate question, from a fairly new poster, that really shouldn’t be at −4.