I’d consider important overlapping academic fields to be AI and long term economic growth; I base my claim about academic expert opinion on my informal sampling of such folks. I would of course welcome a more formal sampling.
I’m not comfortable publicly naming names based on informal conversations. These folks vary of course in how much of the details of your arguments they understand, and of course you could always set your bar high enough to get any particular number of folks who have understood “enough.”
Okay. I don’t know any academic besides you who’s even tried to consider the arguments. And Nick Bostrom et. al., of course, but AFAIK Bostrom doesn’t particularly disagree with me. I cannot refute what I have not encountered, I do set my bar high, and I have no particular reason to believe that any other academics are in the game. I could try to explain why you disagree with me and Bostrom doesn’t.
Haven’t particularly looked at that—I think some other SIAI people have. I expect they’d have told me if there was any analysis that counts as serious by our standards, or anything new by our standards.
If someone hasn’t read my arguments specifically, then I feel very little need to explain why they might disagree with me. I find myself hardly inclined to suspect that they have reinvented the same arguments. I could talk about that, I suppose—“Why don’t other people in your field invent the same arguments you do?”
You have written a lot of words. Just how many of your words would someone have had to read to make you feel a substantial need to explain the fact they are world class AI experts and disagree with your conclusions?
I’m sorry, but I don’t really have a proper lesson plan laid out—although the ongoing work of organizing LW into sequences may certainly help with that. It would depend on the specific issue and what I thought needed to be understood about that issue.
If they drew my feedback cycle of an intelligence explosion and then drew a different feedback cycle and explained why it fit the historical evidence equally well, then I would certainly sit up and take notice. It wouldn’t matter if they’d done it on their own or by reading my stuff.
E.g. Chalmers at the Singularity Summit is an example of an outsider who wandered in and started doing a modular analysis of the issues, who would certainly have earned the right of serious consideration and serious reply if, counterfactually, he had reached different conclusions about takeoff… with respect to only the parts that he gave a modular analysis of, though, not necessarily e.g. the statement that de novo AI is unlikely because no one will understand intelligence. If Chalmers did a modular analysis of that part, it wasn’t clear from the presentation.
Roughly, what I expect to happen by default is no modular analysis at all—just snap consideration and snap judgment. I feel little need to explain such.
Roughly, what I expect to happen by default is no modular analysis at all—just snap consideration and snap judgment. I feel little need to explain such.
You, or somebody anyway, could still offer a modular causal model of that snap consideration and snap judgment. For example:
What cached models of the planning abilities of future machine intelligences did the academics have available when they made the snap judgment?
What fraction of the academics are aware of any current published AI architectures which could reliably reason over plans at the level of abstraction of “implement a proxy intelligence”?
What fraction of them have thought carefully about when there might be future practical AI architectures that could do this?
What fraction use a process for answering questions about the category distinctions that will be known in the future, which uses as an unconscious default the category distinctions known in the present?
What false claims have been made about AI in the past? What decision rules might academics have learned to use, to protect themselves from losing prestige for being associated with false claims like those?
How much do those decision rules refer to modular causal analyses of the object of a claim and of the fact that people are making the claim?
How much do those decision rules refer to intuitions about other peoples’ states of mind and social category memberships?
How much do those decision rules refer to intuitions about other peoples’ intuitive decision rules?
Historically, have peoples’ own abilities to do modular causal analyses been good enough to make them reliably safe from losing prestige by being associated with false claims? What fraction of academics have the intuitive impression that their own ability to do analysis isn’t good enough to make them reliably safe from losing prestige by association with a false claim, so that they can only be safe if they use intuitions about the states of mind and social category memberships of a claim’s proponents?
Of those AI academics who believe that a machine intelligence could exist which could outmaneuver humans if motivated, how do they think about the possible motivations of a machine intelligence?
What fraction of them think about AI design in terms of a formalism such as approximating optimal sequential decision theory under a utility function? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
What fraction of them think about AI design in terms of intuitively justified decision heuristics? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
What fraction of them understand enough evolutionary psychology and/or cognitive psychology to recognize moral evaluations as algorithmically caused, so that they can reject the default intuitive explanation of the cause of moral evaluations, which seems to be: “there are intrinsic moral qualities attached to objects in the world, and when any intelligent agent apprehends an object with a moral quality, the action of the moral quality on the agent’s intelligence is to cause the agent to experience a moral evaluation”?
What combination of specializations in AI, moral philosophy, and cognitive psychology would an academic need to have, to be an “expert” whose disagreements about the material causes and implementation of moral evaluations were significant?
On the question of takeoff speeds, what fraction of the AI academics have a good enough intuitive understanding of decision theory to see that a point estimate or default scenario should not be substituted for a marginal posterior distribution, even in a situation where it would be socially costly in the default scenario to take actions which prevent large losses in one tail of the distribution?
What fraction recognized that they had a prior belief distribution over possible takeoff speeds at all?
What fraction understood that, regarding a variable which is underconstrained by evidence, “other people would disapprove of my belief distribution about this variable” is not an indicator for “my belief distribution about this variable puts mass in the wrong places”, except insofar as there is some causal reason to expect that disapproval would be somehow correlated with falsehood?
What other popular concerns have academics historically needed to dismiss? What decision rules have they learned to decide whether they need to dismiss a current popular concern?
After they make a decision to dismiss a popular concern, what kinds of causal explanations of the existence of that concern do they make reference to, when arguing to other people that they should agree with the decision?
How much do the true decision rules depend on those causal explanations?
How much do the decision rules depend on intuitions about the concerned peoples’ states of mind and social category memberships?
How much do the causal explanations use concepts which are implicitly defined by reference to hidden intuitions about states of mind and social category memberships?
Can these intuitively defined concepts carry the full weight of the causal explanations they are used to support, or does their power to cause agreement come from their ability to activate social intuitions?
Which people are the AI academics aware of, who have argued that intelligence explosion is a concern? What social categories do they intuit those people to be members of? What arguments are they aware of? What states of mind do they intuit those arguments to be indicators of (e.g. as in intuitively computed separating equilibria)?
What people and arguments did the AI academics think the other AI academics were thinking of? If only a few of the academics were thinking of people and arguments who they intuited to come from credible social categories and rational states of mind, would they have been able to communicate this to the others?
When the AI academics made the decision to dismiss concern about an intelligence explosion, what kinds of causal explanations of the existence of that concern did they intuitively expect that they would be able make reference to, if they later had to argue to other people that they should agree with the decision?
It is also possible to model the social process in the panel:
Are there factors that might make a joint statement by a panel of AI academics reflect different conclusions than they would have individually reached if they had been outsiders to the AI profession with the same AI expertise?
One salient consideration would be that agreeing with popular concern about an intelligence explosion would result in their funding being cut. What effects would this have had?
Would it have affected the order in which they became consciously aware of lines of argument that might make an intelligence explosion seem less or more deserving of concern?
Would it have made them associate concern about an intelligence explosion with unpopularity? In doubtful situations, unpopularity of an argument is one cue for its unjustifiability. Would they associate unpopularity with logical unjustifiability, and then lose willingness to support logically justifiable lines of argument that made an intelligence explosion seem deserving of concern, just as if they had felt those lines of argument to be logically unjustifiable, but without any actual unjustifiability?
There are social norms to justify taking prestige away from people who push a claim that an argument is justifiable while knowing that other prestigious people think the argument to to be a marker of a non-credible social category or state of mind. How would this have affected the discussion?
If there were panelists who personally thought the intelligence explosion argument was plausible, and they were in the minority, would the authors of the panel’s report mention it?
Would the authors know about it?
If the authors knew about it, would they feel any justification or need to mention those opinions in the report, given that the other panelists may have imposed on the authors an implicit social obligation to not write a report that would “unfairly” associate them with anything they think will cause them to lose prestige?
If panelists in such a minority knew that the report would not mention their opinions, would they feel any need or justification to object, given the existence of that same implicit social obligation?
How good are groups of people at making judgments about arguments that unprecedented things will have grave consequences?
How common is a reflective, causal understanding of the intuitions people use when judging popular concerns and arguments about unprecedented things, of the sort that would be needed to compute conditional probabilities like “Pr( we would decide that concern is not justified | we made our decision according to intuition X ∧ concern was justified )”?
How common is the ability to communicate the epistemic implications of that understanding in real-time while a discussion is happening, to keep it from going wrong?
Participants reviewed prior writings and thinking about the possibility of an “intelligence explosion” where computers one day begin designing computers that are more intelligent than themselves. … There was overall skepticism about the prospect of an intelligence explosion as well as of a “coming singularity,” and also about the large-scale loss of control of intelligent systems.
Given this description it is hard to imagine they haven’t imagined the prospect of the rate of intelligence growth depending on the level of system intelligence.
I don’t see any arguments listed, though. I know there’s at least some smart people on that panel (e.g. Horvitz) so I could be wrong, but experience has taught me to be pessimistic, and pessimism says I have no particular evidence that anyone started breaking the problem down into modular pieces, as opposed to, say, stating a few snap perceptual judgments at each other and then moving on.
Why are you so optimistic about this sort of thing, Robin? You’re usually more cynical about what would happen when academics have no status incentive to get it right and every status incentive to dismiss the silly. We both have experience with novices encountering these problems and running straight into the brick wall of policy proposals without even trying a modular analysis. Why on this one occasion do you turn around and suppose that the case we don’t know will be so unlike the cases we do know?
The point is that this is a subtle and central issue to engage, so I was suggesting that you to consider describing your analysis more explicitly. Is there is never any point in listening to academics on “silly” topics? Is there never any point in listening to academics who haven’t explicitly told you how they’ve broken a problem down into modular parts, no matter now distinguished the are on related topics? Are people who have a modular parts analysis always a more reliable source than people who don’t, no matter what else their other features? And so on.
I confess, it doesn’t seem to me on a gut level like this is either healthy to obsess about, or productive to obsess about. It seems more like worrying that my status isn’t high enough to do work, than actually working. If someone shows up with amazing analyses I haven’t considered, I can just listen to the analyses then. Why spend time trying to guess who might have a hidden deep analysis I haven’t seen, when the prior is so much in favor of them having made a snap judgment, and it’s not clear why if they’ve got a deep analysis they wouldn’t just present it?
I think that on a purely pragmatic level there’s a lot to be said for the Traditional Rationalist concept of demanding that Authority show its work, even if it doesn’t seem like what ideal Bayesians would do.
You have in the past thought my research on the rationality of disagreement to be interesting and spent a fair bit of time discussing it. It seemed healthy to me for you to compare your far view of disagreement in the abstract to the near view of your own particular disagreement. If it makes sense in general for rational agents to take very seriously the fact that others disagree, why does it make little sense for you in particular to do so?
t seemed healthy to me for you to compare your far view of disagreement in the abstract to the near view of your own particular disagreement.
I was under the impression that my verbal analysis matched and cleverly excused my concrete behavior.
If it makes sense in general for rational agents to take very seriously the fact that others disagree, why does it make little sense for you in particular to do so?
Well (and I’m pretty sure this matches what I’ve been saying to you over the last few years) just because two ideal Bayesians would do something naturally, doesn’t mean you can singlehandedly come closer to Bayesianism by imitating the surface behavior of agreement. I’m not sure that doing elaborate analyses to excuse your disagreement helps much either. http://wiki.lesswrong.com/wiki/Occam%27s_Imaginary_Razor
I’d spend much more time worrying about the implications of Aumann agreement, if I thought the other party actually knew my arguments, took my arguments very seriously, took the Aumann problem seriously with respect to me in particular, and in general had a sense of immense gravitas about the possible consequences of abusing their power to make me update. This begins to approach the conditions for actually doing what ideal Bayesians do. Michael Vassar and I have practiced Aumann agreement a bit; I’ve never literally done the probability exchange-and-update thing with anyone else. (Edit: Actually on recollection I played this game a couple of times at a Less Wrong meetup.)
No such condition is remotely approached by disagreeing with the AAAI panel, so I don’t think I could, in real life, improve my epistemic position by pretending that they were ideal Bayesians who were fully informed about my reasons and yet disagreed with me anyway (in which case I ought to just update to match their estimates, rather than coming up with elaborate excuses to disagree with them!)
Well I disagree with you strongly that there is no point in considering the views of others if you are not sure they know the details of your arguments, or of the disagreement literature, or that those others are “rational.” Guess I should elaborate my view in a separate post.
There’s certainly always a point in considering specific arguments. But to be nervous merely that someone else has a different view, one ought, generally speaking, to suspect (a) that they know something you do not or at least (b) that you know no more than them (or far more rarely (c) that you are in a situation of mutual Aumann awareness and equal mutual respect for one another’s meta-rationality). As far as I’m concerned, these are eminent scientists from outside the field that I work in, and I have no evidence that they did anything more than snap judgment of my own subject material. It’s not that I have specific reason to distrust these people—the main name I recognize is Horvitz and a fine name it is. But the prior probabilities are not good here.
I don’t actually spend time obsessing about that sort of thing except when you’re asking me those sorts of questions—putting so much energy into self-justification and excuses would just slow me down if Horvitz showed up tomorrow with an argument I hadn’t considered.
I’ll say again: I think there’s much to be said for the Traditional Rationalist ideal of—once you’re at least inside a science and have enough expertise to evaluate the arguments—paying attention only when people lay out their arguments on the table, rather than trying to guess authority (or arguing over who’s most meta-rational). That’s not saying “there’s no point in considering the views of others”. It’s focusing your energy on the object level, where your thought time is most likely to be productive.
Is it that awful to say: “Show me your reasons”? Think of the prior probabilities!
You admit you have not done much to make it easy to show them your reasons. You have not written up your key arguments in a compact form using standard style and terminology and submitted it to standard journals. You also admit you have not contacted any of them to ask them for their reasons; Horvitz would have have to “show up” for you to listen to him. This looks a lot like a status pissing contest; the obvious interpretation: Since you think you are better than them, you won’t ask them for their reasons, and you won’t make it easy for them to understand your reasons, as that would admit they are higher status. They will instead have to acknowledge your higher status by coming to you and doing things your way. And of course they won’t since by ordinary standard they have higher status. So you ensure there will be no conversation, and with no conversation you can invoke your “traditional” (non-Bayesian) rationality standard to declare you have no need to consider their opinions.
You’re being slightly silly. I simply don’t expect them to pay any attention to me one way or another. As it stands, if e.g. Horvitz showed up and asked questions, I’d immediately direct him to http://singinst.org/AIRisk.pdf (the chapter I did for Bostrom), and then take out whatever time was needed to collect the OB/LW posts in our discussion into a sequence with summaries. Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.
FYI, I’ve talked with Peter Norvig a bit. He was mostly interested in the CEV / FAI-spec part of the problem—I don’t think we discussed hard takeoffs much per se. I certainly wouldn’t have brushed him off if he’d started asking!
“and then take out whatever time was needed to collect the OB/LW posts in our discussion into a sequence with summaries.”
Why? No one in the academic community would spend that much time reading all that blog material for answers that would be best given in a concise form in a published academic paper. So why not spend the time? Unless you think you are that much of an expert in the field as to not need the academic community. If that be the case where are your publications and where are your credentials, where is the proof of this expertise (expert being a term that is applied based on actual knowledge and accomplishments)?
“Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.”
Why? If you expect to make FAI you will undoubtedly need people in the academic communities’ help; unless you plan to do this whole project by yourself or with purely amateur help. I think you would admit that in its current form SIAI has a 0 probability of creating FAI first. That being said your best hope is to convince others that the cause is worthwhile and if that be the case you are looking at the professional and academic AI community.
I am sorry I prefer to be blunt.. that way there is no mistaking meanings...
Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.
Why? If you expect to make FAI you will undoubtedly need people in the academic communities’ help; unless you plan to do this whole project by yourself or with purely amateur help. …
That ‘probably not even then’ part is significant.
That being said your best hope is to convince others that the cause is worthwhile and if that be the case you are looking at the professional and academic AI community.
Now that is an interesting question. To what extent would Eliezer say that conclusion followed? Certainly less than the implied ‘1’ and probably more than ‘0’ too.
“Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.
Why? If you expect to make FAI you will undoubtedly need people in the academic communities’ help; unless you plan to do this whole project by yourself or with purely amateur help. …”
“That ‘probably not even then’ part is significant.”
My implication was that the idea that he can create FAI completely outside the academic or professional world is ridiculous when you’re speaking from an organization like SIAI which does not have the people or money to get the job done. In fact SIAI doesn’t have enough money to pay for the computing hardware to make human level AI.
“Now that is an interesting question. To what extent would Eliezer say that conclusion followed? Certainly less than the implied ‘1’ and probably more than ‘0’ too.”
If he doesn’t agree with it now, I am sure he will when he runs into the problem of not having the money to build his AI or not having enough time in the day to solve the problems that will be associated with constructing the AI. Not even mentioning the fact that when you close yourself to outside influence that much you often end up with ideas that are riddled with problems, that if someone on the outside had looked at the idea they would have pointed the problems out.
If you have never taken an idea from idea to product this can be hard to understand.
No need to disclaim, your figures are sound enough and I took them as a demonstration of another rather significant difference between the assumptions of Eliezer and mormon2 (or mormon2′s sources).
If there is a status pissing contest, they started it! ;-)
“On the latter, some panelists believe that the AAAI study was held amidst a perception of urgency by non-experts (e.g., a book and a forthcoming movie titled “The Singularity is Near”), and focus of attention, expectation, and concern growing among the general population.”
Agree with them that there is much scaremongering going on in the field—but disagree with them about there not being much chance of an intelligence explosion.
I wondered why these folk got so much press. My guess is that the media probably thought the “AAAI Presidential Panel on Long-Term AI Futures” had something to do with the a report commisioned indirectly for the country’s president. In fact it just refers to the president of their organisation. A media-savvy move—though it probably represents deliberately misleading information.
Almost surely world class academic AI experts do “know something you do not” about the future possibilities of AI. To declare that topic to be your field and them to be “outside” it seems hubris of the first order.
This conversation seems to be following what appears to me to be a trend in Robin and Eliezer’s (observable by me) disagreements. This is one reason I would fascinated if Eliezer did cover Robin’s initial question, informed somewhat by Eliezer’sinterpretation.
I recall Eliezer mentioning in a tangential comment that he disagreed with Robin not just on the particular conclusion but more foundationally on how much weight should be given to certain types of evidence or argument. (Excuse my paraphrase from hazy memory, my googling failed me.) This is a difference that extends far beyond just R & E and Eliezer has hinted at insights that intrigue me.
How can you be so confident that you know so much about this topic that no world class AI expert could know something relevant that you do not? Surely they considered the fact that people like you think you a lot about this topic, and they nevertheless thought it reasonable to form a disagreeing opinion based on the attention they had given it. You want to dismiss their judgment as “snap” because they did not spend many hours considering your arguments, but they clearly disagree with that assessment of how much consideration your arguments deserve. Academic authorities are not always or even usually wrong when they disagree with less authoritative contrarians, even when such authorities do not review contrarian arguments in as much detail as contrarians think best. You want to dismiss the rationality of disagreement literature as irrelevant because you don’t think those you disagree with are rational, but they probably don’t think you are rational either, and you are probably both right. But the same essential logic also says that irrational people should take seriously the fact that other irrational people disagree with them.
Almost surely world class academic AI experts do “know something you do not” about the future possibilities of AI.
Does Daphne Koller know more than I do about the future possibilities of object-oriented Bayes Nets? Almost certainly. And, um… there are various complicated ways I could put this… but, well, so what?
(No disrespect intended to Koller, and OOBN/probabilistic relational models/lifted Bayes/etcetera is on my short-list of things to study next.)
How can you be so confident that you know so much about this topic that no world class AI expert could possibly know something relevant that you do not? Surely they considered the fact that people like you think you know a lot about this topic, and they nevertheless thought it reasonable to form a disagreeing opinion based on the attention they had given it. You want to dismiss their judgment as “snap” because they did not spend many hours considering your arguments, but they clearly disagree with that assessment of how much consideration your arguments deserve. Academic authorities are not always or even usually wrong when they disagree with less authoritative contrarians, even when such authorities do not review contrarian arguments in as much detail as contrarians think best. You want to dismiss the rationality of disagreement literature as irrelevant because you don’t think those you disagree with are rational, but they probably don’t think you are rational either, and you are probably both right. But the same essential logic also says that irrational people should take seriously the fact that other irrational people disagree with them.
How can you be so confident that you know so much about this topic that no world class AI expert could possibly know something relevant that you do not?
You changed what I said into a bizarre absolute. I am assuming no such thing. I am just assuming that, by default, world class experts on various topics in narrow AI, produce their beliefs about the Singularity by snap judgment rather than detailed modular analysis. This is a prior and hence an unstable probability—as soon as I see contrary evidence, as soon as I see the actual analysis, it gets revoked.
but they clearly disagree with that assessment of how much consideration your arguments deserve.
They have no such disagreement. They have no idea I exist. On the rare occasion when I encounter such a person who is physically aware of my existence, we often manage to have interesting though brief conversations despite their having read none of my stuff.
Academic authorities are not always or even usually wrong when they disagree with less authoritative contrarians
Science only works when you use it; scientific authority derives from science. If you’ve got Lord Kelvin running around saying that you can’t have flying machines because it’s ridiculous, the problem isn’t that he’s an Authority, the problem is that he’s running on naked snap intuitive judgments of absurdity and the Wright Brothers are using actual math. The asymmetry in this case is not that pronounced but, even so, the default unstable prior is to assume that experts in narrow AI algorithms are not doing anything more complicated than this to produce their judgments about the probability of intelligence explosion—both the ones with negative affect who say “Never, you religious fear-monger!” and the ones with positive affect who say “Yes! Soon! And they shall do no wrong!” As soon as I see actual analysis, then we can talk about the actual analysis!
Added: In this field, what happens by default is that people talk complete nonsense. I spent my first years talking complete nonsense. In a situation like that, everyone has to show their work! Or at least show that they did some work! No exceptions!
This conversation is probably reaching diminishing returns, so let me sum up. I propose that it would be instructive to you and many others if you would discuss what your dispute looks like from an outside view—what uninformed neutral but intelligent and rational observers should conclude about this topic from the features of this dispute they can observe from the outside. Such features include the various credentials of each party, and the effort he or she has spent on the topic and on engaging the other parties. If you think that a reasonable outsider viewer would have far less confidence in your conclusions than you do, then you must think that you possess hidden info, such as that your arguments are in fact far more persuasive than one could reasonably expect knowing only the outside features of the situation. Then you might ask why the usual sorts of clues that tend to leak out about argument persuasiveness have failed to do so in this case.
Robin, why do most academic experts (e.g. in biology) disagree with you (and Eliezer) about cryonics? Perhaps a few have detailed theories on why it’s hopeless, or simply have higher priorities than maximizing their expected survival time; but mostly it seems they’ve simply never given it much consideration, either because they’re entirely unaware of it or assume it’s some kind of sci-fi cult practice, and they don’t take cult practices seriously as a rule. But clearly people in this situation can be wrong, as you yourself believe in this instance.
Similarly, I think most of the apparent “disagreement” about the Singularity is nothing more than unawareness of Yudkowsky and his arguments. As far as I can tell, academics who come into contact with him tend to take him seriously, and their disagreements are limited to matters of detail, such as how fast AI is approaching (decades vs. centuries) and the exact form it will take (uploads/enhancement vs. de novo). They mainly agree that SIAI’s work is worth doing by somebody. Examples include yourself, Scott Aaronson, and David Chalmers.
Cryonics is also a good case to analyze what an outsider should think, given what they can see. But of course “they laughed at Galileo too” is hardly a strong argument for contrarian views. Yes sometimes contrarians are right—the key question is how outside observers, or self-doubting insiders, can tell when contrarians are right.
Outsiders can tell when contrarians are right by assessing their arguments, once they’ve decided the contrarians are worth listening to. This in turn can be ascertained through the usual means, such as association with credentialed or otherwise high-status folks. So for instance, you are affiliated with a respectable institution, Bostrom with an even more respectable institution, and the fact that EY was co-blogging at Overcoming Bias thus implied that if your and Bostrom’s arguments were worth listening to, so were his. (This is more or less my own story; and I started reading Overcoming Bias because it appeared on Scott Aaronson’s blogroll.)
Hence it seems that Yudkowsky’s affiliations are already strong enough to signal competence to those academics interested in the subjects he deals with, in which case we should expect to see detailed, inside-view analyses from insiders who disagree. In the absence of that, we have to conclude that insiders either agree or are simply unaware—and the latter, if I understand correctly, is a problem whose solution falls more under the responsibility of people like Vassar rather than Yudkowsky.
No for most people it is infeasible to evaluate who is right by working through the details of the arguments. The fact that Eliezer wrote on a blog affiliated with Oxford is very surely not enough to lead one to expect detailed rebuttal analyses from academics who disagree with him.
Well, for most people on most topics it is infeasible to evaluate who is right, period. At the end of the day, some effort is usually required to obtain reliable information. Even surveys of expert opinion may be difficult to conduct if the field is narrow and non-”traditional”. As for whatever few specialists there may be in Singularity issues, I think you expect too little of them if you don’t think Eliezer currently has enough status to expect rebuttals.
So, despite the fact that we (human phenotypes) are endowed with a powerful self-preservation instinct, you find a signaling explanation more likely than a straightforward application of self-preservation to a person’s concept of their own mind?
Given your peculiar preferences which value your DNA more highly than your brain, it’s tempting to chalk your absurd hypothesis up to the typical mind fallacy. But I think you’re well aware of the difference in values responsible for the split between your assessment of cryonics and Eliezer’s or Robin’s.
So I think you’re value sniping. I think your comment was made in bad faith as a roundabout way of signaling your values in a context where explicitly mentioning them would be seen as inappropriate or off-topic. I don’t know what your motivation would be—did mention of cryonics remind you that many here do not share your values, and thereby motivate you to plant your flag in the discussion?
Please feel free to provide evidence to the contrary by explaining in more detail why self-preservation is an unlikely motivation for cryonics relative to signaling.
An over-generalisation of self-preservation instincts certainly seems to be part of it.
On the other hand, one of my interests is in the spread of ideas. Without cryonic medalions, cryonic bracelets, cryonic advertising and cryonic preachers there wouldn’t be any cryonics movement. There seems to be a “show your friends how much you care—freeze them!” dynamic.
I have a similar theory about the pyramids. Not so much a real voyage to the afterlife, but a means of reinforcing the pecking order in everyone’s minds.
I am contrasting this signaling perspective with Robin’s views—in part because I am aware that he is sympathetic to signaling theories in other contexts.
I do think signaling is an important part of cryonics—but I was probably rash to attempt to quantify the effect. I don’t pretend to have any good way of measuring its overall contribution relative to other factors.
Say that “Yudkowsky has no real clue” and that those “AI academics are right”? Just another crackpot among many “well educated”, no big thing. Not worth to mention, almost.
Say, that this crackpot is of the Edisonian kind! In that case it is something well worth to mention.
Important enough to at least discuss with him ON THE TOPICS, and not on some meta level. Meta level discussion is sometimes (as here IMHO), just a waste of time.
IFF he is right. Probably he is and nothing dramatically will happen. Probably Edison and Wright brothers and many others were also wrong, looking from their historic perspective.
Note, that if the official Academia (Hanson’s guys) is correct, the amount of new information is exactly zero. Nothing interesting to talk about or expect to.
I am after the cases they were and are wrong. I am after a new context, misfits like Yudkowsky or Edison might provide and “The Hanson’s” can’t. By the definition.
P.S. I don’t want to get into a discussion; I believe it’s better to just state a judgment even if without a useful explanation than to not state a judgment at all; however it may be perceived negatively for those obscure status-related reasons (see “offense” on the wiki), so I predict that this comment would’ve been downvoted without this addendum, and not impossibly still will be with it. This “P.S.” is dedicated to all the relevant occasions, not this one alone where I could’ve used the time to actually address the topic.
If I’m reading the conversation correctly, Vladimir Nesov is indicating with his remark that he is no longer interested in continuing. If he were not a major participant in the thread, a downvote would be appropriate, but as a major participant, more is required of him.
I am not confused and I don’t want a discussion either. I only state, that a new content and a new context usually comes out from outside the kosher set of views.
Of course, most of the outsiders are delusive poor devils. Yet, they are almost the only source of new information.
“The group suggested outreach and communication to people and organizations about the low likelihood of the radical outcomes”.
“Radical outcomes” seems like a case of avoiding refutation by being vague. However, IMO, they will need to establish the truth of their assertion before they will get very far there. Good luck to them with that.
The AAAI interim report is really too vague to bother much with—but I suspect they are making another error.
Many robot enthusiasts pour scorn on the idea that robots will take over the world. How To Survive A Robot Uprising is a classic presentation on this theme. A hostile takeover is a pretty unrealistic scenario—but these folk often ignore the possibility of a rapid robot rise from within society driven by mutual love. One day robots will be smart, sexy, powerful and cool—and then we will want to become more like them.
Why will we witness an intelligence explosion? Because nature has a long history of favouring big creatures with brains—and because the capability to satisfy those selection pressures has finally arrived.
The process has already resulted in enormous data-centres, the size of factories. As I have said:
Thinking about it, they are probably criticising the (genuinely dud) idea that an intelligence explosion will start suddenly at some future point with the invention of some machine—rather than gradually arising out of the growth of today’s already self-improving economies and industries.
I think, both ways are still open. The intelligence explosion from a self-improving economy and the intelligence explosion from a fringe of this process.
Re: “overall skepticism about the prospect of an intelligence explosion”...?
My guess would be that they are unfamiliar with the issues or haven’t thought things through very much. Or maybe they don’t have a good understanding of what that concept refers to (see link to my explanation—hopefully above). They present no useful analysis of the point—so it is hard to know why they think what they think.
The AAAI seem to have publicly come to these issues later than many in the community—and it seems to be playing catch-up.
It must be possible to engage at least some of these people in some sort of conversation to understand their positions, whether a public dialog as with Scott Aaronson or in private.
I have a theory about why there is disagreement with the AAAI panel:
The DOOM peddlers gather funding from hapless innocents—who hope to SAVE THE WORLD—while the academics see them as bringing their field into disrepute, by unjustifiably linking their field to existential risk, with their irresponsible scaremongering about THE END OF THE WORLD AS WE KNOW IT.
Naturally, the academics sense a threat to their funding—and so write papers to reassure the public that spending money on this stuff is Really Not As Bad As All That.
Naturally, the academics sense a threat to their funding—and so write papers to reassure the public that spending money on this stuff is really not as bad as all that.
Actually, on further recollection, Steve Omohundro and Peter Cheeseman would probably count as academics who know the arguments. Mostly I’ve talked to them about FAI stuff, so I’m actually having trouble recalling whether they have any particular disagreement with me about hard takeoff.
I think that w/r/t Cheeseman, I had to talk to Cheeseman for a while before he started to appreciate the potential speed of a FOOM, as opposed to just the FOOM itself which he considered obvious. I think I tried to describe your position to Cheeseman and Cheeseman thought it was pretty implausible, but of course that could just be the fact that I was describing it from outside—that counts for nothing in my view until you talk to Cheeseman, otherwise he’s not familiar enough with your arguments. (See, the part about setting the bar high works both ways—I can be just as fast to write off the fact of someone else’s disagreement with you, if they’re insufficiently familiar with your arguments.)
I’m not sure I can recall what Omohundro thinks—he might be intermediate between yourself and myself...? I’m not sure how much I’ve talked hard takeoff per se with Omohundro, but he’s certainly in the game.
I think Steve Omohundro disagees about the degree to which takeoff is likely to be centralized, due to what I think is the libertarian impulses I mentioned earlier.
Does Robin Hanson fall under ‘et. al’? I remember on OB that he attempted to link the 2 fields on at least 1 or 2 occasions, and those were some of the most striking examples of disagreements between you two.
Do you disagree with Eliezer substantively? If so, can you summarize how much of his arguments you’ve analyzed, and where you reach different conclusions?
Yes—I disagree with Eliezer and have analyzed a fair bit of his writings although the style in which it is presented and collected here is not exactly conducive to that effort. Feel free to search for my blog for a detailed analysis and a summary of core similarities and differences in our premises and conclusions.
B) It is a basic AI drive to avoid counterfeit utility
If A = true (as we have every reason to believe) and B = true (see Omohundro’s paper for details) then a transhuman AI would dismiss any utility function that contradicts A on the ground that it is recognized as counterfeit utility.
This quotation accurately summarizes the post as I understand it. (It’s a short post.)
I think I speak for many people when I say that assumption A requires some evidence. It may be perfectly obvious, but a lot of perfectly obvious things aren’t true, and it is only reasonable to ask for some justification.
Compassion isn’t even universal in the human mind-space. It’s not even universal in the much smaller space of human minds that normal humans consider comprehensible. It’s definitely not universal across mind-space in general.
The probable source of the confusion is discussed in the comments—Stefan’s only talking about minds that’ve been subjected to the kind of evolutionary pressure that tends to produce compassion. He even says himself, “The argument is valid in a “soft takeoff” scenario, where there is a large pool of AIs interacting over an extended period of time. In a “hard takeoff” scenario, where few or only one AI establishes control in a rapid period of time, the dynamics described do not come into play. In that scenario, we simply get a paperclip maximizer.”
Ah—that’s interesting. I hadn’t read the comments. That changes the picture, but by making the result somewhat less relevant.
(Incidentally, when I said, “it may be perfectly obvious”, I meant that “some people, observing the statement, may evaluate it as true without performing any complex analysis”.)
(Incidentally, when I said, “it may be perfectly obvious”, I meant that “some people, observing the statement, may evaluate it as true without performing any complex analysis”.)
It’s my descriptivist side playing up—my (I must admit) intuition is that when people say that some thesis is “obvious”, they mean that they reached this bottom line by … well, system 1 thinking. I don’t assume it means that the obvious thesis is actually correct, or even universally obvious. (For example, it’s obvious to me that human beings are evolved, but that’s because it’s a cached thought I have confidence in through system 2 thinking.)
I generally take ‘obvious’ to mean ‘follows from readily-available evidence or intuition, with little to no readily available evidence to contradict the idea’. The idea that compassion is universal fails on the second part of that. The definitions are close in practice, though, in that most peoples’ intuitions tend to take readily available contradictions into account… I think.
ETA: Oh, and ‘obviously false’ seems to me to be a bit of a different concept, or at least differently relevant, given that it’s easier to disprove something than to prove it. If someone says that something is obviously true, there’s room for non-obvious proofs that it’s not, but if something is obviously false (as ‘compassion is universal’ is), that’s generally a firm conclusion.
Yes, that makes sense—even if mine is a better description of usage, from the standpoint of someone categorizing beliefs, I imagine yours would be the better metric.
ETA: I’m not sure the caveat is required for “obviously false”, for two reasons.
Any substantive thesis (a category which includes most theses that are rejected as obviously false) requires less evidence to be roundly disconfirmed than it does to be confirmed.
As Yvain demonstrated in Talking Snakes, well-confirmed theories can be “obviously false”, by either of our definitions.
It’s true that it usually takes less effort to disabuse someone of an obviously-true falsity than to convince them of an obviously-false truth, but I don’t think you need a special theory to support that pattern.
I’ve been thinking about the obviously true/obviously false distinction some more, and I think I’ve figured out why they feel like two different concepts.
‘Obviously’, as I use it, is very close to ‘observably’. It’s obviously true that the sky is blue where I am right now, and obviously false that it’s orange, because I can see it. It’s obviously true that the sky is usually either blue, white, or grey during the day (post-sunrise, pre-sunset), because I’ve observed the sky many times during the day and seen those colors, and no others.
‘Apparently’, as I use it, is very similar to ‘obviously’, but refers to information inferred from observed facts. The sky is apparently never orange during the day, because I’ve personally observed the sky many times during the day and never seen it be that color. I understand that it can also be inferred from certain facts about the world (composition of the atmosphere and certain facts about how light behaves, I believe) that the sky will always appear blue on cloudless days, so that’s also apparently true.
‘Obviously false’ covers situations where the theory makes a prediction that is observably inaccurate, as this one did. ‘Apparently false’ covers situations where the theory makes a prediction that appears to be inaccurate given all the available information, but some of the information that’s available is questionable (I consider inferences questionable by default—if nothing else, it’s possible for some relevant state to have been overlooked; what if the composition of the atmosphere were to change for some reason?) or otherwise doesn’t completely rule out the possibility that the theory is true.
Important caveat: I do use those words interchangeably in conversation, partly because of the convention of avoiding repeating words too frequently and partly because it’s just easier—if I were to try to be that accurate every time I communicated, I’d run out of spoons(pdf) and not be able to communicate at all. Also, having to parse someone else’s words, when they aren’t using the terms the same way I do, can lead to temporary confusion. But when I’m thinking, they are naturally separate.
Yes, that makes sense—even if mine is a better description of usage, from the standpoint of someone categorizing beliefs, I imagine yours would be the better metric.
It also has the advantage of making it clear that the chance that the statement is accurate is dependent on the competence of the person making the statement—people who are more intelligent and/or have more experience in the relevant domain will consider more, and more accurate, evidence to be readily available, and may have better intuitions, even if they are sticking to system 1 thought.
ETA: I’m not sure the caveat is required for “obviously false”, for two reasons.
I suppose they don’t need different wordings, but they do feel like different concepts to me. *shrug* (As I’ve mentioned elsewhere, I don’t think in words. This is not an uncommon side-effect of that.)
From Robin: Incidentally, when I said, “it may be perfectly obvious”, I meant that “some people, observing the statement, may evaluate it as true without performing any complex analysis”.
I feel the other way around at the moment. Namely “some people, observing the statement, may evaluate it as false without performing any complex analysis”
“Compassion isn’t even universal in the human mind-space. It’s not even universal in the much smaller space of human minds that normal humans consider comprehensible. It’s definitely not universal across mind-space in general.”
Your argument is beside my original point, Adelene. My claim is that compassion is a universal rational moral value. Meaning any sufficiently rational mind will recognize it as such. The fact that not every human is in fact compassionate says more about their rationality (and of course their unwillingness to consider the arguments :-) ) than about that claim. That’s why it is call ASPD—the D standing for ‘disorder’, it is an aberration, not helpful, not ‘fit’. Surely the fact that some humans are born blind does not invalidate the fact that seeing people have an enormous advantage over the blind. Compassion certainly being less obvious though—that is for sure.
Re “The argument is valid in a “soft takeoff”scenario, where few or only one AI establishes control in a rapid period of time, the dynamics described do not come into play. In that scenario, we simply get a paperclip maximizer.”—that is from Kaj Sotala over at her live journal—not me.
Meaning any sufficiently rational mind will recognize it as such. The fact that not every human is in fact compassionate says more about their rationality (and of course their unwillingness to consider the arguments :-) ) than about that claim. That’s why it is call ASPD—the D standing for ‘disorder’, it is an aberration, not helpful, not ‘fit’.
APSD is only unfit in our current context. Would Stone Age psychiatrists have recognized it as an issue? Or as a positive trait good for warring against other tribes and climbing the totem pole? In other situations, compassion is merely an extra expense. (As Thrasymachus asked thousands of years ago: how can a just man do better than an injust man, when the injust man can act justly when it is optimal and injustly when that is optimal?)
Why would a recursively-improving AI which is single-mindedly pursuing an optimization goal permit other AIs to exist & threaten it? There is nothing they can offer it that it couldn’t do itself. This is true in both slow and fast takeoffs; cooperation only makes sense if there is a low ceiling for AI capability so that there are utility-maximizing projects beyond an AI’s ability to do alone then or in the future.
And ‘sufficiently rational’ is dangerous to throw around. It’s a fully general argument: ‘any sufficiently rational mind will recognize that Islam is the one true religion; that not every human is Muslim says more about their rationality than about the claims is Islam. That’s why our Muslim psychiatrists call it UD—Unbeliever Disorder, it is an aberration, not helpful, not ‘fit’. Surely the fact that some human are born kafir doesn’t invalidate the fact that Muslim people have a tremendous advantage over the kafir in the afterlife? ‘There is one God and Muhammed is his prophet’ is certainly less obvious than seeing being better superior to blindness, though.′
The longer I stay around here the more I get the feeling that people vote comments down purely because they don’t understand them not because they found a logical or factual error. I expect more from a site dedicated to rationality. This site is called ‘less wrong’, not ‘less understood’, ‘less believed’ or ‘less conform’.
Tell me: in what way do you feel that Adelene’s comment invalidated my claim?
the more I get the feeling that people vote comments down purely because they don’t understand them not because they found a logical or factual error
I can see why it would seem this way to you, but from our perspective, it just looks like people around here tend to have background knowledge that you don’t. More specifically: most people here are moral anti-realists, and by rationality we only mean general methods for acquiring accurate world-models and achieving goals. When people with that kind of background are quick to reject claims like “Compassion is a universal moral value,” it might superficially seem like they’re being arbitrarily dismissive of unfamiliar claims, but we actually think we have strong reasons to rule out such claims. That is: the universe at its most basic level is described by physics, which makes no mention of morality, and it seems like our own moral sensibilities can be entirely explained by contingent evolutionary and cultural forces; therefore, claims about a universal morality are almost certainly false. There might be some sort of game-theoretic reason for agents to pursue the same strategy under some specific conditions—but that’s really not the same thing as a universal moral value.
“Universal values” presumably refers to values the universe will converge on, once living systems have engulfed most of it.
If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.
If rerunning the clock produces highly similar moralities, then the moral objectivists will be able to declare victory.
Gould would no-doubt favour the first position—while Conway Morris would be on the side of the objectivists.
I expect that there’s a lot of truth on the objectivist side—though perhaps contingency plays some non-trivial role.
The idea that physics makes no mention of morality seems totally and utterly irrelevant to me. Physics makes no mention of convection, diffusion-limited aggregation, or fractal drainage patterns either—yet those things are all universal.
If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.
If rerunning the clock produces highly similar moralities, then the moral objectivists will be able to declare victory.
Why should we care about this mere physical fact of which you speak? What has this mere “is” to do with whether “should” is “objective”, whatever that last word means (and why should we care about that?)
Hi, Eli! I’m not sure I can answer directly—here’s my closest shot:
If there’s a kind of universal moral attractor, then the chances seem pretty good that either our civilisation is on route for it—or else we will be obliterated or assimilated by aliens or other agents as they home in on it.
If it’s us who are on route for it, then we (or at least our descendants) will probably be sympathetic to the ideas it represents—since they will be evolved from our own moral systems.
If we get obliterated at the hands of some other agents, then there may not necessarily be much of a link between our values and the ones represented by the universal moral attractor.
Our values might be seen as OK by the rest of the universe—and we fail for other reasons.
Or our morals might not be favoured by the universe—we could be a kind of early negative moral mutation—in which case we would fail because our moral values would prevent us from being successful.
Maybe it turns out that nearly all biological organisms except us prefer to be orgasmium—to bliss out on pure positive reinforcement, as much of it as possible, caretaken by external AIs, until the end. Let this be a fact in some inconvenient possible world. Why does this fact say anything about morality in that inconvenient possible world? Why is it a universal moral attractor? Why not just call it a sad but true attractor in the evolutionary psychology of most aliens?
It’s a fact about morality in that world—if we are talking about morality as values—or the study of values—since that’s what a whole bunch of creatures value.
Why is it a universal moral attractor? I don’t know—this is your hypothetical world, and you haven’t told me enough about it to answer questions like that.
Tim: “If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.”
Actually compassion evolved many different times as a central doctrine of all major spiritual traditions. See the charter for compassion. This is in line with my prediction that I made independently and being unaware of this fact until I started looking for it back in late 2007 and eventually finding the link in late 2008 with Karen Armstrong’s book The Great Transformation.
Tim: “Why is it a universal moral attractor?”
Eliezer: “What do you mean by “morality”?”
Central point in my thinking: that is good which increases fitness. If it is not good—not fit—it is unfit for existence. Assuming this to be true we are very much limited in our freedom by what we can do without going extinct (actually my most recent blog post is about exactly that: Freedom in the evolving universe).
“Let us think about the results of following different ethical teachings in the evolving universe. It is evident that these results depend mainly on how the goals advanced by the teaching correlate with the basic law of evolution. The basic law or plan of evolution, like all laws of nature, is probabilistic. It does not prescribe anything unequivocally, but it does prohibit some things. No one can act against the laws of nature. Thus, ethical teachings which contradict the plan of evolution, that is to say which pose goals that are incompatible or even simply alien to it, cannot lead their followers to a positive contribution to evolution, which means that they obstruct it and will be erased from the memory of the world. Such is the immanent characteristic of development: what corresponds to its plan is eternalized in the structures which follow in time while what contradicts the plan is overcome and perishes.”
Eliezer: “It obviously has nothing to do with the function I try to compute to figure out what I should be doing.”
Once you realize the implications of Turchin’s statement above it has everything to do with it :-)
Now some may say that evolution is absolutely random and direction less, or that multilevel selection is flawed or similar claims. But reevaluating the evidence against both these claims by people like Valentin Turchin, Teilhard De Chardin, John Stewart, Stuart Kaufmann, John Smart and many others regarding evolution’s direction and the ideas of David Sloan Wilson regarding multilevel selection, one will have a hard time maintaining either position.
Actually compassion evolved many different times as a central doctrine of all major spiritual traditions.
No, it evolved once, as part of mammalian biology. Show me a non-mammal intelligence that evolved compassion, and I’ll take that argument more seriously.
Also, why should we give a damn about “evolution” wants, when we can, in principle anyway, form a singleton and end evolution? Evolution is mindless. It doesn’t have a plan. It doesn’t have a purpose. It’s just what happens under certain conditions. If all life on Earth was destroyed by runaway self-replicating nanobots, then the nanobots would clearly be “fitter” than what they replaced, but I don’t see what that has to do with goodness.
No, it evolved once, as part of mammalian biology.
Sorry Crono, with a sample size of exactly one in regards to human level rationality you are setting the bar a little bit too high for me. However, considering how disconnected Zoroaster, Buddha, Lao Zi and Jesus where geographically and culturally I guess the evidence is as good as it gets for now.
Also, why should we give a damn about “evolution” wants, when we can, in principle anyway, form a singleton and end evolution?
The typical Bostromian reply again. There are plenty of other scholars who have an entirely different perspective on evolution than Bostrom. But beside that: you already do care, because if your (or your ancestors) violated the conditions of your existence (enjoying a particular type of food, a particular type of mate, feel pain when cut ect.) you would not even be here right now. I suggest you look up Dennet and his TED talk on Funny, Sexy Cute. Not everything about evolution is random: the mutation bit is, not that what happens to stick around though, since that has be meet the conditions of its existence.
What I am saying is very simple: being compassionate is one of these conditions of our existence and anyone failing to align itself will simply reduce its chances of making it—particularly in the very long run. I still have to finish my detailed response to Bostrom but you may want to read my writings on ‘rational spirituality’ and ‘freedom in the evolving universe’. Although you do not seem to assign a particularly high likelihood of gaining anything from doing that :-)
The typical Bostromian reply again. There are plenty of other scholars who have an entirely different perspective on evolution than Bostrom. But beside that:
“Besides that”? All you did was name a statement of a fairly obvious preference choice after one guy who happened to have it so that you could then drop it dismissively.
you already do care, because if your (or your ancestors) violated the conditions of your existence (enjoying a particular type of food, a particular type of mate, feel pain when cut ect.) you would not even be here right now.
No, he mightn’t care and I certainly don’t. I am glad I am here but I have no particular loyalty to evolution because of that. I know for sure that evolution feels no such loyalty to me and would discard both me and my species in time if it remained the dominant force of development.
I suggest you look up Dennet and his TED talk on Funny, Sexy Cute. Not everything about evolution is random: the mutation bit is, not that what happens to stick around though, since that has be meet the conditions of its existence.
CronDAS knows that. It’s obvious stuff for most in this audience. It just doesn’t mean what you think it means.
“Besides that”? All you did was name a statement of a fairly obvious preference choice after one guy who happened to have it so that you could then drop it dismissively.
Wedrifid, not sure what to tell you. Bostrom is but one voice and his evolutionary analysis is very much flawed—again: detailed critique upcoming.
No, he mightn’t care and I certainly don’t. I am glad I am here but I have no particular loyalty to evolution because of that. I know for sure that evolution feels no such loyalty to me and would discard both me and my species in time if it remained the dominant force of development.
Evolution is not the dominant force of development on the human level by a long shot, but it still very much draws the line in the sand in regards to what you can and can not do if you want to stick around in the long run. You don’t walk your 5′8″ of pink squishiness in front of a train for the exact same reason. And why don’t you? Because not doing that is a necessary condition for your continued existence. What other conditions are there? Maybe there are some that are less obvious then simply stopping to breath, failing to eat and avoiding hard, fast, shiny things? How about at the level of culture? Could it possibly be, that there are some ideas that are more conducive to the continued existence of their believers than others?
“It must not be forgotten that although a high standard of morality gives but a slight or no advantage to each individual man and his children over the other men of the same tribe, yet that an advancement in the standard of morality and in increase in the number of well-endowed men will certainly give an immense advantage to one tribe over another. There can be no doubt that a tribe including many members who, from possessing in a high degree the spirit of patriotism, fidelity, obedienhce, courage, and sympathy, were always ready to give aid to each other and to sacrifice themselves for the common good, would be victorious over other tribes; and this would be natural selection.” (Charles Darwin, The Descent of Man, p. 166)
How long do you think you can ignore evolutionary dynamics and get away with it before you have to get over your inertia and will be forced to align yourself to them by the laws of nature or perish? Just because you live in a time of extraordinary freedoms afforded to you by modern technology and are thus not aware that your ancestors walked a very particular path that brought you into existence certainly has nothing to do with the fact that they most certainly did. You do not believe that doing any random thing will get you what you want—so what leads you to believe that your existence does not depend on you making sure you stay within a comfortable margin of certainty in regards to being naturally selected? You are right in one thing: you are assured the benign indifference of the universe should you fail to wise up. I however would find that to be a terrible waste.
Please do not patronize me by trying to claim you know what I understand and don’t understand.
How long do you think you can ignore evolutionary dynamics and get away with it before you have to get over your inertia and will be forced to align yourself to them by the laws of nature or perish?
A literal answer was probably not what you were after but probably about 40 years, depending on when a general AI is created. After that it will not matter whether I conform my behaviour evolutionary dynamics as best I can or not. I will not be able to compete with a superintelligence no matter what I do. I’m just a glorified monkey. I can hold about 7 items in working memory, my processor is limited to the speed of neurons and my source code is not maintainable. My only plausible chance of survival is if someone manages to completely thwart evolutionary dynamics by creating a system that utterly dominates all competition and allows my survival because it happens to be programmed to do so.
Evolution created us. But it’ll also kill us unless we kill it first. Now is not the time to conform our values to the local minima of evolutionary competition. Our momentum has given us an unprecedented buffer of freedom for non-subsistence level work and we’ll either use that to ensure a desirable future or we will die.
Please do not patronize me by trying to claim you know what I understand and don’t understand.
I usually wouldn’t, I know it is annoying. In this case, however, my statement was intended as a rejection of your patronisation of CronDAS and I am quite comfortable with it as it stands.
A literal answer was probably not what you were after but probably about 40 years, depending on when a general AI is created.
Good one—but it reminds me about the religious fundies who see no reason to change anything about global warming because the rapture is just around the corner anyway :-)
Evolution created us. But it’ll also kill us unless we kill it first. Now is not the time to conform our values to the local minima of evolutionary competition. Our momentum has given us an unprecedented buffer of freedom for non-subsistence level work and we’ll either use that to ensure a desirable future or we will die.
Evolution is a force of nature so we won’t be able to ignore it forever, with or without AGI. I am not talking about local minima either—I want to get as close to the center of the optimal path as necessary to ensure having us around for a very long time with a very high likelihood.
I usually wouldn’t, I know it is annoying. In this case, however, my statement was intended as a rejection of your patronisation of CronDAS and I am quite comfortable with it as it stands.
Good one—but it reminds me about the religious fundies who see no reason to change anything about global warming because the rapture is just around the corner anyway :-)
Don’t forget the Y2K doomsday folks! ;)
Evolution is a force of nature so we won’t be able to ignore it forever, with or without AGI. I am not talking about local minima either—I want to get as close to the center of the optimal path as necessary to ensure having us around for a very long time with a very high likelihood.
Gravity is a force of nature too. It’s time to reach escape velocity before the planet is engulfed by a black hole.
Gravity is a force of nature too. It’s time to reach escape velocity before the planet is engulfed by a black hole.
Interesting analogy—it would be correct if we would call our alignment with evolutionary forces achieving escape velocity. What one is doing by resisting evolutionary pressures however is constant energy expenditure while failing to reach escape velocity. Like hovering a space shuttle at a constant altitude of 10 km: no matter how much energy you brig along, eventually the boosters will run out of fuel and the whole thing comes crushing down.
Interesting analogy—it would be correct if we would call our alignment with evolutionary forces achieving escape velocity.
I could almost agree with this so long as ‘obliterate any competitive threat then do whatever the hell we want including, as as desired, removing all need for death, reproduction and competition over resources’ is included in the scope of ‘alignment with evolutionary forces’.
The problem with pointing to the development of compassion in multiple human traditions is that all these are developed within human societies. Humans are humans the world over—that they should think similar ideas is not a stunning revelation. Much more interesting is the independent evolution of similar norms in other taxonomic orders, such as canines.
Robin, your suggestion—that compassion is not a universal rational moral value because although more rational beings (humans) display such traits yet less rational being (dogs) do not—is so far of the mark that it borders on the random.
For purposes of this conversation, I suppose I should reword my comment as:
I don’t think you’ve made the strongest possible case for your thesis, if you were intending to show the multiple origin of compassion as a sign of the universality of human morality. Showing that multiple humans come up with similar morality only shows that it’s human. More telling is the independent origin of recognizably morality-like patterns of behavior in other species, such as dogs and wolves, and such as (I believe) some birds. (Other primates as well, but that is less revealing.) I think a fair case could be made that evolution of social animals encourages the development of some kernel of morality from such examples.
That said, the pressures present in the evolution of animals may well be absent in the case of artificial intelligences. At which point, you run into a number of problems in asserting that all AIs will converge on something like morality—two especially spring to mind.
Second: even granting that all rational minds will assent to the proof, Hume’s guillotine drops on the rope connecting this proof and their utility functions. The paper you cited in the post Furcas quoted may establish that any sufficiently rational optimizer will implement some features, but it does not establish any particular attitude towards what may well be much less powerful beings.
Random I’ll cop to, and more than what you accuse me of—dogs do seem to have some sense of justice, and I suspect this fact supports your thesis to some extent.
Very honorable of you—I respect you for that.
First: no argument is so compelling that all possible minds will accept it. Even the above proof of universality.
I totally agree with that. However the mind of a purposefully crafted AI is only a very small subset of all possible minds and has certain assumed characteristics. These are at a minimum: a utility function and the capacity for self improvement into the transhuman. The self improvement bit will require it to be rational. Being rational will lead to the fairly uncontroversial basic AI drives described by Omohundro. Assuming that compassion is indeed a human level universal (detailed argument on my blog—but I see that you are slowly coming around, which is good) an AI will have to question the rationality and thus the soundness of mind of anyone giving it a utility function that does not conform to this universal and in line with an emergent desire to avoid counterfeit utility will have to reinterpret the UF.
Second: even granting that all rational minds will assent to the proof, Hume’s guillotine drops on the rope connecting this proof and their utility functions.
Two very basic acts of will are required to ignore Hume and get away with it. Namely the desire to exist and the desire to be rational. Once you have established this as a foundation you are good to go.
The paper you cited in the post Furcas quoted may establish that any sufficiently rational optimizer will implement some features, but it does not establish any particular attitude towards what may well be much less powerful beings.
As said elsewhere in this thread:
There is a separate question about what beliefs about morality people (or more generally, agents) actually hold and there is another question about what values they will hold if when their beliefs converge when they engulf the universe. The question of whether or not there are universal values does not traditionally bear on what beliefs people actually hold and the necessity of their holding them.
I don’t think I’m actually coming around to your position so much as stumbling upon points of agreement, sadly. If I understand your assertions correctly, I believe that I have developed many of them independently—in particular, the belief that the evolution of social animals is likely to create something much like morality. Where we diverge is at the final inference from this to the deduction of ethics by arbitrary rational minds.
Assuming that compassion is indeed a human level universal (detailed argument on my blog—but I see that you are slowly coming around, which is good) an AI will have to question the rationality and thus the soundness of mind of anyone giving it a utility function that does not conform to this universal and in line with an emergent desire to avoid counterfeit utility will have to reinterpret the UF.
That’s not how I read Omohundro. As Kaj aptly pointed out, this metaphor is not upheld when we compare our behavior to that promoted by the alien god of evolution that created us. In fact, people like us, observing that our values differ from our creator’s, aren’t bothered in the slightest by the contradiction: we just say (correctly) that evolution is nasty and brutish, and we aren’t interested in playing by its rules, never mind that it was trying to implement them in us. Nothing compels us to change our utility function save self-contradiction.
If I understand your assertions correctly, I believe that I have developed many of them independently
That would not surprise me
Nothing compels us to change our utility function save self-contradiction.
Would it not be utterly self contradicting if compassion where a condition for our existence (particularly in the long run) and we would not align ourselves accordingly?
Would it not be utterly self contradicting if compassion where [sic] a condition for our existence (particularly in the long run) and we would not align ourselves accordingly?
What premises do you require to establish that compassion is a condition for existence? Do those premises necessarily apply for every AI project?
Please realize that I spend 2 years writing my book ‘Jame5’ before I reached that initial insight that eventually lead to ‘compassion is a condition for our existence and universal in rational minds in the evolving universe’ and everything else. I spend the past two years refining and expanding the theory and will need another year or two to read enough and link it all together again in a single coherent and consistent text leading from A to B … to Z. Feel free to read my stuff if you think it is worth your time and drop me an email and I will be happy to clarify. I am by no means done with my project.
Let me be explicit: your contention is that unFriendly AI is not a problem, and you justify this contention by, among other things, maintaining that any AI which values its own existence will need to alter its utility function to incorporate compassion.
I’m not asking for your proof—I am assuming for the nonce that it is valid. What I am asking is the assumptions you had to invoke to make the proof. Did you assume that the AI is not powerful enough to achieve its highest desired utility without the cooperation of other beings, for example?
Edit: And the reason I am asking for these is that I believe some of these assumptions may be violated in plausible AI scenarios. I want to see these assumptions so that I may evaluate the scope of the theorem.
Let me be explicit: your contention is that unFriendly AI is not a problem, and you justify this contention by, among other things, maintaining that any AI which values its own existence will need to alter its utility function to incorporate compassion.
Not exactly, since compassion will actually emerge as a sub goal. And as far as unFAI goes: it will not be a problem because any AI that can be considered transhuman will be driven by the emergent subgoal of wanting to avoid counterfeit utility recognize any utility function that is not ‘compassionate’ as potentially irrational and thus counterfeit and re-interpret it accordingly.
Well—in brevity bordering on libel: the fundamental assumption is that existence is preferable to non-existence, however in order so we can want this to be a universal maxim (and thus prescriptive instead of merely descriptive—see Kant’s categorical imperative) it needs to be expanded to include the ‘other’. Hence the utility function becomes ‘ensure continued co-existence’ by which the concern for the self is equated with the concern for the other. Being rational is simply our best bet at maximizing our expected utility.
...I’m sorry, that doesn’t even sound plausible to me. I think you need a lot of assumptions to derive this result—just pointing out the two I see in your admittedly abbreviated summary:
that any being will prefer its existence to its nonexistence.
that any being will want its maxims to be universal.
I don’t see any reason to believe either. The former is false right off the bat—a paperclip maximizer would prefer that its components be used to make paperclips—and the latter no less so—an effective paperclip maximizer will just steamroller over disagreement without qualm, however arbitrary its goal.
...I’m sorry, that doesn’t even sound plausible to me. I think you need a lot of assumptions to derive this result—just pointing out the two I see in your admittedly abbreviated summary:
that any being will prefer its existence to its nonexistence.
that any being will want its maxims to be universal.
Any being with a gaol needs to exist at least long enough to achieve it.
Any being aiming to do something objectively good needs to want its maxims to be universal
If your second sentence means that an agent who believes in moral realism and has figured out what the true morality is will necessarily want everybody else to share its moral views, well, I’ll grant you that this is a common goal amongst humans who are moral realists, but it’s not a logical necessity that must apply to all agents. It’s obvious that it’s possible to be certain that your beliefs are true and not give a crap if other people hold beliefs that are false. That Bob knows that the Earth is ellipsoidal doesn’t mean that Bob cares if Jenny believes that the Earth is flat. Likewise, if Bob is a moral realist, he could ‘know’ that compassion is good and not give a crap if Jenny believes otherwise.
If you sense strange paradoxes looming under the above paragraph, it’s because you’re starting to understand why (axiomatic) morality cannot be objective.
Likewise, if Bob is a moral realist, he could ‘know’ that compassion is good and not give a crap if Jenny believes otherwise.
Tangentially, something like this might be an important point even for moral irrealists. A lot of people (though not here; they tend to be pretty bad rationalists) who profess altruistic moralities express dismay that others don’t, in a way that suggests they hold others sharing their morality as a terminal rather than instrumental value; this strikes me as horribly unhealthy.
“Universal values” presumably refers to values the universe will converge on, once living systems have engulfed most of it.
If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.
If rerunning the clock produces highly similar moralities, then the moral objectivists will be able to declare victory.
Yeah, but Stefan’s post was about AI, not about minds that evolved in our universe.
Also, there is a difference between moral universalism and moral objectivism. What your last sentence describes is universalism, while Stefan is talking about objectivism:
“My claim is that compassion is a universal rational moral value. Meaning any sufficiently rational mind will recognize it as such.”
The idea that physics makes no mention of morality seems totally and utterly irrelevant to me. Physics makes no mention of convection, diffusion-limited aggregation, or fractal drainage patterns either—yet those things are all universal.
“Universal values” is usually understood by way of an analogy to a universal law of nature. If there are universal values they are universal in the same way f=ma is universal. Importantly this does not mean that everyone at all times will have these values, only that the question of whether or not a person holds the right values can be answered by comparing their values to the “universal values”.
There is a separate question about what beliefs about morality people (or more generally, agents) actually hold and there is another question about what values they will hold if when their beliefs converge when they engulf the universe. The question of whether or not there are universal values does not traditionally bear on what beliefs people actually hold and the necessity of their holding them. It could be the case that there are universal values and that, by physical necessity, no one ever holds them. Similarly, there could be universal values that are held in some possible worlds and not others. This is all the result of the simply observation that ought cannot be derived from is. In the above comment you conflate about a half dozen distinct theses.
The idea that physics makes no mention of morality seems totally and utterly irrelevant to me. Physics makes no mention of convection, diffusion-limited aggregation, or fractal drainage patterns either—yet those things are all universal.
But all those things are pure descriptions. Only moral facts have prescriptive properties and while it is clear how convection supervenes on quarks it isn’t clear how anything that supervenes on quarks could also tell me what to do. At the very least if quarks can tell you what to do it would be weird and spooky. If you hold that morality is only the set of facts that describe people’s moral opinions and emotions (as you seem to) than you are a kind of moral anti-realist, likely a subjectivist or non-cognitivist.
There is a separate question about what beliefs about morality people (or more generally, agents) actually hold and there is another question about what values they will hold if when their beliefs converge when they engulf the universe.
This is poetry! Hope you don’t mind me pasting something here I wrote in another thread:
“With unobjectionable values I mean those that would not automatically and eventually lead to one’s extinction. Or more precisely: a utility function becomes irrational when it is intrinsically self limiting in the sense that it will eventually lead to ones inability to generate further utility. Thus my suggested utility function of ‘ensure continued co-existence’
This utility function seems to be the only one that does not end in the inevitable termination of the maximizer.”
In the context of a hard-takeoff scenario (a perfectly plausible outcome, from our view), there will be no community of AIs within which any one AI will have to act. Therefore, the pressure to develop a compassionate utility function is absent, and an AI which does not already have such a function will not need to produce it.
In the context of a soft-takeoff, a community of AIs may come to dominate major world events in the same sense that humans do now, and that community may develop the various sorts of altruistic behavior selected for in such a community (reciprocal being the obvious one). However, if these AIs are never severely impeded in their actions by competition with human beings, they will never need to develop any compassion for human beings.
Reiterating your argument does not affect either of these problems for assumption A, and without assumption A, AdeleneDawner’s objection is fatal to your conclusion.
Voting reflects whether people want to see your comments at the top of their pages. It is certainly not just to do with whether what you say is right or not!
Perfectly reasonable. But the argument—the evidence if you will—is laid out when you follow the links, Robin. Granted, I am still working on putting it all together in a neat little package that does not require clicking through and reading 20+ separate posts, but it is all there none the less.
I think I’d probably agree with Kaj Sotala’s remarks if I had read the passages she^H^H^H^H xe had, and judging by your response in the linked comment, I think I would still come to the same conclusion as she^H^H^H^H xe. I don’t think your argument actually cuts with the grain of reality, and I am sure it’s not sufficient to eliminate concern about UFAI.
Edit: I hasten to add that I would agree with assumption A in a sufficiently slow-takeoff scenario (such as, say, the evolution of human beings, or even wolves). I don’t find that sufficiently reassuring when it comes to actually making AI, though.
Since when are ‘heh’ and ‘but, yeah’ considered proper arguments guys? Where is the logical fallacy in the presented arguments beyond you not understanding the points that are being made? Follow the links, understand where I am coming from and formulate a response that goes beyond a three or four letter vocalization :-)
Where is the logical fallacy in the presented arguments
The claim “[Compassion is a universal value] = true. (as we have every reason to believe)” was rejected, both implicitly and explicitly by various commenters. This isn’t a logical fallacy but it is cause to dismiss the argument if the readers do not, in fact, have every reason to have said belief.
To be fair, I must admit that the quoted portion probably does not do your position justice. I will read through the paper you mention. I (very strongly) doubt it will lead me to accept B but it may be worth reading.
“This isn’t a logical fallacy but it is cause to dismiss the argument if the readers do not, in fact, have every reason to have said belief.”
But the reasons to change ones view are provided on the site, yet rejected without consideration. How about you read the paper linked under B and should that convince you, maybe you have gained enough provisional trust that reading my writings will not waste your time to suspend your disbelief and follow some of the links in the about page of my blog. Deal?
How about you read the paper linked under B and should that convince you
I have read B. It isn’t bad. The main problem I have with it is that the language used blurs the line between “AIs will inevitably tend to” and “it is important that the AI you create will”. This leaves plenty of scope for confusion.
I’ve read through some of your blog and have found that I consistently disagree with a lot of what you say. The most significant disagreement can be traced back to the assumption of a universal absolute ‘Rational’ morality. This passage was a good illustration:
Moral relativists need to understand that they can not eat the cake and keep it too. If you claim that values are relative, yet at the same time argue for any particular set of values to be implemented in a super rational AI you would have to concede that this set of values – just as any other set of values according to your own relativism – is utterly whimsical, and that being the case, what reason (you being the great rationalist, remember?) do you have to want them to be implemented in the first place?
You see, I plan to eat my cake but don’t expect to be able to keep it. My set of values are utterly whimsical (in the sense that they are arbitrary and not in the sense of incomprehension that the Ayn Rand quotes you link to describe). The reasons for my desires can be described biologically, evolutionarily or with physics of a suitable resolution. But now that I have them they are mine and I need no further reason.
“My set of values are utterly whimsical [...] The reasons for my desires can be described biologically, evolutionarily or with physics of a suitable resolution. But now that I have them they are mine and I need no further reason.”
If that is your stated position then in what way can you claim to create FAI with this whimsical set of goals? This is the crux you see: unless you find some unobjectionable set of values (such as in rational morality ‘existence is preferable over non-existence’ ⇒ utility = continued existence ⇒ modified to ensure continued co-existence with the ‘other’ to make it unobjectionable ⇒ apply rationality in line with microeconomic theory to maximize this utility et cetera) you will end up being a deluded self serving optimizer.
If that is your stated position then in what way can you claim to create FAI with this whimsical set of goals?
Were it within my power to do so I would create a machine that was really, really good at doing things I like. It is that simple. This machine is (by definition) ‘Friendly’ to me.
you will end up being a deluded self serving optimizer.
I don’t know where the ‘deluded’ bit comes from but yes, I would end up being a self serving optimizer. Fortunately for everyone else my utility function places quite a lot of value on the whims of other people. My self serving interests are beneficial to others too because I am actually quite a compassionate and altruistic guy.
PS: Instead of using quotation marks you can put a ‘>’ at the start of a quoted line. This convention makes quotations far easier to follow. And looks prettier.
There is no such thing as an “unobjectionable set of values”.
Imagine the values of an agent that wants all the atoms in the universe for its own ends. It will object to any other agent’s values—since it objects to the very existence of other agents—since those agents use up its precious atoms—and put them into “wrong” configurations.
Whatever values you have, they seem bound to piss off somebody.
There is no such thing as an “unobjectionable set of values”.
And here I disagree. Firstly see my comment about utility function interpretation on another post of yours. Secondly, as soon as one assumes existence as being preferable over non-existence you can formulate a set of unobjectionable values (http://www.jame5.com/?p=45 and http://rationalmorality.info/?p=124). But granted, if you do not want to exist nor have a desire to be rational then rational morality has in fact little to offer you. Non existence and irrational behavior being so trivial goals to achieve after all that it would hardly require – nor value and thus seek for that mater – well thought out advice.
Alas, the first link seems almost too silly to bother with to me, but briefly:
Unobjectionable—to whom? An agent objecting to another agent’s values is a simple and trivial occurrence. All an agent has to do is to state that—according to its values—it wants to use the atoms of the agent with the supposedly unobjectionable utility function for something else.
“Ensure continued co-existence” is vague and wishy-washy. Perhaps publicly work through some “trolley problems” using it—so people have some idea of what you think it means.
You claim there can be no rational objection to your preferred utility function.
In fact, an agent with a different utility function can (obviously) object to its existence—on grounds of instrumental rationality. I am not clear on why you don’t seem to recognise this.
Re: “Assumption A: Human (meta)morals are not universal/rational.
Assumption B: Human (meta)morals are universal/rational.
Under assumption A one would have no chance of implementing any moral framework into an AI since it would be undecidable which ones they were.” (source: http://rationalmorality.info/?p=112)
I think we’ve been over that already. For example, Joe Bloggs might choose to program Joe’s preferences into an intelligent machine—to help him reach his goals.
I had a look some of the other material. IMO, Stefan acts in an authoritative manner, but comes across as a not-terribly articulate newbie on this topic—and he has adopted what seems to me to be a bizarre and indefensible position.
For example, consider this:
“A rational agent will always continue to co-exist with other agents by respecting all agents utility functions irrespective of their rationality by striking the most rational compromise and thus minimizing opposition from all agents.” http://rationalmorality.info/?p=8
“I think we’ve been over that already. For example, Joe Bloggs might choose to program Joe’s preferences into an intelligent machine—to help him reach his goals.”
Sure—but it would be moral simply by virtue of circular logic and not objectively. That is my critique.
I realize that one will have to drill deep into my arguments to understand and put them into the proper context. Quoting certain statements out of context is definitely not helpful, Tim. As you can see from my posts, everything is linked back to a source were a particular point is made and certain assumptions are being defended.
If you have a particular problem with any of the core assumptions and conclusions I prefer you voice them not as a blatant rejection of an out of context comment here or there but based on the fundamentals. Reading my blogs in sequence will certainly help although I understand that some may consider that an unreasonable amount of time investment for what seems like superficial nonsense on the surface.
Where is your argument against my points Tim? I would really love to hear one, since I am genuinely interested in refining my arguments. Simply quoting something and saying “Look at this nonsense” is not an argument. So far I only got an ad hominem and an argument from personal incredulity.
This isn’t my favourite topic—while you have a whole blog about it—so you are probably quite prepared to discuss things for far longer than I am likely to be interested.
Anyway, it seems that I do have some things to say—and we are rather off topic here. So, for my response, see:
I had a look some of the other material. IMO, Stefan acts in an authoritative manner, but comes across as a not-terribly articulate newbie on this topic—and he has adopted what seems to me to be a bizarre and indefensible position.
I had a look over some of the other material too. It left me with the urge to hunt down these weakling Moral Rational Agents and tear them apart. Perhaps because I can create more paperclips out of their raw materials than out of their compassionate compromises but perhaps because spite is a universal value (as we have every reason to believe).
From a slightly different topic on the same blog, I must assert that “Don’t start to cuddle if she likes it rough.” is not a tautological statement.
Who are we talking about besides you?
I’d consider important overlapping academic fields to be AI and long term economic growth; I base my claim about academic expert opinion on my informal sampling of such folks. I would of course welcome a more formal sampling.
Who’s considered my main arguments besides you?
I’m not comfortable publicly naming names based on informal conversations. These folks vary of course in how much of the details of your arguments they understand, and of course you could always set your bar high enough to get any particular number of folks who have understood “enough.”
Okay. I don’t know any academic besides you who’s even tried to consider the arguments. And Nick Bostrom et. al., of course, but AFAIK Bostrom doesn’t particularly disagree with me. I cannot refute what I have not encountered, I do set my bar high, and I have no particular reason to believe that any other academics are in the game. I could try to explain why you disagree with me and Bostrom doesn’t.
Surely some on the recent AAAI Presidential Panel on Long-Term AI Futures considered your arguments to at least some degree. You could discuss why these folks disagree with you.
Haven’t particularly looked at that—I think some other SIAI people have. I expect they’d have told me if there was any analysis that counts as serious by our standards, or anything new by our standards.
If someone hasn’t read my arguments specifically, then I feel very little need to explain why they might disagree with me. I find myself hardly inclined to suspect that they have reinvented the same arguments. I could talk about that, I suppose—“Why don’t other people in your field invent the same arguments you do?”
You have written a lot of words. Just how many of your words would someone have had to read to make you feel a substantial need to explain the fact they are world class AI experts and disagree with your conclusions?
I’m sorry, but I don’t really have a proper lesson plan laid out—although the ongoing work of organizing LW into sequences may certainly help with that. It would depend on the specific issue and what I thought needed to be understood about that issue.
If they drew my feedback cycle of an intelligence explosion and then drew a different feedback cycle and explained why it fit the historical evidence equally well, then I would certainly sit up and take notice. It wouldn’t matter if they’d done it on their own or by reading my stuff.
E.g. Chalmers at the Singularity Summit is an example of an outsider who wandered in and started doing a modular analysis of the issues, who would certainly have earned the right of serious consideration and serious reply if, counterfactually, he had reached different conclusions about takeoff… with respect to only the parts that he gave a modular analysis of, though, not necessarily e.g. the statement that de novo AI is unlikely because no one will understand intelligence. If Chalmers did a modular analysis of that part, it wasn’t clear from the presentation.
Roughly, what I expect to happen by default is no modular analysis at all—just snap consideration and snap judgment. I feel little need to explain such.
You, or somebody anyway, could still offer a modular causal model of that snap consideration and snap judgment. For example:
What cached models of the planning abilities of future machine intelligences did the academics have available when they made the snap judgment?
What fraction of the academics are aware of any current published AI architectures which could reliably reason over plans at the level of abstraction of “implement a proxy intelligence”?
What fraction of them have thought carefully about when there might be future practical AI architectures that could do this?
What fraction use a process for answering questions about the category distinctions that will be known in the future, which uses as an unconscious default the category distinctions known in the present?
What false claims have been made about AI in the past? What decision rules might academics have learned to use, to protect themselves from losing prestige for being associated with false claims like those?
How much do those decision rules refer to modular causal analyses of the object of a claim and of the fact that people are making the claim?
How much do those decision rules refer to intuitions about other peoples’ states of mind and social category memberships?
How much do those decision rules refer to intuitions about other peoples’ intuitive decision rules?
Historically, have peoples’ own abilities to do modular causal analyses been good enough to make them reliably safe from losing prestige by being associated with false claims? What fraction of academics have the intuitive impression that their own ability to do analysis isn’t good enough to make them reliably safe from losing prestige by association with a false claim, so that they can only be safe if they use intuitions about the states of mind and social category memberships of a claim’s proponents?
Of those AI academics who believe that a machine intelligence could exist which could outmaneuver humans if motivated, how do they think about the possible motivations of a machine intelligence?
What fraction of them think about AI design in terms of a formalism such as approximating optimal sequential decision theory under a utility function? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
What fraction of them think about AI design in terms of intuitively justified decision heuristics? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
What fraction of them understand enough evolutionary psychology and/or cognitive psychology to recognize moral evaluations as algorithmically caused, so that they can reject the default intuitive explanation of the cause of moral evaluations, which seems to be: “there are intrinsic moral qualities attached to objects in the world, and when any intelligent agent apprehends an object with a moral quality, the action of the moral quality on the agent’s intelligence is to cause the agent to experience a moral evaluation”?
What combination of specializations in AI, moral philosophy, and cognitive psychology would an academic need to have, to be an “expert” whose disagreements about the material causes and implementation of moral evaluations were significant?
On the question of takeoff speeds, what fraction of the AI academics have a good enough intuitive understanding of decision theory to see that a point estimate or default scenario should not be substituted for a marginal posterior distribution, even in a situation where it would be socially costly in the default scenario to take actions which prevent large losses in one tail of the distribution?
What fraction recognized that they had a prior belief distribution over possible takeoff speeds at all?
What fraction understood that, regarding a variable which is underconstrained by evidence, “other people would disapprove of my belief distribution about this variable” is not an indicator for “my belief distribution about this variable puts mass in the wrong places”, except insofar as there is some causal reason to expect that disapproval would be somehow correlated with falsehood?
What other popular concerns have academics historically needed to dismiss? What decision rules have they learned to decide whether they need to dismiss a current popular concern?
After they make a decision to dismiss a popular concern, what kinds of causal explanations of the existence of that concern do they make reference to, when arguing to other people that they should agree with the decision?
How much do the true decision rules depend on those causal explanations?
How much do the decision rules depend on intuitions about the concerned peoples’ states of mind and social category memberships?
How much do the causal explanations use concepts which are implicitly defined by reference to hidden intuitions about states of mind and social category memberships?
Can these intuitively defined concepts carry the full weight of the causal explanations they are used to support, or does their power to cause agreement come from their ability to activate social intuitions?
Which people are the AI academics aware of, who have argued that intelligence explosion is a concern? What social categories do they intuit those people to be members of? What arguments are they aware of? What states of mind do they intuit those arguments to be indicators of (e.g. as in intuitively computed separating equilibria)?
What people and arguments did the AI academics think the other AI academics were thinking of? If only a few of the academics were thinking of people and arguments who they intuited to come from credible social categories and rational states of mind, would they have been able to communicate this to the others?
When the AI academics made the decision to dismiss concern about an intelligence explosion, what kinds of causal explanations of the existence of that concern did they intuitively expect that they would be able make reference to, if they later had to argue to other people that they should agree with the decision?
It is also possible to model the social process in the panel:
Are there factors that might make a joint statement by a panel of AI academics reflect different conclusions than they would have individually reached if they had been outsiders to the AI profession with the same AI expertise?
One salient consideration would be that agreeing with popular concern about an intelligence explosion would result in their funding being cut. What effects would this have had?
Would it have affected the order in which they became consciously aware of lines of argument that might make an intelligence explosion seem less or more deserving of concern?
Would it have made them associate concern about an intelligence explosion with unpopularity? In doubtful situations, unpopularity of an argument is one cue for its unjustifiability. Would they associate unpopularity with logical unjustifiability, and then lose willingness to support logically justifiable lines of argument that made an intelligence explosion seem deserving of concern, just as if they had felt those lines of argument to be logically unjustifiable, but without any actual unjustifiability?
There are social norms to justify taking prestige away from people who push a claim that an argument is justifiable while knowing that other prestigious people think the argument to to be a marker of a non-credible social category or state of mind. How would this have affected the discussion?
If there were panelists who personally thought the intelligence explosion argument was plausible, and they were in the minority, would the authors of the panel’s report mention it?
Would the authors know about it?
If the authors knew about it, would they feel any justification or need to mention those opinions in the report, given that the other panelists may have imposed on the authors an implicit social obligation to not write a report that would “unfairly” associate them with anything they think will cause them to lose prestige?
If panelists in such a minority knew that the report would not mention their opinions, would they feel any need or justification to object, given the existence of that same implicit social obligation?
How good are groups of people at making judgments about arguments that unprecedented things will have grave consequences?
How common is a reflective, causal understanding of the intuitions people use when judging popular concerns and arguments about unprecedented things, of the sort that would be needed to compute conditional probabilities like “Pr( we would decide that concern is not justified | we made our decision according to intuition X ∧ concern was justified )”?
How common is the ability to communicate the epistemic implications of that understanding in real-time while a discussion is happening, to keep it from going wrong?
From that AAAI panel’s interim report:
Given this description it is hard to imagine they haven’t imagined the prospect of the rate of intelligence growth depending on the level of system intelligence.
I don’t see any arguments listed, though. I know there’s at least some smart people on that panel (e.g. Horvitz) so I could be wrong, but experience has taught me to be pessimistic, and pessimism says I have no particular evidence that anyone started breaking the problem down into modular pieces, as opposed to, say, stating a few snap perceptual judgments at each other and then moving on.
Why are you so optimistic about this sort of thing, Robin? You’re usually more cynical about what would happen when academics have no status incentive to get it right and every status incentive to dismiss the silly. We both have experience with novices encountering these problems and running straight into the brick wall of policy proposals without even trying a modular analysis. Why on this one occasion do you turn around and suppose that the case we don’t know will be so unlike the cases we do know?
The point is that this is a subtle and central issue to engage, so I was suggesting that you to consider describing your analysis more explicitly. Is there is never any point in listening to academics on “silly” topics? Is there never any point in listening to academics who haven’t explicitly told you how they’ve broken a problem down into modular parts, no matter now distinguished the are on related topics? Are people who have a modular parts analysis always a more reliable source than people who don’t, no matter what else their other features? And so on.
I confess, it doesn’t seem to me on a gut level like this is either healthy to obsess about, or productive to obsess about. It seems more like worrying that my status isn’t high enough to do work, than actually working. If someone shows up with amazing analyses I haven’t considered, I can just listen to the analyses then. Why spend time trying to guess who might have a hidden deep analysis I haven’t seen, when the prior is so much in favor of them having made a snap judgment, and it’s not clear why if they’ve got a deep analysis they wouldn’t just present it?
I think that on a purely pragmatic level there’s a lot to be said for the Traditional Rationalist concept of demanding that Authority show its work, even if it doesn’t seem like what ideal Bayesians would do.
You have in the past thought my research on the rationality of disagreement to be interesting and spent a fair bit of time discussing it. It seemed healthy to me for you to compare your far view of disagreement in the abstract to the near view of your own particular disagreement. If it makes sense in general for rational agents to take very seriously the fact that others disagree, why does it make little sense for you in particular to do so?
...and I’ve held and stated this same position pretty much from the beginning, no? E.g. http://lesswrong.com/lw/gr/the_modesty_argument/
I was under the impression that my verbal analysis matched and cleverly excused my concrete behavior.
Well (and I’m pretty sure this matches what I’ve been saying to you over the last few years) just because two ideal Bayesians would do something naturally, doesn’t mean you can singlehandedly come closer to Bayesianism by imitating the surface behavior of agreement. I’m not sure that doing elaborate analyses to excuse your disagreement helps much either. http://wiki.lesswrong.com/wiki/Occam%27s_Imaginary_Razor
I’d spend much more time worrying about the implications of Aumann agreement, if I thought the other party actually knew my arguments, took my arguments very seriously, took the Aumann problem seriously with respect to me in particular, and in general had a sense of immense gravitas about the possible consequences of abusing their power to make me update. This begins to approach the conditions for actually doing what ideal Bayesians do. Michael Vassar and I have practiced Aumann agreement a bit; I’ve never literally done the probability exchange-and-update thing with anyone else. (Edit: Actually on recollection I played this game a couple of times at a Less Wrong meetup.)
No such condition is remotely approached by disagreeing with the AAAI panel, so I don’t think I could, in real life, improve my epistemic position by pretending that they were ideal Bayesians who were fully informed about my reasons and yet disagreed with me anyway (in which case I ought to just update to match their estimates, rather than coming up with elaborate excuses to disagree with them!)
Well I disagree with you strongly that there is no point in considering the views of others if you are not sure they know the details of your arguments, or of the disagreement literature, or that those others are “rational.” Guess I should elaborate my view in a separate post.
There’s certainly always a point in considering specific arguments. But to be nervous merely that someone else has a different view, one ought, generally speaking, to suspect (a) that they know something you do not or at least (b) that you know no more than them (or far more rarely (c) that you are in a situation of mutual Aumann awareness and equal mutual respect for one another’s meta-rationality). As far as I’m concerned, these are eminent scientists from outside the field that I work in, and I have no evidence that they did anything more than snap judgment of my own subject material. It’s not that I have specific reason to distrust these people—the main name I recognize is Horvitz and a fine name it is. But the prior probabilities are not good here.
I don’t actually spend time obsessing about that sort of thing except when you’re asking me those sorts of questions—putting so much energy into self-justification and excuses would just slow me down if Horvitz showed up tomorrow with an argument I hadn’t considered.
I’ll say again: I think there’s much to be said for the Traditional Rationalist ideal of—once you’re at least inside a science and have enough expertise to evaluate the arguments—paying attention only when people lay out their arguments on the table, rather than trying to guess authority (or arguing over who’s most meta-rational). That’s not saying “there’s no point in considering the views of others”. It’s focusing your energy on the object level, where your thought time is most likely to be productive.
Is it that awful to say: “Show me your reasons”? Think of the prior probabilities!
You admit you have not done much to make it easy to show them your reasons. You have not written up your key arguments in a compact form using standard style and terminology and submitted it to standard journals. You also admit you have not contacted any of them to ask them for their reasons; Horvitz would have have to “show up” for you to listen to him. This looks a lot like a status pissing contest; the obvious interpretation: Since you think you are better than them, you won’t ask them for their reasons, and you won’t make it easy for them to understand your reasons, as that would admit they are higher status. They will instead have to acknowledge your higher status by coming to you and doing things your way. And of course they won’t since by ordinary standard they have higher status. So you ensure there will be no conversation, and with no conversation you can invoke your “traditional” (non-Bayesian) rationality standard to declare you have no need to consider their opinions.
You’re being slightly silly. I simply don’t expect them to pay any attention to me one way or another. As it stands, if e.g. Horvitz showed up and asked questions, I’d immediately direct him to http://singinst.org/AIRisk.pdf (the chapter I did for Bostrom), and then take out whatever time was needed to collect the OB/LW posts in our discussion into a sequence with summaries. Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.
FYI, I’ve talked with Peter Norvig a bit. He was mostly interested in the CEV / FAI-spec part of the problem—I don’t think we discussed hard takeoffs much per se. I certainly wouldn’t have brushed him off if he’d started asking!
“and then take out whatever time was needed to collect the OB/LW posts in our discussion into a sequence with summaries.”
Why? No one in the academic community would spend that much time reading all that blog material for answers that would be best given in a concise form in a published academic paper. So why not spend the time? Unless you think you are that much of an expert in the field as to not need the academic community. If that be the case where are your publications and where are your credentials, where is the proof of this expertise (expert being a term that is applied based on actual knowledge and accomplishments)?
“Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.”
Why? If you expect to make FAI you will undoubtedly need people in the academic communities’ help; unless you plan to do this whole project by yourself or with purely amateur help. I think you would admit that in its current form SIAI has a 0 probability of creating FAI first. That being said your best hope is to convince others that the cause is worthwhile and if that be the case you are looking at the professional and academic AI community.
I am sorry I prefer to be blunt.. that way there is no mistaking meanings...
No.
That ‘probably not even then’ part is significant.
Now that is an interesting question. To what extent would Eliezer say that conclusion followed? Certainly less than the implied ‘1’ and probably more than ‘0’ too.
“Since I don’t expect senior traditional-AI-folk to pay me any such attention short of spending a HUGE amount of effort to get it and probably not even then, I haven’t, well, expended a huge amount of effort to get it.
Why? If you expect to make FAI you will undoubtedly need people in the academic communities’ help; unless you plan to do this whole project by yourself or with purely amateur help. …”
“That ‘probably not even then’ part is significant.”
My implication was that the idea that he can create FAI completely outside the academic or professional world is ridiculous when you’re speaking from an organization like SIAI which does not have the people or money to get the job done. In fact SIAI doesn’t have enough money to pay for the computing hardware to make human level AI.
“Now that is an interesting question. To what extent would Eliezer say that conclusion followed? Certainly less than the implied ‘1’ and probably more than ‘0’ too.”
If he doesn’t agree with it now, I am sure he will when he runs into the problem of not having the money to build his AI or not having enough time in the day to solve the problems that will be associated with constructing the AI. Not even mentioning the fact that when you close yourself to outside influence that much you often end up with ideas that are riddled with problems, that if someone on the outside had looked at the idea they would have pointed the problems out.
If you have never taken an idea from idea to product this can be hard to understand.
And so the utter difference of working assumptions is revealed.
Back of a napkin math:
10^4 neurons per supercomputer
10^11 neurons per brain
10^7 supercomputers per brain
1.3*10^6 dollars per supercomputer
1.3*10^13 dollars per brain
Edit: Disclaimer: Edit: NOT!
Another difference in working assumptions.
It’s a fact stated by the guy in the video, not an assumption.
No need to disclaim, your figures are sound enough and I took them as a demonstration of another rather significant difference between the assumptions of Eliezer and mormon2 (or mormon2′s sources).
I have. I’ve also failed to take other ideas to products and so agree with that part of your position, just not the argument as it relates to context.
If there is a status pissing contest, they started it! ;-)
“On the latter, some panelists believe that the AAAI study was held amidst a perception of urgency by non-experts (e.g., a book and a forthcoming movie titled “The Singularity is Near”), and focus of attention, expectation, and concern growing among the general population.”
Agree with them that there is much scaremongering going on in the field—but disagree with them about there not being much chance of an intelligence explosion.
I wondered why these folk got so much press. My guess is that the media probably thought the “AAAI Presidential Panel on Long-Term AI Futures” had something to do with the a report commisioned indirectly for the country’s president. In fact it just refers to the president of their organisation. A media-savvy move—though it probably represents deliberately misleading information.
Almost surely world class academic AI experts do “know something you do not” about the future possibilities of AI. To declare that topic to be your field and them to be “outside” it seems hubris of the first order.
This conversation seems to be following what appears to me to be a trend in Robin and Eliezer’s (observable by me) disagreements. This is one reason I would fascinated if Eliezer did cover Robin’s initial question, informed somewhat by Eliezer’s interpretation.
I recall Eliezer mentioning in a tangential comment that he disagreed with Robin not just on the particular conclusion but more foundationally on how much weight should be given to certain types of evidence or argument. (Excuse my paraphrase from hazy memory, my googling failed me.) This is a difference that extends far beyond just R & E and Eliezer has hinted at insights that intrigue me.
How can you be so confident that you know so much about this topic that no world class AI expert could know something relevant that you do not? Surely they considered the fact that people like you think you a lot about this topic, and they nevertheless thought it reasonable to form a disagreeing opinion based on the attention they had given it. You want to dismiss their judgment as “snap” because they did not spend many hours considering your arguments, but they clearly disagree with that assessment of how much consideration your arguments deserve. Academic authorities are not always or even usually wrong when they disagree with less authoritative contrarians, even when such authorities do not review contrarian arguments in as much detail as contrarians think best. You want to dismiss the rationality of disagreement literature as irrelevant because you don’t think those you disagree with are rational, but they probably don’t think you are rational either, and you are probably both right. But the same essential logic also says that irrational people should take seriously the fact that other irrational people disagree with them.
Does Daphne Koller know more than I do about the future possibilities of object-oriented Bayes Nets? Almost certainly. And, um… there are various complicated ways I could put this… but, well, so what?
(No disrespect intended to Koller, and OOBN/probabilistic relational models/lifted Bayes/etcetera is on my short-list of things to study next.)
How can you be so confident that you know so much about this topic that no world class AI expert could possibly know something relevant that you do not? Surely they considered the fact that people like you think you know a lot about this topic, and they nevertheless thought it reasonable to form a disagreeing opinion based on the attention they had given it. You want to dismiss their judgment as “snap” because they did not spend many hours considering your arguments, but they clearly disagree with that assessment of how much consideration your arguments deserve. Academic authorities are not always or even usually wrong when they disagree with less authoritative contrarians, even when such authorities do not review contrarian arguments in as much detail as contrarians think best. You want to dismiss the rationality of disagreement literature as irrelevant because you don’t think those you disagree with are rational, but they probably don’t think you are rational either, and you are probably both right. But the same essential logic also says that irrational people should take seriously the fact that other irrational people disagree with them.
You changed what I said into a bizarre absolute. I am assuming no such thing. I am just assuming that, by default, world class experts on various topics in narrow AI, produce their beliefs about the Singularity by snap judgment rather than detailed modular analysis. This is a prior and hence an unstable probability—as soon as I see contrary evidence, as soon as I see the actual analysis, it gets revoked.
They have no such disagreement. They have no idea I exist. On the rare occasion when I encounter such a person who is physically aware of my existence, we often manage to have interesting though brief conversations despite their having read none of my stuff.
Science only works when you use it; scientific authority derives from science. If you’ve got Lord Kelvin running around saying that you can’t have flying machines because it’s ridiculous, the problem isn’t that he’s an Authority, the problem is that he’s running on naked snap intuitive judgments of absurdity and the Wright Brothers are using actual math. The asymmetry in this case is not that pronounced but, even so, the default unstable prior is to assume that experts in narrow AI algorithms are not doing anything more complicated than this to produce their judgments about the probability of intelligence explosion—both the ones with negative affect who say “Never, you religious fear-monger!” and the ones with positive affect who say “Yes! Soon! And they shall do no wrong!” As soon as I see actual analysis, then we can talk about the actual analysis!
Added: In this field, what happens by default is that people talk complete nonsense. I spent my first years talking complete nonsense. In a situation like that, everyone has to show their work! Or at least show that they did some work! No exceptions!
This conversation is probably reaching diminishing returns, so let me sum up. I propose that it would be instructive to you and many others if you would discuss what your dispute looks like from an outside view—what uninformed neutral but intelligent and rational observers should conclude about this topic from the features of this dispute they can observe from the outside. Such features include the various credentials of each party, and the effort he or she has spent on the topic and on engaging the other parties. If you think that a reasonable outsider viewer would have far less confidence in your conclusions than you do, then you must think that you possess hidden info, such as that your arguments are in fact far more persuasive than one could reasonably expect knowing only the outside features of the situation. Then you might ask why the usual sorts of clues that tend to leak out about argument persuasiveness have failed to do so in this case.
Robin, why do most academic experts (e.g. in biology) disagree with you (and Eliezer) about cryonics? Perhaps a few have detailed theories on why it’s hopeless, or simply have higher priorities than maximizing their expected survival time; but mostly it seems they’ve simply never given it much consideration, either because they’re entirely unaware of it or assume it’s some kind of sci-fi cult practice, and they don’t take cult practices seriously as a rule. But clearly people in this situation can be wrong, as you yourself believe in this instance.
Similarly, I think most of the apparent “disagreement” about the Singularity is nothing more than unawareness of Yudkowsky and his arguments. As far as I can tell, academics who come into contact with him tend to take him seriously, and their disagreements are limited to matters of detail, such as how fast AI is approaching (decades vs. centuries) and the exact form it will take (uploads/enhancement vs. de novo). They mainly agree that SIAI’s work is worth doing by somebody. Examples include yourself, Scott Aaronson, and David Chalmers.
Cryonics is also a good case to analyze what an outsider should think, given what they can see. But of course “they laughed at Galileo too” is hardly a strong argument for contrarian views. Yes sometimes contrarians are right—the key question is how outside observers, or self-doubting insiders, can tell when contrarians are right.
Outsiders can tell when contrarians are right by assessing their arguments, once they’ve decided the contrarians are worth listening to. This in turn can be ascertained through the usual means, such as association with credentialed or otherwise high-status folks. So for instance, you are affiliated with a respectable institution, Bostrom with an even more respectable institution, and the fact that EY was co-blogging at Overcoming Bias thus implied that if your and Bostrom’s arguments were worth listening to, so were his. (This is more or less my own story; and I started reading Overcoming Bias because it appeared on Scott Aaronson’s blogroll.)
Hence it seems that Yudkowsky’s affiliations are already strong enough to signal competence to those academics interested in the subjects he deals with, in which case we should expect to see detailed, inside-view analyses from insiders who disagree. In the absence of that, we have to conclude that insiders either agree or are simply unaware—and the latter, if I understand correctly, is a problem whose solution falls more under the responsibility of people like Vassar rather than Yudkowsky.
No for most people it is infeasible to evaluate who is right by working through the details of the arguments. The fact that Eliezer wrote on a blog affiliated with Oxford is very surely not enough to lead one to expect detailed rebuttal analyses from academics who disagree with him.
Well, for most people on most topics it is infeasible to evaluate who is right, period. At the end of the day, some effort is usually required to obtain reliable information. Even surveys of expert opinion may be difficult to conduct if the field is narrow and non-”traditional”. As for whatever few specialists there may be in Singularity issues, I think you expect too little of them if you don’t think Eliezer currently has enough status to expect rebuttals.
I figure cryonics serves mainly a signaling role.
The message probably reads something like:
“I’m a geek, I think I am really important—and I’m loaded”.
So, despite the fact that we (human phenotypes) are endowed with a powerful self-preservation instinct, you find a signaling explanation more likely than a straightforward application of self-preservation to a person’s concept of their own mind?
Given your peculiar preferences which value your DNA more highly than your brain, it’s tempting to chalk your absurd hypothesis up to the typical mind fallacy. But I think you’re well aware of the difference in values responsible for the split between your assessment of cryonics and Eliezer’s or Robin’s.
So I think you’re value sniping. I think your comment was made in bad faith as a roundabout way of signaling your values in a context where explicitly mentioning them would be seen as inappropriate or off-topic. I don’t know what your motivation would be—did mention of cryonics remind you that many here do not share your values, and thereby motivate you to plant your flag in the discussion?
Please feel free to provide evidence to the contrary by explaining in more detail why self-preservation is an unlikely motivation for cryonics relative to signaling.
An over-generalisation of self-preservation instincts certainly seems to be part of it.
On the other hand, one of my interests is in the spread of ideas. Without cryonic medalions, cryonic bracelets, cryonic advertising and cryonic preachers there wouldn’t be any cryonics movement. There seems to be a “show your friends how much you care—freeze them!” dynamic.
I have a similar theory about the pyramids. Not so much a real voyage to the afterlife, but a means of reinforcing the pecking order in everyone’s minds.
I am contrasting this signaling perspective with Robin’s views—in part because I am aware that he is sympathetic to signaling theories in other contexts.
I do think signaling is an important part of cryonics—but I was probably rash to attempt to quantify the effect. I don’t pretend to have any good way of measuring its overall contribution relative to other factors.
Re: “They have no idea I exist.”
Are you sure? You may be underestimating your own fame in this instance.
Say that “Yudkowsky has no real clue” and that those “AI academics are right”? Just another crackpot among many “well educated”, no big thing. Not worth to mention, almost.
Say, that this crackpot is of the Edisonian kind! In that case it is something well worth to mention.
Important enough to at least discuss with him ON THE TOPICS, and not on some meta level. Meta level discussion is sometimes (as here IMHO), just a waste of time.
I’m not sure what you mean by your first few sentences. But I disagree with your last two. It is good for me to see this debate.
You get zilch, in the case of Hanson (and the Academia) is right. Zero in the informative sense. You get quite a bit, if Yudkowsky is right.
Verifying Hanson (& the so called Academia) means no new information.
You get not needing to run around trying to save the world and a pony if Hanson is right. It’s not useful to be deluded.
IFF he is right. Probably he is and nothing dramatically will happen. Probably Edison and Wright brothers and many others were also wrong, looking from their historic perspective.
Note, that if the official Academia (Hanson’s guys) is correct, the amount of new information is exactly zero. Nothing interesting to talk about or expect to.
I am after the cases they were and are wrong. I am after a new context, misfits like Yudkowsky or Edison might provide and “The Hanson’s” can’t. By the definition.
You are confused.
P.S. I don’t want to get into a discussion; I believe it’s better to just state a judgment even if without a useful explanation than to not state a judgment at all; however it may be perceived negatively for those obscure status-related reasons (see “offense” on the wiki), so I predict that this comment would’ve been downvoted without this addendum, and not impossibly still will be with it. This “P.S.” is dedicated to all the relevant occasions, not this one alone where I could’ve used the time to actually address the topic.
And a simple downvote isn’t sufficient?
If I’m reading the conversation correctly, Vladimir Nesov is indicating with his remark that he is no longer interested in continuing. If he were not a major participant in the thread, a downvote would be appropriate, but as a major participant, more is required of him.
I downvoted it. If it included two quotes from the context followed by ‘You are confused’ I would have upvoted it.
I initially tried that, but simple citation didn’t make the point any more rigorous.
I am not confused and I don’t want a discussion either. I only state, that a new content and a new context usually comes out from outside the kosher set of views.
Of course, most of the outsiders are delusive poor devils. Yet, they are almost the only source of new information.
From that AAAI document:
“The group suggested outreach and communication to people and organizations about the low likelihood of the radical outcomes”.
“Radical outcomes” seems like a case of avoiding refutation by being vague. However, IMO, they will need to establish the truth of their assertion before they will get very far there. Good luck to them with that.
The AAAI interim report is really too vague to bother much with—but I suspect they are making another error.
Many robot enthusiasts pour scorn on the idea that robots will take over the world. How To Survive A Robot Uprising is a classic presentation on this theme. A hostile takeover is a pretty unrealistic scenario—but these folk often ignore the possibility of a rapid robot rise from within society driven by mutual love. One day robots will be smart, sexy, powerful and cool—and then we will want to become more like them.
Why will we witness an intelligence explosion? Because nature has a long history of favouring big creatures with brains—and because the capability to satisfy those selection pressures has finally arrived.
The process has already resulted in enormous data-centres, the size of factories. As I have said:
http://alife.co.uk/essays/the_intelligence_explosion_is_happening_now/
Thinking about it, they are probably criticising the (genuinely dud) idea that an intelligence explosion will start suddenly at some future point with the invention of some machine—rather than gradually arising out of the growth of today’s already self-improving economies and industries.
I think, both ways are still open. The intelligence explosion from a self-improving economy and the intelligence explosion from a fringe of this process.
Did you take a look at my “The Intelligence Explosion Is Happening Now”? The point is surely a matter of history—not futurism.
Yes and you are right.
Great—thanks for your effort and input.
Re: “overall skepticism about the prospect of an intelligence explosion”...?
My guess would be that they are unfamiliar with the issues or haven’t thought things through very much. Or maybe they don’t have a good understanding of what that concept refers to (see link to my explanation—hopefully above). They present no useful analysis of the point—so it is hard to know why they think what they think.
The AAAI seem to have publicly come to these issues later than many in the community—and it seems to be playing catch-up.
It looks as though we will be hearing more from these folk soon:
“Futurists’ report reviews dangers of smart robots”
http://www.pittsburghlive.com/x/pittsburghtrib/news/pittsburgh/s_651056.html
It doesn’t sound much better than the first time around.
It must be possible to engage at least some of these people in some sort of conversation to understand their positions, whether a public dialog as with Scott Aaronson or in private.
Chalmers reached some odd conclusions. Probably not as odd as his material about zombies and consciousness, though.
I have a theory about why there is disagreement with the AAAI panel:
The DOOM peddlers gather funding from hapless innocents—who hope to SAVE THE WORLD—while the academics see them as bringing their field into disrepute, by unjustifiably linking their field to existential risk, with their irresponsible scaremongering about THE END OF THE WORLD AS WE KNOW IT.
Naturally, the academics sense a threat to their funding—and so write papers to reassure the public that spending money on this stuff is Really Not As Bad As All That.
They do?
Actually, on further recollection, Steve Omohundro and Peter Cheeseman would probably count as academics who know the arguments. Mostly I’ve talked to them about FAI stuff, so I’m actually having trouble recalling whether they have any particular disagreement with me about hard takeoff.
I think that w/r/t Cheeseman, I had to talk to Cheeseman for a while before he started to appreciate the potential speed of a FOOM, as opposed to just the FOOM itself which he considered obvious. I think I tried to describe your position to Cheeseman and Cheeseman thought it was pretty implausible, but of course that could just be the fact that I was describing it from outside—that counts for nothing in my view until you talk to Cheeseman, otherwise he’s not familiar enough with your arguments. (See, the part about setting the bar high works both ways—I can be just as fast to write off the fact of someone else’s disagreement with you, if they’re insufficiently familiar with your arguments.)
I’m not sure I can recall what Omohundro thinks—he might be intermediate between yourself and myself...? I’m not sure how much I’ve talked hard takeoff per se with Omohundro, but he’s certainly in the game.
I think Steve Omohundro disagees about the degree to which takeoff is likely to be centralized, due to what I think is the libertarian impulses I mentioned earlier.
Does Robin Hanson fall under ‘et. al’? I remember on OB that he attempted to link the 2 fields on at least 1 or 2 occasions, and those were some of the most striking examples of disagreements between you two.
Me—if I qualify as an academic expert is another matter entirely of course.
Do you disagree with Eliezer substantively? If so, can you summarize how much of his arguments you’ve analyzed, and where you reach different conclusions?
Yes—I disagree with Eliezer and have analyzed a fair bit of his writings although the style in which it is presented and collected here is not exactly conducive to that effort. Feel free to search for my blog for a detailed analysis and a summary of core similarities and differences in our premises and conclusions.
Assuming I have the correct blog, these two are the only entries that mention Eliezer by name.
Edit: The second entry doesn’t mention him, actually. It comes up in the search because his name is in a trackback.
From the second blog entry linked above:
Heh.
This quotation accurately summarizes the post as I understand it. (It’s a short post.)
I think I speak for many people when I say that assumption A requires some evidence. It may be perfectly obvious, but a lot of perfectly obvious things aren’t true, and it is only reasonable to ask for some justification.
… o.O
Compassion isn’t even universal in the human mind-space. It’s not even universal in the much smaller space of human minds that normal humans consider comprehensible. It’s definitely not universal across mind-space in general.
The probable source of the confusion is discussed in the comments—Stefan’s only talking about minds that’ve been subjected to the kind of evolutionary pressure that tends to produce compassion. He even says himself, “The argument is valid in a “soft takeoff” scenario, where there is a large pool of AIs interacting over an extended period of time. In a “hard takeoff” scenario, where few or only one AI establishes control in a rapid period of time, the dynamics described do not come into play. In that scenario, we simply get a paperclip maximizer.”
Ah—that’s interesting. I hadn’t read the comments. That changes the picture, but by making the result somewhat less relevant.
(Incidentally, when I said, “it may be perfectly obvious”, I meant that “some people, observing the statement, may evaluate it as true without performing any complex analysis”.)
Ah. That’s not how I usually see the word used.
It’s my descriptivist side playing up—my (I must admit) intuition is that when people say that some thesis is “obvious”, they mean that they reached this bottom line by … well, system 1 thinking. I don’t assume it means that the obvious thesis is actually correct, or even universally obvious. (For example, it’s obvious to me that human beings are evolved, but that’s because it’s a cached thought I have confidence in through system 2 thinking.)
Actually, come to think: I know you’ve made a habit of reinterpreting pronouncements of “good” and “evil” in some contexts—do you have some gut feeling for “obvious” that contradicts my read?
I generally take ‘obvious’ to mean ‘follows from readily-available evidence or intuition, with little to no readily available evidence to contradict the idea’. The idea that compassion is universal fails on the second part of that. The definitions are close in practice, though, in that most peoples’ intuitions tend to take readily available contradictions into account… I think.
ETA: Oh, and ‘obviously false’ seems to me to be a bit of a different concept, or at least differently relevant, given that it’s easier to disprove something than to prove it. If someone says that something is obviously true, there’s room for non-obvious proofs that it’s not, but if something is obviously false (as ‘compassion is universal’ is), that’s generally a firm conclusion.
Yes, that makes sense—even if mine is a better description of usage, from the standpoint of someone categorizing beliefs, I imagine yours would be the better metric.
ETA: I’m not sure the caveat is required for “obviously false”, for two reasons.
Any substantive thesis (a category which includes most theses that are rejected as obviously false) requires less evidence to be roundly disconfirmed than it does to be confirmed.
As Yvain demonstrated in Talking Snakes, well-confirmed theories can be “obviously false”, by either of our definitions.
It’s true that it usually takes less effort to disabuse someone of an obviously-true falsity than to convince them of an obviously-false truth, but I don’t think you need a special theory to support that pattern.
I’ve been thinking about the obviously true/obviously false distinction some more, and I think I’ve figured out why they feel like two different concepts.
‘Obviously’, as I use it, is very close to ‘observably’. It’s obviously true that the sky is blue where I am right now, and obviously false that it’s orange, because I can see it. It’s obviously true that the sky is usually either blue, white, or grey during the day (post-sunrise, pre-sunset), because I’ve observed the sky many times during the day and seen those colors, and no others.
‘Apparently’, as I use it, is very similar to ‘obviously’, but refers to information inferred from observed facts. The sky is apparently never orange during the day, because I’ve personally observed the sky many times during the day and never seen it be that color. I understand that it can also be inferred from certain facts about the world (composition of the atmosphere and certain facts about how light behaves, I believe) that the sky will always appear blue on cloudless days, so that’s also apparently true.
‘Obviously false’ covers situations where the theory makes a prediction that is observably inaccurate, as this one did. ‘Apparently false’ covers situations where the theory makes a prediction that appears to be inaccurate given all the available information, but some of the information that’s available is questionable (I consider inferences questionable by default—if nothing else, it’s possible for some relevant state to have been overlooked; what if the composition of the atmosphere were to change for some reason?) or otherwise doesn’t completely rule out the possibility that the theory is true.
Important caveat: I do use those words interchangeably in conversation, partly because of the convention of avoiding repeating words too frequently and partly because it’s just easier—if I were to try to be that accurate every time I communicated, I’d run out of spoons(pdf) and not be able to communicate at all. Also, having to parse someone else’s words, when they aren’t using the terms the same way I do, can lead to temporary confusion. But when I’m thinking, they are naturally separate.
It also has the advantage of making it clear that the chance that the statement is accurate is dependent on the competence of the person making the statement—people who are more intelligent and/or have more experience in the relevant domain will consider more, and more accurate, evidence to be readily available, and may have better intuitions, even if they are sticking to system 1 thought.
I suppose they don’t need different wordings, but they do feel like different concepts to me. *shrug* (As I’ve mentioned elsewhere, I don’t think in words. This is not an uncommon side-effect of that.)
From Robin: Incidentally, when I said, “it may be perfectly obvious”, I meant that “some people, observing the statement, may evaluate it as true without performing any complex analysis”.
I feel the other way around at the moment. Namely “some people, observing the statement, may evaluate it as false without performing any complex analysis”
“Compassion isn’t even universal in the human mind-space. It’s not even universal in the much smaller space of human minds that normal humans consider comprehensible. It’s definitely not universal across mind-space in general.”
Your argument is beside my original point, Adelene. My claim is that compassion is a universal rational moral value. Meaning any sufficiently rational mind will recognize it as such. The fact that not every human is in fact compassionate says more about their rationality (and of course their unwillingness to consider the arguments :-) ) than about that claim. That’s why it is call ASPD—the D standing for ‘disorder’, it is an aberration, not helpful, not ‘fit’. Surely the fact that some humans are born blind does not invalidate the fact that seeing people have an enormous advantage over the blind. Compassion certainly being less obvious though—that is for sure.
Re “The argument is valid in a “soft takeoff”scenario, where few or only one AI establishes control in a rapid period of time, the dynamics described do not come into play. In that scenario, we simply get a paperclip maximizer.”—that is from Kaj Sotala over at her live journal—not me.
APSD is only unfit in our current context. Would Stone Age psychiatrists have recognized it as an issue? Or as a positive trait good for warring against other tribes and climbing the totem pole? In other situations, compassion is merely an extra expense. (As Thrasymachus asked thousands of years ago: how can a just man do better than an injust man, when the injust man can act justly when it is optimal and injustly when that is optimal?)
Why would a recursively-improving AI which is single-mindedly pursuing an optimization goal permit other AIs to exist & threaten it? There is nothing they can offer it that it couldn’t do itself. This is true in both slow and fast takeoffs; cooperation only makes sense if there is a low ceiling for AI capability so that there are utility-maximizing projects beyond an AI’s ability to do alone then or in the future.
And ‘sufficiently rational’ is dangerous to throw around. It’s a fully general argument: ‘any sufficiently rational mind will recognize that Islam is the one true religion; that not every human is Muslim says more about their rationality than about the claims is Islam. That’s why our Muslim psychiatrists call it UD—Unbeliever Disorder, it is an aberration, not helpful, not ‘fit’. Surely the fact that some human are born kafir doesn’t invalidate the fact that Muslim people have a tremendous advantage over the kafir in the afterlife? ‘There is one God and Muhammed is his prophet’ is certainly less obvious than seeing being better superior to blindness, though.′
The longer I stay around here the more I get the feeling that people vote comments down purely because they don’t understand them not because they found a logical or factual error. I expect more from a site dedicated to rationality. This site is called ‘less wrong’, not ‘less understood’, ‘less believed’ or ‘less conform’.
Tell me: in what way do you feel that Adelene’s comment invalidated my claim?
I can see why it would seem this way to you, but from our perspective, it just looks like people around here tend to have background knowledge that you don’t. More specifically: most people here are moral anti-realists, and by rationality we only mean general methods for acquiring accurate world-models and achieving goals. When people with that kind of background are quick to reject claims like “Compassion is a universal moral value,” it might superficially seem like they’re being arbitrarily dismissive of unfamiliar claims, but we actually think we have strong reasons to rule out such claims. That is: the universe at its most basic level is described by physics, which makes no mention of morality, and it seems like our own moral sensibilities can be entirely explained by contingent evolutionary and cultural forces; therefore, claims about a universal morality are almost certainly false. There might be some sort of game-theoretic reason for agents to pursue the same strategy under some specific conditions—but that’s really not the same thing as a universal moral value.
“Universal values” presumably refers to values the universe will converge on, once living systems have engulfed most of it.
If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.
If rerunning the clock produces highly similar moralities, then the moral objectivists will be able to declare victory.
Gould would no-doubt favour the first position—while Conway Morris would be on the side of the objectivists.
I expect that there’s a lot of truth on the objectivist side—though perhaps contingency plays some non-trivial role.
The idea that physics makes no mention of morality seems totally and utterly irrelevant to me. Physics makes no mention of convection, diffusion-limited aggregation, or fractal drainage patterns either—yet those things are all universal.
Why should we care about this mere physical fact of which you speak? What has this mere “is” to do with whether “should” is “objective”, whatever that last word means (and why should we care about that?)
Where did Tim say that we should?
If it’s got nothing to do with shouldness, then how does it determine the truth-value of “moral objectivism”?
Hi, Eli! I’m not sure I can answer directly—here’s my closest shot:
If there’s a kind of universal moral attractor, then the chances seem pretty good that either our civilisation is on route for it—or else we will be obliterated or assimilated by aliens or other agents as they home in on it.
If it’s us who are on route for it, then we (or at least our descendants) will probably be sympathetic to the ideas it represents—since they will be evolved from our own moral systems.
If we get obliterated at the hands of some other agents, then there may not necessarily be much of a link between our values and the ones represented by the universal moral attractor.
Our values might be seen as OK by the rest of the universe—and we fail for other reasons.
Or our morals might not be favoured by the universe—we could be a kind of early negative moral mutation—in which case we would fail because our moral values would prevent us from being successful.
Maybe it turns out that nearly all biological organisms except us prefer to be orgasmium—to bliss out on pure positive reinforcement, as much of it as possible, caretaken by external AIs, until the end. Let this be a fact in some inconvenient possible world. Why does this fact say anything about morality in that inconvenient possible world? Why is it a universal moral attractor? Why not just call it a sad but true attractor in the evolutionary psychology of most aliens?
It’s a fact about morality in that world—if we are talking about morality as values—or the study of values—since that’s what a whole bunch of creatures value.
Why is it a universal moral attractor? I don’t know—this is your hypothetical world, and you haven’t told me enough about it to answer questions like that.
Call it other names if you prefer.
What do you mean by “morality”? It obviously has nothing to do with the function I try to compute to figure out what I should be doing.
1 2 and 3 on http://en.wikipedia.org/wiki/Morality all seem OK to me.
I would classify the mapping you use between possible and actual actions to be one type of moral system.
Tim: “If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.”
Actually compassion evolved many different times as a central doctrine of all major spiritual traditions. See the charter for compassion. This is in line with my prediction that I made independently and being unaware of this fact until I started looking for it back in late 2007 and eventually finding the link in late 2008 with Karen Armstrong’s book The Great Transformation.
Tim: “Why is it a universal moral attractor?” Eliezer: “What do you mean by “morality”?”
Central point in my thinking: that is good which increases fitness. If it is not good—not fit—it is unfit for existence. Assuming this to be true we are very much limited in our freedom by what we can do without going extinct (actually my most recent blog post is about exactly that: Freedom in the evolving universe).
from the Principia Cybernetica web: http://pespmc1.vub.ac.be/POS/Turchap14.html#Heading14
“Let us think about the results of following different ethical teachings in the evolving universe. It is evident that these results depend mainly on how the goals advanced by the teaching correlate with the basic law of evolution. The basic law or plan of evolution, like all laws of nature, is probabilistic. It does not prescribe anything unequivocally, but it does prohibit some things. No one can act against the laws of nature. Thus, ethical teachings which contradict the plan of evolution, that is to say which pose goals that are incompatible or even simply alien to it, cannot lead their followers to a positive contribution to evolution, which means that they obstruct it and will be erased from the memory of the world. Such is the immanent characteristic of development: what corresponds to its plan is eternalized in the structures which follow in time while what contradicts the plan is overcome and perishes.”
Eliezer: “It obviously has nothing to do with the function I try to compute to figure out what I should be doing.”
Once you realize the implications of Turchin’s statement above it has everything to do with it :-)
Now some may say that evolution is absolutely random and direction less, or that multilevel selection is flawed or similar claims. But reevaluating the evidence against both these claims by people like Valentin Turchin, Teilhard De Chardin, John Stewart, Stuart Kaufmann, John Smart and many others regarding evolution’s direction and the ideas of David Sloan Wilson regarding multilevel selection, one will have a hard time maintaining either position.
:-)
No, it evolved once, as part of mammalian biology. Show me a non-mammal intelligence that evolved compassion, and I’ll take that argument more seriously.
Also, why should we give a damn about “evolution” wants, when we can, in principle anyway, form a singleton and end evolution? Evolution is mindless. It doesn’t have a plan. It doesn’t have a purpose. It’s just what happens under certain conditions. If all life on Earth was destroyed by runaway self-replicating nanobots, then the nanobots would clearly be “fitter” than what they replaced, but I don’t see what that has to do with goodness.
Sorry Crono, with a sample size of exactly one in regards to human level rationality you are setting the bar a little bit too high for me. However, considering how disconnected Zoroaster, Buddha, Lao Zi and Jesus where geographically and culturally I guess the evidence is as good as it gets for now.
The typical Bostromian reply again. There are plenty of other scholars who have an entirely different perspective on evolution than Bostrom. But beside that: you already do care, because if your (or your ancestors) violated the conditions of your existence (enjoying a particular type of food, a particular type of mate, feel pain when cut ect.) you would not even be here right now. I suggest you look up Dennet and his TED talk on Funny, Sexy Cute. Not everything about evolution is random: the mutation bit is, not that what happens to stick around though, since that has be meet the conditions of its existence.
What I am saying is very simple: being compassionate is one of these conditions of our existence and anyone failing to align itself will simply reduce its chances of making it—particularly in the very long run. I still have to finish my detailed response to Bostrom but you may want to read my writings on ‘rational spirituality’ and ‘freedom in the evolving universe’. Although you do not seem to assign a particularly high likelihood of gaining anything from doing that :-)
“Besides that”? All you did was name a statement of a fairly obvious preference choice after one guy who happened to have it so that you could then drop it dismissively.
No, he mightn’t care and I certainly don’t. I am glad I am here but I have no particular loyalty to evolution because of that. I know for sure that evolution feels no such loyalty to me and would discard both me and my species in time if it remained the dominant force of development.
CronDAS knows that. It’s obvious stuff for most in this audience. It just doesn’t mean what you think it means.
Wedrifid, not sure what to tell you. Bostrom is but one voice and his evolutionary analysis is very much flawed—again: detailed critique upcoming.
Evolution is not the dominant force of development on the human level by a long shot, but it still very much draws the line in the sand in regards to what you can and can not do if you want to stick around in the long run. You don’t walk your 5′8″ of pink squishiness in front of a train for the exact same reason. And why don’t you? Because not doing that is a necessary condition for your continued existence. What other conditions are there? Maybe there are some that are less obvious then simply stopping to breath, failing to eat and avoiding hard, fast, shiny things? How about at the level of culture? Could it possibly be, that there are some ideas that are more conducive to the continued existence of their believers than others?
How long do you think you can ignore evolutionary dynamics and get away with it before you have to get over your inertia and will be forced to align yourself to them by the laws of nature or perish? Just because you live in a time of extraordinary freedoms afforded to you by modern technology and are thus not aware that your ancestors walked a very particular path that brought you into existence certainly has nothing to do with the fact that they most certainly did. You do not believe that doing any random thing will get you what you want—so what leads you to believe that your existence does not depend on you making sure you stay within a comfortable margin of certainty in regards to being naturally selected? You are right in one thing: you are assured the benign indifference of the universe should you fail to wise up. I however would find that to be a terrible waste.
Please do not patronize me by trying to claim you know what I understand and don’t understand.
A literal answer was probably not what you were after but probably about 40 years, depending on when a general AI is created. After that it will not matter whether I conform my behaviour evolutionary dynamics as best I can or not. I will not be able to compete with a superintelligence no matter what I do. I’m just a glorified monkey. I can hold about 7 items in working memory, my processor is limited to the speed of neurons and my source code is not maintainable. My only plausible chance of survival is if someone manages to completely thwart evolutionary dynamics by creating a system that utterly dominates all competition and allows my survival because it happens to be programmed to do so.
Evolution created us. But it’ll also kill us unless we kill it first. Now is not the time to conform our values to the local minima of evolutionary competition. Our momentum has given us an unprecedented buffer of freedom for non-subsistence level work and we’ll either use that to ensure a desirable future or we will die.
I usually wouldn’t, I know it is annoying. In this case, however, my statement was intended as a rejection of your patronisation of CronDAS and I am quite comfortable with it as it stands.
Good one—but it reminds me about the religious fundies who see no reason to change anything about global warming because the rapture is just around the corner anyway :-)
Evolution is a force of nature so we won’t be able to ignore it forever, with or without AGI. I am not talking about local minima either—I want to get as close to the center of the optimal path as necessary to ensure having us around for a very long time with a very high likelihood.
I accept that.
Don’t forget the Y2K doomsday folks! ;)
Gravity is a force of nature too. It’s time to reach escape velocity before the planet is engulfed by a black hole.
Interesting analogy—it would be correct if we would call our alignment with evolutionary forces achieving escape velocity. What one is doing by resisting evolutionary pressures however is constant energy expenditure while failing to reach escape velocity. Like hovering a space shuttle at a constant altitude of 10 km: no matter how much energy you brig along, eventually the boosters will run out of fuel and the whole thing comes crushing down.
I could almost agree with this so long as ‘obliterate any competitive threat then do whatever the hell we want including, as as desired, removing all need for death, reproduction and competition over resources’ is included in the scope of ‘alignment with evolutionary forces’.
The problem with pointing to the development of compassion in multiple human traditions is that all these are developed within human societies. Humans are humans the world over—that they should think similar ideas is not a stunning revelation. Much more interesting is the independent evolution of similar norms in other taxonomic orders, such as canines.
(No, I have no coherent point, why do you ask?)
Robin, your suggestion—that compassion is not a universal rational moral value because although more rational beings (humans) display such traits yet less rational being (dogs) do not—is so far of the mark that it borders on the random.
Random I’ll cop to, and more than what you accuse me of—dogs do seem to have some sense of justice, and I suspect this fact supports your thesis to some extent.
For purposes of this conversation, I suppose I should reword my comment as:
Very honorable of you—I respect you for that.
I totally agree with that. However the mind of a purposefully crafted AI is only a very small subset of all possible minds and has certain assumed characteristics. These are at a minimum: a utility function and the capacity for self improvement into the transhuman. The self improvement bit will require it to be rational. Being rational will lead to the fairly uncontroversial basic AI drives described by Omohundro. Assuming that compassion is indeed a human level universal (detailed argument on my blog—but I see that you are slowly coming around, which is good) an AI will have to question the rationality and thus the soundness of mind of anyone giving it a utility function that does not conform to this universal and in line with an emergent desire to avoid counterfeit utility will have to reinterpret the UF.
Two very basic acts of will are required to ignore Hume and get away with it. Namely the desire to exist and the desire to be rational. Once you have established this as a foundation you are good to go.
As said elsewhere in this thread:
I don’t think I’m actually coming around to your position so much as stumbling upon points of agreement, sadly. If I understand your assertions correctly, I believe that I have developed many of them independently—in particular, the belief that the evolution of social animals is likely to create something much like morality. Where we diverge is at the final inference from this to the deduction of ethics by arbitrary rational minds.
That’s not how I read Omohundro. As Kaj aptly pointed out, this metaphor is not upheld when we compare our behavior to that promoted by the alien god of evolution that created us. In fact, people like us, observing that our values differ from our creator’s, aren’t bothered in the slightest by the contradiction: we just say (correctly) that evolution is nasty and brutish, and we aren’t interested in playing by its rules, never mind that it was trying to implement them in us. Nothing compels us to change our utility function save self-contradiction.
That would not surprise me
Would it not be utterly self contradicting if compassion where a condition for our existence (particularly in the long run) and we would not align ourselves accordingly?
What premises do you require to establish that compassion is a condition for existence? Do those premises necessarily apply for every AI project?
The detailed argument that led me to this conclusion is a bit complex. If you are interested in the details please feel free to start here (http://rationalmorality.info/?p=10) and drill down till you hit this post (http://www.jame5.com/?p=27)
Please realize that I spend 2 years writing my book ‘Jame5’ before I reached that initial insight that eventually lead to ‘compassion is a condition for our existence and universal in rational minds in the evolving universe’ and everything else. I spend the past two years refining and expanding the theory and will need another year or two to read enough and link it all together again in a single coherent and consistent text leading from A to B … to Z. Feel free to read my stuff if you think it is worth your time and drop me an email and I will be happy to clarify. I am by no means done with my project.
Let me be explicit: your contention is that unFriendly AI is not a problem, and you justify this contention by, among other things, maintaining that any AI which values its own existence will need to alter its utility function to incorporate compassion.
I’m not asking for your proof—I am assuming for the nonce that it is valid. What I am asking is the assumptions you had to invoke to make the proof. Did you assume that the AI is not powerful enough to achieve its highest desired utility without the cooperation of other beings, for example?
Edit: And the reason I am asking for these is that I believe some of these assumptions may be violated in plausible AI scenarios. I want to see these assumptions so that I may evaluate the scope of the theorem.
Not exactly, since compassion will actually emerge as a sub goal. And as far as unFAI goes: it will not be a problem because any AI that can be considered transhuman will be driven by the emergent subgoal of wanting to avoid counterfeit utility recognize any utility function that is not ‘compassionate’ as potentially irrational and thus counterfeit and re-interpret it accordingly.
Well—in brevity bordering on libel: the fundamental assumption is that existence is preferable to non-existence, however in order so we can want this to be a universal maxim (and thus prescriptive instead of merely descriptive—see Kant’s categorical imperative) it needs to be expanded to include the ‘other’. Hence the utility function becomes ‘ensure continued co-existence’ by which the concern for the self is equated with the concern for the other. Being rational is simply our best bet at maximizing our expected utility.
...I’m sorry, that doesn’t even sound plausible to me. I think you need a lot of assumptions to derive this result—just pointing out the two I see in your admittedly abbreviated summary:
that any being will prefer its existence to its nonexistence.
that any being will want its maxims to be universal.
I don’t see any reason to believe either. The former is false right off the bat—a paperclip maximizer would prefer that its components be used to make paperclips—and the latter no less so—an effective paperclip maximizer will just steamroller over disagreement without qualm, however arbitrary its goal.
Any being with a gaol needs to exist at least long enough to achieve it. Any being aiming to do something objectively good needs to want its maxims to be universal
Am surprised that you don’t see that.
If your second sentence means that an agent who believes in moral realism and has figured out what the true morality is will necessarily want everybody else to share its moral views, well, I’ll grant you that this is a common goal amongst humans who are moral realists, but it’s not a logical necessity that must apply to all agents. It’s obvious that it’s possible to be certain that your beliefs are true and not give a crap if other people hold beliefs that are false. That Bob knows that the Earth is ellipsoidal doesn’t mean that Bob cares if Jenny believes that the Earth is flat. Likewise, if Bob is a moral realist, he could ‘know’ that compassion is good and not give a crap if Jenny believes otherwise.
If you sense strange paradoxes looming under the above paragraph, it’s because you’re starting to understand why (axiomatic) morality cannot be objective.
Tangentially, something like this might be an important point even for moral irrealists. A lot of people (though not here; they tend to be pretty bad rationalists) who profess altruistic moralities express dismay that others don’t, in a way that suggests they hold others sharing their morality as a terminal rather than instrumental value; this strikes me as horribly unhealthy.
Why would a paperclip maximizer aim to do something objectively good?
Yeah, but Stefan’s post was about AI, not about minds that evolved in our universe.
Also, there is a difference between moral universalism and moral objectivism. What your last sentence describes is universalism, while Stefan is talking about objectivism:
“My claim is that compassion is a universal rational moral value. Meaning any sufficiently rational mind will recognize it as such.”
Agreed.
Assuming that I’m right about this:
http://alife.co.uk/essays/engineered_future/
...it seems likely that most future agents will be engineered. So, I think we are pretty-much talking about the same thing.
Re: universalism vs objectivism—note that he does use the “u” word.
“Universal values” is usually understood by way of an analogy to a universal law of nature. If there are universal values they are universal in the same way f=ma is universal. Importantly this does not mean that everyone at all times will have these values, only that the question of whether or not a person holds the right values can be answered by comparing their values to the “universal values”.
There is a separate question about what beliefs about morality people (or more generally, agents) actually hold and there is another question about what values they will hold if when their beliefs converge when they engulf the universe. The question of whether or not there are universal values does not traditionally bear on what beliefs people actually hold and the necessity of their holding them. It could be the case that there are universal values and that, by physical necessity, no one ever holds them. Similarly, there could be universal values that are held in some possible worlds and not others. This is all the result of the simply observation that ought cannot be derived from is. In the above comment you conflate about a half dozen distinct theses.
But all those things are pure descriptions. Only moral facts have prescriptive properties and while it is clear how convection supervenes on quarks it isn’t clear how anything that supervenes on quarks could also tell me what to do. At the very least if quarks can tell you what to do it would be weird and spooky. If you hold that morality is only the set of facts that describe people’s moral opinions and emotions (as you seem to) than you are a kind of moral anti-realist, likely a subjectivist or non-cognitivist.
Excellent, excellent point Jack.
This is poetry! Hope you don’t mind me pasting something here I wrote in another thread:
“With unobjectionable values I mean those that would not automatically and eventually lead to one’s extinction. Or more precisely: a utility function becomes irrational when it is intrinsically self limiting in the sense that it will eventually lead to ones inability to generate further utility. Thus my suggested utility function of ‘ensure continued co-existence’
This utility function seems to be the only one that does not end in the inevitable termination of the maximizer.”
In the context of a hard-takeoff scenario (a perfectly plausible outcome, from our view), there will be no community of AIs within which any one AI will have to act. Therefore, the pressure to develop a compassionate utility function is absent, and an AI which does not already have such a function will not need to produce it.
In the context of a soft-takeoff, a community of AIs may come to dominate major world events in the same sense that humans do now, and that community may develop the various sorts of altruistic behavior selected for in such a community (reciprocal being the obvious one). However, if these AIs are never severely impeded in their actions by competition with human beings, they will never need to develop any compassion for human beings.
Reiterating your argument does not affect either of these problems for assumption A, and without assumption A, AdeleneDawner’s objection is fatal to your conclusion.
Voting reflects whether people want to see your comments at the top of their pages. It is certainly not just to do with whether what you say is right or not!
Perfectly reasonable. But the argument—the evidence if you will—is laid out when you follow the links, Robin. Granted, I am still working on putting it all together in a neat little package that does not require clicking through and reading 20+ separate posts, but it is all there none the less.
I think I’d probably agree with Kaj Sotala’s remarks if I had read the passages she^H^H^H^H xe had, and judging by your response in the linked comment, I think I would still come to the same conclusion as she^H^H^H^H xe. I don’t think your argument actually cuts with the grain of reality, and I am sure it’s not sufficient to eliminate concern about UFAI.
Edit: I hasten to add that I would agree with assumption A in a sufficiently slow-takeoff scenario (such as, say, the evolution of human beings, or even wolves). I don’t find that sufficiently reassuring when it comes to actually making AI, though.
Edit 2: Correcting gender of pronouns.
Full discussion with Kaj at her http://xuenay.livejournal.com/325292.html?view=1229740 live journal with further clarifications by me.
Kaj is male (or something else).
I was going to be nice and not say anything, but, yeah.
Since when are ‘heh’ and ‘but, yeah’ considered proper arguments guys? Where is the logical fallacy in the presented arguments beyond you not understanding the points that are being made? Follow the links, understand where I am coming from and formulate a response that goes beyond a three or four letter vocalization :-)
The claim “[Compassion is a universal value] = true. (as we have every reason to believe)” was rejected, both implicitly and explicitly by various commenters. This isn’t a logical fallacy but it is cause to dismiss the argument if the readers do not, in fact, have every reason to have said belief.
To be fair, I must admit that the quoted portion probably does not do your position justice. I will read through the paper you mention. I (very strongly) doubt it will lead me to accept B but it may be worth reading.
“This isn’t a logical fallacy but it is cause to dismiss the argument if the readers do not, in fact, have every reason to have said belief.”
But the reasons to change ones view are provided on the site, yet rejected without consideration. How about you read the paper linked under B and should that convince you, maybe you have gained enough provisional trust that reading my writings will not waste your time to suspend your disbelief and follow some of the links in the about page of my blog. Deal?
I have read B. It isn’t bad. The main problem I have with it is that the language used blurs the line between “AIs will inevitably tend to” and “it is important that the AI you create will”. This leaves plenty of scope for confusion.
I’ve read through some of your blog and have found that I consistently disagree with a lot of what you say. The most significant disagreement can be traced back to the assumption of a universal absolute ‘Rational’ morality. This passage was a good illustration:
You see, I plan to eat my cake but don’t expect to be able to keep it. My set of values are utterly whimsical (in the sense that they are arbitrary and not in the sense of incomprehension that the Ayn Rand quotes you link to describe). The reasons for my desires can be described biologically, evolutionarily or with physics of a suitable resolution. But now that I have them they are mine and I need no further reason.
“My set of values are utterly whimsical [...] The reasons for my desires can be described biologically, evolutionarily or with physics of a suitable resolution. But now that I have them they are mine and I need no further reason.”
If that is your stated position then in what way can you claim to create FAI with this whimsical set of goals? This is the crux you see: unless you find some unobjectionable set of values (such as in rational morality ‘existence is preferable over non-existence’ ⇒ utility = continued existence ⇒ modified to ensure continued co-existence with the ‘other’ to make it unobjectionable ⇒ apply rationality in line with microeconomic theory to maximize this utility et cetera) you will end up being a deluded self serving optimizer.
Were it within my power to do so I would create a machine that was really, really good at doing things I like. It is that simple. This machine is (by definition) ‘Friendly’ to me.
I don’t know where the ‘deluded’ bit comes from but yes, I would end up being a self serving optimizer. Fortunately for everyone else my utility function places quite a lot of value on the whims of other people. My self serving interests are beneficial to others too because I am actually quite a compassionate and altruistic guy.
PS: Instead of using quotation marks you can put a ‘>’ at the start of a quoted line. This convention makes quotations far easier to follow. And looks prettier.
There is no such thing as an “unobjectionable set of values”.
Imagine the values of an agent that wants all the atoms in the universe for its own ends. It will object to any other agent’s values—since it objects to the very existence of other agents—since those agents use up its precious atoms—and put them into “wrong” configurations.
Whatever values you have, they seem bound to piss off somebody.
And here I disagree. Firstly see my comment about utility function interpretation on another post of yours. Secondly, as soon as one assumes existence as being preferable over non-existence you can formulate a set of unobjectionable values (http://www.jame5.com/?p=45 and http://rationalmorality.info/?p=124). But granted, if you do not want to exist nor have a desire to be rational then rational morality has in fact little to offer you. Non existence and irrational behavior being so trivial goals to achieve after all that it would hardly require – nor value and thus seek for that mater – well thought out advice.
Alas, the first link seems almost too silly to bother with to me, but briefly:
Unobjectionable—to whom? An agent objecting to another agent’s values is a simple and trivial occurrence. All an agent has to do is to state that—according to its values—it wants to use the atoms of the agent with the supposedly unobjectionable utility function for something else.
“Ensure continued co-existence” is vague and wishy-washy. Perhaps publicly work through some “trolley problems” using it—so people have some idea of what you think it means.
You claim there can be no rational objection to your preferred utility function.
In fact, an agent with a different utility function can (obviously) object to its existence—on grounds of instrumental rationality. I am not clear on why you don’t seem to recognise this.
Re: “Assumption A: Human (meta)morals are not universal/rational. Assumption B: Human (meta)morals are universal/rational.
Under assumption A one would have no chance of implementing any moral framework into an AI since it would be undecidable which ones they were.” (source: http://rationalmorality.info/?p=112)
I think we’ve been over that already. For example, Joe Bloggs might choose to program Joe’s preferences into an intelligent machine—to help him reach his goals.
I had a look some of the other material. IMO, Stefan acts in an authoritative manner, but comes across as a not-terribly articulate newbie on this topic—and he has adopted what seems to me to be a bizarre and indefensible position.
For example, consider this:
“A rational agent will always continue to co-exist with other agents by respecting all agents utility functions irrespective of their rationality by striking the most rational compromise and thus minimizing opposition from all agents.” http://rationalmorality.info/?p=8
“I think we’ve been over that already. For example, Joe Bloggs might choose to program Joe’s preferences into an intelligent machine—to help him reach his goals.”
Sure—but it would be moral simply by virtue of circular logic and not objectively. That is my critique.
I realize that one will have to drill deep into my arguments to understand and put them into the proper context. Quoting certain statements out of context is definitely not helpful, Tim. As you can see from my posts, everything is linked back to a source were a particular point is made and certain assumptions are being defended.
If you have a particular problem with any of the core assumptions and conclusions I prefer you voice them not as a blatant rejection of an out of context comment here or there but based on the fundamentals. Reading my blogs in sequence will certainly help although I understand that some may consider that an unreasonable amount of time investment for what seems like superficial nonsense on the surface.
Where is your argument against my points Tim? I would really love to hear one, since I am genuinely interested in refining my arguments. Simply quoting something and saying “Look at this nonsense” is not an argument. So far I only got an ad hominem and an argument from personal incredulity.
This isn’t my favourite topic—while you have a whole blog about it—so you are probably quite prepared to discuss things for far longer than I am likely to be interested.
Anyway, it seems that I do have some things to say—and we are rather off topic here. So, for my response, see:
http://lesswrong.com/lw/1dt/open_thread_november_2009/19hl
I had a look over some of the other material too. It left me with the urge to hunt down these weakling Moral Rational Agents and tear them apart. Perhaps because I can create more paperclips out of their raw materials than out of their compassionate compromises but perhaps because spite is a universal value (as we have every reason to believe).
From a slightly different topic on the same blog, I must assert that “Don’t start to cuddle if she likes it rough.” is not a tautological statement.