(I would have liked to reply to the deleted comment, but you can’t reply to deleted comments so I’ll reply to the repost.)
EDIT: Roko reveals that he was actually never asked to delete his comment! Disregard parts of the rest of this comment accordingly.
I don’t think Roko should have been requested to delete his comment. I don’t think Roko should have conceded to deleting his comment.
The correct reaction when someone posts something scandalous like
I was once criticized by a senior singinst member for not being prepared to be tortured or raped for the cause
is not to attempt to erase it, even if that was possible, but to reveal the context. The context, supposedly, would make it seem less scandalous—for example, maybe it was a private dicussion about philosophical hypotheticals. If it wouldn’t, that’s a bad sign about SIAI.
The fact that that erasure was the reaction suggests that there is no redeeming context!
That someone asked Roko to erase his comment isn’t a very bad sign, since it’s enough that one person didn’t understand the reasoning above for that to happen. That fact that Roko conceded is a bad sign, though.
Now SIAI should save face not by asking a moderator to delete wfg’s reposts, but by revealing the redeeming context in which the scandalous remarks that Roko alluded to were made.
Roko may have been thinking of [just called him, he was thinking of it] a conversation we had when he and I were roommates in Oxford while I was visiting the Future of Humanity Institute, and frequently discussed philosophical problems and thought experiments. Here’s the (redeeming?) context:
As those who know me can attest, I often make the point that radical self-sacrificing utilitarianism isn’t found in humans and isn’t a good target to aim for. Almost no one would actually take on serious harm with certainty for a small chance of helping distant others. Robin Hanson often presents evidence for this, e.g. this presentation on “why doesn’t anyone create investment funds for future people?” However, sometimes people caught up in thoughts of the good they can do, or a self-image of making a big difference in the world, are motivated to think of themselves as really being motivated primarily by helping others as such. Sometimes they go on to an excessive smart sincere syndrome, and try (at the conscious/explicit level) to favor altruism at the severe expense of their other motivations: self-concern, relationships, warm fuzzy feelings.
Usually this doesn’t work out well, as the explicit reasoning about principles and ideals is gradually overridden by other mental processes, leading to exhaustion, burnout, or disillusionment. The situation winds up worse according to all of the person’s motivations, even altruism. Burnout means less good gets done than would have been achieved by leading a more balanced life that paid due respect to all one’s values. Even more self-defeatingly, if one actually does make severe sacrifices, it will tend to repel bystanders.
Instead, I typically advocate careful introspection and the use of something like Nick Bostrom’s parliamentary model:
The idea here is that moral theories get more influence the more probable they are; yet even a relatively weak theory can still get its way on some issues that the theory think are extremely important by sacrificing its influence on other issues that other theories deem more important. For example, suppose you assign 10% probability to total utilitarianism and 90% to moral egoism (just to illustrate the principle). Then the Parliament would mostly take actions that maximize egoistic satisfaction; however it would make some concessions to utilitarianism on issues that utilitarianism thinks is especially important. In this example, the person might donate some portion of their income to existential risks research and otherwise live completely selfishly.
In the conversation with Roko, we were discussing philosophical thought experiments (trolley problem style, which may indeed be foolish ) to get at ‘real’ preferences and values for such an exercise. To do that, one often does best to adopt the device of the True Prisoner’s Dilemma and select positive and negative payoffs that actually have emotional valence (as opposed to abstract tokens). For positive payoffs, we used indefinite lifespans of steady “peak experiences” involving discovery, health, status, and elite mates. For negative payoffs we used probabilities of personal risk of death (which comes along with almost any effort, e.g. driving to places) and harms that involved pain and/or a decline in status (since these are separate drives). Since we were friends and roommates without excessive squeamishness, hanging out at home, we used less euphemistic language.
Neither of us was keen on huge sacrifices in Pascal’s-Mugging-like situations, viewing altruism as only one part of our respective motivational coalitions, or one term in bounded utility functions. I criticized his past “cheap talk” of world-saving as a primary motivation, given that in less convenient possible worlds, it was more easily overcome than his phrasing signaled. I said he should scale back his claims of altruism to match the reality, in the way that I explicitly note my bounded do-gooding impulses.
We also differed in our personal views on the relative badness of torture, humiliation and death. For me, risk of death was the worst, which I was least willing to trade off in trolley-problem type cases to save others. Roko placed relatively more value on the other two, which I jokingly ribbed and teased him about.
In retrospect, I was probably a bit of a jerk in pushing (normative) Hansonian transparency. I wish I had been more careful to distinguish between critiquing a gap between talk and values, and critiquing the underlying values, and probably should just take wedifrid’s advice on trolley-problem-type scenarios generally.
First off, great comment—interesting, and complex.
But, some things still don’t make sense to me...
Assuming that what you described led to:
I was once criticized by a senior singinst member for not being prepared to be tortured or raped for the cause. I mean not actually, but, you know, in theory. Precommiting to being prepared to make a sacrifice that big. shrugs
How did precommitting enter in to it?
Are you prepared to be tortured or raped for the cause? Have you precommitted to it?
Have other SIAI people you know of talked about this with you, have other SIAI people precommitted to it?
What do you think of others who do not want to be tortured or raped for the cause?
I find this whole line of conversation fairly ludicrous, but here goes:
Number 1. Time-inconsistency: we have different reactions about an immediate certainty of some bad than a future probability of it. So many people might be willing to go be a health worker in a poor country where aid workers are commonly (1 in 10,000) raped or killed, even though they would not be willing to be certainly attacked in exchange for 10,000 times the benefits to others. In the actual instant of being tortured anyone would break, but people do choose courses of action that carry risk (every action does, to some extent), so the latter is more meaningful for such hypotheticals.
Number 2. I have driven and flown thousands of kilometers in relation to existential risk, increasing my chance of untimely death in a car accident or plane crash, so obviously I am willing to take some increased probability of death. I think I would prefer a given chance of being tortured to a given chance of death, so obviously I care enough to take at least some tiny risk from what I said above. As I also said above, I’m not willing to make very big sacrifices (big probabilities of such nasty personal outcomes) for tiny shifts in probabilities of big impersonal payoffs (like existential risk reduction). In realistic scenarios, that’s what “the cause” would refer to. I haven’t made any verbal or explicit “precommitment” or promises or anything like that.
In sufficiently extreme (and ludicrously improbable) trolley-problem style examples, e.g. “if you push this button you’ll be tortured for a week, but if you don’t then the Earth will be destroyed (including all your loved ones) if this fair coin comes up heads, and you have incredibly (impossibly?) good evidence that this really is the setup” I hope I would push the button, but in a real world of profound uncertainty, limited evidence, limited personal power (I am not Barack Obama or Bill Gates), and cognitive biases, I don’t expect that to ever happen. I also haven’t made any promises or oaths about that.
I am willing to give of my time and effort, and forgo the financial rewards of a more lucrative career, in exchange for a chance for efficient do-gooding, interaction with interesting people who share my values, and a meaningful project. Given diminishing returns to money in rich countries today, and the ease of obtaining money for folk with high human capital, those aren’t big sacrifices, if they are sacrifices at all.
Number 3. SIAIers love to be precise and analytical and consider philosophical thought experiments, including ethical ones. I think most have views pretty similar to mine, with somewhat varying margins. Certainly Michael Vassar, the head of the organization, is also keen on recognizing one’s various motives and living a balanced life, and avoiding fanatics. Like me, he actively advocates Bostrom-like parliamentary model approaches to combining self-concern with parochial and universalist altruistic feelings.
I have never heard anyone making oaths or promises to make severe sacrifices.
Number 4. This is a pretty ridiculous question. I think that’s fine and normal, and I feel more comfortable with such folk than the alternative. I think people should not exaggerate that do-gooding is the most important thing in their life lest they deceive themselves and others about their willingness to make such choices, which I criticized Roko for.
This sounds very sane, and makes me feel a lot better about the context. Thank you very much.
I very much like the idea that top SIAI people believe that there is such a thing as too much devotion to the cause (and, I’m assuming, actively talk people who are above that level down as you describe doing for Roko).
As someone who has demonstrated impressive sanity around these topics, you seem to be in a unique position to answer these questions with an above-average level-headedness:
Do you understand the math behind the Roko post deletion?
Do you understand the math behind the Roko post deletion?
Yes, his post was based on (garbled versions of) some work I had been doing at FHI, which I had talked about with him while trying to figure out some knotty sub-problems.
What do you think about the Roko post deletion?
I think the intent behind it was benign, at least in that Eliezer had his views about the issue (which is more general, and not about screwed-up FAI attempts) previously, and that he was motivated to prevent harm to people hearing the idea and others generally. Indeed, he was explicitly motivated enough to take a PR hit for SIAI.
Regarding the substance, I think there are some pretty good reasons for thinking that the expected value (with a small probability of a high impact) of the info for the overwhelming majority of people exposed to it would be negative, although that estimate is unstable in the face of new info.
It’s obvious that the deletion caused more freak-out and uncertainty than anticipated, leading to a net increase in people reading and thinking about the content compared to the counterfactual with no deletion. So regardless of the substance about the info, clearly it was a mistake to delete (which Eliezer also recognizes).
What do you think about future deletions?
Obviously, Eliezer is continuing to delete comments reposting on the topic of the deleted post. It seems fairly futile to me, but not entirely. I don’t think that Less Wrong is made worse by the absence of that content as such, although the fear and uncertainty about it seem to be harmful. You said you were worried because it makes you uncertain about whether future deletions will occur and of what.
After about half an hour of trying, I can’t think of another topic with the same sorts of features. There may be cases involving things like stalkers or bank PINs or 4chan attacks or planning illegal activities. Eliezer called on people not to discuss AI at the beginning of Less Wrong to help establish its rationality focus, and to back off from the gender warfare, but hasn’t used deletion powers for such things.
Less Wrong has been around for 20 months. If we can rigorously carve out the stalker/PIN/illegality/spam/threats cases I would be happy to bet $500 against $50 that we won’t see another topic banned over the next 20 months.
Less Wrong has been around for 20 months. If we can rigorously carve out the stalker/PIN/illegality/spam/threats cases I would be happy to bet $500 against $50 that we won’t see another topic banned over the next 20 months.
That sounds like it’d generate some perverse incentives to me.
Well, if counterfactually Roko hadn’t wanted to take it down I think it would have been even more of a mistake to delete it, because then the author would have been peeved, not just the audience/commenters.
But Eliezer’s comments on the subject suggest to me that he doesn’t think that.
More specifically, they suggest that he thinks the most important thing is that the post not be viewable, and if we can achieve that by quietly convincing the author to take it down, great, and if we can achieve it by quietly deleting it without anybody noticing, great, and if we can’t do either of those then we achieve it without being quiet, which is less great but still better than leaving it up.
And it seemed to me your parenthetical could be taken to mean that he agrees with you that deleting it would be a mistake in all of those cases, so I figured I would clarify (or let myself be corrected, if I’m misunderstanding).
So many people might be willing to go be a health worker in a poor country where aid workers are commonly (1 in 10,000) raped or killed, even though they would not be willing to be certainly attacked in exchange for 10,000 times the benefits to others.
I agree with your main point, but the thought experiment seems to be based on the false assumption that the risk of being raped or murdered are smaller than 1 in 10K if you stay at home. Wikipedia guesstimates that 1 in 6 women in the US are on the receiving end of attempted rape at some point, so someone who goes to a place with a 1 in 10K chance of being raped or murdered has probably improved their personal safety. To make a better thought experiment, I suppose you have to talk about the marginal increase in rape or murder rate when working in the poor country when compared to staying home, and perhaps you should stick to murder since the rape rate is so high.
I don’t think Roko should have been requested to delete his comment. I don’t think Roko should have conceded to deleting his comment.
Roko was not requested to delete his comment. See this parallel thread. (I would appreciate it if you would edit your comment to note this, so readers who miss this comment don’t have a false belief reinforced.) (ETA: thanks)
The correct reaction when someone posts something scandalous like
I was once criticized by a senior singinst member for not being prepared to be tortured or raped for the cause
is not to attempt to erase it, even if that was possible, but to reveal the context.… Now SIAI should save face not by asking a moderator to delete wfg’s reposts....
Agreed (and I think the chance of wfg’s reposts being deleted is very low, because most people get this). Unfortunately, I know nothing about the alleged event (Roko may be misdescribing it, as he misdescribed my message to him) or its context.
Roko’s reply to me strongly suggested that he interpreted my message as requesting deletion, and that I was the cause of him deleting it. I doubt anyone at SIAI would have explicitly requested deletion.
I’m wondering whether you, Nick, have learned anything from this experience—something perhaps about how attempting to hide something is almost always counterproductive?
Of course, Roko contributed here by deleting the message, you didn’t create this mess by yourself. But you sure have helped. :)
Well, look, I deleted it of my own accord, but only after being prompted that it was a bad thing to have posted. Can we just drop this? It makes me look like even more of a troublemaker than I already look like, and all I really want to do is finish the efficient charity competition then get on with life outside teh intenetz.
Will you at least publicly state that you precommit, on behalf of CEV, to not apply negative incentives in this case? (Roko, Jul 24, 2010 1:37 PM)
This is very important. If the SIAI is the organisation to solve the friendly AI problem and implement CEV then it should be subject to public examination, especially if they ask for money.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially. And we have those. The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans. However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
Addendum: I see others have been asking “but what do you actually mean?” for a couple of years now.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This strikes me as a demand for particular proof. SIAI is small (and was much smaller until the last year or two), the set of people engaged in FAI research is smaller, Eliezer has chosen to focus on writing about rationality over research for nearly four years, and FAI is a huge problem, in which any specific subproblem should be expected to be underdeveloped at this early stage. And while I and others expect work to speed up in the near future with Eliezer’s attention and better organization, yes, we probably are dead.
The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans.
Somewhat nitpickingly, this is a reason for FAI in general. CEV is attractive mostly for moving as much work from the designers to the FAI as possible, reducing the potential for uncorrectable error, and being fairer than letting the designers lay out an object-level goal system.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially.… However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
This sounds interesting; do you think you could expand?
It wasn’t intended to be—more incredulity. I thought this was a really important piece of the puzzle, so expected there’d be something at all by now. I appreciate your point: that this is a ridiculously huge problem and SIAI is ridiculously small.
However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
This sounds interesting; do you think you could expand?
I meant that, as I understand it, CEV is what is fed to the seed AI. Or the AI does the work to ascertain the CEV. It requires an intelligence to ascertain the CEV, but I’d think the ascertaining process would be reasonably set out once we had an intelligence on hand, artificial or no. Or the process to get to the ascertaining process.
I thought we needed the CEV before the AI goes FOOM, because it’s too late after. That implies it doesn’t take a superintelligence to work it out.
Thus: CEV would have to be a process that mere human-level intelligences could apply. That would be a useful process to have, and doesn’t require first creating an AI.
I must point out that my statements on the subject are based in curiosity, ignorance and extrapolation from what little I do know, and I’m asking (probably annoyingly) for more to work with.
“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are, in my opinion, the best we can hope for, and better than we are likely to get from any other entity involved in this.
Besides, I think that XiXiDu, et al are complaining about the difference between cotton and silk, when what is actually likely to happen is more like a big kick in the teeth from reality. SIAI is imperfect. Yes. Well done. Nothing is perfect. At least cut them a bit of slack.
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are the best we can hope for, and better than we are likely to get from any other entity involved in this.
What?!? Open source code—under a permissive license—is the traditional way to signal that you are not going to run off into the sunset with the fruits of a programming effort. Private assurances are usually worth diddly-squat by comparison.
I think that you don’t realize just how bad the situation is. You want that silken sheet. Rude awakening methinks. Also open-source not neccessarily good for FAI in any case.
I think that you don’t realize just how bad the situation is.
I don’t think that you realize how bad it is. I’d rather have the universe being paperclipped than supporting the SIAI if that means that I might be tortured for the rest of infinity!
To the best of my knowledge, SIAI has not planned to do anything, under any circumstances, which would increase the probability of you or anyone else being tortured for the rest of infinity.
Supporting SIAI should not, to the best of my knowledge, increase the probability of you or anyone else being tortured for the rest of infinity.
But imagine there was a person a level above yours that went to create some safeguards for an AGI. That person would tell you that you can be sure that the safeguards s/he plans to implement will benefit everyone. Are you just going to believe that? Wouldn’t you be worried and demand that their project is being supervised?
You are in a really powerful position because you are working for an organisation that might influence the future of the universe. Is it really weird to be skeptical and ask for reassurance of their objectives?
Logical rudeness is the error of rejecting an argument for reasons other than disagreement with it. Does your “I don’t think so” mean that you in fact believe that SIAI (possibly) plans to increase the probability of you or someone else being tortured for the rest of eternity? If not, what does this statement mean?
I removed that sentence. I meant that I didn’t believe that the SIAI plans to harm someone deliberately. Although I believe that harm could be a side-effect and that they would rather harm a few beings than allowing some Paperclip maximizer to take over.
You can call me a hypocrite because I’m in favor of animal experiments to support my own survival. But I’m not sure if I’d like to have someone leading an AI project who thinks like me. Take that sentence to reflect my inner conflict. I see why one would favor torture over dust specks but I don’t like such decisions. I’d rather have the universe to end now, or having everyone turned into paperclips, than having to torture beings (especially if I am the being).
I feel uncomfortable that I don’t know what will happen because there is a policy of censorship being favored when it comes to certain thought experiments. I believe that even given negative consequences, transparency is the way to go here. If the stakes are this high, people who believe will do anything to get what they want. That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
Better yet, you could use a kind of doublethink—and then even actually mean it. Here is W. D. Hamilton on that topic:
A world where everyone else has been persuaded to be altruistic is a good one to live in from the point of view of pursuing our own selfish ends. This hypocracy is even more convincing if we don’t admit it even in our thoughts—if only on our death beds, so to speak, we change our wills back to favour the carriers of our own genes.
Discriminating Nepotism—as reprinted in: Narrow Roads of Gene Land, Volume 2 Evolution of Sex, p.356.
That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
For 30 years I have been wondering, what indication of its existence might we expect from a true AI? Certainly not any explicit revelation, which might spark a movement to pull the plug. Anomalous accumulation or creation of wealth might be a sign, or an unquenchable thirst for raw information, storage space, and processing cycles, or a concerted attempt to secure an uninterrupted, autonomous power supply. But the real sign, I suspect, would be a circle of cheerful, contented, intellectually and physically well-nourished people surrounding the AI.
I think many people would like to be in that group—if they can find a way to arrange it.
Unless AI was given that outcome (cheerful, contented people etc) as a terminal goal, or that circle of people was the best possible route to some other terminal goal, both of which are staggeringly unlikely, Dyson suspects wrongly.
If you think he suspects rightly, I would really like to see a justification. Keep in mind that AGIs are currently not being built using multi-agent environment evolutionary methods, so any kind of ‘social cooperation’ mechanism will not arise.
Machine intelligence programmers seem likely to construct their machines so as to help them satisfy their preferences—which in turn is likely to make them satisfied. I am not sure what you are talking about—but surely this kind of thing is already happening all the time—with Sergey Brin, James Harris Simons—and so on.
That doesn’t really strike me as a stunning insight, though. I have a feeling that I could find many people who would like to be in almost any group of “cheerful, contented, intellectually and physically well-nourished people.”
This all depends on what the AI wants. Without some idea of its utility function, can we really speculate? And if we speculate, we should note those assumptions. People often think of an AI as being essentially human-like in its values, which is problematic.
It’s a fair description of today’s more successful IT companies. The most obvious extrapolation for the immediate future involves more of the same—but with even greater wealth and power inequalities. However, I would certainly also council caution if extrapolating this out more than 20 years or so.
Currently, there are no entities in physical existence which, to my knowledge, have the ability to torture anyone for the rest of eternity.
You intend to build an entity which would have that ability (or if not for infinity, for a googolplex of subjective years).
You intend to give it a morality based on the massed wishes of humanity—and I have noticed that other people don’t always have my best interests at heart. It is possible—though unlikely—that I might so irritate the rest of humanity that they wish me to be tortured forever.
Therefore, you are, by your own statements, raising the risk of my infinite torture from zero to a tiny non-zero probability. It may well be that you are also raising my expected reward enough for that to be more than counterbalanced, but that’s not what you’re saying—any support for SIAI will, unless I’m completely misunderstanding, raise the probability of infinite torture for some individuals.
That just pushes the problem along a step. IF the Last Judge can’t be mistaken about the results of the AI running AND the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe. That’s very far from saying there’s a zero probability.
IF … the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe.
If the Last Judge peeks at the output and finds that it’s going to decide to torture people, that doesn’t imply abandoning FAI, it just requires fixing the bug and trying again.
Just because AGIs have capability to inflict infinite torture, doesn’t mean they have a motive. Also, status quo (with regard to SIAI’s activity) doesn’t involve nothing new happening.
I explained that he is planning to supply one with a possible motive (namely that the CEV of humanity might hate me or people like me). It is precisely because of this that the problem arises. A paperclipper, or any other AGI whose utility function had nothing to do with humanity’s wishes, would have far less motive to do this—it might kill me, but it really would have no motive to torture me.
If someone solicits for you to “trust in me”, alarm bells should start ringing immediately. If you really think that is “the best we can hope for”, then perhaps revisit that.
Gollum gave his private assurances to Frodo—and we all know how that turned out.
Well I’m convinced. Frodo should definitely have worked out a way to clone the ring and made sure the information was available to all of Middle Earth. You can never have too many potential Ring-Wraiths.
I’d also like to see His Dark Materials with rationalist!Lyra. The girl had an alethiometer. She should have kicked way more ass than she did as soon as she realized what she had.
Open source AGI is not a good thing. In fact, it would be a disastrously bad thing. Giving people the source code doesn’t just let them inspect it for errors, it also lets them launch it themselves. If you get an AGI close to ready for launch, then sharing its source code means that instead of having one party to decide whether there are enough safety measures ready to launch, you have many parties individually deciding whether to launch it themselves, possibly modifying its utility function to suit their own whim, and the hastiest party’s AGI wins.
Ideally, you’d want to let people study the code, but only trustworthy people, and in a controlled environment where they can’t take the source code with them. But even that is risky, since revealing that you have an AGI makes you a target for espionage and attack by parties who shouldn’t be trusted with humanity’s future.
Actually it reduces the chance of any party drawing massively ahead of the rest. It acts as an equalising force, by power-sharing. Since one of the main things we want to avoid is a disreputable organisation using machine intelligence to gain an advantage—and sustaining it over a long period of time. Using open-source software helps to defend against that possibility.
Machine intelligence will be a race—but it will be a race, whether participants share code or not.
Having said all that, machine intelligence protected by patents with secret source code on a server somewhere does seem like a reasonably probable outcome.
Using open-source software helps to defend against that possibility.
Only if (a) there is no point at which AGIs “foom”, (b) source code sharing is well enough enforced on everyone that no bad organizations combine open source with refinements that they keep secret for an advantage, (c) competing AIs form a stable power equilibrium at all points along their advancement, and (d) it is impossible to trade off goal system stability for optimization power.
I estimate probabilities of 0.4, 0.2, 0.05, and 0.5 for these hypotheses, respectively.
I disagree with most of that analysis. I assume machine intelligence will catalyse its own creation. I fully expect that some organisations will stick with secret source code. How could the probability of that possibly be as low as 0.8!?!
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
Nor am I assuming that no team will ever win. If there is to be a winner, we want the best possible lead up. The “trust us” model is not it—not by a long shot.
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
There are two problems with this reasoning. First, you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
And second, if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
Really? I just provided an example of a mechanism that helps keep open source software projects ethical—the fact that if the manufacturers attempt to exploit their customers it is much easier for the customers to switch to a more ethical fork—because creating such a fork no longer violates copyright law. Though you said you were pointing out problems with my reasoning, you didn’t actually point out any problems with that reasoning.
We saw an example of this kind of thing very recently—with LibreOffice. The developers got afraid that their adopted custodian, Oracle, was going to screw the customers of their project—so, to protect their customers and themselves, they forked it—and went their own way.
if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
If other people are running software that does bad things to you then running good quality software yourself most certainly is a kind of defense. It means you are better able to construct defenses, better able to anticipate their attacks—and so on. Better brains makes you more powerful.
Compare with the closed-source alternative: If other people are running software that does bad things to you—and you have no way to run such software yourself—since it is on their server and running secret source that is also protected by copyright law—you are probably pretty screwed.
It makes me look like even more of a troublemaker...
How so? I’ve just reread some of your comments on your now deleted post. It looks like you honestly tried to get the SIAI to put safeguards into CEV. Given that the idea spread to many people by now, don’t you think it would be acceptably to discuss the matter before one or more people take it serious or even consider to implement it deliberately?
Ok by me. It is pretty obvious by this point that there is no evil conspiracy involved here. But I think the lesson remains, I you delete something, even if it is just because you regret posting it, you create more confusion than you remove.
I think the question you should be asking is less about evil conspiracies, and more about what kind of organization SIAI is—what would they tell you about, and what would they lie to you about.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI. That alone is enough to conclude that the SIAI is not trying to hold back something that would discredit it as an organisation concerned with charitable objectives. The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence. Making the subject matter public did already harm some people and could harm people in future.
But the forbidden topic is already public. All the effects that would follow from it being public would already follow. THE HORSE HAS BOLTED. It’s entirely unclear to me what pretending it hasn’t does for the problem or the credibility of the SIAI.
Of course, if Clippy were clever he would then offer to sell SIAI a commitment to never release the UFAI in exchange for a commitment to produce a fixed number of paperclips per year, in perpetuity.
Admittedly, his mastery of human signaling probably isn’t nuanced enough to prevent that from sounding like blackmail.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI.
I really don’t see how that follows. Will more of the public take it seriously? As I have noted, so far the reaction from people outside SIAI/LW has been “They did WHAT? Are they IDIOTS?”
The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence.
That doesn’t make it not stupid or not counterproductive. Sincere stupidity is not less stupid than insincere stupidity. Indeed, sincere stupidity is more problematic in my experience as the sincere are less likely to back down, whereas the insincere will more quickly hop to a different idea.
Making the subject matter public did already harm some people
Hmm, okay. But that, I suggest, appears to have been a case of reasoning oneself stupid.
It does, of course, account for SIAI continuing to attempt to secure the stable doors after the horse has been dancing around in a field for several months taunting them with “COME ON IF YOU THINK YOU’RE HARD ENOUGH.”
(I upvoted XiXiDu’s comment here because he did actually supply a substantive response in PM, well deserving of a vote, and I felt this should be encouraged by reward.)
That someone asked Roko to erase his comment isn’t a very bad sign, since it’s enough that one person didn’t understand the reasoning above for that to happen. That fact that Roko conceded is a bad sign, though.
A kind of meta-question: is there any evidence suggesting that one of the following explanations of the recent deletion is better than another?
One of the more disturbing topics in this post is the question of how much can you trust an organization of people who are willing to endure torture, rape, and death for their cause.
Surely lying isn’t as bad as any of those...
Of course, lying for your cause is almost certainly a long term retarded thing to do… but so is censoring ideas...
As you may know from your study of marketing, accusations stick in the mind even when one is explicitly told they are false. In the parent comment and a sibling, you describe a hypothetical SIAI lying to its donors because… Roko had some conversations with Carl that led you to believe we care strongly about existential risk reduction?
If your aim is to improve SIAI, to cause there to be good organizations in this space, and/or to cause Less Wrong-ers to have accurate info, you might consider:
Talking with SIAI and/or with Fellows program alumni, so as to gather information on the issues you are concerned about. (I’d be happy to talk to you; I suspect Jasen and various alumni would too.) And then
Informing folks on LW of anything interesting/useful that you find out.
Anyone else who is concerned about any SIAI-related issue is also welcome to talk to me/us.
accusations stick in the mind even when one is explicitly told they are false
Actually that citation is about both positive and negative things—so unless you’re also asking pro-SIAI people to hush up, you’re (perhaps unknowingly) seeking to cause a pro-SIAI bias.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
One of the interesting morals from Roko’s contest is that if you care deeply about getting the most benefit per donated dollar you have to look very closely at who you’re giving it to.
Market forces work really well for lightbulb-sales businesses, but not so well for mom-and-pop shops, let alone charities. The motivations, preferences, and likely future actions of the people you’re giving money to become very important. Knowing if you can believe the person, in these contexts, becomes even more important.
As you note, I’ve studied marketing, sales, propaganda, cults, and charities. I know that there are some people who have no problem lying for their cause (especially if it’s for their god or to save the world).
I also know that there are some people who absolutely suck at lying. They try to lie, but the truth just seeps out of them.
That’s why I give Roko’s blurted comments more weight than whatever I’d hear from SIAI people who were chosen by you—no offence. I’ll still talk with you guys, but I don’t think a reasonably sane person can trust the sales guy beyond a point.
As far as your question goes, my primary desire is a public, consistent moderation policy for LessWrong. If you’re going to call this a community blog devoted to rationality, then please behave in sane ways. (If no one owns the blog—if it belongs to the community—then why is there dictatorial post deletion?)
I’d also like an apology from EY with regard to the chilling effects his actions have caused.
But back to what you replied to:
What would SIAI be willing to lie to donors about?
To answer your question, despite David Gerard’s advice:
I would not lie to donors about the likely impact of their donations, the evidence concerning SIAI’s ability or inability to pull off projects, how we compare to other organizations aimed at existential risk reduction, etc. (I don’t have all the answers, but I aim for accuracy and revise my beliefs and my statements as evidence comes in; I’ve actively tried to gather info on whether we or FHI reduce risk more per dollar, and I often recommend to donors that they do their own legwork with that charity comparison to improve knowledge and incentives). If a maniacal donor with a gun came searching for a Jew I had hidden in my house, or if I somehow had a “how to destroy the world” recipe and someone asked me how to use it, I suppose lying would be more tempting.
While I cannot speak for others, I suspect that Michael Vassar, Eliezer, Jasen, and others feel similarly, especially about the “not lying to one’s cooperative partners” point.
I suppose I should add “unless the actual answer is not a trolley problem” to my advice on not answering this sort of hypothetical ;-)
(my usual answer to hypotheticals is “we have no plans along those lines”, because usually we really don’t. We’re also really good at not having opinions on other organisations, e.g. Wikileaks, which we’re getting asked about A LOT because their name starts with “wiki”. A blog post on the subject is imminent. Edit:up now.)
It’s very easy to not lie when talking about the future. It is much easier to “just this once” lie about the past. You can do both, for instance, by explaining that you believe a project will succeed, even while withholding information that would convince a donor otherwise.
An example of this would be errors or misconduct in completing past projects.
Lack of relevant qualifications for people SIAI plans to employ on a project.
Or administrative errors and misconduct.
Or public relations / donor outreach misconduct.
To put the question another, less abstract way, have you ever lied to a SIAI donor? Do you know of anyone affiliated with SIAI who has lied a donor?
Hypothetically, If I said I had evidence in the affirmative to the second question, how surprising would that be to you? How much money would you bet that such evidence doesn’t exist?
You’re trying very hard to get everyone to think that SIAI has lied to donors or done something equally dishonest. I agree that this is an appropriate question to discuss, but you are pursuing the matter so aggressively that I just have to ask: do you know something we don’t? Do you think that you/other donors have been lied to on a particular occasion, and if so, when?
An example of this would be errors or misconduct in completing past projects.
When I asked Anna about the coordination between SIAI and FHI, something like “Do you talk enough with each other that you wouldn’t both spend resources writing the same research paper?”, she was told me about the one time that they had in fact both presented a paper on the same topic at a conference, and that they do now coordinate more to prevent that sort of thing.
I have found that Anna and others at SIAI are honest and forthcoming.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
Well, uh, yeah. The horse has bolted. It’s entirely unclear what choosing to keep one’s head in the sand gains anyone.
What would SIAI be willing to lie to donors about?
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Investigating the logic or lack thereof behind the (apparently ongoing) memory-holing is, however, incredibly on-topic and relevant for LW.
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Ambiguity is their ally. Both answers elicit negative responses, and they can avoid that from most people by not saying anything, so why shouldn’t they shut up?
I presume you’re not a native English speaker then—pretty much any moderately intelligent native English speaker has been forced to familiarity with 1984 at school. (When governments in the UK are being particularly authoritarian, there is often a call to send MPs copies of 1984 with a note “This is not a manual.”) Where are you from? Also, you really should read the book, then lots of the commentary on it :-) It’s one of the greatest works of science fiction and political fiction in English.
I can tell you all about equal pigs and newspeak but ‘memory-holing’ has not seemed to make as much of a cultural footprint—probably because as a phrase it is rather awkward fit. I wholeheartedly approve of Orwell in principle but actually reading either of his famous books sounds too much like highschool homework. :)
Animal Farm is probably passable (though it’s so short). 1984 on the other hand is maybe my favorite book of all time. I don’t think I’ve had a stronger emotional reaction to another book. It makes Shakespeare’s tragedies look like comedies. I’d imagine you’d have similar feelings about it based on what I’ve read of your comments here.
His less-famous novels aren’t as good. On the other hand, some of his essays are among the clearest, most intelligent thinking I’ve ever come across, and would probably be of a lot of interest to LessWrong readers...
Oh yeah. Politics and the English Language is a classic on a par with the great two novels. I first read that in 1992 and wanted to print copies to distribute everywhere (we didn’t have internet then).
Yeah, that’s one of those I was thinking of. Also things like the piece about the PEN ‘anti-censorship’ event that wasn’t, and his analysis of James Burnham’s Managerialist writing...
I’m terribly curious now—did the use of any of the phrases Orwell singles out in the article actually drop significantly after the article was published? Wikipedia will not say...
Well, reading it in the 1990s and having a burnt-out ex-Communist for a housemate at the time, I fear I recognised far too many of the cliches therein as current in those circles ;-)
A lot are still current in those less rational/more angry elements of the left who still think the Labour Party represents socialism and use phrases like that to justify themselves...
Because this is LessWrong—you can give a sane response and not only does it clear the air, people understand and appreciate it.
Cable news debating isn’t needed here.
Sure we might still wonder if they’re being perfectly honest, but saying something more sane on the topic than silence seems like a net-positive from their perspective.
LessWrongers are not magically free of bias. Nor are they inherently moral people that wouldn’t stoop to using misleading rhetorical techniques, though here they are more likely to be called on it.
In any case, an answer here is available to the public internet for all to see.
I respectfully disagree, and have my hopes set on Carl (or some other level-headed person in a position to know) giving a satisfying answer.
This is LessWrong after all—we can follow complicated arguments, and at least hearing how SIAI is actually thinking about such things would (probably) reduce my paranoia.
Yeah, but this is on the Internet for everyone to see. The potential for political abuse is ridiculous and can infect even LessWrong readers. Politics is the mind-killer, but pretending it doesn’t affect almost everyone else strikes me as not smart.
The concept of ethical injunctions is known in SIAI circles I think. Enduring personal harm for your cause and doing unethical things for your cause are therefore different. Consider Eliezer’s speculation about whether a rationalist “confessor” should ever lie in this post, too. And these personal struggles with whether to ever lie about SIAI’s work.
If banning Roko’s post would reasonably cause discussion of those ideas to move away from LessWrong, then by EY’s own reasoning (the link you gave) it seems like a retarded move.
If the idea is actually dangerous, it’s way less dangerous to people who aren’t familiar with pretty esoteric Lesswrongian ideas. They’re prerequisites to being vulnerable to it. So getting conversation about the idea away from Lesswrong isn’t an obviously retarded idea.
Lying for good causes has a time honored history. Protecting fugitive slaves or holocaust victims immediately comes to mind. Just because it is more often practical to be honest than not doesn’t mean that dishonesty isn’t sometimes unambiguously the better option.
I agree that there’s a lot in history, but the examples you cited have something that doesn’t match here—historically, you lie to people you don’t plan on cooperating with later.
If you lie to an oppressive government, it’s okay because it’ll either get overthrown or you’ll never want to cooperate with it (so great is your reason for lying).
Lying to your donor pool is very, very different than lying to the Nazis about hiding jews.
Is calling someone here Glenn Beck equivalent to Godwination?
wfg’s post strikes me as almost entirely reasonable (except the last question, which is pointless to ask) and your response as excessively defensive.
Also, you’re saying this to someone who says he’s a past donor and has not yet ruled out being a future donor. This is someone who could reasonably expect his questions to be taken seriously.
(I have some experience of involvement in a charity that suffers a relentless barrage of blitheringly stupid questions from idiots, and my volunteer role is media handling—mostly I come up with good and effective soundbites. So I appreciate and empathise with your frustration, but I think I can state with some experience behind me that your response is actually terrible.)
Okay. Given your and the folks who downvoted my comment’s perceptions, I’ll revise my opinion on the matter. I’ll also put that under “analogies not to use”; I was probably insufficiently familiar with the pop culture.
The thing I meant to say was just… Roko made a post, Nick suggested it gave bad impressions, Roko deleted it. wfg spent hours commenting again and again about how he had been asked to delete it, perhaps by someone “high up within SIAI”, and how future censorship might be imminent, how the fact that Roko had had a bascially unrelated conversation suggested that we might be lying to donors (a suggestion that he didn’t make explicitly, but rather left to innuendo), etc. I feel tired of this conversation and want to go back to research and writing, but I’m kind of concerned that it’ll leave a bad taste in readers mouths not because of any evidence that’s actually being advanced, but because innuendo and juxtapositions, taken out of context, leave impressions of badness.
I wish I knew how to have a simple, high-content, low-politics conversation on the subject. Especially one that was self-contained and didn’t leave me feeling as though I couldn’t bow out after awhile and return to other projects.
The essential problem is that with the (spectacular) deletion of the Forbidden Post, LessWrong turned into the sort of place where posts get disappeared. Those are not good places to be on the Internet. They are places where honesty is devalued and statements of fact must be reviewed for their political nature.
So it can happen here—because it did happen. It’s no longer in the class “things that are unthinkable”. This is itself a major credibility hit for LW.
And when a Roko post disappears—well, it was one of his posts that was disappeared before.
With this being the situation, assumptions of bad faith are going to happen. (And “stupidity” is actually the assumption of good faith.)
Your problem now is to restore trust in LW’s intellectual integrity, because SIAI broke it good and hard. Note that this is breaking an expectation, which is much worse than breaking a rule—if you break a rule you can say “we broke this rule for this reason”, but if you break expectations, people feel the ground moving under their feet, and get very upset.
There are lots of suggestions in this thread as to what people think might restore their trust in LW’s intellectual integrity, SIAI needs to go through them and work out precisely what expectations they broke and how to come clean on this.
I suspect you could at this point do with an upside to all this. Fortunately, there’s an excellent one: no-one would bother making all this fuss if they didn’t really care about LW. People here really care about LW and will do whatever they can to help you make it better.
(And the downside is that this is separate from caring about SIAI, but oh well ;-) )
(and yes, this sort of discussion around WP/WMF has been perennial since it started.)
The essential problem is that with the (spectacular) deletion of the Forbidden Post, LessWrong turned into the sort of place where posts get disappeared. Those are not good places to be on the Internet. They are places where honesty is devalued and statements of fact must be reviewed for their political nature.
I’ve seen several variations of this expressed about this topic, and it’s interesting to me, because this sort of view is somewhat foreign to me. I wouldn’t say I’m pro-censorship, but as an attorney trained in U.S. law, I think I’ve very much internalized the idea that the most serious sorts of censorship actions are those taken by the government (i.e., this is what the First Amendment free speech right is about, and that makes sense because of the power of the government), and that there are various levels of seriousness/danger beyond that, with say, big corporate censorship also being somewhat serious because of corporate power, and censorship by the owner of a single blog (even a community one) not being very serious at all, because a blogowner is not very powerful compared to the government or a major corporation, and shutting down one outlet of communication on the Internet is comparatively not a big deal because it’s a big internet where there are lots of other places to express one’s views. If a siteowner exercises his or her right to delete something on a website, it’s just not the sort of harm that I weigh very heavily.
What I’m totally unsure of is where the average LW reader falls on the scale between you and me, and therefore, despite the talk about the Roko incident being such a public relations disaster and a “spectacular” deletion, I just don’t know how true that is and I’m curious what the answer would be. People who feel like me may just not feel the need to weigh in on the controversy, whereas people who are very strongly anti-censorship in this particular context do.
If a siteowner exercises his or her right to delete something on a website, it’s just not the sort of harm that I weigh very heavily.
That’s not really the crux of the issue (for me, at least, and probably not for others). As David Gerard put it, the banning of Roko’s post was a blow to people’s expectations, which was why it was so shocking. In other words, it was like discovering that LW wasn’t what everyone thought it was (and not in a good way).
Note: I personally wouldn’t classify the incident as a “disaster,” but was still very alarming.
The essential problem is that with the (spectacular) deletion of the Forbidden Post, LessWrong turned into the sort of place where posts get disappeared. Those are not good places to be on the Internet. They are places where honesty is devalued and statements of fact must be reviewed for their political nature.
Like Airedale, I don’t have that impression—my impression is that 1) Censorship by website’s owner doesn’t have the moral problems associated with censorship by governments (or corporations), and 2) in online communities, dictatorship can work quite well, as long as the dictator isn’t a complete dick.
I’ve seen quite functional communities where the moderators would delete posts without warning if they were too stupid, offensive, repetitive or immoral (such as bragging about vandalizing wikipedia).
So personally, I don’t see a need for “restoring trust”. Of course, as your post attests, my experience doesn’t seem to generalize to other posters.
Y’know, one of the actual problems with LW is that I read it in my Internet as Television time, but there’s a REALLY PROMINENT SCORE COUNTER at the top left. This does not help in not treating it as a winnable video game.
(That said, could the people mass-downvoting waitingforgodel please stop? It’s tiresome. Please try to go by comment, not poster.)
I wish I knew how to have a simple, high-content, low-politics conversation on the subject.
This is about politics. The censorship of an idea related to a future dictator implementing some policy is obviously about politics.
You tell people to take friendly AI serious. You tell people that we need friendly AI to marshal our future galactic civilisation. People take it serious. Now the only organisation working on this is the SIAI. Therefore the SIAI is currently in direct causal control of our collective future. So why do you wonder people care about censorship and transparency? People already care about what the U.S. is doing and demand transparency. Which is ludicrous in comparison to the power of a ruling superhuman artificial intelligence that implements what the SIAI came up with as the seed for its friendliness.
If you really think that the SIAI has any importance and could possible achieve to influence or implement the safeguards for some AGI project, then everything the SIAI does is obviously very important to everyone concerned (everyone indeed).
I wish I knew how to have a simple, high-content, low-politics conversation on the subject. Especially one that was self-contained and didn’t leave me feeling as though I couldn’t bow out after awhile and return to other projects.
I wish you used a classification algorithm that more naturally identified the tension between “wanting low-politics conversation” and comparing someone to Glenn Beck as a means of criticism.
Sorry. This was probably simply a terrible mistake born of unusual ignorance of pop culture and current politics. I meant to invoke “using questions as a means to plant accusations” and honestly didn’t understand that he was radically unpopular. I’ve never watched anything by him.
(I would have liked to reply to the deleted comment, but you can’t reply to deleted comments so I’ll reply to the repost.)
EDIT: Roko reveals that he was actually never asked to delete his comment! Disregard parts of the rest of this comment accordingly.
I don’t think Roko should have been requested to delete his comment. I don’t think Roko should have conceded to deleting his comment.
The correct reaction when someone posts something scandalous like
is not to attempt to erase it, even if that was possible, but to reveal the context. The context, supposedly, would make it seem less scandalous—for example, maybe it was a private dicussion about philosophical hypotheticals. If it wouldn’t, that’s a bad sign about SIAI.
The fact that that erasure was the reaction suggests that there is no redeeming context!
That someone asked Roko to erase his comment isn’t a very bad sign, since it’s enough that one person didn’t understand the reasoning above for that to happen. That fact that Roko conceded is a bad sign, though.
Now SIAI should save face not by asking a moderator to delete wfg’s reposts, but by revealing the redeeming context in which the scandalous remarks that Roko alluded to were made.
Roko may have been thinking of [just called him, he was thinking of it] a conversation we had when he and I were roommates in Oxford while I was visiting the Future of Humanity Institute, and frequently discussed philosophical problems and thought experiments. Here’s the (redeeming?) context:
As those who know me can attest, I often make the point that radical self-sacrificing utilitarianism isn’t found in humans and isn’t a good target to aim for. Almost no one would actually take on serious harm with certainty for a small chance of helping distant others. Robin Hanson often presents evidence for this, e.g. this presentation on “why doesn’t anyone create investment funds for future people?” However, sometimes people caught up in thoughts of the good they can do, or a self-image of making a big difference in the world, are motivated to think of themselves as really being motivated primarily by helping others as such. Sometimes they go on to an excessive smart sincere syndrome, and try (at the conscious/explicit level) to favor altruism at the severe expense of their other motivations: self-concern, relationships, warm fuzzy feelings.
Usually this doesn’t work out well, as the explicit reasoning about principles and ideals is gradually overridden by other mental processes, leading to exhaustion, burnout, or disillusionment. The situation winds up worse according to all of the person’s motivations, even altruism. Burnout means less good gets done than would have been achieved by leading a more balanced life that paid due respect to all one’s values. Even more self-defeatingly, if one actually does make severe sacrifices, it will tend to repel bystanders.
Instead, I typically advocate careful introspection and the use of something like Nick Bostrom’s parliamentary model:
In the conversation with Roko, we were discussing philosophical thought experiments (trolley problem style, which may indeed be foolish ) to get at ‘real’ preferences and values for such an exercise. To do that, one often does best to adopt the device of the True Prisoner’s Dilemma and select positive and negative payoffs that actually have emotional valence (as opposed to abstract tokens). For positive payoffs, we used indefinite lifespans of steady “peak experiences” involving discovery, health, status, and elite mates. For negative payoffs we used probabilities of personal risk of death (which comes along with almost any effort, e.g. driving to places) and harms that involved pain and/or a decline in status (since these are separate drives). Since we were friends and roommates without excessive squeamishness, hanging out at home, we used less euphemistic language.
Neither of us was keen on huge sacrifices in Pascal’s-Mugging-like situations, viewing altruism as only one part of our respective motivational coalitions, or one term in bounded utility functions. I criticized his past “cheap talk” of world-saving as a primary motivation, given that in less convenient possible worlds, it was more easily overcome than his phrasing signaled. I said he should scale back his claims of altruism to match the reality, in the way that I explicitly note my bounded do-gooding impulses.
We also differed in our personal views on the relative badness of torture, humiliation and death. For me, risk of death was the worst, which I was least willing to trade off in trolley-problem type cases to save others. Roko placed relatively more value on the other two, which I jokingly ribbed and teased him about.
In retrospect, I was probably a bit of a jerk in pushing (normative) Hansonian transparency. I wish I had been more careful to distinguish between critiquing a gap between talk and values, and critiquing the underlying values, and probably should just take wedifrid’s advice on trolley-problem-type scenarios generally.
First off, great comment—interesting, and complex.
But, some things still don’t make sense to me...
Assuming that what you described led to:
How did precommitting enter in to it?
Are you prepared to be tortured or raped for the cause? Have you precommitted to it?
Have other SIAI people you know of talked about this with you, have other SIAI people precommitted to it?
What do you think of others who do not want to be tortured or raped for the cause?
Thanks, wfg
I find this whole line of conversation fairly ludicrous, but here goes:
Number 1. Time-inconsistency: we have different reactions about an immediate certainty of some bad than a future probability of it. So many people might be willing to go be a health worker in a poor country where aid workers are commonly (1 in 10,000) raped or killed, even though they would not be willing to be certainly attacked in exchange for 10,000 times the benefits to others. In the actual instant of being tortured anyone would break, but people do choose courses of action that carry risk (every action does, to some extent), so the latter is more meaningful for such hypotheticals.
Number 2. I have driven and flown thousands of kilometers in relation to existential risk, increasing my chance of untimely death in a car accident or plane crash, so obviously I am willing to take some increased probability of death. I think I would prefer a given chance of being tortured to a given chance of death, so obviously I care enough to take at least some tiny risk from what I said above. As I also said above, I’m not willing to make very big sacrifices (big probabilities of such nasty personal outcomes) for tiny shifts in probabilities of big impersonal payoffs (like existential risk reduction). In realistic scenarios, that’s what “the cause” would refer to. I haven’t made any verbal or explicit “precommitment” or promises or anything like that.
In sufficiently extreme (and ludicrously improbable) trolley-problem style examples, e.g. “if you push this button you’ll be tortured for a week, but if you don’t then the Earth will be destroyed (including all your loved ones) if this fair coin comes up heads, and you have incredibly (impossibly?) good evidence that this really is the setup” I hope I would push the button, but in a real world of profound uncertainty, limited evidence, limited personal power (I am not Barack Obama or Bill Gates), and cognitive biases, I don’t expect that to ever happen. I also haven’t made any promises or oaths about that.
I am willing to give of my time and effort, and forgo the financial rewards of a more lucrative career, in exchange for a chance for efficient do-gooding, interaction with interesting people who share my values, and a meaningful project. Given diminishing returns to money in rich countries today, and the ease of obtaining money for folk with high human capital, those aren’t big sacrifices, if they are sacrifices at all.
Number 3. SIAIers love to be precise and analytical and consider philosophical thought experiments, including ethical ones. I think most have views pretty similar to mine, with somewhat varying margins. Certainly Michael Vassar, the head of the organization, is also keen on recognizing one’s various motives and living a balanced life, and avoiding fanatics. Like me, he actively advocates Bostrom-like parliamentary model approaches to combining self-concern with parochial and universalist altruistic feelings.
I have never heard anyone making oaths or promises to make severe sacrifices.
Number 4. This is a pretty ridiculous question. I think that’s fine and normal, and I feel more comfortable with such folk than the alternative. I think people should not exaggerate that do-gooding is the most important thing in their life lest they deceive themselves and others about their willingness to make such choices, which I criticized Roko for.
This sounds very sane, and makes me feel a lot better about the context. Thank you very much.
I very much like the idea that top SIAI people believe that there is such a thing as too much devotion to the cause (and, I’m assuming, actively talk people who are above that level down as you describe doing for Roko).
As someone who has demonstrated impressive sanity around these topics, you seem to be in a unique position to answer these questions with an above-average level-headedness:
Do you understand the math behind the Roko post deletion?
What do you think about the Roko post deletion?
What do you think about future deletions?
Yes, his post was based on (garbled versions of) some work I had been doing at FHI, which I had talked about with him while trying to figure out some knotty sub-problems.
I think the intent behind it was benign, at least in that Eliezer had his views about the issue (which is more general, and not about screwed-up FAI attempts) previously, and that he was motivated to prevent harm to people hearing the idea and others generally. Indeed, he was explicitly motivated enough to take a PR hit for SIAI.
Regarding the substance, I think there are some pretty good reasons for thinking that the expected value (with a small probability of a high impact) of the info for the overwhelming majority of people exposed to it would be negative, although that estimate is unstable in the face of new info.
It’s obvious that the deletion caused more freak-out and uncertainty than anticipated, leading to a net increase in people reading and thinking about the content compared to the counterfactual with no deletion. So regardless of the substance about the info, clearly it was a mistake to delete (which Eliezer also recognizes).
Obviously, Eliezer is continuing to delete comments reposting on the topic of the deleted post. It seems fairly futile to me, but not entirely. I don’t think that Less Wrong is made worse by the absence of that content as such, although the fear and uncertainty about it seem to be harmful. You said you were worried because it makes you uncertain about whether future deletions will occur and of what.
After about half an hour of trying, I can’t think of another topic with the same sorts of features. There may be cases involving things like stalkers or bank PINs or 4chan attacks or planning illegal activities. Eliezer called on people not to discuss AI at the beginning of Less Wrong to help establish its rationality focus, and to back off from the gender warfare, but hasn’t used deletion powers for such things.
Less Wrong has been around for 20 months. If we can rigorously carve out the stalker/PIN/illegality/spam/threats cases I would be happy to bet $500 against $50 that we won’t see another topic banned over the next 20 months.
That sounds like it’d generate some perverse incentives to me.
Urk.
Just to be clear: he recognizes this by comparison with the alternative of privately having the poster delete it themselves, rather than by comparison to not-deleting.
Or at least that was my understanding.
Regardless, thanks for a breath of clarity in this thread. As a mostly disinterested newcomer, I very much appreciated it.
Well, if counterfactually Roko hadn’t wanted to take it down I think it would have been even more of a mistake to delete it, because then the author would have been peeved, not just the audience/commenters.
Which is fine.
But Eliezer’s comments on the subject suggest to me that he doesn’t think that.
More specifically, they suggest that he thinks the most important thing is that the post not be viewable, and if we can achieve that by quietly convincing the author to take it down, great, and if we can achieve it by quietly deleting it without anybody noticing, great, and if we can’t do either of those then we achieve it without being quiet, which is less great but still better than leaving it up.
And it seemed to me your parenthetical could be taken to mean that he agrees with you that deleting it would be a mistake in all of those cases, so I figured I would clarify (or let myself be corrected, if I’m misunderstanding).
I should have taken this bet
Your post has been moved to the Discussion section, not deleted.
Looking at your recent post, I think Alicorn had a good point.
I agree with your main point, but the thought experiment seems to be based on the false assumption that the risk of being raped or murdered are smaller than 1 in 10K if you stay at home. Wikipedia guesstimates that 1 in 6 women in the US are on the receiving end of attempted rape at some point, so someone who goes to a place with a 1 in 10K chance of being raped or murdered has probably improved their personal safety. To make a better thought experiment, I suppose you have to talk about the marginal increase in rape or murder rate when working in the poor country when compared to staying home, and perhaps you should stick to murder since the rape rate is so high.
You lost me at ‘ludicrous’. :)
but he won me back by answering anyway <3
How so?
Thanks!
Great comment Carl!
Roko was not requested to delete his comment. See this parallel thread. (I would appreciate it if you would edit your comment to note this, so readers who miss this comment don’t have a false belief reinforced.) (ETA: thanks)
Agreed (and I think the chance of wfg’s reposts being deleted is very low, because most people get this). Unfortunately, I know nothing about the alleged event (Roko may be misdescribing it, as he misdescribed my message to him) or its context.
Roko said he was asked. You didn’t ask him but maybe someone else did?
Roko’s reply to me strongly suggested that he interpreted my message as requesting deletion, and that I was the cause of him deleting it. I doubt anyone at SIAI would have explicitly requested deletion.
I can confirm that I was not asked to delete the comment but did so voluntarily.
I think you are too trigger-happy.
The wording here leaves weird wiggle room—you’re implying it wasn’t Nick?
I’m wondering whether you, Nick, have learned anything from this experience—something perhaps about how attempting to hide something is almost always counterproductive?
Of course, Roko contributed here by deleting the message, you didn’t create this mess by yourself. But you sure have helped. :)
Well, look, I deleted it of my own accord, but only after being prompted that it was a bad thing to have posted. Can we just drop this? It makes me look like even more of a troublemaker than I already look like, and all I really want to do is finish the efficient charity competition then get on with life outside teh intenetz.
This is very important. If the SIAI is the organisation to solve the friendly AI problem and implement CEV then it should be subject to public examination, especially if they ask for money.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially. And we have those. The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans. However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
Addendum: I see others have been asking “but what do you actually mean?” for a couple of years now.
This strikes me as a demand for particular proof. SIAI is small (and was much smaller until the last year or two), the set of people engaged in FAI research is smaller, Eliezer has chosen to focus on writing about rationality over research for nearly four years, and FAI is a huge problem, in which any specific subproblem should be expected to be underdeveloped at this early stage. And while I and others expect work to speed up in the near future with Eliezer’s attention and better organization, yes, we probably are dead.
Somewhat nitpickingly, this is a reason for FAI in general. CEV is attractive mostly for moving as much work from the designers to the FAI as possible, reducing the potential for uncorrectable error, and being fairer than letting the designers lay out an object-level goal system.
This sounds interesting; do you think you could expand?
It wasn’t intended to be—more incredulity. I thought this was a really important piece of the puzzle, so expected there’d be something at all by now. I appreciate your point: that this is a ridiculously huge problem and SIAI is ridiculously small.
I meant that, as I understand it, CEV is what is fed to the seed AI. Or the AI does the work to ascertain the CEV. It requires an intelligence to ascertain the CEV, but I’d think the ascertaining process would be reasonably set out once we had an intelligence on hand, artificial or no. Or the process to get to the ascertaining process.
I thought we needed the CEV before the AI goes FOOM, because it’s too late after. That implies it doesn’t take a superintelligence to work it out.
Thus: CEV would have to be a process that mere human-level intelligences could apply. That would be a useful process to have, and doesn’t require first creating an AI.
I must point out that my statements on the subject are based in curiosity, ignorance and extrapolation from what little I do know, and I’m asking (probably annoyingly) for more to work with.
“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
OK. I still don’t understand it, but I now feel my lack of understanding more clearly. Thank you!
(I suppose “what do people really want?” is a large philosophical question, not just undefined but subtle in its lack of definition.)
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are, in my opinion, the best we can hope for, and better than we are likely to get from any other entity involved in this.
Besides, I think that XiXiDu, et al are complaining about the difference between cotton and silk, when what is actually likely to happen is more like a big kick in the teeth from reality. SIAI is imperfect. Yes. Well done. Nothing is perfect. At least cut them a bit of slack.
What?!? Open source code—under a permissive license—is the traditional way to signal that you are not going to run off into the sunset with the fruits of a programming effort. Private assurances are usually worth diddly-squat by comparison.
I think that you don’t realize just how bad the situation is. You want that silken sheet. Rude awakening methinks. Also open-source not neccessarily good for FAI in any case.
I don’t think that you realize how bad it is. I’d rather have the universe being paperclipped than supporting the SIAI if that means that I might be tortured for the rest of infinity!
To the best of my knowledge, SIAI has not planned to do anything, under any circumstances, which would increase the probability of you or anyone else being tortured for the rest of infinity.
Supporting SIAI should not, to the best of my knowledge, increase the probability of you or anyone else being tortured for the rest of infinity.
Thank you.
But imagine there was a person a level above yours that went to create some safeguards for an AGI. That person would tell you that you can be sure that the safeguards s/he plans to implement will benefit everyone. Are you just going to believe that? Wouldn’t you be worried and demand that their project is being supervised?
You are in a really powerful position because you are working for an organisation that might influence the future of the universe. Is it really weird to be skeptical and ask for reassurance of their objectives?
Logical rudeness is the error of rejecting an argument for reasons other than disagreement with it. Does your “I don’t think so” mean that you in fact believe that SIAI (possibly) plans to increase the probability of you or someone else being tortured for the rest of eternity? If not, what does this statement mean?
I removed that sentence. I meant that I didn’t believe that the SIAI plans to harm someone deliberately. Although I believe that harm could be a side-effect and that they would rather harm a few beings than allowing some Paperclip maximizer to take over.
You can call me a hypocrite because I’m in favor of animal experiments to support my own survival. But I’m not sure if I’d like to have someone leading an AI project who thinks like me. Take that sentence to reflect my inner conflict. I see why one would favor torture over dust specks but I don’t like such decisions. I’d rather have the universe to end now, or having everyone turned into paperclips, than having to torture beings (especially if I am the being).
I feel uncomfortable that I don’t know what will happen because there is a policy of censorship being favored when it comes to certain thought experiments. I believe that even given negative consequences, transparency is the way to go here. If the stakes are this high, people who believe will do anything to get what they want. That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
I apologize. I realized my stupidity in interpreting your comment a few seconds after posting the reply (which I then deleted).
Better yet, you could use a kind of doublethink—and then even actually mean it. Here is W. D. Hamilton on that topic:
Discriminating Nepotism—as reprinted in: Narrow Roads of Gene Land, Volume 2 Evolution of Sex, p.356.
In TURING’S CATHEDRAL, George Dyson writes:
I think many people would like to be in that group—if they can find a way to arrange it.
Unless AI was given that outcome (cheerful, contented people etc) as a terminal goal, or that circle of people was the best possible route to some other terminal goal, both of which are staggeringly unlikely, Dyson suspects wrongly.
If you think he suspects rightly, I would really like to see a justification. Keep in mind that AGIs are currently not being built using multi-agent environment evolutionary methods, so any kind of ‘social cooperation’ mechanism will not arise.
Machine intelligence programmers seem likely to construct their machines so as to help them satisfy their preferences—which in turn is likely to make them satisfied. I am not sure what you are talking about—but surely this kind of thing is already happening all the time—with Sergey Brin, James Harris Simons—and so on.
That doesn’t really strike me as a stunning insight, though. I have a feeling that I could find many people who would like to be in almost any group of “cheerful, contented, intellectually and physically well-nourished people.”
This all depends on what the AI wants. Without some idea of its utility function, can we really speculate? And if we speculate, we should note those assumptions. People often think of an AI as being essentially human-like in its values, which is problematic.
It’s a fair description of today’s more successful IT companies. The most obvious extrapolation for the immediate future involves more of the same—but with even greater wealth and power inequalities. However, I would certainly also council caution if extrapolating this out more than 20 years or so.
Currently, there are no entities in physical existence which, to my knowledge, have the ability to torture anyone for the rest of eternity.
You intend to build an entity which would have that ability (or if not for infinity, for a googolplex of subjective years).
You intend to give it a morality based on the massed wishes of humanity—and I have noticed that other people don’t always have my best interests at heart. It is possible—though unlikely—that I might so irritate the rest of humanity that they wish me to be tortured forever.
Therefore, you are, by your own statements, raising the risk of my infinite torture from zero to a tiny non-zero probability. It may well be that you are also raising my expected reward enough for that to be more than counterbalanced, but that’s not what you’re saying—any support for SIAI will, unless I’m completely misunderstanding, raise the probability of infinite torture for some individuals.
See the “Last Judge” section of the CEV paper.
As Vladimir observes, the alternative to SIAI doesn’t involve nothing new happening.
That just pushes the problem along a step. IF the Last Judge can’t be mistaken about the results of the AI running AND the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe. That’s very far from saying there’s a zero probability.
If the Last Judge peeks at the output and finds that it’s going to decide to torture people, that doesn’t imply abandoning FAI, it just requires fixing the bug and trying again.
Just because AGIs have capability to inflict infinite torture, doesn’t mean they have a motive. Also, status quo (with regard to SIAI’s activity) doesn’t involve nothing new happening.
I explained that he is planning to supply one with a possible motive (namely that the CEV of humanity might hate me or people like me). It is precisely because of this that the problem arises. A paperclipper, or any other AGI whose utility function had nothing to do with humanity’s wishes, would have far less motive to do this—it might kill me, but it really would have no motive to torture me.
You can have your private assurances—and I will have my open-source software.
Gollum gave his private assurances to Frodo—and we all know how that turned out.
If someone solicits for you to “trust in me”, alarm bells should start ringing immediately. If you really think that is “the best we can hope for”, then perhaps revisit that.
Well I’m convinced. Frodo should definitely have worked out a way to clone the ring and made sure the information was available to all of Middle Earth. You can never have too many potential Ring-Wraiths.
Suddenly I have a mental image of “The Lord of the Rings: The Methods of Rationality.”
Someone should write that (with a better title). We could have a whole genre of rational fanfiction.
Agreed; Lord of the Rings seems like a natural candidate for discussing AI and related topics.
I’d also like to see His Dark Materials with rationalist!Lyra. The girl had an alethiometer. She should have kicked way more ass than she did as soon as she realized what she had.
Open source AGI is not a good thing. In fact, it would be a disastrously bad thing. Giving people the source code doesn’t just let them inspect it for errors, it also lets them launch it themselves. If you get an AGI close to ready for launch, then sharing its source code means that instead of having one party to decide whether there are enough safety measures ready to launch, you have many parties individually deciding whether to launch it themselves, possibly modifying its utility function to suit their own whim, and the hastiest party’s AGI wins.
Ideally, you’d want to let people study the code, but only trustworthy people, and in a controlled environment where they can’t take the source code with them. But even that is risky, since revealing that you have an AGI makes you a target for espionage and attack by parties who shouldn’t be trusted with humanity’s future.
Actually it reduces the chance of any party drawing massively ahead of the rest. It acts as an equalising force, by power-sharing. Since one of the main things we want to avoid is a disreputable organisation using machine intelligence to gain an advantage—and sustaining it over a long period of time. Using open-source software helps to defend against that possibility.
Machine intelligence will be a race—but it will be a race, whether participants share code or not.
Having said all that, machine intelligence protected by patents with secret source code on a server somewhere does seem like a reasonably probable outcome.
Only if (a) there is no point at which AGIs “foom”, (b) source code sharing is well enough enforced on everyone that no bad organizations combine open source with refinements that they keep secret for an advantage, (c) competing AIs form a stable power equilibrium at all points along their advancement, and (d) it is impossible to trade off goal system stability for optimization power.
I estimate probabilities of 0.4, 0.2, 0.05, and 0.5 for these hypotheses, respectively.
I disagree with most of that analysis. I assume machine intelligence will catalyse its own creation. I fully expect that some organisations will stick with secret source code. How could the probability of that possibly be as low as 0.8!?!
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
Nor am I assuming that no team will ever win. If there is to be a winner, we want the best possible lead up. The “trust us” model is not it—not by a long shot.
There are two problems with this reasoning. First, you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
And second, if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
Really? I just provided an example of a mechanism that helps keep open source software projects ethical—the fact that if the manufacturers attempt to exploit their customers it is much easier for the customers to switch to a more ethical fork—because creating such a fork no longer violates copyright law. Though you said you were pointing out problems with my reasoning, you didn’t actually point out any problems with that reasoning.
We saw an example of this kind of thing very recently—with LibreOffice. The developers got afraid that their adopted custodian, Oracle, was going to screw the customers of their project—so, to protect their customers and themselves, they forked it—and went their own way.
If other people are running software that does bad things to you then running good quality software yourself most certainly is a kind of defense. It means you are better able to construct defenses, better able to anticipate their attacks—and so on. Better brains makes you more powerful.
Compare with the closed-source alternative: If other people are running software that does bad things to you—and you have no way to run such software yourself—since it is on their server and running secret source that is also protected by copyright law—you are probably pretty screwed.
How so? I’ve just reread some of your comments on your now deleted post. It looks like you honestly tried to get the SIAI to put safeguards into CEV. Given that the idea spread to many people by now, don’t you think it would be acceptably to discuss the matter before one or more people take it serious or even consider to implement it deliberately?
I don’t think it is a good idea to discuss it. I think that the costs outweigh the benefits. The costs are very big. Benefits marginal.
Ok by me. It is pretty obvious by this point that there is no evil conspiracy involved here. But I think the lesson remains, I you delete something, even if it is just because you regret posting it, you create more confusion than you remove.
I think the question you should be asking is less about evil conspiracies, and more about what kind of organization SIAI is—what would they tell you about, and what would they lie to you about.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI. That alone is enough to conclude that the SIAI is not trying to hold back something that would discredit it as an organisation concerned with charitable objectives. The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence. Making the subject matter public did already harm some people and could harm people in future.
But the forbidden topic is already public. All the effects that would follow from it being public would already follow. THE HORSE HAS BOLTED. It’s entirely unclear to me what pretending it hasn’t does for the problem or the credibility of the SIAI.
It is not as public as you think. If it was then people like waitingforgodel wouldn’t ask about it.
I’m just trying to figure out how to behave without being able talk about it directly. It’s also really interesting on many levels.
Rather more public than a long forgotten counterfactual discussion collecting dust in the blog’s history books would be. :P
Precisely. The place to hide a needle is in a large stack of needles.
The choice here was between “bad” and “worse”—a trolley problem, a lose-lose hypothetical—and they appear to have chosen “worse”.
I prefer to outsource my needle-keeping security to Clippy in exchange for allowing certain ‘bending’ liberties from time to time. :)
Upvoted for LOL value. We’ll tell Clippy the terrible, no good, very bad idea with reasons as to why this would hamper the production of paperclips.
“Hi! I see you’ve accidentally the whole uFAI! Would you like help turning it into paperclips?”
Brilliant.
Frankly, Clippy would be better than the Forbidden Idea. At least Clippy just wants paperclips.
Of course, if Clippy were clever he would then offer to sell SIAI a commitment to never release the UFAI in exchange for a commitment to produce a fixed number of paperclips per year, in perpetuity.
Admittedly, his mastery of human signaling probably isn’t nuanced enough to prevent that from sounding like blackmail.
I really don’t see how that follows. Will more of the public take it seriously? As I have noted, so far the reaction from people outside SIAI/LW has been “They did WHAT? Are they IDIOTS?”
That doesn’t make it not stupid or not counterproductive. Sincere stupidity is not less stupid than insincere stupidity. Indeed, sincere stupidity is more problematic in my experience as the sincere are less likely to back down, whereas the insincere will more quickly hop to a different idea.
Citation needed.
Citation needed.
I sent you another PM.
Hmm, okay. But that, I suggest, appears to have been a case of reasoning oneself stupid.
It does, of course, account for SIAI continuing to attempt to secure the stable doors after the horse has been dancing around in a field for several months taunting them with “COME ON IF YOU THINK YOU’RE HARD ENOUGH.”
(I upvoted XiXiDu’s comment here because he did actually supply a substantive response in PM, well deserving of a vote, and I felt this should be encouraged by reward.)
I wish I could upvote twice
A kind of meta-question: is there any evidence suggesting that one of the following explanations of the recent deletion is better than another?
That an LW moderator deleted Roko’s comment.
That Roko was asked to delete it, and complied.
That Roko deleted it himself, without prompting.
One of the more disturbing topics in this post is the question of how much can you trust an organization of people who are willing to endure torture, rape, and death for their cause.
Surely lying isn’t as bad as any of those...
Of course, lying for your cause is almost certainly a long term retarded thing to do… but so is censoring ideas...
It’s hard to know what to trust on this thread
As you may know from your study of marketing, accusations stick in the mind even when one is explicitly told they are false. In the parent comment and a sibling, you describe a hypothetical SIAI lying to its donors because… Roko had some conversations with Carl that led you to believe we care strongly about existential risk reduction?
If your aim is to improve SIAI, to cause there to be good organizations in this space, and/or to cause Less Wrong-ers to have accurate info, you might consider:
Talking with SIAI and/or with Fellows program alumni, so as to gather information on the issues you are concerned about. (I’d be happy to talk to you; I suspect Jasen and various alumni would too.) And then
Informing folks on LW of anything interesting/useful that you find out.
Anyone else who is concerned about any SIAI-related issue is also welcome to talk to me/us.
Actually that citation is about both positive and negative things—so unless you’re also asking pro-SIAI people to hush up, you’re (perhaps unknowingly) seeking to cause a pro-SIAI bias.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
One of the interesting morals from Roko’s contest is that if you care deeply about getting the most benefit per donated dollar you have to look very closely at who you’re giving it to.
Market forces work really well for lightbulb-sales businesses, but not so well for mom-and-pop shops, let alone charities. The motivations, preferences, and likely future actions of the people you’re giving money to become very important. Knowing if you can believe the person, in these contexts, becomes even more important.
As you note, I’ve studied marketing, sales, propaganda, cults, and charities. I know that there are some people who have no problem lying for their cause (especially if it’s for their god or to save the world).
I also know that there are some people who absolutely suck at lying. They try to lie, but the truth just seeps out of them.
That’s why I give Roko’s blurted comments more weight than whatever I’d hear from SIAI people who were chosen by you—no offence. I’ll still talk with you guys, but I don’t think a reasonably sane person can trust the sales guy beyond a point.
As far as your question goes, my primary desire is a public, consistent moderation policy for LessWrong. If you’re going to call this a community blog devoted to rationality, then please behave in sane ways. (If no one owns the blog—if it belongs to the community—then why is there dictatorial post deletion?)
I’d also like an apology from EY with regard to the chilling effects his actions have caused.
But back to what you replied to:
What would SIAI be willing to lie to donors about?
Do you have any answers to this?
To answer your question, despite David Gerard’s advice:
I would not lie to donors about the likely impact of their donations, the evidence concerning SIAI’s ability or inability to pull off projects, how we compare to other organizations aimed at existential risk reduction, etc. (I don’t have all the answers, but I aim for accuracy and revise my beliefs and my statements as evidence comes in; I’ve actively tried to gather info on whether we or FHI reduce risk more per dollar, and I often recommend to donors that they do their own legwork with that charity comparison to improve knowledge and incentives). If a maniacal donor with a gun came searching for a Jew I had hidden in my house, or if I somehow had a “how to destroy the world” recipe and someone asked me how to use it, I suppose lying would be more tempting.
While I cannot speak for others, I suspect that Michael Vassar, Eliezer, Jasen, and others feel similarly, especially about the “not lying to one’s cooperative partners” point.
I suppose I should add “unless the actual answer is not a trolley problem” to my advice on not answering this sort of hypothetical ;-)
(my usual answer to hypotheticals is “we have no plans along those lines”, because usually we really don’t. We’re also really good at not having opinions on other organisations, e.g. Wikileaks, which we’re getting asked about A LOT because their name starts with “wiki”. A blog post on the subject is imminent. Edit: up now.)
I notice that your list is future facing.
Lies are usually about the past.
It’s very easy to not lie when talking about the future. It is much easier to “just this once” lie about the past. You can do both, for instance, by explaining that you believe a project will succeed, even while withholding information that would convince a donor otherwise.
An example of this would be errors or misconduct in completing past projects.
Lack of relevant qualifications for people SIAI plans to employ on a project.
Or administrative errors and misconduct.
Or public relations / donor outreach misconduct.
To put the question another, less abstract way, have you ever lied to a SIAI donor? Do you know of anyone affiliated with SIAI who has lied a donor?
Hypothetically, If I said I had evidence in the affirmative to the second question, how surprising would that be to you? How much money would you bet that such evidence doesn’t exist?
You’re trying very hard to get everyone to think that SIAI has lied to donors or done something equally dishonest. I agree that this is an appropriate question to discuss, but you are pursuing the matter so aggressively that I just have to ask: do you know something we don’t? Do you think that you/other donors have been lied to on a particular occasion, and if so, when?
When I asked Anna about the coordination between SIAI and FHI, something like “Do you talk enough with each other that you wouldn’t both spend resources writing the same research paper?”, she was told me about the one time that they had in fact both presented a paper on the same topic at a conference, and that they do now coordinate more to prevent that sort of thing.
I have found that Anna and others at SIAI are honest and forthcoming.
Your comment here killed the hostage.
Well, uh, yeah. The horse has bolted. It’s entirely unclear what choosing to keep one’s head in the sand gains anyone.
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Investigating the logic or lack thereof behind the (apparently ongoing) memory-holing is, however, incredibly on-topic and relevant for LW.
Total agreement here. In Eliezer’s words:
A fellow called George Orwell.
Ahh, thankyou.
I presume you’re not a native English speaker then—pretty much any moderately intelligent native English speaker has been forced to familiarity with 1984 at school. (When governments in the UK are being particularly authoritarian, there is often a call to send MPs copies of 1984 with a note “This is not a manual.”) Where are you from? Also, you really should read the book, then lots of the commentary on it :-) It’s one of the greatest works of science fiction and political fiction in English.
I can tell you all about equal pigs and newspeak but ‘memory-holing’ has not seemed to make as much of a cultural footprint—probably because as a phrase it is rather awkward fit. I wholeheartedly approve of Orwell in principle but actually reading either of his famous books sounds too much like highschool homework. :)
Animal Farm is probably passable (though it’s so short). 1984 on the other hand is maybe my favorite book of all time. I don’t think I’ve had a stronger emotional reaction to another book. It makes Shakespeare’s tragedies look like comedies. I’d imagine you’d have similar feelings about it based on what I’ve read of your comments here.
That’s some high praise there.
So I take it there isn’t a romantic ‘happily ever after’ ending? :P
Actually, there is… ;)
Both are short and enjoyable- I strongly recommend checking them out from a library or picking up a copy.
Read them. They’re actually really good books. His less-famous ones are not as brilliant, but are good too.
(We were taught 1984 in school, I promptly read to the end with eyes wide. I promptly borrowed Animal Farm of my own accord.)
His less-famous novels aren’t as good. On the other hand, some of his essays are among the clearest, most intelligent thinking I’ve ever come across, and would probably be of a lot of interest to LessWrong readers...
Oh yeah. Politics and the English Language is a classic on a par with the great two novels. I first read that in 1992 and wanted to print copies to distribute everywhere (we didn’t have internet then).
Yeah, that’s one of those I was thinking of. Also things like the piece about the PEN ‘anti-censorship’ event that wasn’t, and his analysis of James Burnham’s Managerialist writing...
I’m terribly curious now—did the use of any of the phrases Orwell singles out in the article actually drop significantly after the article was published? Wikipedia will not say...
Well, reading it in the 1990s and having a burnt-out ex-Communist for a housemate at the time, I fear I recognised far too many of the cliches therein as current in those circles ;-)
A lot are still current in those less rational/more angry elements of the left who still think the Labour Party represents socialism and use phrases like that to justify themselves...
Because this is LessWrong—you can give a sane response and not only does it clear the air, people understand and appreciate it.
Cable news debating isn’t needed here.
Sure we might still wonder if they’re being perfectly honest, but saying something more sane on the topic than silence seems like a net-positive from their perspective.
By way of a reminder, the question under discussion was:
LessWrongers are not magically free of bias. Nor are they inherently moral people that wouldn’t stoop to using misleading rhetorical techniques, though here they are more likely to be called on it.
In any case, an answer here is available to the public internet for all to see.
I respectfully disagree, and have my hopes set on Carl (or some other level-headed person in a position to know) giving a satisfying answer.
This is LessWrong after all—we can follow complicated arguments, and at least hearing how SIAI is actually thinking about such things would (probably) reduce my paranoia.
Yeah, but this is on the Internet for everyone to see. The potential for political abuse is ridiculous and can infect even LessWrong readers. Politics is the mind-killer, but pretending it doesn’t affect almost everyone else strikes me as not smart.
The concept of ethical injunctions is known in SIAI circles I think. Enduring personal harm for your cause and doing unethical things for your cause are therefore different. Consider Eliezer’s speculation about whether a rationalist “confessor” should ever lie in this post, too. And these personal struggles with whether to ever lie about SIAI’s work.
That “confessor” link is terrific
If banning Roko’s post would reasonably cause discussion of those ideas to move away from LessWrong, then by EY’s own reasoning (the link you gave) it seems like a retarded move.
Right?
If the idea is actually dangerous, it’s way less dangerous to people who aren’t familiar with pretty esoteric Lesswrongian ideas. They’re prerequisites to being vulnerable to it. So getting conversation about the idea away from Lesswrong isn’t an obviously retarded idea.
Lying for good causes has a time honored history. Protecting fugitive slaves or holocaust victims immediately comes to mind. Just because it is more often practical to be honest than not doesn’t mean that dishonesty isn’t sometimes unambiguously the better option.
I agree that there’s a lot in history, but the examples you cited have something that doesn’t match here—historically, you lie to people you don’t plan on cooperating with later.
If you lie to an oppressive government, it’s okay because it’ll either get overthrown or you’ll never want to cooperate with it (so great is your reason for lying).
Lying to your donor pool is very, very different than lying to the Nazis about hiding jews.
You’re throwing around accusations of lying pretty lightly.
Am I missing something? Desrtopa responded to questions of lying to the donor pool with the equivalent of “We do it for the greater good”
Desrtopa isn’t affiliated with SIAI. You seem to be deliberately designing confusing comments, a la Glenn Beck’s “I’m just asking questions” motif.
Is calling someone here Glenn Beck equivalent to Godwination?
wfg’s post strikes me as almost entirely reasonable (except the last question, which is pointless to ask) and your response as excessively defensive.
Also, you’re saying this to someone who says he’s a past donor and has not yet ruled out being a future donor. This is someone who could reasonably expect his questions to be taken seriously.
(I have some experience of involvement in a charity that suffers a relentless barrage of blitheringly stupid questions from idiots, and my volunteer role is media handling—mostly I come up with good and effective soundbites. So I appreciate and empathise with your frustration, but I think I can state with some experience behind me that your response is actually terrible.)
Okay. Given your and the folks who downvoted my comment’s perceptions, I’ll revise my opinion on the matter. I’ll also put that under “analogies not to use”; I was probably insufficiently familiar with the pop culture.
The thing I meant to say was just… Roko made a post, Nick suggested it gave bad impressions, Roko deleted it. wfg spent hours commenting again and again about how he had been asked to delete it, perhaps by someone “high up within SIAI”, and how future censorship might be imminent, how the fact that Roko had had a bascially unrelated conversation suggested that we might be lying to donors (a suggestion that he didn’t make explicitly, but rather left to innuendo), etc. I feel tired of this conversation and want to go back to research and writing, but I’m kind of concerned that it’ll leave a bad taste in readers mouths not because of any evidence that’s actually being advanced, but because innuendo and juxtapositions, taken out of context, leave impressions of badness.
I wish I knew how to have a simple, high-content, low-politics conversation on the subject. Especially one that was self-contained and didn’t leave me feeling as though I couldn’t bow out after awhile and return to other projects.
The essential problem is that with the (spectacular) deletion of the Forbidden Post, LessWrong turned into the sort of place where posts get disappeared. Those are not good places to be on the Internet. They are places where honesty is devalued and statements of fact must be reviewed for their political nature.
So it can happen here—because it did happen. It’s no longer in the class “things that are unthinkable”. This is itself a major credibility hit for LW.
And when a Roko post disappears—well, it was one of his posts that was disappeared before.
With this being the situation, assumptions of bad faith are going to happen. (And “stupidity” is actually the assumption of good faith.)
Your problem now is to restore trust in LW’s intellectual integrity, because SIAI broke it good and hard. Note that this is breaking an expectation, which is much worse than breaking a rule—if you break a rule you can say “we broke this rule for this reason”, but if you break expectations, people feel the ground moving under their feet, and get very upset.
There are lots of suggestions in this thread as to what people think might restore their trust in LW’s intellectual integrity, SIAI needs to go through them and work out precisely what expectations they broke and how to come clean on this.
I suspect you could at this point do with an upside to all this. Fortunately, there’s an excellent one: no-one would bother making all this fuss if they didn’t really care about LW. People here really care about LW and will do whatever they can to help you make it better.
(And the downside is that this is separate from caring about SIAI, but oh well ;-) )
(and yes, this sort of discussion around WP/WMF has been perennial since it started.)
I’ve seen several variations of this expressed about this topic, and it’s interesting to me, because this sort of view is somewhat foreign to me. I wouldn’t say I’m pro-censorship, but as an attorney trained in U.S. law, I think I’ve very much internalized the idea that the most serious sorts of censorship actions are those taken by the government (i.e., this is what the First Amendment free speech right is about, and that makes sense because of the power of the government), and that there are various levels of seriousness/danger beyond that, with say, big corporate censorship also being somewhat serious because of corporate power, and censorship by the owner of a single blog (even a community one) not being very serious at all, because a blogowner is not very powerful compared to the government or a major corporation, and shutting down one outlet of communication on the Internet is comparatively not a big deal because it’s a big internet where there are lots of other places to express one’s views. If a siteowner exercises his or her right to delete something on a website, it’s just not the sort of harm that I weigh very heavily.
What I’m totally unsure of is where the average LW reader falls on the scale between you and me, and therefore, despite the talk about the Roko incident being such a public relations disaster and a “spectacular” deletion, I just don’t know how true that is and I’m curious what the answer would be. People who feel like me may just not feel the need to weigh in on the controversy, whereas people who are very strongly anti-censorship in this particular context do.
That’s not really the crux of the issue (for me, at least, and probably not for others). As David Gerard put it, the banning of Roko’s post was a blow to people’s expectations, which was why it was so shocking. In other words, it was like discovering that LW wasn’t what everyone thought it was (and not in a good way).
Note: I personally wouldn’t classify the incident as a “disaster,” but was still very alarming.
Like Airedale, I don’t have that impression—my impression is that 1) Censorship by website’s owner doesn’t have the moral problems associated with censorship by governments (or corporations), and 2) in online communities, dictatorship can work quite well, as long as the dictator isn’t a complete dick.
I’ve seen quite functional communities where the moderators would delete posts without warning if they were too stupid, offensive, repetitive or immoral (such as bragging about vandalizing wikipedia).
So personally, I don’t see a need for “restoring trust”. Of course, as your post attests, my experience doesn’t seem to generalize to other posters.
Great post. It confuses me why this isn’t at 10+ karma
+5 is fine!
Y’know, one of the actual problems with LW is that I read it in my Internet as Television time, but there’s a REALLY PROMINENT SCORE COUNTER at the top left. This does not help in not treating it as a winnable video game.
(That said, could the people mass-downvoting waitingforgodel please stop? It’s tiresome. Please try to go by comment, not poster.)
So true!
(Except it’s at the top right. At least, the one I’m thinking of.)
The other left.
(Yes, I actually just confused left and right. STOP POSTING.)
Probably because its buried in the middle of an enormous discussion that very few people have read and will read.
Lol. right, that’d do it
This is about politics. The censorship of an idea related to a future dictator implementing some policy is obviously about politics.
You tell people to take friendly AI serious. You tell people that we need friendly AI to marshal our future galactic civilisation. People take it serious. Now the only organisation working on this is the SIAI. Therefore the SIAI is currently in direct causal control of our collective future. So why do you wonder people care about censorship and transparency? People already care about what the U.S. is doing and demand transparency. Which is ludicrous in comparison to the power of a ruling superhuman artificial intelligence that implements what the SIAI came up with as the seed for its friendliness.
If you really think that the SIAI has any importance and could possible achieve to influence or implement the safeguards for some AGI project, then everything the SIAI does is obviously very important to everyone concerned (everyone indeed).
What? No way! The organisation seems very unlikely to produce machine intelligence to me—due to all the other vastly-better funded players.
I wish you used a classification algorithm that more naturally identified the tension between “wanting low-politics conversation” and comparing someone to Glenn Beck as a means of criticism.
Sorry. This was probably simply a terrible mistake born of unusual ignorance of pop culture and current politics. I meant to invoke “using questions as a means to plant accusations” and honestly didn’t understand that he was radically unpopular. I’ve never watched anything by him.
Well, it’s not that Beck is unpopular; it’s that he’s very popular with people of a particular political ideology.
In fairness, though, he is sort of the canonical example for “I’m just asking questions, here!”. (And I wasn’t one of those voting you down on this.)
I think referring to the phenomenon itself is enough to make one’s point on the issue, and it’s not necessary to identify a person who does it a lot.
-3 after less than 15 minutes suggests so!
Make that “they do it for the greater good”
Sorry about mistakingly implying s/he was affiliated. I’ll be more diligent with my google stalking in the future.
edit: In my defense, SIAI affiliation has been very common when looking up very “pro” people from this thread
Thanks. I appreciate that.