As you may know from your study of marketing, accusations stick in the mind even when one is explicitly told they are false. In the parent comment and a sibling, you describe a hypothetical SIAI lying to its donors because… Roko had some conversations with Carl that led you to believe we care strongly about existential risk reduction?
If your aim is to improve SIAI, to cause there to be good organizations in this space, and/or to cause Less Wrong-ers to have accurate info, you might consider:
Talking with SIAI and/or with Fellows program alumni, so as to gather information on the issues you are concerned about. (I’d be happy to talk to you; I suspect Jasen and various alumni would too.) And then
Informing folks on LW of anything interesting/useful that you find out.
Anyone else who is concerned about any SIAI-related issue is also welcome to talk to me/us.
accusations stick in the mind even when one is explicitly told they are false
Actually that citation is about both positive and negative things—so unless you’re also asking pro-SIAI people to hush up, you’re (perhaps unknowingly) seeking to cause a pro-SIAI bias.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
One of the interesting morals from Roko’s contest is that if you care deeply about getting the most benefit per donated dollar you have to look very closely at who you’re giving it to.
Market forces work really well for lightbulb-sales businesses, but not so well for mom-and-pop shops, let alone charities. The motivations, preferences, and likely future actions of the people you’re giving money to become very important. Knowing if you can believe the person, in these contexts, becomes even more important.
As you note, I’ve studied marketing, sales, propaganda, cults, and charities. I know that there are some people who have no problem lying for their cause (especially if it’s for their god or to save the world).
I also know that there are some people who absolutely suck at lying. They try to lie, but the truth just seeps out of them.
That’s why I give Roko’s blurted comments more weight than whatever I’d hear from SIAI people who were chosen by you—no offence. I’ll still talk with you guys, but I don’t think a reasonably sane person can trust the sales guy beyond a point.
As far as your question goes, my primary desire is a public, consistent moderation policy for LessWrong. If you’re going to call this a community blog devoted to rationality, then please behave in sane ways. (If no one owns the blog—if it belongs to the community—then why is there dictatorial post deletion?)
I’d also like an apology from EY with regard to the chilling effects his actions have caused.
But back to what you replied to:
What would SIAI be willing to lie to donors about?
To answer your question, despite David Gerard’s advice:
I would not lie to donors about the likely impact of their donations, the evidence concerning SIAI’s ability or inability to pull off projects, how we compare to other organizations aimed at existential risk reduction, etc. (I don’t have all the answers, but I aim for accuracy and revise my beliefs and my statements as evidence comes in; I’ve actively tried to gather info on whether we or FHI reduce risk more per dollar, and I often recommend to donors that they do their own legwork with that charity comparison to improve knowledge and incentives). If a maniacal donor with a gun came searching for a Jew I had hidden in my house, or if I somehow had a “how to destroy the world” recipe and someone asked me how to use it, I suppose lying would be more tempting.
While I cannot speak for others, I suspect that Michael Vassar, Eliezer, Jasen, and others feel similarly, especially about the “not lying to one’s cooperative partners” point.
I suppose I should add “unless the actual answer is not a trolley problem” to my advice on not answering this sort of hypothetical ;-)
(my usual answer to hypotheticals is “we have no plans along those lines”, because usually we really don’t. We’re also really good at not having opinions on other organisations, e.g. Wikileaks, which we’re getting asked about A LOT because their name starts with “wiki”. A blog post on the subject is imminent. Edit:up now.)
It’s very easy to not lie when talking about the future. It is much easier to “just this once” lie about the past. You can do both, for instance, by explaining that you believe a project will succeed, even while withholding information that would convince a donor otherwise.
An example of this would be errors or misconduct in completing past projects.
Lack of relevant qualifications for people SIAI plans to employ on a project.
Or administrative errors and misconduct.
Or public relations / donor outreach misconduct.
To put the question another, less abstract way, have you ever lied to a SIAI donor? Do you know of anyone affiliated with SIAI who has lied a donor?
Hypothetically, If I said I had evidence in the affirmative to the second question, how surprising would that be to you? How much money would you bet that such evidence doesn’t exist?
You’re trying very hard to get everyone to think that SIAI has lied to donors or done something equally dishonest. I agree that this is an appropriate question to discuss, but you are pursuing the matter so aggressively that I just have to ask: do you know something we don’t? Do you think that you/other donors have been lied to on a particular occasion, and if so, when?
An example of this would be errors or misconduct in completing past projects.
When I asked Anna about the coordination between SIAI and FHI, something like “Do you talk enough with each other that you wouldn’t both spend resources writing the same research paper?”, she was told me about the one time that they had in fact both presented a paper on the same topic at a conference, and that they do now coordinate more to prevent that sort of thing.
I have found that Anna and others at SIAI are honest and forthcoming.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
Well, uh, yeah. The horse has bolted. It’s entirely unclear what choosing to keep one’s head in the sand gains anyone.
What would SIAI be willing to lie to donors about?
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Investigating the logic or lack thereof behind the (apparently ongoing) memory-holing is, however, incredibly on-topic and relevant for LW.
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Ambiguity is their ally. Both answers elicit negative responses, and they can avoid that from most people by not saying anything, so why shouldn’t they shut up?
I presume you’re not a native English speaker then—pretty much any moderately intelligent native English speaker has been forced to familiarity with 1984 at school. (When governments in the UK are being particularly authoritarian, there is often a call to send MPs copies of 1984 with a note “This is not a manual.”) Where are you from? Also, you really should read the book, then lots of the commentary on it :-) It’s one of the greatest works of science fiction and political fiction in English.
I can tell you all about equal pigs and newspeak but ‘memory-holing’ has not seemed to make as much of a cultural footprint—probably because as a phrase it is rather awkward fit. I wholeheartedly approve of Orwell in principle but actually reading either of his famous books sounds too much like highschool homework. :)
Animal Farm is probably passable (though it’s so short). 1984 on the other hand is maybe my favorite book of all time. I don’t think I’ve had a stronger emotional reaction to another book. It makes Shakespeare’s tragedies look like comedies. I’d imagine you’d have similar feelings about it based on what I’ve read of your comments here.
His less-famous novels aren’t as good. On the other hand, some of his essays are among the clearest, most intelligent thinking I’ve ever come across, and would probably be of a lot of interest to LessWrong readers...
Oh yeah. Politics and the English Language is a classic on a par with the great two novels. I first read that in 1992 and wanted to print copies to distribute everywhere (we didn’t have internet then).
Yeah, that’s one of those I was thinking of. Also things like the piece about the PEN ‘anti-censorship’ event that wasn’t, and his analysis of James Burnham’s Managerialist writing...
I’m terribly curious now—did the use of any of the phrases Orwell singles out in the article actually drop significantly after the article was published? Wikipedia will not say...
Well, reading it in the 1990s and having a burnt-out ex-Communist for a housemate at the time, I fear I recognised far too many of the cliches therein as current in those circles ;-)
A lot are still current in those less rational/more angry elements of the left who still think the Labour Party represents socialism and use phrases like that to justify themselves...
Because this is LessWrong—you can give a sane response and not only does it clear the air, people understand and appreciate it.
Cable news debating isn’t needed here.
Sure we might still wonder if they’re being perfectly honest, but saying something more sane on the topic than silence seems like a net-positive from their perspective.
LessWrongers are not magically free of bias. Nor are they inherently moral people that wouldn’t stoop to using misleading rhetorical techniques, though here they are more likely to be called on it.
In any case, an answer here is available to the public internet for all to see.
I respectfully disagree, and have my hopes set on Carl (or some other level-headed person in a position to know) giving a satisfying answer.
This is LessWrong after all—we can follow complicated arguments, and at least hearing how SIAI is actually thinking about such things would (probably) reduce my paranoia.
Yeah, but this is on the Internet for everyone to see. The potential for political abuse is ridiculous and can infect even LessWrong readers. Politics is the mind-killer, but pretending it doesn’t affect almost everyone else strikes me as not smart.
As you may know from your study of marketing, accusations stick in the mind even when one is explicitly told they are false. In the parent comment and a sibling, you describe a hypothetical SIAI lying to its donors because… Roko had some conversations with Carl that led you to believe we care strongly about existential risk reduction?
If your aim is to improve SIAI, to cause there to be good organizations in this space, and/or to cause Less Wrong-ers to have accurate info, you might consider:
Talking with SIAI and/or with Fellows program alumni, so as to gather information on the issues you are concerned about. (I’d be happy to talk to you; I suspect Jasen and various alumni would too.) And then
Informing folks on LW of anything interesting/useful that you find out.
Anyone else who is concerned about any SIAI-related issue is also welcome to talk to me/us.
Actually that citation is about both positive and negative things—so unless you’re also asking pro-SIAI people to hush up, you’re (perhaps unknowingly) seeking to cause a pro-SIAI bias.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
One of the interesting morals from Roko’s contest is that if you care deeply about getting the most benefit per donated dollar you have to look very closely at who you’re giving it to.
Market forces work really well for lightbulb-sales businesses, but not so well for mom-and-pop shops, let alone charities. The motivations, preferences, and likely future actions of the people you’re giving money to become very important. Knowing if you can believe the person, in these contexts, becomes even more important.
As you note, I’ve studied marketing, sales, propaganda, cults, and charities. I know that there are some people who have no problem lying for their cause (especially if it’s for their god or to save the world).
I also know that there are some people who absolutely suck at lying. They try to lie, but the truth just seeps out of them.
That’s why I give Roko’s blurted comments more weight than whatever I’d hear from SIAI people who were chosen by you—no offence. I’ll still talk with you guys, but I don’t think a reasonably sane person can trust the sales guy beyond a point.
As far as your question goes, my primary desire is a public, consistent moderation policy for LessWrong. If you’re going to call this a community blog devoted to rationality, then please behave in sane ways. (If no one owns the blog—if it belongs to the community—then why is there dictatorial post deletion?)
I’d also like an apology from EY with regard to the chilling effects his actions have caused.
But back to what you replied to:
What would SIAI be willing to lie to donors about?
Do you have any answers to this?
To answer your question, despite David Gerard’s advice:
I would not lie to donors about the likely impact of their donations, the evidence concerning SIAI’s ability or inability to pull off projects, how we compare to other organizations aimed at existential risk reduction, etc. (I don’t have all the answers, but I aim for accuracy and revise my beliefs and my statements as evidence comes in; I’ve actively tried to gather info on whether we or FHI reduce risk more per dollar, and I often recommend to donors that they do their own legwork with that charity comparison to improve knowledge and incentives). If a maniacal donor with a gun came searching for a Jew I had hidden in my house, or if I somehow had a “how to destroy the world” recipe and someone asked me how to use it, I suppose lying would be more tempting.
While I cannot speak for others, I suspect that Michael Vassar, Eliezer, Jasen, and others feel similarly, especially about the “not lying to one’s cooperative partners” point.
I suppose I should add “unless the actual answer is not a trolley problem” to my advice on not answering this sort of hypothetical ;-)
(my usual answer to hypotheticals is “we have no plans along those lines”, because usually we really don’t. We’re also really good at not having opinions on other organisations, e.g. Wikileaks, which we’re getting asked about A LOT because their name starts with “wiki”. A blog post on the subject is imminent. Edit: up now.)
I notice that your list is future facing.
Lies are usually about the past.
It’s very easy to not lie when talking about the future. It is much easier to “just this once” lie about the past. You can do both, for instance, by explaining that you believe a project will succeed, even while withholding information that would convince a donor otherwise.
An example of this would be errors or misconduct in completing past projects.
Lack of relevant qualifications for people SIAI plans to employ on a project.
Or administrative errors and misconduct.
Or public relations / donor outreach misconduct.
To put the question another, less abstract way, have you ever lied to a SIAI donor? Do you know of anyone affiliated with SIAI who has lied a donor?
Hypothetically, If I said I had evidence in the affirmative to the second question, how surprising would that be to you? How much money would you bet that such evidence doesn’t exist?
You’re trying very hard to get everyone to think that SIAI has lied to donors or done something equally dishonest. I agree that this is an appropriate question to discuss, but you are pursuing the matter so aggressively that I just have to ask: do you know something we don’t? Do you think that you/other donors have been lied to on a particular occasion, and if so, when?
When I asked Anna about the coordination between SIAI and FHI, something like “Do you talk enough with each other that you wouldn’t both spend resources writing the same research paper?”, she was told me about the one time that they had in fact both presented a paper on the same topic at a conference, and that they do now coordinate more to prevent that sort of thing.
I have found that Anna and others at SIAI are honest and forthcoming.
Your comment here killed the hostage.
Well, uh, yeah. The horse has bolted. It’s entirely unclear what choosing to keep one’s head in the sand gains anyone.
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Investigating the logic or lack thereof behind the (apparently ongoing) memory-holing is, however, incredibly on-topic and relevant for LW.
Total agreement here. In Eliezer’s words:
A fellow called George Orwell.
Ahh, thankyou.
I presume you’re not a native English speaker then—pretty much any moderately intelligent native English speaker has been forced to familiarity with 1984 at school. (When governments in the UK are being particularly authoritarian, there is often a call to send MPs copies of 1984 with a note “This is not a manual.”) Where are you from? Also, you really should read the book, then lots of the commentary on it :-) It’s one of the greatest works of science fiction and political fiction in English.
I can tell you all about equal pigs and newspeak but ‘memory-holing’ has not seemed to make as much of a cultural footprint—probably because as a phrase it is rather awkward fit. I wholeheartedly approve of Orwell in principle but actually reading either of his famous books sounds too much like highschool homework. :)
Animal Farm is probably passable (though it’s so short). 1984 on the other hand is maybe my favorite book of all time. I don’t think I’ve had a stronger emotional reaction to another book. It makes Shakespeare’s tragedies look like comedies. I’d imagine you’d have similar feelings about it based on what I’ve read of your comments here.
That’s some high praise there.
So I take it there isn’t a romantic ‘happily ever after’ ending? :P
Actually, there is… ;)
Both are short and enjoyable- I strongly recommend checking them out from a library or picking up a copy.
Read them. They’re actually really good books. His less-famous ones are not as brilliant, but are good too.
(We were taught 1984 in school, I promptly read to the end with eyes wide. I promptly borrowed Animal Farm of my own accord.)
His less-famous novels aren’t as good. On the other hand, some of his essays are among the clearest, most intelligent thinking I’ve ever come across, and would probably be of a lot of interest to LessWrong readers...
Oh yeah. Politics and the English Language is a classic on a par with the great two novels. I first read that in 1992 and wanted to print copies to distribute everywhere (we didn’t have internet then).
Yeah, that’s one of those I was thinking of. Also things like the piece about the PEN ‘anti-censorship’ event that wasn’t, and his analysis of James Burnham’s Managerialist writing...
I’m terribly curious now—did the use of any of the phrases Orwell singles out in the article actually drop significantly after the article was published? Wikipedia will not say...
Well, reading it in the 1990s and having a burnt-out ex-Communist for a housemate at the time, I fear I recognised far too many of the cliches therein as current in those circles ;-)
A lot are still current in those less rational/more angry elements of the left who still think the Labour Party represents socialism and use phrases like that to justify themselves...
Because this is LessWrong—you can give a sane response and not only does it clear the air, people understand and appreciate it.
Cable news debating isn’t needed here.
Sure we might still wonder if they’re being perfectly honest, but saying something more sane on the topic than silence seems like a net-positive from their perspective.
By way of a reminder, the question under discussion was:
LessWrongers are not magically free of bias. Nor are they inherently moral people that wouldn’t stoop to using misleading rhetorical techniques, though here they are more likely to be called on it.
In any case, an answer here is available to the public internet for all to see.
I respectfully disagree, and have my hopes set on Carl (or some other level-headed person in a position to know) giving a satisfying answer.
This is LessWrong after all—we can follow complicated arguments, and at least hearing how SIAI is actually thinking about such things would (probably) reduce my paranoia.
Yeah, but this is on the Internet for everyone to see. The potential for political abuse is ridiculous and can infect even LessWrong readers. Politics is the mind-killer, but pretending it doesn’t affect almost everyone else strikes me as not smart.