Damnit! My smug self assurance that I could postpone thinking about these issues seriously because I’m an SIAI donor …. GONE! How am I supposed to get any work done now?
Seriously though, I do wish the SIAI toned down its self importance and incredible claims, however true they are. I realize, of course, that dulling some claims to appear more credible is approaching a Dark Side type strategy, but… well, no buts. I’m just confused.
Edit: I misunderstood what Jordan was trying to say—the previous version of this comment is irrelevant to the present discussion and so I’ve deleted it.
Deciding that the truth unconditionally deserves top priority seems to me to be an overly convenient easy way out of confronting the challenges of demanded by instrumental rationality.
No one is claiming that honesty deserves top priority. I would lie to save someone’s life, or to make a few million dollars, etc. In the context of SIAI though, or any organization, being manipulative can severely discredit you.
I believe that when one takes into account unintended consequences, when Eliezer makes his most incredible claims he lowers overall levels of epistemic rationality rather than raising overall levels of epistemic rationality.
If he were to go back on his incredible claims, or even only make more credible claims in the future, how would he reconcile the two when confronted? If someone new to Eliezer read his tame claims, then went back and read his older, more extreme claims, what would they think? To many people this would enforce the idea that SIAI is a cult, and that they are refining their image to be more attractive.
All of that said, I do understand where you’re coming from intuitively, and I’m not convinced that scaling back some of the SIAI claims would ever have a negative effect. Certainly, though, a public policy conversation about it would cast a pretty manipulative shade over SIAI. Hell, even this conversation could cast a nasty shade to some onlookers (to many people trying to judge SIAI, the two of us might be a sufficiently close proxy, even though we have no direct connections).
Okay, I misunderstood where you were coming from earlier, I thought you were making a general statement about the importance of stating one’s beliefs. Sorry about that.
In response to your present comments, I would say that though the phenomenon that you have in mind may be a PR issue, I think it would be less of a PR issue than what’s going on right now.
One thing that I would say is that I think that Eliezer would come across as much more credible simply by accompanying his weird sounding statements with disclaimers of the type “I know that what I’m saying probably sounds pretty ‘out there’ and understand if you don’t believe me, but I’ve thought about this hard, and I think...” See my remark here.
I mostly agree, although I’m still mulling it and think the issue is more complicated than it appears. One nitpick:
“I know that what I’m saying probably sounds pretty ‘out there’ and understand if you don’t believe me, but I’ve thought about this hard, and I think...”
Personally these kind of qualifiers rarely do anything to allay my doubt, and can easily increase them. I prefer to see incredulity. For instance, when a scientist has an amazing result, rather than seeing that they fully believe it but recognizing it’s difficult for me to believe, I’d rather see them doubtful of their own conclusion but standing by it nonetheless because of the strength of the evidence.
“I know it’s hard to believe, but it’s likely an AI will kill us all in the future.”
could become
“It’s hard for me to come to terms with, but there doesn’t seem to be any natural safeguards preventing an AI from doing serious damage.”
Personally these kind of qualifiers rarely do anything to allay my doubt, and can easily increase them. I prefer to see incredulity. For instance, when a scientist has an amazing result, rather than seeing that they fully believe it but recognizing it’s difficult for me to believe, I’d rather see them doubtful of their own conclusion but standing by it nonetheless because of the strength of the evidence.
Sure, I totally agree with this—I prefer your formulation to my own. My point was just that there ought to be some disclaimer—the one that I suggested is a weak example.
Edit: Well, okay, actually I prefer:
“It took me a long time to come to terms with, but there don’t seem to be any natural safeguards preventing an AI from doing serious damage.”
If one has actually become convinced of a position, it sounds disingenuous to say that it’s hard for one to come to terms with at present, but any apparently absurd position should at some point have been hard to come to terms with.
Adding such a qualifier is a good caution against appearing to be placing oneself above the listener. It carries the message “I know how you must be feeling about these things, I’ve been there too.”
Promoting less than maximally accurate beliefs is an act of sabotage. Don’t do it to anyone unless you’d also slash their tires, because they’re Nazis or whatever.
set against Eliezer’s more stringent, TDT/mutually source code aware strategy of speaking the truth, even when the fate of the world is at stake.
Mblume merely presented the question without a recommended solution, but Alicorn came closest to your position in the comments.
Damnit! My smug self assurance that I could postpone thinking about these issues seriously because I’m an SIAI donor …. GONE! How am I supposed to get any work done now?
Seriously though, I do wish the SIAI toned down its self importance and incredible claims, however true they are. I realize, of course, that dulling some claims to appear more credible is approaching a Dark Side type strategy, but… well, no buts. I’m just confused.
Edit: I misunderstood what Jordan was trying to say—the previous version of this comment is irrelevant to the present discussion and so I’ve deleted it.
No one is claiming that honesty deserves top priority. I would lie to save someone’s life, or to make a few million dollars, etc. In the context of SIAI though, or any organization, being manipulative can severely discredit you.
If he were to go back on his incredible claims, or even only make more credible claims in the future, how would he reconcile the two when confronted? If someone new to Eliezer read his tame claims, then went back and read his older, more extreme claims, what would they think? To many people this would enforce the idea that SIAI is a cult, and that they are refining their image to be more attractive.
All of that said, I do understand where you’re coming from intuitively, and I’m not convinced that scaling back some of the SIAI claims would ever have a negative effect. Certainly, though, a public policy conversation about it would cast a pretty manipulative shade over SIAI. Hell, even this conversation could cast a nasty shade to some onlookers (to many people trying to judge SIAI, the two of us might be a sufficiently close proxy, even though we have no direct connections).
Okay, I misunderstood where you were coming from earlier, I thought you were making a general statement about the importance of stating one’s beliefs. Sorry about that.
In response to your present comments, I would say that though the phenomenon that you have in mind may be a PR issue, I think it would be less of a PR issue than what’s going on right now.
One thing that I would say is that I think that Eliezer would come across as much more credible simply by accompanying his weird sounding statements with disclaimers of the type “I know that what I’m saying probably sounds pretty ‘out there’ and understand if you don’t believe me, but I’ve thought about this hard, and I think...” See my remark here.
I mostly agree, although I’m still mulling it and think the issue is more complicated than it appears. One nitpick:
Personally these kind of qualifiers rarely do anything to allay my doubt, and can easily increase them. I prefer to see incredulity. For instance, when a scientist has an amazing result, rather than seeing that they fully believe it but recognizing it’s difficult for me to believe, I’d rather see them doubtful of their own conclusion but standing by it nonetheless because of the strength of the evidence.
“I know it’s hard to believe, but it’s likely an AI will kill us all in the future.”
could become
“It’s hard for me to come to terms with, but there doesn’t seem to be any natural safeguards preventing an AI from doing serious damage.”
Sure, I totally agree with this—I prefer your formulation to my own. My point was just that there ought to be some disclaimer—the one that I suggested is a weak example.
Edit: Well, okay, actually I prefer:
“It took me a long time to come to terms with, but there don’t seem to be any natural safeguards preventing an AI from doing serious damage.”
If one has actually become convinced of a position, it sounds disingenuous to say that it’s hard for one to come to terms with at present, but any apparently absurd position should at some point have been hard to come to terms with.
Adding such a qualifier is a good caution against appearing to be placing oneself above the listener. It carries the message “I know how you must be feeling about these things, I’ve been there too.”
Mblume one critical analysis of honesty, quoting Steven0461:
set against Eliezer’s more stringent, TDT/mutually source code aware strategy of speaking the truth, even when the fate of the world is at stake.
Mblume merely presented the question without a recommended solution, but Alicorn came closest to your position in the comments.