Deciding that the truth unconditionally deserves top priority seems to me to be an overly convenient easy way out of confronting the challenges of demanded by instrumental rationality.
No one is claiming that honesty deserves top priority. I would lie to save someone’s life, or to make a few million dollars, etc. In the context of SIAI though, or any organization, being manipulative can severely discredit you.
I believe that when one takes into account unintended consequences, when Eliezer makes his most incredible claims he lowers overall levels of epistemic rationality rather than raising overall levels of epistemic rationality.
If he were to go back on his incredible claims, or even only make more credible claims in the future, how would he reconcile the two when confronted? If someone new to Eliezer read his tame claims, then went back and read his older, more extreme claims, what would they think? To many people this would enforce the idea that SIAI is a cult, and that they are refining their image to be more attractive.
All of that said, I do understand where you’re coming from intuitively, and I’m not convinced that scaling back some of the SIAI claims would ever have a negative effect. Certainly, though, a public policy conversation about it would cast a pretty manipulative shade over SIAI. Hell, even this conversation could cast a nasty shade to some onlookers (to many people trying to judge SIAI, the two of us might be a sufficiently close proxy, even though we have no direct connections).
Okay, I misunderstood where you were coming from earlier, I thought you were making a general statement about the importance of stating one’s beliefs. Sorry about that.
In response to your present comments, I would say that though the phenomenon that you have in mind may be a PR issue, I think it would be less of a PR issue than what’s going on right now.
One thing that I would say is that I think that Eliezer would come across as much more credible simply by accompanying his weird sounding statements with disclaimers of the type “I know that what I’m saying probably sounds pretty ‘out there’ and understand if you don’t believe me, but I’ve thought about this hard, and I think...” See my remark here.
I mostly agree, although I’m still mulling it and think the issue is more complicated than it appears. One nitpick:
“I know that what I’m saying probably sounds pretty ‘out there’ and understand if you don’t believe me, but I’ve thought about this hard, and I think...”
Personally these kind of qualifiers rarely do anything to allay my doubt, and can easily increase them. I prefer to see incredulity. For instance, when a scientist has an amazing result, rather than seeing that they fully believe it but recognizing it’s difficult for me to believe, I’d rather see them doubtful of their own conclusion but standing by it nonetheless because of the strength of the evidence.
“I know it’s hard to believe, but it’s likely an AI will kill us all in the future.”
could become
“It’s hard for me to come to terms with, but there doesn’t seem to be any natural safeguards preventing an AI from doing serious damage.”
Personally these kind of qualifiers rarely do anything to allay my doubt, and can easily increase them. I prefer to see incredulity. For instance, when a scientist has an amazing result, rather than seeing that they fully believe it but recognizing it’s difficult for me to believe, I’d rather see them doubtful of their own conclusion but standing by it nonetheless because of the strength of the evidence.
Sure, I totally agree with this—I prefer your formulation to my own. My point was just that there ought to be some disclaimer—the one that I suggested is a weak example.
Edit: Well, okay, actually I prefer:
“It took me a long time to come to terms with, but there don’t seem to be any natural safeguards preventing an AI from doing serious damage.”
If one has actually become convinced of a position, it sounds disingenuous to say that it’s hard for one to come to terms with at present, but any apparently absurd position should at some point have been hard to come to terms with.
Adding such a qualifier is a good caution against appearing to be placing oneself above the listener. It carries the message “I know how you must be feeling about these things, I’ve been there too.”
No one is claiming that honesty deserves top priority. I would lie to save someone’s life, or to make a few million dollars, etc. In the context of SIAI though, or any organization, being manipulative can severely discredit you.
If he were to go back on his incredible claims, or even only make more credible claims in the future, how would he reconcile the two when confronted? If someone new to Eliezer read his tame claims, then went back and read his older, more extreme claims, what would they think? To many people this would enforce the idea that SIAI is a cult, and that they are refining their image to be more attractive.
All of that said, I do understand where you’re coming from intuitively, and I’m not convinced that scaling back some of the SIAI claims would ever have a negative effect. Certainly, though, a public policy conversation about it would cast a pretty manipulative shade over SIAI. Hell, even this conversation could cast a nasty shade to some onlookers (to many people trying to judge SIAI, the two of us might be a sufficiently close proxy, even though we have no direct connections).
Okay, I misunderstood where you were coming from earlier, I thought you were making a general statement about the importance of stating one’s beliefs. Sorry about that.
In response to your present comments, I would say that though the phenomenon that you have in mind may be a PR issue, I think it would be less of a PR issue than what’s going on right now.
One thing that I would say is that I think that Eliezer would come across as much more credible simply by accompanying his weird sounding statements with disclaimers of the type “I know that what I’m saying probably sounds pretty ‘out there’ and understand if you don’t believe me, but I’ve thought about this hard, and I think...” See my remark here.
I mostly agree, although I’m still mulling it and think the issue is more complicated than it appears. One nitpick:
Personally these kind of qualifiers rarely do anything to allay my doubt, and can easily increase them. I prefer to see incredulity. For instance, when a scientist has an amazing result, rather than seeing that they fully believe it but recognizing it’s difficult for me to believe, I’d rather see them doubtful of their own conclusion but standing by it nonetheless because of the strength of the evidence.
“I know it’s hard to believe, but it’s likely an AI will kill us all in the future.”
could become
“It’s hard for me to come to terms with, but there doesn’t seem to be any natural safeguards preventing an AI from doing serious damage.”
Sure, I totally agree with this—I prefer your formulation to my own. My point was just that there ought to be some disclaimer—the one that I suggested is a weak example.
Edit: Well, okay, actually I prefer:
“It took me a long time to come to terms with, but there don’t seem to be any natural safeguards preventing an AI from doing serious damage.”
If one has actually become convinced of a position, it sounds disingenuous to say that it’s hard for one to come to terms with at present, but any apparently absurd position should at some point have been hard to come to terms with.
Adding such a qualifier is a good caution against appearing to be placing oneself above the listener. It carries the message “I know how you must be feeling about these things, I’ve been there too.”