So I have some doubt, and so I can be persuaded to change my mind.
I had in mind more the possibility of being persuaded by bad arguments than by good ones, as when an unfriendly AI persuades people to let it out.
ETA: Expanding on that, whenever you deal with another human being, it is like dealing with an artificial intelligence of unknown Friendliness. A human isn’t artificial, and doesn’t have the superfast superintelligence and unbounded capabilities (once out of its box) that are attributed to hypothetical AIs, but you are still at memetic risk unless you are so far above them in rationality that their memes pose no threat. But when dealing with someone of whom you know very little, how sure can you be of that? That they believe something that you have already dismissed as irrational is not a good indication that they must be generally stupid—see the counterexamples mentioned in this thread.
Even if they have good memes, how sure are you, that that is why you accepted them?
As you say, human beings are not superintelligent AIs of unknown friendliness.
I’m accustomed to dealing with human beings, including religious believers who are smarter than me (I’m related to a few). I think it quite likely that calcsam is smarter than me, but—and I can’t get past this—HE’S A MORMON. What on Earth could he possibly say to make that turd seem like spun gold? We’re close to two hundred years since Joseph Smith accomplished his amazing con job. In that time, there have been a lot of smart, diligent Mormons trying desperately to reconcile their faith with reality. They have come up with nothing, except that the mainstream has backtracked from some of the more painfully horrible aspects of their sect, like polygamy and racism. But it was a con and a lie from the beginning, and nothing will change that. Calcsam would have to have thought of some thing really, really new—something like the equivalent of a cold fusion reactor in his garage—to change my mind. I didn’t know about his blog at the time, but now it’s quite obvious he has nothing new at all.
In the unlikely event that I am ever put in the position of being the liason between humanity and a real superintelligent AI in a box, I would be terrified. People don’t scare me so much.
I had in mind more the possibility of being persuaded by bad arguments than by good ones, as when an unfriendly AI persuades people to let it out.
ETA: Expanding on that, whenever you deal with another human being, it is like dealing with an artificial intelligence of unknown Friendliness. A human isn’t artificial, and doesn’t have the superfast superintelligence and unbounded capabilities (once out of its box) that are attributed to hypothetical AIs, but you are still at memetic risk unless you are so far above them in rationality that their memes pose no threat. But when dealing with someone of whom you know very little, how sure can you be of that? That they believe something that you have already dismissed as irrational is not a good indication that they must be generally stupid—see the counterexamples mentioned in this thread.
Even if they have good memes, how sure are you, that that is why you accepted them?
As you say, human beings are not superintelligent AIs of unknown friendliness.
I’m accustomed to dealing with human beings, including religious believers who are smarter than me (I’m related to a few). I think it quite likely that calcsam is smarter than me, but—and I can’t get past this—HE’S A MORMON. What on Earth could he possibly say to make that turd seem like spun gold? We’re close to two hundred years since Joseph Smith accomplished his amazing con job. In that time, there have been a lot of smart, diligent Mormons trying desperately to reconcile their faith with reality. They have come up with nothing, except that the mainstream has backtracked from some of the more painfully horrible aspects of their sect, like polygamy and racism. But it was a con and a lie from the beginning, and nothing will change that. Calcsam would have to have thought of some thing really, really new—something like the equivalent of a cold fusion reactor in his garage—to change my mind. I didn’t know about his blog at the time, but now it’s quite obvious he has nothing new at all.
In the unlikely event that I am ever put in the position of being the liason between humanity and a real superintelligent AI in a box, I would be terrified. People don’t scare me so much.