This makes an interesting parallel to the AI Box challenge. It seems obvious to me (without ever having participated in that challenge) that on no account should anyone let an AI of unknown friendliness out of a box merely on account of having had a conversation with it. And yet, many participants in that challenge do let it out, so if I engaged with the AI in that experiment, I cannot be sure a priori of what I would actually do.
You may be sure that no mere conversation with calcsam or anyone else could convince you to convert to a religion, but if your sureness is based only on the arguments you have seen, how sure can you really be that there isn’t an argument you have not seen, that you would accept as refuting all of that?
...but if your sureness is based only on the arguments you have seen, how sure can you really be that there isn’t an argument that you would accept as refuting all of that?
If Joseph Smith was a prophet, then I desire to believe that Joseph Smith was a prophet.
Right now, I’m very confident that Joseph Smith was a lecherous, manipulative, lying charlatan who patched together his church doctrine out of whatever superstitions he happened to have come across in his life. But I can’t prove this to a 100% degree of certainty. So I have some doubt, and so I can be persuaded to change my mind.
So I have some doubt, and so I can be persuaded to change my mind.
I had in mind more the possibility of being persuaded by bad arguments than by good ones, as when an unfriendly AI persuades people to let it out.
ETA: Expanding on that, whenever you deal with another human being, it is like dealing with an artificial intelligence of unknown Friendliness. A human isn’t artificial, and doesn’t have the superfast superintelligence and unbounded capabilities (once out of its box) that are attributed to hypothetical AIs, but you are still at memetic risk unless you are so far above them in rationality that their memes pose no threat. But when dealing with someone of whom you know very little, how sure can you be of that? That they believe something that you have already dismissed as irrational is not a good indication that they must be generally stupid—see the counterexamples mentioned in this thread.
Even if they have good memes, how sure are you, that that is why you accepted them?
As you say, human beings are not superintelligent AIs of unknown friendliness.
I’m accustomed to dealing with human beings, including religious believers who are smarter than me (I’m related to a few). I think it quite likely that calcsam is smarter than me, but—and I can’t get past this—HE’S A MORMON. What on Earth could he possibly say to make that turd seem like spun gold? We’re close to two hundred years since Joseph Smith accomplished his amazing con job. In that time, there have been a lot of smart, diligent Mormons trying desperately to reconcile their faith with reality. They have come up with nothing, except that the mainstream has backtracked from some of the more painfully horrible aspects of their sect, like polygamy and racism. But it was a con and a lie from the beginning, and nothing will change that. Calcsam would have to have thought of some thing really, really new—something like the equivalent of a cold fusion reactor in his garage—to change my mind. I didn’t know about his blog at the time, but now it’s quite obvious he has nothing new at all.
In the unlikely event that I am ever put in the position of being the liason between humanity and a real superintelligent AI in a box, I would be terrified. People don’t scare me so much.
This makes an interesting parallel to the AI Box challenge. It seems obvious to me (without ever having participated in that challenge) that on no account should anyone let an AI of unknown friendliness out of a box merely on account of having had a conversation with it. And yet, many participants in that challenge do let it out, so if I engaged with the AI in that experiment, I cannot be sure a priori of what I would actually do.
You may be sure that no mere conversation with calcsam or anyone else could convince you to convert to a religion, but if your sureness is based only on the arguments you have seen, how sure can you really be that there isn’t an argument you have not seen, that you would accept as refuting all of that?
If Joseph Smith was a prophet, then I desire to believe that Joseph Smith was a prophet.
Right now, I’m very confident that Joseph Smith was a lecherous, manipulative, lying charlatan who patched together his church doctrine out of whatever superstitions he happened to have come across in his life. But I can’t prove this to a 100% degree of certainty. So I have some doubt, and so I can be persuaded to change my mind.
Note: this is never a relevant shortcoming to concern one’s self with.
I had in mind more the possibility of being persuaded by bad arguments than by good ones, as when an unfriendly AI persuades people to let it out.
ETA: Expanding on that, whenever you deal with another human being, it is like dealing with an artificial intelligence of unknown Friendliness. A human isn’t artificial, and doesn’t have the superfast superintelligence and unbounded capabilities (once out of its box) that are attributed to hypothetical AIs, but you are still at memetic risk unless you are so far above them in rationality that their memes pose no threat. But when dealing with someone of whom you know very little, how sure can you be of that? That they believe something that you have already dismissed as irrational is not a good indication that they must be generally stupid—see the counterexamples mentioned in this thread.
Even if they have good memes, how sure are you, that that is why you accepted them?
As you say, human beings are not superintelligent AIs of unknown friendliness.
I’m accustomed to dealing with human beings, including religious believers who are smarter than me (I’m related to a few). I think it quite likely that calcsam is smarter than me, but—and I can’t get past this—HE’S A MORMON. What on Earth could he possibly say to make that turd seem like spun gold? We’re close to two hundred years since Joseph Smith accomplished his amazing con job. In that time, there have been a lot of smart, diligent Mormons trying desperately to reconcile their faith with reality. They have come up with nothing, except that the mainstream has backtracked from some of the more painfully horrible aspects of their sect, like polygamy and racism. But it was a con and a lie from the beginning, and nothing will change that. Calcsam would have to have thought of some thing really, really new—something like the equivalent of a cold fusion reactor in his garage—to change my mind. I didn’t know about his blog at the time, but now it’s quite obvious he has nothing new at all.
In the unlikely event that I am ever put in the position of being the liason between humanity and a real superintelligent AI in a box, I would be terrified. People don’t scare me so much.