Maybe this is a silly question I should already know the answer to and/or not quite appropriate for this thread, but I can’t think of anywhere better to ask. Apologies also if it was already directly addressed.
What’s the point of suppressing discussion of certain concepts for fear of a superintelligent AI discovering them? Isn’t it implicit that said superintelligent AI will independently figure out anything we humans have come up with? Furthermore, what was said in Roko’s post will exist forever; the idea was released to the interwebs and its subsequent banning can’t undo this.
It’s not the attempt to hide an idea from a future AI, but something else entirely. I’ll reply (in vague terms) by private message.
EDIT: People keep asking me privately; and after thinking for a while, I’m confident that it’s not harmful to post my analogy of why a Topic could be rightly Banned (in addition to the very good reason that it can give people nightmares), without giving away what Banned Topic consists of. (If our benevolent overlords disagree on the wisdom of this, I’m happy to edit or delete my comment, and will gladly accept having it done for me.)
It should go without saying that if you want to turn this into a thread about Banned Topic or about the fitness of the analogy to Banned Topic, I cordially entreat you to go play the one-player Prisoner’s Dilemma instead.
Anyway, here’s my analogy:
Let’s say you’re a spy, and an enemy discovers your real identity but not your assumed one. They could then blackmail you by threatening people you know. But if you, in advance, made sure that you’d be genuinely incommunicado—that no message from or about your family could reach you while you were doing your duty—and if your enemy knew this about you, then they’d have no incentive to do so. (I mean, they could hurt your friends just to be a jerk, but it wouldn’t rationally gain them anything.)
So hiding information from yourself could be a really important move to make.
EDIT 2: I guess this still freaks people out, and I apologize. Just let it be clear that worrying about what Banned Topic could be is not equivalent to Banned Topic, this analogy won’t lead you to Banned Topic, and if you don’t already know what Banned Topic is, you’re not going to stumble on it by worrying. Honestly.
I can’t stop thinking about what the Banned Topic might be!
As I’m almost certain to get nightmares of some kind now—thanks—please, if you could, tell me I’m not likely to figure it out by accident, while dreaming. Because, um, I’m getting a very creeped-out feeling from all this.
Edit: I tried to stop thinking about what it might be. It’s not working. I’ve come up with several options, some of the less horrifying thanks to cousin it, but those.. not saying.
Maybe this is a silly question I should already know the answer to and/or not quite appropriate for this thread, but I can’t think of anywhere better to ask. Apologies also if it was already directly addressed.
What’s the point of suppressing discussion of certain concepts for fear of a superintelligent AI discovering them? Isn’t it implicit that said superintelligent AI will independently figure out anything we humans have come up with? Furthermore, what was said in Roko’s post will exist forever; the idea was released to the interwebs and its subsequent banning can’t undo this.
It’s not the attempt to hide an idea from a future AI, but something else entirely. I’ll reply (in vague terms) by private message.
EDIT: People keep asking me privately; and after thinking for a while, I’m confident that it’s not harmful to post my analogy of why a Topic could be rightly Banned (in addition to the very good reason that it can give people nightmares), without giving away what Banned Topic consists of. (If our benevolent overlords disagree on the wisdom of this, I’m happy to edit or delete my comment, and will gladly accept having it done for me.)
It should go without saying that if you want to turn this into a thread about Banned Topic or about the fitness of the analogy to Banned Topic, I cordially entreat you to go play the one-player Prisoner’s Dilemma instead.
Anyway, here’s my analogy:
Let’s say you’re a spy, and an enemy discovers your real identity but not your assumed one. They could then blackmail you by threatening people you know. But if you, in advance, made sure that you’d be genuinely incommunicado—that no message from or about your family could reach you while you were doing your duty—and if your enemy knew this about you, then they’d have no incentive to do so. (I mean, they could hurt your friends just to be a jerk, but it wouldn’t rationally gain them anything.)
So hiding information from yourself could be a really important move to make.
EDIT 2: I guess this still freaks people out, and I apologize. Just let it be clear that worrying about what Banned Topic could be is not equivalent to Banned Topic, this analogy won’t lead you to Banned Topic, and if you don’t already know what Banned Topic is, you’re not going to stumble on it by worrying. Honestly.
I can’t stop thinking about what the Banned Topic might be!
As I’m almost certain to get nightmares of some kind now—thanks—please, if you could, tell me I’m not likely to figure it out by accident, while dreaming. Because, um, I’m getting a very creeped-out feeling from all this.
Edit: I tried to stop thinking about what it might be. It’s not working. I’ve come up with several options, some of the less horrifying thanks to cousin it, but those.. not saying.
You’re not going to figure it out by accident, or even come close. Sorry for freaking you out, but you’re safe.
And also to me, if you would.
Banned post?! Send your private message to me too please.