I don’t have an issue bringing up MNT in these discussions, because our goal is to convince people that incautiously designed machine intelligence is a problem, and a major failure mode for people is that they say really stupid things like ’well, the machine won’t be able to do anything on its own because it’s just a computer—it’ll need humanity, therefore, it’ll never kill us all.” Even if MNT is impossible, that’s still true—but bringing up MNT provides people with an obvious intuitive path to the apocalypse. It isn’t guaranteed to happen, but it’s also not unlikely, and it’s a powerful educational tool for showing people the sorts of things that strong AI may be capable of.
That’s… not a strong criticism. There are compelling reasons not to believe that God is going to be a major force in steering the direction the future takes. The exact opposite is true for MNT—I’d bet at better-than-even odds that MNT will be a major factor in how things play out basically no matter what happens.
All we’re doing is providing people with a plausible scenario that contradicts flawed intuitions that they might have, in an effort to get them to revisit those intuitions and reconsider them. There’s nothing wrong with that. Would we need to do it if people were rational agents? No—but, as you may be aware, we definitely don’t live in that universe.
Of course there is. For starters, most of the good arguments are much more difficult to concisely explain, or invite more arguments from flawed intuitions. Remember, we’re not trying to feel smug in our rational superiority here; we’re trying to save the world.
It isn’t the sort of bad argument that gets refuted. The best someone can do is point out that there’s no guarantee that MNT is possible. In which case, the response is ‘Are you prepared to bet the human species on that? Besides, it doesn’t actually matter, because [insert more sophisticated argument about optimization power here].’ It doesn’t hurt you, and with the overwhelming majority of semi-literate audiences, it helps.
I don’t have an issue bringing up MNT in these discussions, because our goal is to convince people that incautiously designed machine intelligence is a problem, and a major failure mode for people is that they say really stupid things like ’well, the machine won’t be able to do anything on its own because it’s just a computer—it’ll need humanity, therefore, it’ll never kill us all.” Even if MNT is impossible, that’s still true—but bringing up MNT provides people with an obvious intuitive path to the apocalypse. It isn’t guaranteed to happen, but it’s also not unlikely, and it’s a powerful educational tool for showing people the sorts of things that strong AI may be capable of.
This is not a great argument, given that it works equally well if you replace MNT with God/Devil in the above.
That’s… not a strong criticism. There are compelling reasons not to believe that God is going to be a major force in steering the direction the future takes. The exact opposite is true for MNT—I’d bet at better-than-even odds that MNT will be a major factor in how things play out basically no matter what happens.
All we’re doing is providing people with a plausible scenario that contradicts flawed intuitions that they might have, in an effort to get them to revisit those intuitions and reconsider them. There’s nothing wrong with that. Would we need to do it if people were rational agents? No—but, as you may be aware, we definitely don’t live in that universe.
There is no need to use known bad arguments when there are so many good ones.
Of course there is. For starters, most of the good arguments are much more difficult to concisely explain, or invite more arguments from flawed intuitions. Remember, we’re not trying to feel smug in our rational superiority here; we’re trying to save the world.
if your bad argument gets refuted, you lose whatever credibility you may have had.
It isn’t the sort of bad argument that gets refuted. The best someone can do is point out that there’s no guarantee that MNT is possible. In which case, the response is ‘Are you prepared to bet the human species on that? Besides, it doesn’t actually matter, because [insert more sophisticated argument about optimization power here].’ It doesn’t hurt you, and with the overwhelming majority of semi-literate audiences, it helps.