My primary thesis is that the iterated Pascal’s mugging is much more likely to approximate any given real-world situation than the one-shot Pascal’s mugging, and that focusing on the latter is likely to lead by availability heuristic bias to people making bad decisions on important issues.
My primary thesis is that if you have programmed a purported god-like and friendly AI that you know will do poorly in one-shot Pascal’s mugging, then you should not turn it on. Even if you know it will do well in other variations on Pascal’s mugging.
My secondary thesis comes from Polya: “If there’s a problem that you can’t solve, then there’s a simpler problem that you can solve. Find it!” Solutions to, failed solutions to, and ideas about one-shot Pascal’s mugging will illuminate features about iterated Pascal’s mugging and also about many given real-world situations.
(“One-shot”, “iterated”...If these are even good names!)
Forget it. I’m just weirded out that you would respond to “here’s a tentative formalization of a simple version of Pascal’s mugging” with “even thinking about it is dangerous.” I don’t agree and I don’t understand the mindset.
I don’t mean to say that thinking about the one-shot is dangerous, only that grossly overemphasizing it relative to the iterated might be.
I hear about the one-shot all the time, and the iterated not at all, and I think the iterated is more likely to come up than the one-shot, and I think the iterated is easier to solve than the one-shot, so in all I think it’s completely reasonable for me to want to emphasize the iterated.
The iterated has an easy-to-accept-intuitively solution: don’t just randomly accept blackmail from anyone who offers it, but rather investigate first to see if they constitute a credible threat.
The one-shot Pascal’s Mugging, like most one-shot games discussed in game theory, has a harder-to-stomach dominant strategy: pay the ransom, because the mere claim, considered as Bayesian evidence, promotes the threat to much more likely than the reciprocal of its utility-magnitude.
My primary thesis is that the iterated Pascal’s mugging is much more likely to approximate any given real-world situation than the one-shot Pascal’s mugging, and that focusing on the latter is likely to lead by availability heuristic bias to people making bad decisions on important issues.
My primary thesis is that if you have programmed a purported god-like and friendly AI that you know will do poorly in one-shot Pascal’s mugging, then you should not turn it on. Even if you know it will do well in other variations on Pascal’s mugging.
My secondary thesis comes from Polya: “If there’s a problem that you can’t solve, then there’s a simpler problem that you can solve. Find it!” Solutions to, failed solutions to, and ideas about one-shot Pascal’s mugging will illuminate features about iterated Pascal’s mugging and also about many given real-world situations.
(“One-shot”, “iterated”...If these are even good names!)
I’m not persuaded that paying the ransom is doing poorly on the one-shot. And if it predictably does the wrong thing, in what sense is it Friendly?
Forget it. I’m just weirded out that you would respond to “here’s a tentative formalization of a simple version of Pascal’s mugging” with “even thinking about it is dangerous.” I don’t agree and I don’t understand the mindset.
I don’t mean to say that thinking about the one-shot is dangerous, only that grossly overemphasizing it relative to the iterated might be.
I hear about the one-shot all the time, and the iterated not at all, and I think the iterated is more likely to come up than the one-shot, and I think the iterated is easier to solve than the one-shot, so in all I think it’s completely reasonable for me to want to emphasize the iterated.
Granted! And
tell me more.
The iterated has an easy-to-accept-intuitively solution: don’t just randomly accept blackmail from anyone who offers it, but rather investigate first to see if they constitute a credible threat.
The one-shot Pascal’s Mugging, like most one-shot games discussed in game theory, has a harder-to-stomach dominant strategy: pay the ransom, because the mere claim, considered as Bayesian evidence, promotes the threat to much more likely than the reciprocal of its utility-magnitude.