The counter-counter argument is then that you should indeed assign a zero probability to anyone’s ability to produce arbitrary amounts of utility.
Yes, I know it is rhetorically claimed that 0 and 1 are not probabilities. I suggest that this example refutes that claim. You must assign zero probability to such things, otherwise you get money-pumped, and lose.
Well, as someone else suggested, you could just ignore all probabilities below a certain noise floor. You don’t necessarily have to assign 0 probability to those things, you could just make it a heuristic to ignore them.
All that does is adopt a different decision theory but not call it that, sidestepping the requirement to formalise and justify it. It’s a patch, not a solution, like solving FAI by saying we can just keep the AI in a box.
Right, that was pretty much my counter-argument against his argument.
The counter-counter argument is then that you should indeed assign a zero probability to anyone’s ability to produce arbitrary amounts of utility.
Yes, I know it is rhetorically claimed that 0 and 1 are not probabilities. I suggest that this example refutes that claim. You must assign zero probability to such things, otherwise you get money-pumped, and lose.
Well, as someone else suggested, you could just ignore all probabilities below a certain noise floor. You don’t necessarily have to assign 0 probability to those things, you could just make it a heuristic to ignore them.
All that does is adopt a different decision theory but not call it that, sidestepping the requirement to formalise and justify it. It’s a patch, not a solution, like solving FAI by saying we can just keep the AI in a box.