I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong
I think most people use “update” colloquially, i.e. something along the lines of “what you just said appears to constitute evidence that I wasn’t previously aware of, and I should change my beliefs accordingly”. I don’t know how often rationalists actually plug these things into a formula.
I don’t understand why you can’t just ignore some possible outcomes.
This is the problem of Pascal’s Mugging—I think it’s something that people are still confused about. In general, if someone tells you about a really weird possibility you should assign it a probability of a lot less than 50%, as it would essentially be a conjunction of a lot of unlikely events. The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
I’m still waiting for an answer to that one, but in the meantime it seems worth attacking the problems that lie in the grey area between “easily tractable expected utility calculations” and “classic Pascal’s mugging”. For me, AI risk mitigation is still in that grey area.
The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
Don’t worry about it; if you decline a Pascal’s mugging I’ll cause positive utility equal to twice the amount of negative utility you were threatened with, and if you accept one I’ll cause negative utility equal to twice what you were threatened with.
Why did I never think of this? I mean I have thought of very similar things in a thought experiment sense and I’ve even used it to explain to people that paying the mugging cannot be something correct but unpalatable, but it never occurred to me to use it on someone.
I think most people use “update” colloquially, i.e. something along the lines of “what you just said appears to constitute evidence that I wasn’t previously aware of, and I should change my beliefs accordingly”. I don’t know how often rationalists actually plug these things into a formula.
This is the problem of Pascal’s Mugging—I think it’s something that people are still confused about. In general, if someone tells you about a really weird possibility you should assign it a probability of a lot less than 50%, as it would essentially be a conjunction of a lot of unlikely events. The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
I’m still waiting for an answer to that one, but in the meantime it seems worth attacking the problems that lie in the grey area between “easily tractable expected utility calculations” and “classic Pascal’s mugging”. For me, AI risk mitigation is still in that grey area.
Don’t worry about it; if you decline a Pascal’s mugging I’ll cause positive utility equal to twice the amount of negative utility you were threatened with, and if you accept one I’ll cause negative utility equal to twice what you were threatened with.
Trust me.
Excuse me, I’m going to go and use this on all my friends.
Why did I never think of this? I mean I have thought of very similar things in a thought experiment sense and I’ve even used it to explain to people that paying the mugging cannot be something correct but unpalatable, but it never occurred to me to use it on someone.