“You are not perfectly rational” is certainly an understatement, and it does seem to be an excellent catch-all for ways in which a non-brain-melting truth might be dangerous to me… but by that token, a utility-improving falsehood might be quite dangerous to me too, no? It’s unlikely that my current preferences can accurately be represented by a self-consistent utility function, and since my volition hasn’t been professionally extrapolated yet, it’s easy to imagine false utopias that might be an improvement by the metric of my current “utility function” but turn out to be dystopian upon actual experience.
Suppose someone’s been brainwashed to the point that their utility function is “I want to obey The Leader as best as I can”—do you think that after reflection they’d be better off with a utility-maximizing falsehood or with a current-utility-minimizing truth?
The problem does not concern itself with merely ‘better off’, since a metric like ‘better off’ instead of ‘utility’ implies ‘better off’ as defined by someone else. Since Omega knows everything you know and don’t know (by the definition of the problem, since it’s presenting (dis)optimal information based on it’s knowledge of your knowledge), it is in a position to extrapolate your utility function. Accordingly, it maximizes/minimizes for your current utility function, not its own, and certainly not some arbitrary utility function deemed to be optimal for humans by whomever. If your utility function is such that you hold the well-being of another above yourself (maybe you’re a cultist of some kind, true… but maybe you’re just a radically altruistic utilitarian), then the results of optimizing your utility will not necessarily leave you any better off. If you bind your utility function the aggregate utility of all humanity, then maximizing that is something good for all humanity. If you bind it to one specific non-you person, then that person gets a maximized utility. Omega does not discriminate between the cases… but if it is trying to minimize your long-term utility, a handy way to do so is to get you to act against your current utility function.
Accordingly, yes; a current-utility-minimizing truth could possibly be ‘better’ by most definitions for a cultist then a current-utility-maximizing falsehood. Beware, though; reversed stupidity is not intelligence. Being convinced to ruin Great Leader’s life or even murder him outright might be better for you than blindly serving him and making him dictator of everything, but that hardly means there’s nothing better you could be doing. The fact that there exists a class of perverse utility functions which have negative consequences for those adopting them (and which can thus be positively reversed) does not imply that it’s a good idea to try inverting your utility function in general.
“You are not perfectly rational” is certainly an understatement, and it does seem to be an excellent catch-all for ways in which a non-brain-melting truth might be dangerous to me… but by that token, a utility-improving falsehood might be quite dangerous to me too, no? It’s unlikely that my current preferences can accurately be represented by a self-consistent utility function, and since my volition hasn’t been professionally extrapolated yet, it’s easy to imagine false utopias that might be an improvement by the metric of my current “utility function” but turn out to be dystopian upon actual experience.
Suppose someone’s been brainwashed to the point that their utility function is “I want to obey The Leader as best as I can”—do you think that after reflection they’d be better off with a utility-maximizing falsehood or with a current-utility-minimizing truth?
The problem does not concern itself with merely ‘better off’, since a metric like ‘better off’ instead of ‘utility’ implies ‘better off’ as defined by someone else. Since Omega knows everything you know and don’t know (by the definition of the problem, since it’s presenting (dis)optimal information based on it’s knowledge of your knowledge), it is in a position to extrapolate your utility function. Accordingly, it maximizes/minimizes for your current utility function, not its own, and certainly not some arbitrary utility function deemed to be optimal for humans by whomever. If your utility function is such that you hold the well-being of another above yourself (maybe you’re a cultist of some kind, true… but maybe you’re just a radically altruistic utilitarian), then the results of optimizing your utility will not necessarily leave you any better off. If you bind your utility function the aggregate utility of all humanity, then maximizing that is something good for all humanity. If you bind it to one specific non-you person, then that person gets a maximized utility. Omega does not discriminate between the cases… but if it is trying to minimize your long-term utility, a handy way to do so is to get you to act against your current utility function.
Accordingly, yes; a current-utility-minimizing truth could possibly be ‘better’ by most definitions for a cultist then a current-utility-maximizing falsehood. Beware, though; reversed stupidity is not intelligence. Being convinced to ruin Great Leader’s life or even murder him outright might be better for you than blindly serving him and making him dictator of everything, but that hardly means there’s nothing better you could be doing. The fact that there exists a class of perverse utility functions which have negative consequences for those adopting them (and which can thus be positively reversed) does not imply that it’s a good idea to try inverting your utility function in general.