Thoughts about moral uncertainty (I am giving up on writing long coherent posts, somebody help me with my ADHD):
What are the sources of moral uncertainty?
Moral realism is actually true and your moral uncertainty reflects your ignorance about moral truth. It seems to me that there is no much empirical evidence for resolving uncertainty-about-moral-truth and this kind of uncertainty is purely logical? I don’t believe in moral realism and what do you mean by talking about moral truth anyway, but I should mention it.
Identity uncertainty: you are not sure what kind of person you are. Here is a ton of embedding agency problems. For example, let’s say that you are 50% sure in utility function U1 and 50% sure in U2, and you need to choose between actions a1 and a2. Let’s suppose that U1 favors a1 and U2 favors a2. But expected value w.r.t moral uncertainty says that a1 is preferable. But Bayesian inference concludes that a1 is decisive evidence for U1 and updates towards 100% confidence in U1. It would be nice to find good way to deal with identity uncertainty.
Indirect normativity is a source of sort-of normative uncertainty: we know that we should, for example, implement CEV, but we don’t know details of CEV implementation. EDIT: I realized that this kind of uncertainty can be named “uncertainty by unactionable definition”—you know the description of your preference, but it is, for example, computationally untractable, so you need to discover efficiently computable proxies.
I think trying to be an EU maximizer without knowing a utility function is a bad idea. And without that, things like boundary-respecting norms and their acausal negotiation make more sense as primary concerns. Making decisions only within some scope of robustness where things make sense rather than in full generality, and defending a habitat (to remain) within that scope.
Right. I’m trying to find a decision theoretic frame for boundary norms for basically the same reason. Both situations are where agents might put themselves before they know what global preference they should endorse. But uncertainty never fully resolves, superintelligence or not, so anchoring to global expected utility maximization is not obviously relevant to anything. I’m currently guessing that the usual moral uncertainty frame is less sensible than building from a foundation of decision making in a simpler familiar environment (platonic environment, not directly part of the world), towards capability in wider environments.
Thoughts about moral uncertainty (I am giving up on writing long coherent posts, somebody help me with my ADHD):
What are the sources of moral uncertainty?
Moral realism is actually true and your moral uncertainty reflects your ignorance about moral truth. It seems to me that there is no much empirical evidence for resolving uncertainty-about-moral-truth and this kind of uncertainty is purely logical? I don’t believe in moral realism and what do you mean by talking about moral truth anyway, but I should mention it.
Identity uncertainty: you are not sure what kind of person you are. Here is a ton of embedding agency problems. For example, let’s say that you are 50% sure in utility function U1 and 50% sure in U2, and you need to choose between actions a1 and a2. Let’s suppose that U1 favors a1 and U2 favors a2. But expected value w.r.t moral uncertainty says that a1 is preferable. But Bayesian inference concludes that a1 is decisive evidence for U1 and updates towards 100% confidence in U1. It would be nice to find good way to deal with identity uncertainty.
Indirect normativity is a source of sort-of normative uncertainty: we know that we should, for example, implement CEV, but we don’t know details of CEV implementation. EDIT: I realized that this kind of uncertainty can be named “uncertainty by unactionable definition”—you know the description of your preference, but it is, for example, computationally untractable, so you need to discover efficiently computable proxies.
I think trying to be an EU maximizer without knowing a utility function is a bad idea. And without that, things like boundary-respecting norms and their acausal negotiation make more sense as primary concerns. Making decisions only within some scope of robustness where things make sense rather than in full generality, and defending a habitat (to remain) within that scope.
I am trying to study moral uncertainty foremost to clarify question about reflexion of superintelligence on its values and sharp left turn.
Right. I’m trying to find a decision theoretic frame for boundary norms for basically the same reason. Both situations are where agents might put themselves before they know what global preference they should endorse. But uncertainty never fully resolves, superintelligence or not, so anchoring to global expected utility maximization is not obviously relevant to anything. I’m currently guessing that the usual moral uncertainty frame is less sensible than building from a foundation of decision making in a simpler familiar environment (platonic environment, not directly part of the world), towards capability in wider environments.