More like operationalization of a binding rule is not something even superintelligence can do for you, even when it gives the correct operationalization. Because if you follow operationalizations merely because they are formally correct and given by a superintelligence, then you follow arbitrary operationalizations, not ones you’ve decided to follow yourself. How would the superintelligence even know what you decide, if you shirk that responsibility and wait for the superintelligence to tell you?
This post is mostly a reaction to Bostrom’s podcasts about his new book. I think making your own decisions is a central example of something that can’t be solved by others, no matter how capable, and also this activity is very close to what it means to define values, so plans in the vicinity of external imposition of CEV might be missing the point.
One thought about the rule/exception discussion you’ve linked (in the next paragraph; this one sets up the framing). Rules/norms are primitives of acausal coordination, especially interesting when they are agents in their own right. They mostly live in other minds, and are occasionally directly incarnated in the world, outside other minds. When a norm lives in many minds, it exerts influence on the world through its decisions made inside its hosts. It has no influence where it has no hosts, and also where the hosts break the norm. Where it does have influence, it speaks in synchrony in many voices, through all of its instances, thus it can have surprising power even when it only weakly compels the minds that host its individual instances.
So there is a distinction between an exception that isn’t part of a rule, and an exception that the rule doesn’t plan for. Rules are often not very intelligent, so they can fail to plan for most things, and thus suggest stupid decisions that don’t take those things into account. This can be patched by adding those things into the rule, hardcoding the knowledge. But not every thing needs to be hardcoded in order for the rule to be able to adequately anticipate it in its decision making. This includes hosts (big agents) making an exception to the rule (not following it in a particular situation): some rules are able to anticipate when specifically that happens, and so don’t need those conditions to become part of their formulation.
If I understand you correctly, you are describing the same sort of thing as I mentioned in the footnote to this comment, yes?
More like operationalization of a binding rule is not something even superintelligence can do for you, even when it gives the correct operationalization. Because if you follow operationalizations merely because they are formally correct and given by a superintelligence, then you follow arbitrary operationalizations, not ones you’ve decided to follow yourself. How would the superintelligence even know what you decide, if you shirk that responsibility and wait for the superintelligence to tell you?
This post is mostly a reaction to Bostrom’s podcasts about his new book. I think making your own decisions is a central example of something that can’t be solved by others, no matter how capable, and also this activity is very close to what it means to define values, so plans in the vicinity of external imposition of CEV might be missing the point.
One thought about the rule/exception discussion you’ve linked (in the next paragraph; this one sets up the framing). Rules/norms are primitives of acausal coordination, especially interesting when they are agents in their own right. They mostly live in other minds, and are occasionally directly incarnated in the world, outside other minds. When a norm lives in many minds, it exerts influence on the world through its decisions made inside its hosts. It has no influence where it has no hosts, and also where the hosts break the norm. Where it does have influence, it speaks in synchrony in many voices, through all of its instances, thus it can have surprising power even when it only weakly compels the minds that host its individual instances.
So there is a distinction between an exception that isn’t part of a rule, and an exception that the rule doesn’t plan for. Rules are often not very intelligent, so they can fail to plan for most things, and thus suggest stupid decisions that don’t take those things into account. This can be patched by adding those things into the rule, hardcoding the knowledge. But not every thing needs to be hardcoded in order for the rule to be able to adequately anticipate it in its decision making. This includes hosts (big agents) making an exception to the rule (not following it in a particular situation): some rules are able to anticipate when specifically that happens, and so don’t need those conditions to become part of their formulation.