Humans are not immediately prepared to solve many decision problems, and one of the hardest problems is formulation of preference for a consequentialist agent. In expanding the scope of well-defined/reasonable decisions, formulating our goals well enough for use in a formal decision theory is perhaps the last milestone, far outside of what can be reached with a lot of work!
Indirect normativity (after distillation) can make the timeline for reaching this milestone mostly irrelevant, as long as there is sufficient capability to compute the outcome, and amplification is about capability. It’s unclear how the scope of reasonable decisions is related to capability within that scope, amplification seems ambiguous between the two, perhaps the scope of reasonable decisions is just another kind of stuff that can be improved. And it’s corrigibility’s aspect to keep AI within the scope of well-defined decisions.
But with these principles in place, it’s unclear if formulating goals for consequentialist agents remains a thing, when instead it’s possible to just continue to expand the scope of reasonable decisions and to distill/amplify them.
formulating our goals well enough for use in a formal decision theory is perhaps the last milestone, far outside of what can be reached with a lot of work!
Did you mean to say “without a lot of work”?
(Or did you really mean to say that we can’t reach it, even with a lot of work?)
The latter, where “a lot of work” is the kind of thing humanity can manage in subjective centuries. In an indirect normativity design, doing much more work than that should still be feasible, since it’s only specified abstractly, to be predicted by an AI, enabling distillation. So we can still reach it, if there is an AI to compute the result. But if there is already such an AI, perhaps the work is pointless, because the AI can carry out the work’s purpose in a different way.
Humans are not immediately prepared to solve many decision problems, and one of the hardest problems is formulation of preference for a consequentialist agent. In expanding the scope of well-defined/reasonable decisions, formulating our goals well enough for use in a formal decision theory is perhaps the last milestone, far outside of what can be reached with a lot of work!
Indirect normativity (after distillation) can make the timeline for reaching this milestone mostly irrelevant, as long as there is sufficient capability to compute the outcome, and amplification is about capability. It’s unclear how the scope of reasonable decisions is related to capability within that scope, amplification seems ambiguous between the two, perhaps the scope of reasonable decisions is just another kind of stuff that can be improved. And it’s corrigibility’s aspect to keep AI within the scope of well-defined decisions.
But with these principles in place, it’s unclear if formulating goals for consequentialist agents remains a thing, when instead it’s possible to just continue to expand the scope of reasonable decisions and to distill/amplify them.
Did you mean to say “without a lot of work”?
(Or did you really mean to say that we can’t reach it, even with a lot of work?)
The latter, where “a lot of work” is the kind of thing humanity can manage in subjective centuries. In an indirect normativity design, doing much more work than that should still be feasible, since it’s only specified abstractly, to be predicted by an AI, enabling distillation. So we can still reach it, if there is an AI to compute the result. But if there is already such an AI, perhaps the work is pointless, because the AI can carry out the work’s purpose in a different way.