I’m not sure you’re talking about the same thing I am, or maybe I’m just not following you?
There is only C1 and C2. C2 serves as a grounding that checks to see if what it would pick given the experiences it went through is acceptable to C1′s initial state. C1 would not have the “virgin imagination”, it would be the one hooked up to the wirehead machine.
Really I was thinking about the “Last Judge” idea from the CEV, which (as I understand it, but it is super vague so maybe I don’t) basically somehow has someone peek at the solution given by the CEV and decide whether the outcome is acceptable from the outside.
Aside from my accidental swapping of the terms (C1 as the judge, not C2), I still stand by my (unclear, possibly?) opinion. In the situation you are describing, the “judge” would never allow the agent to change beyond a very small distance that the judge is comfortable with, and additional checks would never be necessary, as it would only be logical that the judge’s opinion would be the same every time that an improvement was considered. Whichever of the states that the judge finds acceptable the first time, should become the new base state for the judge. Similarly, in real life, you don’t hold your preferences to the same standards that you had when you were five years old. The gradual improvements in cognition usually justify the risks of updating one’s values, in my opinion.
I’m not sure you’re talking about the same thing I am, or maybe I’m just not following you? There is only C1 and C2. C2 serves as a grounding that checks to see if what it would pick given the experiences it went through is acceptable to C1′s initial state. C1 would not have the “virgin imagination”, it would be the one hooked up to the wirehead machine.
Really I was thinking about the “Last Judge” idea from the CEV, which (as I understand it, but it is super vague so maybe I don’t) basically somehow has someone peek at the solution given by the CEV and decide whether the outcome is acceptable from the outside.
Aside from my accidental swapping of the terms (C1 as the judge, not C2), I still stand by my (unclear, possibly?) opinion. In the situation you are describing, the “judge” would never allow the agent to change beyond a very small distance that the judge is comfortable with, and additional checks would never be necessary, as it would only be logical that the judge’s opinion would be the same every time that an improvement was considered. Whichever of the states that the judge finds acceptable the first time, should become the new base state for the judge. Similarly, in real life, you don’t hold your preferences to the same standards that you had when you were five years old. The gradual improvements in cognition usually justify the risks of updating one’s values, in my opinion.