Not only does this seem wrong, but if I believed it I would want SI to look for the correct decision theory (roughly what Eliezer says he’s doing anyway). It fails to stress the possibility that Eliezer’s whole approach is wrong. In fact it seems willfully (heh) ignorant of the planning fallacy and similar concerns: even formalizing the ‘correct’ prior seems tricky to me, so why would it be feasible to formalize “correct” meta-ethics even if it exists in the sense you mean? And what reason do we have to believe that a version with no pointers to brains exists at all?
At least with reflective decision theory I see no good reason to think that a transparently-written AGI is impossible in principle (our neurons don’t just fire randomly, nor does evolution seem like a particularly good searcher of mindspace), so a theory of decisions that can describe said AGI’s actions should be mathematically possible barring some alternative to math. (Whether, eg, the description would fit in our observable universe seems like another question.)
Not only does this seem wrong, but if I believed it I would want SI to look for the correct decision theory (roughly what Eliezer says he’s doing anyway). It fails to stress the possibility that Eliezer’s whole approach is wrong. In fact it seems willfully (heh) ignorant of the planning fallacy and similar concerns: even formalizing the ‘correct’ prior seems tricky to me, so why would it be feasible to formalize “correct” meta-ethics even if it exists in the sense you mean? And what reason do we have to believe that a version with no pointers to brains exists at all?
At least with reflective decision theory I see no good reason to think that a transparently-written AGI is impossible in principle (our neurons don’t just fire randomly, nor does evolution seem like a particularly good searcher of mindspace), so a theory of decisions that can describe said AGI’s actions should be mathematically possible barring some alternative to math. (Whether, eg, the description would fit in our observable universe seems like another question.)