Style suggestion: give an informal overview of the idea, like your original comment, before going into the details. New readers need to see the basic idea before they’ll be willing to wade into code.
Content suggestion: The main reason that I find your idea intriguing is something that you barely mention above: that because you’re giving the AI an optimization target that only cares about its immediate progeny, it won’t start cooperating with its later descendants (which would pretty clearly lead to un-boxing itself), nor upgrade to a decision theory that would cooperate further down the line. I think that part deserves more discussion.
Style suggestion: give an informal overview of the idea, like your original comment, before going into the details. New readers need to see the basic idea before they’ll be willing to wade into code.
Content suggestion: The main reason that I find your idea intriguing is something that you barely mention above: that because you’re giving the AI an optimization target that only cares about its immediate progeny, it won’t start cooperating with its later descendants (which would pretty clearly lead to un-boxing itself), nor upgrade to a decision theory that would cooperate further down the line. I think that part deserves more discussion.
Thanks, I’ve added a small overview section. I might edit this a little more later.