I agree with you on what is the inner optimiser. I might not have been able to make myself super clear in the OP, but I see the “outer” alignment as some version “propagate our genes”, and I find it curious that that outer goal produced a very robust “want to have kids” inner alignment. I did also try to make the point that the alignment isn’t maximal in some way, as in yeah, we don’t have 16 kids, and men don’t donate to sperm banks as much as possible and other things that might maximise gene propagation, but even that I find interesting: we fulfill evolution’s “outer goal” somewhat, without going into paperclip-maximiser-style propagate genes at all cost. This seems to me like something we would want out of an AGI.
I agree with you on what is the inner optimiser. I might not have been able to make myself super clear in the OP, but I see the “outer” alignment as some version “propagate our genes”, and I find it curious that that outer goal produced a very robust “want to have kids” inner alignment. I did also try to make the point that the alignment isn’t maximal in some way, as in yeah, we don’t have 16 kids, and men don’t donate to sperm banks as much as possible and other things that might maximise gene propagation, but even that I find interesting: we fulfill evolution’s “outer goal” somewhat, without going into paperclip-maximiser-style propagate genes at all cost. This seems to me like something we would want out of an AGI.