I think the title should be rephrased, “If alignment is hard, then so is self-replication”.
Linear self-improvement seems a tenable proposition to me.
Your argument assumes (perhaps correctly) that a FOOM would require continual offloading of ‘greatest agency’ from one agent to another, as opposed to upgrading-in-place.
I think the title should be rephrased, “If alignment is hard, then so is self-replication”.
Linear self-improvement seems a tenable proposition to me.
Your argument assumes (perhaps correctly) that a FOOM would require continual offloading of ‘greatest agency’ from one agent to another, as opposed to upgrading-in-place.