I think the title should be rephrased, “If alignment is hard, then so is self-replication”.
Linear self-improvement seems a tenable proposition to me.
Your argument assumes (perhaps correctly) that a FOOM would require continual offloading of ‘greatest agency’ from one agent to another, as opposed to upgrading-in-place.
yea that’s cool to see. Very similar attempt at categorization. I feel we get caught up often in the potential / theoretical capabilities of systems. But there are already plenty of systems that fulfill self-replicating, harmful, intelligent behaviors. It’s entirely a question of degrees. That’s why a visual ranking of all systems’ metrics is in order I think.
Defining what comprises a ‘system’ would be the other big challenge. Is a hostile government a system? That’s fairly intelligent and self-replicating. etc.