I think my main point is that “CHAI’s agenda depends strongly on an agent assumption” seems only true of the specific mathematical formalization that currently exists; I would not be surprised if the work could then be generalized to optimizing systems instead of agents / EU maximizers in particular.
Ah, very interesting, yeah I agree this seems plausible, and also this is very encouraging to me!
Ah, very interesting, yeah I agree this seems plausible, and also this is very encouraging to me!