Daniel Kokotajlo comments on Definitions of “objective” should be Probable and Predictive

[Daniel Kokotajlo]

OK, I agree that what I said was probably a bit too pessimistic. But still, I wanna say “citation needed” for this claim:

Only paperclip maximizers prefer this on reflection, not anything remotely person-like, such as an LLM originating in training on human culture.

[Vladimir_Nesov]

The practical implication of this hunch (for unfortunately I don’t see how this could get a meaningfully clearer justification) is that clever alignment architectures are a risk, if they lead to more alien AGIs. Too much tuning and we might get that penny-pinching cannibal.