tailcalled comments on Why The Focus on Expected Utility Maximisers?

[tailcalled]

I don’t think consequentialism is related to utility maximisation in the way you try to present it. There are many consequentialistic agent architectures that are explicitly not utility maximising, e. g. Active Inference, JEPA, ReduNets.

JEPA seems like it is basically utility maximizing to me. What distinction are you referring to?

I keep getting confused about Active Inference (I think I understood it once based on an equivalence to utility maximization, but it’s a while ago and you seem to be saying that this equivalence doesn’t hold), and I’m not familiar with ReduNets, so I would appreciate a link or an explainer to catch up.

Then you seem to switch your response to discussing that consequentialism is important for reaching the far-superhuman AI level. This looks at least plausible to me, but first, these far-superhuman AIs could have a non-UM consequentialistic agent architecture (see above), and second, DragonGod didn’t say that the risk is necessarily from far-superhuman AIs (even though non-UM ones): I believe he argued for that here. It’s possible even that far-superhuman intelligence is not a thing at all (except for the speed of cognition and the size of memory), but the risks that he highlights: human disempowerment and dystopian scenarios, still absolutely stand.

I was sort of addressing alternative risks in this paragraph:

Risk that arises purely from language models or non-consequentialist RLHF might be quite interesting and important to study. I feel less able to predict it, though, partly because I don’t know what the models will be deployed to do, or how much they can be coerced into doing, or what kinds of witchcraft are necessary to coerce the models into doing those things.