DragonGod comments on Normative vs Descriptive Models of Agency

[DragonGod]

I’m a student of the descriptive school of agent foundations I think.


I suspect normative agent foundations research is just largely misguided/​mistaken. Quoting myself from my research philosophy draft:

Furthermore in worlds where there’s a paucity of empirical data, I don’t actually believe that we can necessarily develop towers of abstraction out of the void and have them remain rooted in reality^[16]. I expect theory developed in the absence of empirical data to validate it to largely just fail to describe reality^[17]. Our theories of safety must be rooted in descriptive models of the intelligent systems we’re dealing with to be useful at all.

And the relevant footnotes:

16. You cannot I posit develop an adequate theory of alignment from the null string as input.

17. I do not think there’s a simple theory of intelligence that can be derived even in principle from pure computer science (in the limiting case, no intelligence is feasible in max entropic universes).

To what extent is a particular universe learnable? What inductive biases and (hyper)priors are best for learning it? What efficient approximations exist for common algorithmic problems? How well do learning algorithms generalise for common problems? Etc. All seem like empirical questions about the reality we live in. And I expect the answers to these empirical questions to constrain what intelligent systems in our universe look like.

I am pessimistic about normative theories of intelligent systems that are divorced from the behaviour of intelligent systems in our universe.

E.g. I suspect that expected utility maximisation is anti-natural to generally intelligent behaviour in our universe. More generally, I suspect large portions of normative agent foundations research would fail to describe the systems that matter in the real world and thus end up being inapplicable/​unusable.

This is a crux behind my strong scepticism of theories constructed from void.

[mattmacdermott]

Do you expect useful generic descriptive models of agency to exist?

[DragonGod]

Yes, and I expect that agenda to be significantly more tractable for designing systems that are effective in the real world than normative agendas.

All that said, my preference is to draw the distinction around “models of intelligent systems” not “models of agency” as agents aren’t the only type of intelligent systems that matter (foundation models aren’t well described as agents).