Robin, I still don’t understand why economic models predict only modest changes in agency problems, as you claimed here, when the principal is very limited and the agent is fully rational. I attempted to look through the literature, but did not find any models of this form. This is very likely because my literature search was not very good, as I am not an economist, so I would appreciate references.
That said, I would be very surprised if these references convinced me that a strongly superintelligent expected utility maximizer with a misaligned utility function (like “maximize the number of paperclips”) would not destroy almost all of the value from our source (assuming the AI itself is not valuable). To me, this is the extreme example of a principal-agent problem where the principal is limited and the agent is very capable. When I hear “principal-agent problems are not much worse with a smarter agent”, I hear “a paperclip maximizer wouldn’t destroy most of the value”, which seems crazy. Perhaps that is not what you mean though.
(Of course, you can argue that this scenario is not very likely, and I agree with that. I point to it mainly as a crystallization of the disagreement about principal-agent problems.)
In almost all applications, researchers assume that the agent (she) behaves according to one psychologically based model, while the principal (he) is fully rational and has a classical goal (usually profit maximization).
Optimal Delegation and Limited Awareness is relevant insofar as you consider an agent knowing more facts about the world is akin to them being more capable. Papers which consider contracting scenarios with bounded rationality, though not exactly principal-agent problems include Cognition and Incomplete Contracts and Satisfying Contracts. There are also some papers where the principal and agent have heterogenous priors, but the agent typically has the false prior. I’ve talked to a few economists about this, and they weren’t able to suggest anything I hadn’t seen (I don’t think my literature review is totally thorough, though).
Robin, I still don’t understand why economic models predict only modest changes in agency problems, as you claimed here, when the principal is very limited and the agent is fully rational. I attempted to look through the literature, but did not find any models of this form. This is very likely because my literature search was not very good, as I am not an economist, so I would appreciate references.
That said, I would be very surprised if these references convinced me that a strongly superintelligent expected utility maximizer with a misaligned utility function (like “maximize the number of paperclips”) would not destroy almost all of the value from our source (assuming the AI itself is not valuable). To me, this is the extreme example of a principal-agent problem where the principal is limited and the agent is very capable. When I hear “principal-agent problems are not much worse with a smarter agent”, I hear “a paperclip maximizer wouldn’t destroy most of the value”, which seems crazy. Perhaps that is not what you mean though.
(Of course, you can argue that this scenario is not very likely, and I agree with that. I point to it mainly as a crystallization of the disagreement about principal-agent problems.)
I’ve also found it hard to find relevant papers.
Behavioural Contract Theory reviews papers based on psychology findings and notes:
Optimal Delegation and Limited Awareness is relevant insofar as you consider an agent knowing more facts about the world is akin to them being more capable. Papers which consider contracting scenarios with bounded rationality, though not exactly principal-agent problems include Cognition and Incomplete Contracts and Satisfying Contracts. There are also some papers where the principal and agent have heterogenous priors, but the agent typically has the false prior. I’ve talked to a few economists about this, and they weren’t able to suggest anything I hadn’t seen (I don’t think my literature review is totally thorough, though).
Most models have an agent who is fully rational, but I’m not sure what you mean by “principal is very limited”.