We do have some models of [boundedly] rational principals with perfectly rational agents, and those models don’t display huge added agency rents. If you want to claim that relative intelligence creates large agency problems, you should offer concrete models that show such an effect.
The conclusions of those models seem very counterintuitive to me. I think the most likely explanation is that they make some assumptions that I do not expect to apply to the default scenarios involving humans and AGI. To check this, can you please reference some of the models that you had in mind when you wrote the above? (This might also help people construct concrete models that they would consider more realistic.)
Robin, I’m very confused by your response. The question I asked was for references to the specific models you talked about (with boundedly rational principals and perfectly rational agents), not how to find academic papers with the words “principal” and “agent” in them.
Did you misunderstand my question, or is this your way of saying “look it up yourself”? I have searched through the 5 review papers you cited in your blog post for mentions of models of this kind, and also searched on Google Scholar, with negative results. I can try to do more extensive searches but surely it’s a lot easier at this point if you could just tell me which models you were talking about?
Note that all three of the linked paper are about “boundedly rational agents with perfectly rational principals” or about “equally boundedly rational agents and principals”. I have been so far unable to find any papers that follow the described pattern of “boundedly rational principals and perfectly rational agents”.
It seems you consider previous AI booms to be a useful reference class for today’s progress in AI.
Suppose we will learn that the fraction of global GDP that currently goes into AI research is at least X times higher than in any previous AI boom. What is roughly the smallest X for which you’ll change your mind (i.e. no longer consider previous AI booms to be a useful reference class for today’s progress in AI)?
I’d also want to know that ratio X for each of the previous booms. There isn’t a discrete threshold, because analogies go on a continuum from more to less relevant. An unusually high X would be noteworthy and relevant, but not make prior analogies irrelevant.
Other than, say looking at our computers and comparing them to insects, what other signposts should we look for, if we want to calibrate progress towards domain-general artificial intelligence?
Mostly unrelated to your point about AI, your comments about the 100,000 fans having the potential to cause harm rang true to me.
Are there other areas in which you think the many non-expert fans problem is especially bad (as opposed to computer security, which you view as healthy in this respect)?
Then the experts can be reasonable and people can say, “Okay,” and take their word seriously, although they might not feel too much pressure to listen and do anything. If you can say that about computer security today, for example, the public doesn’t scream a bunch about computer security.
Would you consider progress on image recognition and machine translation as outside view evidence for lumpiness? Accuracies on ImageNet, an image classification benchmark, dropped by >10% over a 4-year period (graph below) mostly due to the successful scaling up of a type of neural network.
This also seems relevant to your point about AI researchers who have been in the field for a long time being more skeptical. My understanding is that most AI researchers would not have predicted such rapid progress on this benchmark before it happened.
That said, I can see how you still might argue this is an example of over-emphasizing a simple form of perception, which in reality is much more complicated and involves a bunch of different interlocking pieces.
My understanding is that this progress looks much less of a trend deviation when you scale it against the hardware and other resources devoted to these tasks. And of course in any larger area there are always subareas which happen to progress faster. So we have to judge how large is a subarea that is going faster, and is that size unusually large.
Life extension also suffers from the 100,000 fans hype problem.
I’ll respond to comments here, at least for a few days.
You previously wrote:
The conclusions of those models seem very counterintuitive to me. I think the most likely explanation is that they make some assumptions that I do not expect to apply to the default scenarios involving humans and AGI. To check this, can you please reference some of the models that you had in mind when you wrote the above? (This might also help people construct concrete models that they would consider more realistic.)
The literature is vast, but this gets you started: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C47&q=%22principal+agent%22&btnG=
Robin, I’m very confused by your response. The question I asked was for references to the specific models you talked about (with boundedly rational principals and perfectly rational agents), not how to find academic papers with the words “principal” and “agent” in them.
Did you misunderstand my question, or is this your way of saying “look it up yourself”? I have searched through the 5 review papers you cited in your blog post for mentions of models of this kind, and also searched on Google Scholar, with negative results. I can try to do more extensive searches but surely it’s a lot easier at this point if you could just tell me which models you were talking about?
If you specifically want models with “bounded rationality”, why do add in that search term: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C47&as_vis=1&q=bounded+rationality+principal+agent&btnG=
See also:
https://onlinelibrary.wiley.com/doi/abs/10.1111/geer.12111
https://www.mdpi.com/2073-4336/4/3/508
https://etd.ohiolink.edu/!etd.send_file?accession=miami153299521737861&disposition=inline
Note that all three of the linked paper are about “boundedly rational agents with perfectly rational principals” or about “equally boundedly rational agents and principals”. I have been so far unable to find any papers that follow the described pattern of “boundedly rational principals and perfectly rational agents”.
It seems you consider previous AI booms to be a useful reference class for today’s progress in AI.
Suppose we will learn that the fraction of global GDP that currently goes into AI research is at least X times higher than in any previous AI boom. What is roughly the smallest X for which you’ll change your mind (i.e. no longer consider previous AI booms to be a useful reference class for today’s progress in AI)?
[EDIT: added “at least”]
I’d also want to know that ratio X for each of the previous booms. There isn’t a discrete threshold, because analogies go on a continuum from more to less relevant. An unusually high X would be noteworthy and relevant, but not make prior analogies irrelevant.
Other than, say looking at our computers and comparing them to insects, what other signposts should we look for, if we want to calibrate progress towards domain-general artificial intelligence?
The % of world income that goes to computer hardware & software, and the % of useful tasks that are done by them.
Recent paper that might be relevant:
https://arxiv.org/abs/1911.01547
Mostly unrelated to your point about AI, your comments about the 100,000 fans having the potential to cause harm rang true to me.
Are there other areas in which you think the many non-expert fans problem is especially bad (as opposed to computer security, which you view as healthy in this respect)?
Would you consider progress on image recognition and machine translation as outside view evidence for lumpiness? Accuracies on ImageNet, an image classification benchmark, dropped by >10% over a 4-year period (graph below) mostly due to the successful scaling up of a type of neural network.
This also seems relevant to your point about AI researchers who have been in the field for a long time being more skeptical. My understanding is that most AI researchers would not have predicted such rapid progress on this benchmark before it happened.
That said, I can see how you still might argue this is an example of over-emphasizing a simple form of perception, which in reality is much more complicated and involves a bunch of different interlocking pieces.
My understanding is that this progress looks much less of a trend deviation when you scale it against the hardware and other resources devoted to these tasks. And of course in any larger area there are always subareas which happen to progress faster. So we have to judge how large is a subarea that is going faster, and is that size unusually large.
Life extension also suffers from the 100,000 fans hype problem.