It’s nice to separate the levels between modeling a decision theory, analyzing multiple theories, and actually implementing decisions. I don’t think I’d number them that way, or imply they’re single-dimensional.
It seems level 1 is a prereq for level 4, and 2 and 3 are somewhat inter-dependent, but it’s not clear that an agent needs to understand the formal theory in order to actually avoid D,D. The programmer of the agent does, but even then “understand” may be too much of a stretch, if evolutionary or lucky algorithms manage it.
but it’s not clear that an agent needs to understand the formal theory in order to actually avoid D,D.
They definitely don’t, in many cases—humans in PDs cooperate all the time, without actually understanding decision theory.
The hierarchy is meant to express that robustly avoiding (D,D) for decision theory-based reasons, requires either that the agent itself, or its programmers, understand and implement the theory.
Each level is intended to be a pre-requisite for the preceding levels, modulo the point that, in the case of programmed bots in a toy environment, the comprehension can be either in the bot itself, or in the programmers that built the bot.
I don’t see how level 2 depends on anything in level 3 - being at level 2 just means you understand the concept of a Nash equilibrium and why it is an attractor state. You have a desire to avoid it (in fact, you have that desire even at level 1), but you don’t know how to do so, robustly and formally.
It’s nice to separate the levels between modeling a decision theory, analyzing multiple theories, and actually implementing decisions. I don’t think I’d number them that way, or imply they’re single-dimensional.
It seems level 1 is a prereq for level 4, and 2 and 3 are somewhat inter-dependent, but it’s not clear that an agent needs to understand the formal theory in order to actually avoid D,D. The programmer of the agent does, but even then “understand” may be too much of a stretch, if evolutionary or lucky algorithms manage it.
They definitely don’t, in many cases—humans in PDs cooperate all the time, without actually understanding decision theory.
The hierarchy is meant to express that robustly avoiding (D,D) for decision theory-based reasons, requires either that the agent itself, or its programmers, understand and implement the theory.
Each level is intended to be a pre-requisite for the preceding levels, modulo the point that, in the case of programmed bots in a toy environment, the comprehension can be either in the bot itself, or in the programmers that built the bot.
I don’t see how level 2 depends on anything in level 3 - being at level 2 just means you understand the concept of a Nash equilibrium and why it is an attractor state. You have a desire to avoid it (in fact, you have that desire even at level 1), but you don’t know how to do so, robustly and formally.