Ah, MIRI summer fellows! Maybe that’s why there’s so many posts today.
I think that if there’s a dichotomy, it’s “abstract/ideal agents” vs. “physical ‘agents’”.
Physical agents, like humans, don’t have to be anything like agent clusters—there doesn’t have to be any ideal agent hiding inside them. Instead, thinking about them as agents is a descriptive step taken by us, the people modeling them. The key philosophical technology is the intentional stance.
(Yeah, I do feel like “read about the intentional stance” is this year’s “read the sequences”)
On to the meat of the post—agents are already very general, especially if you allow preferences over world-histories, at which point they become really general. Maybe it makes more sense to think of these things as languages in which some things are simple and others are complicated? At which point I think you have a straightforward distance function between languages (how surprising is one language one average to another), but no sense of equivalency aside from identical rankings.
Ah, MIRI summer fellows! Maybe that’s why there’s so many posts today.
I think that if there’s a dichotomy, it’s “abstract/ideal agents” vs. “physical ‘agents’”.
Physical agents, like humans, don’t have to be anything like agent clusters—there doesn’t have to be any ideal agent hiding inside them. Instead, thinking about them as agents is a descriptive step taken by us, the people modeling them. The key philosophical technology is the intentional stance.
(Yeah, I do feel like “read about the intentional stance” is this year’s “read the sequences”)
On to the meat of the post—agents are already very general, especially if you allow preferences over world-histories, at which point they become really general. Maybe it makes more sense to think of these things as languages in which some things are simple and others are complicated? At which point I think you have a straightforward distance function between languages (how surprising is one language one average to another), but no sense of equivalency aside from identical rankings.