My current candidate definitions, with some significant issues in the footnotes:
A fair environment is a probabilistic function from an array of actions to an array of payoffs.
An agent is a random variable
which takes in a fair environment [1] and a list of agents (including itself), and outputs a mixed strategy over its available actions in . [2]
A fair agent is one whose mixed strategy is a function of subjective probabilities[3] that it assigns to [the actions of some finite collection of agents in fair environments, where any agents not appearing in the original problem must themselves be fair].
Formally, if is a fair agent in with a subjective probability estimator , ’s mixed strategy in a fair environment ,
should depend only on a finite collection of ’s subjective probabilities about outcomes
for a set of fair environments and an additional set of fair[4] agents[5] if needed (note that not all agents need to appear in all environments).
A fair problem is a fair environment with one designated player, where all other agents are fair agents.
- ^
I might need to require every to have a default action , so that I don’t need to worry about axiom-of-choice issues when defining an agent over the space of all fair environments.
- ^
I specified a probabilistic environment and mixed strategies because I think there should be a unique fixed point for agents, such that this is well-defined for any fair environment . (By analogy to reflective oracles.) But I might be wrong, or I might need further restrictions on .
- ^
Grossly underspecified. What kinds of properties are required for subjective probabilities here? You can obviously cheat by writing BlueEyedBot into your probability estimator.
- ^
This is an infinite recursion, of course. It works if we require each to have a strictly lower complexity in some sense than (e.g. the rank of an agent is the largest number of environments it can reason about when making any decision, and each needs to be lower-rank than ), but I worry that’s too strong of a restriction and would exclude some well-definable and interesting agents.
- ^
Does the fairness requirement on the suffice to avert the MetaBlueEyedBot problem in general? I’m unsure.
[EDIT: Never mind, this is just Kleene’s second recursion theorem!]
Quick question about Kleene’s recursion theorem:
Let’s say F is a computable function from ℕ^N to ℕ. Is there a single computable function X from ℕ^N to ℕ such that
X = F(X, y_2,..., y_N) for all y_2,...,y_N in ℕ
(taking the X within F as the binary code of X in a fixed encoding) or do there need to be additional conditions?