Allow me to restate the problem in my own words. We are asked to produce a general purpose function named “agent()” which returns a positive integer. The returned result is intended to represent a choice (made by “agent()”) among a discrete collection of options. There is one implicit parameter to agent—the source code for second function named “world()”. This second function returns an integer result as well, but this result is to be interpreted as a kind of payoff. The objective in this exercise is to produce an agent() function which maximizes the result returned by an arbitrary world().
The code of the function world() is expected to include one or more calls to agent(). For example, when modeling a Newcomb or Parfit’s hitchhiker decision, the world() will call agent() at least once on behalf of Omega in order to predict what the agent will do later. Then world calls agent() a second time, this time “for real”. Or, in a symmetric Prisoner’s dilemma, agent() might be called twice, once for each decision-making prisoner.
Slepnev (cousin it?) suggests that the way agent() should be structured is as a theorem-prover, rather than as a simulator. (This approach has the important technical advantage of better control over non-termination.)
Ok, that is my summary—my understanding of what is going on here. Now for my questions:
Do I seem to understand?
Did I leave out any key points?
Is there any reason not to extend this to situations in which world() passes parameters to agent(). For example, if we wanted to do a PD in which the payoffs to the two players are slightly different, world() could pass different actual values in its two calls to agent(), because those two calls represent different players. So we want a parameter to represent “player id”.
Is there any reason not to allow world() to pass parameters representing random numbers to the agent()s? This would allow the agents to choose mixed or correlated strategies, if these kinds of strategies prove to be optimal.
The way the Newcomb example was written, it was pretty clear that agent() was being assumed to be deterministic. How essential is that assumption to this whole enterprise?
That is enough questions for now. Depending on the answers, I may have follow-ups.
Yeah. Sadly I didn’t use my real name when I registered. Sorry if that got anyone confused. Nesov uses my real name because we had many email exchanges, and also because he’s a university friend of my brother’s :-)
In the draft, I used your nickname at first, but in “prior work” section that sounded rather strange, so I replaced it with your real name everywhere. Glad to see that agrees with your preference.
The code of the function world() is expected to include one or more calls to agent()
Not necessarily (or not exactly). Where I used agent2() for a predictor, I could use agent3() for the agent as well. See this comment.
Extensions are possible, but not relevant to the main point of the model. It doesn’t seem natural to model different players with the same agent-program. Observations are not part of the model, but you could consider the agent as producing a program with parameters (instead of an unstructured constant) as a result of its decision.
The other use for inputs is information passed to agents through sensory observation. The extension is extremely natural.
“random numbers”: random numbers in nature are really pseudorandom numbers—numbers we lack the computational power to predict. In a full model of the universe, random numbers are not necessary—but different possible worlds with different laws of physics and some probability mass assigned to each, are.
Any randomness in the agent would really be pseudorandomness.
The other use for inputs is information passed to agents through sensory observation.
I don’t believe you can obtain proper information about the world through observation (unless you are a human and don’t know what you want). Preference is unchanging and is defined subjectively, so formal agents can’t learn new things about it (apart from resolution of logical uncertainty). Observations play a role in how plans play out (plans are prepared to be conditional on observations), or in priority for preparing said plans as observations come in, but not as criteria for making decisions.
On the other hand, observations could probably be naturally seen as constructing new agents from existing ones (without changing their preference/concept of environment). And some notion of observation needs to be introduced at some point, just as a notion of computational time.
I was using “information” loosely. Define it as “that thing you get from observation” if you want.
The point is, you will make different choices if you get different sensory experiences, because the sensory experiences imply something about how you control the world program. You’re right that this could be modeled with different agent functions instead of parameters. Interesting—that seems rather deeply meaningful.
The point is, you will make different choices if you get different sensory experiences
You can see it as unconditionally making a single conditional choice. The choice is itself a program with parameters, and goes different ways depending on observation, but is made without regard for observation. As an option, the choice is a parametrized constructor for new agents, which upon being constructed will make further choices, again without parameters.
There are several different possible formulations here. It seems like they will lead to the same results. The best one is, I suppose, the most elegant one.
So it sounds like you are saying that the agent() program only represents the decision-theory portion of the agent. The Bayesian cognitive portion of the agent and the part of the agent that prefers some things over others are both modeled in world(). Communication with other agents, logic, all these features of agency are in world(), not in agent().
May I suggest that “agent()” has been misnamed? Shouldn’t it be called something more like choice() or decision()? And shouldn’t “world()” be named “expected-value-of-decision-by-agent()”, or something like that?
Note that world() is part of the agent(). Certainly, world() will become preference in the next post (and cease to be necessarily a program, while agent() must remain a program), but historically this part was always “world program”, and preference replaces (subsumes) that, which is not an obvious step.
Allow me to restate the problem in my own words. We are asked to produce a general purpose function named “agent()” which returns a positive integer. The returned result is intended to represent a choice (made by “agent()”) among a discrete collection of options. There is one implicit parameter to agent—the source code for second function named “world()”. This second function returns an integer result as well, but this result is to be interpreted as a kind of payoff. The objective in this exercise is to produce an agent() function which maximizes the result returned by an arbitrary world().
The code of the function world() is expected to include one or more calls to agent(). For example, when modeling a Newcomb or Parfit’s hitchhiker decision, the world() will call agent() at least once on behalf of Omega in order to predict what the agent will do later. Then world calls agent() a second time, this time “for real”. Or, in a symmetric Prisoner’s dilemma, agent() might be called twice, once for each decision-making prisoner.
Slepnev (cousin it?) suggests that the way agent() should be structured is as a theorem-prover, rather than as a simulator. (This approach has the important technical advantage of better control over non-termination.)
Ok, that is my summary—my understanding of what is going on here. Now for my questions:
Do I seem to understand?
Did I leave out any key points?
Is there any reason not to extend this to situations in which world() passes parameters to agent(). For example, if we wanted to do a PD in which the payoffs to the two players are slightly different, world() could pass different actual values in its two calls to agent(), because those two calls represent different players. So we want a parameter to represent “player id”.
Is there any reason not to allow world() to pass parameters representing random numbers to the agent()s? This would allow the agents to choose mixed or correlated strategies, if these kinds of strategies prove to be optimal.
The way the Newcomb example was written, it was pretty clear that agent() was being assumed to be deterministic. How essential is that assumption to this whole enterprise?
That is enough questions for now. Depending on the answers, I may have follow-ups.
Edit: minor cleanups.
Yeah. Sadly I didn’t use my real name when I registered. Sorry if that got anyone confused. Nesov uses my real name because we had many email exchanges, and also because he’s a university friend of my brother’s :-)
In the draft, I used your nickname at first, but in “prior work” section that sounded rather strange, so I replaced it with your real name everywhere. Glad to see that agrees with your preference.
Not necessarily (or not exactly). Where I used agent2() for a predictor, I could use agent3() for the agent as well. See this comment.
Extensions are possible, but not relevant to the main point of the model. It doesn’t seem natural to model different players with the same agent-program. Observations are not part of the model, but you could consider the agent as producing a program with parameters (instead of an unstructured constant) as a result of its decision.
The other use for inputs is information passed to agents through sensory observation. The extension is extremely natural.
“random numbers”: random numbers in nature are really pseudorandom numbers—numbers we lack the computational power to predict. In a full model of the universe, random numbers are not necessary—but different possible worlds with different laws of physics and some probability mass assigned to each, are.
Any randomness in the agent would really be pseudorandomness.
I don’t believe you can obtain proper information about the world through observation (unless you are a human and don’t know what you want). Preference is unchanging and is defined subjectively, so formal agents can’t learn new things about it (apart from resolution of logical uncertainty). Observations play a role in how plans play out (plans are prepared to be conditional on observations), or in priority for preparing said plans as observations come in, but not as criteria for making decisions.
On the other hand, observations could probably be naturally seen as constructing new agents from existing ones (without changing their preference/concept of environment). And some notion of observation needs to be introduced at some point, just as a notion of computational time.
I was using “information” loosely. Define it as “that thing you get from observation” if you want.
The point is, you will make different choices if you get different sensory experiences, because the sensory experiences imply something about how you control the world program. You’re right that this could be modeled with different agent functions instead of parameters. Interesting—that seems rather deeply meaningful.
You can see it as unconditionally making a single conditional choice. The choice is itself a program with parameters, and goes different ways depending on observation, but is made without regard for observation. As an option, the choice is a parametrized constructor for new agents, which upon being constructed will make further choices, again without parameters.
There are several different possible formulations here. It seems like they will lead to the same results. The best one is, I suppose, the most elegant one.
Not sure which is.
So it sounds like you are saying that the agent() program only represents the decision-theory portion of the agent. The Bayesian cognitive portion of the agent and the part of the agent that prefers some things over others are both modeled in world(). Communication with other agents, logic, all these features of agency are in world(), not in agent().
May I suggest that “agent()” has been misnamed? Shouldn’t it be called something more like choice() or decision()? And shouldn’t “world()” be named “expected-value-of-decision-by-agent()”, or something like that?
Note that world() is part of the agent(). Certainly, world() will become preference in the next post (and cease to be necessarily a program, while agent() must remain a program), but historically this part was always “world program”, and preference replaces (subsumes) that, which is not an obvious step.
I want to see it!