I agree that you can learn lots of things on long timescales from short-timescale phenomena; that’s what I was getting at with the “Compositionality and extrapolation” thing. (If you think the existence of buildings tells the AI something that only shows up on long timescales, then we might disagree, but also in that case I don’t know what you are talking about, so you will have to provide an example.)
I think even short episodes encourage accurate long-term dynamics models.
This seems like perhaps the key point of disagreement. I think it encourages accurate long-term dynamics models among the factors that can be observed to vary enough within the episode length, but if some factors are not observable or do not vary (more than infinitesimally), then I don’t think short episodes would encourage accurate long-term dynamics models.
Ah, you’re right, a specific example of what I’m trying to get at would be much clearer, so here’s a chain of reasoning using the existence of buildings that would be useful to a model on short timescales:
Building exists, which is known from image recognition even on short timescales
Humans exist, again from image recognition
Some humans entered the particular building I’m looking at in the past 45 seconds, why?
Ah, they entered their building for their “jobs”, why?
They appear to have something called “families” to support, objects which rank highly in their value function.
Manipulating those humans would help me in my goal over the next 15 minutes, perhaps their families would be an effective point of leverage?
Build model of long-term human family dynamics in order to predict how best to manipulate them, perhaps by threatening to kill their children.
So suddenly the AI is reasoning about families and their dynamics, which are very long-term objects. Buildings are built to last a long time, meaning that you can infer that the agents who built them wish to use them repeatedly, this is a constraint on the long-term dynamics of the environment which is very useful to model. In general in an environment where some agents have value functions which care about long-term effects, it will be instrumentally useful for our short-term agent to model them, and for that it needs to understand the long-term behavior even of those parts of the environment which are not other agents.
Or, for instance, the position of the sun relative to the earth doesn’t vary much within a 15-min timeframe, but again you need to understand the day-night cycle to predict humans accurately, even within a short conversation.
That would indeed be difficult, maybe by downloading human media and reading stories the AI could infer the jobs → supporting families link (this wouldn’t quite be strategy-stealing, since the AI doesn’t plan on having a family itself, just to understand why the human has one). I’m not saying that these long-term dynamics are especially easy to learn if you only have a short time window, what I’m saying is that there is non-zero incentive to learn long-term dynamics even in that case. I’m imagining what a perfect learning algorithm would do if it had to learn from 15 minute episodes (with correspondingly short-term values) starting from the same point in time, and access to video, text, and the internet, and that agent would certainly need to learn a lot about long-term dynamics. Though I completely agree that you’d likely need to achieve very high model competence before it would try to squeeze out the short-term gains of long-term dynamics understanding, and we’re not really close to that point right now. In my view extending the episode length is not required for building long-term dynamics models, merely very useful. And of course I agree that the model wouldn’t magically start planning for itself beyond the 15 minutes.
That would indeed be difficult, maybe by downloading human media and reading stories the AI could infer the jobs → supporting families link (this wouldn’t quite be strategy-stealing, since the AI doesn’t plan on having a family itself, just to understand why the human has one).
I mean it’s the sort of thing I was trying to get at with the strategy-stealing section...
I’m not saying that these long-term dynamics are especially easy to learn if you only have a short time window, what I’m saying is that there is non-zero incentive to learn long-term dynamics even in that case. I’m imagining what a perfect learning algorithm would do if it had to learn from 15 minute episodes (with correspondingly short-term values) starting from the same point in time, and access to video, text, and the internet, and that agent would certainly need to learn a lot about long-term dynamics. Though I completely agree that you’d likely need to achieve very high model competence before it would try to squeeze out the short-term gains of long-term dynamics understanding, and we’re not really close to that point right now. In my view extending the episode length is not required for building long-term dynamics models, merely very useful.
I think we might not really disagree, then. Not sure.
The thing about text is that it tends to contain exactly the sorts of high-level information that would usually be latent, so it is probably very useful to learn from.
And of course I agree that the model wouldn’t magically start planning for itself beyond the 15 minutes.
I should probably add that I think model timescale and planner timescale are mostly independent variables. You can set a planner to work on a very long timescale in a model that has been trained to be accurate for a very short timescale, or a planner to work on a very short timescale in a model that has been trained to be accurate for a very long timescale. Though I assume you’d get the most bang for your buck by having the timescales be roughly proportional.
I agree that you can learn lots of things on long timescales from short-timescale phenomena; that’s what I was getting at with the “Compositionality and extrapolation” thing. (If you think the existence of buildings tells the AI something that only shows up on long timescales, then we might disagree, but also in that case I don’t know what you are talking about, so you will have to provide an example.)
This seems like perhaps the key point of disagreement. I think it encourages accurate long-term dynamics models among the factors that can be observed to vary enough within the episode length, but if some factors are not observable or do not vary (more than infinitesimally), then I don’t think short episodes would encourage accurate long-term dynamics models.
Ah, you’re right, a specific example of what I’m trying to get at would be much clearer, so here’s a chain of reasoning using the existence of buildings that would be useful to a model on short timescales:
Building exists, which is known from image recognition even on short timescales
Humans exist, again from image recognition
Some humans entered the particular building I’m looking at in the past 45 seconds, why?
Ah, they entered their building for their “jobs”, why?
They appear to have something called “families” to support, objects which rank highly in their value function.
Manipulating those humans would help me in my goal over the next 15 minutes, perhaps their families would be an effective point of leverage?
Build model of long-term human family dynamics in order to predict how best to manipulate them, perhaps by threatening to kill their children.
So suddenly the AI is reasoning about families and their dynamics, which are very long-term objects. Buildings are built to last a long time, meaning that you can infer that the agents who built them wish to use them repeatedly, this is a constraint on the long-term dynamics of the environment which is very useful to model. In general in an environment where some agents have value functions which care about long-term effects, it will be instrumentally useful for our short-term agent to model them, and for that it needs to understand the long-term behavior even of those parts of the environment which are not other agents.
Or, for instance, the position of the sun relative to the earth doesn’t vary much within a 15-min timeframe, but again you need to understand the day-night cycle to predict humans accurately, even within a short conversation.
I’m confused about how you expect the AI to be able to answer the “why?” questions.
That would indeed be difficult, maybe by downloading human media and reading stories the AI could infer the jobs → supporting families link (this wouldn’t quite be strategy-stealing, since the AI doesn’t plan on having a family itself, just to understand why the human has one). I’m not saying that these long-term dynamics are especially easy to learn if you only have a short time window, what I’m saying is that there is non-zero incentive to learn long-term dynamics even in that case. I’m imagining what a perfect learning algorithm would do if it had to learn from 15 minute episodes (with correspondingly short-term values) starting from the same point in time, and access to video, text, and the internet, and that agent would certainly need to learn a lot about long-term dynamics. Though I completely agree that you’d likely need to achieve very high model competence before it would try to squeeze out the short-term gains of long-term dynamics understanding, and we’re not really close to that point right now. In my view extending the episode length is not required for building long-term dynamics models, merely very useful. And of course I agree that the model wouldn’t magically start planning for itself beyond the 15 minutes.
I mean it’s the sort of thing I was trying to get at with the strategy-stealing section...
I think we might not really disagree, then. Not sure.
The thing about text is that it tends to contain exactly the sorts of high-level information that would usually be latent, so it is probably very useful to learn from.
I should probably add that I think model timescale and planner timescale are mostly independent variables. You can set a planner to work on a very long timescale in a model that has been trained to be accurate for a very short timescale, or a planner to work on a very short timescale in a model that has been trained to be accurate for a very long timescale. Though I assume you’d get the most bang for your buck by having the timescales be roughly proportional.