Why do you say this? For instance, the simplicity & restricted action space of a gridworld means that there are very limited mechanisms available to steer/point with the agent towards what we want. That’s a pretty extreme limitation compared what to can do in a high resolution sim world. It seems plausible that “pointing at what we want in an informative/high-bandwidth way” would be an important component of a solution.
right, for sure, but a good solution to the high bandwidth case that doesn’t scale down to gridworld correctly is useless, it can’t be trusted to actually be a good solution to the larger problem if it definitely breaks when shrunk. it should work just as well in gridworld, or conway’s life*, or smoothlife*, or mujoco, or a fluid sim, or real life. my point is that a good solution should at an absolute bare minimum scale down. if your base rl algorithm can’t solve gridworld, it definitely just doesn’t work at all. similarly for rl safety.
* although I’m not sure if conway’s or smoothlife have an energy conservation law, and that might be pretty important. also, maybe continuous state spaces are really important somehow, I definitely see some possibilities where that’s the case. I don’t know, though. it seems to me like we want to be able to go all the way down to the smallest units and still have a working solution.
I think I have a lot more uncertainty here. Like, yes it is always nice when solutions scale down all the way into the simplest setting, but there is no guarantee that that’s how things work in this domain. Reality is allowed to say “the minimum requirements for building a FAI are X”, where X entails a high-bandwidth or otherwise highly-specific interface between us and the agent.
if it definitely won’t work in a gridworld, it definitely won’t work in a high resolution sim world
Why do you say this? For instance, the simplicity & restricted action space of a gridworld means that there are very limited mechanisms available to steer/point with the agent towards what we want. That’s a pretty extreme limitation compared what to can do in a high resolution sim world. It seems plausible that “pointing at what we want in an informative/high-bandwidth way” would be an important component of a solution.
right, for sure, but a good solution to the high bandwidth case that doesn’t scale down to gridworld correctly is useless, it can’t be trusted to actually be a good solution to the larger problem if it definitely breaks when shrunk. it should work just as well in gridworld, or conway’s life*, or smoothlife*, or mujoco, or a fluid sim, or real life. my point is that a good solution should at an absolute bare minimum scale down. if your base rl algorithm can’t solve gridworld, it definitely just doesn’t work at all. similarly for rl safety.
* although I’m not sure if conway’s or smoothlife have an energy conservation law, and that might be pretty important. also, maybe continuous state spaces are really important somehow, I definitely see some possibilities where that’s the case. I don’t know, though. it seems to me like we want to be able to go all the way down to the smallest units and still have a working solution.
I think I have a lot more uncertainty here. Like, yes it is always nice when solutions scale down all the way into the simplest setting, but there is no guarantee that that’s how things work in this domain. Reality is allowed to say “the minimum requirements for building a FAI are X”, where X entails a high-bandwidth or otherwise highly-specific interface between us and the agent.