Looking at that map of representations of the DNQ agent playing Space Invaders I can’t help thinking if it really has learned any concepts that are similar to what a human would learn. Does the DNQ agent have the concepts of ship, shield, bullet or space invader? Does it have anything that corresponds to the concept of a group of space invaders? Can it generalize? I’m sure human players could quickly adapt if we changed the game so that the ship would shoot from the top to the bottom instead. Does the DNQ agent have anything analogous to an inner simulator? If we showed it a movie where the ship would fly up to the invaders and collide what would it predict happens next?
My gut feeling is that artificial agents are still far away from having reusable and generalizable concepts. It’s one thing, although an impressive one, to use the same framework with identical parameters for different DNQ agents learning different games than it is to use one framework for one agent that learns to play all the games and abstract concepts across them.
That’s… rather the point of this research, no? Kaj is trying to make an original contribution here. The DNQ agent does have an internal simulator, but is not designed to have a rigorous concept network like Kaj is describing here.
Yes. Also my thought. The space invader concepts are alien (uh, kind of pun) and this can use used to gain some intuition about how alien concepts work.
The concepts of the DNQ agent are surely alien to us—because it’s world is alien. At least if you think of the world as what it is: A bunch of coordinates and movement-patterns. The 2D visualization is just a rendering to even allow humans to deal with it in familiar categories. And that is what we do: We use our pre-existing categories to play it well. Over time an experienced player will improve these toward what the DNQ starts to aquire from scratch.
I’d bet that it possible to map the DNQ representations to player coodinate, enemy hight and something that amounts to the aim of a shot. At least if the NN is deep and wide enough.
Looking at that map of representations of the DNQ agent playing Space Invaders I can’t help thinking if it really has learned any concepts that are similar to what a human would learn.
The DNQ agent has a much simpler visual system that is suitable to the low complexity graphics of the Atari world. It also learns through supervised backprop on the RL signals, whereas the human cortex appears to learn through some more complex mix of RL and UL (more UL the closer one gets to the sensory stream, more RL as one moves up closer to the reward cirucits)
The more complex vision ANNs trained on natural images do produce visual concepts (features) that are reasonably close to those found in various stages of the human visual system. It all depends on the training data and the model architecture.
Looking at that map of representations of the DNQ agent playing Space Invaders I can’t help thinking if it really has learned any concepts that are similar to what a human would learn. Does the DNQ agent have the concepts of ship, shield, bullet or space invader? Does it have anything that corresponds to the concept of a group of space invaders? Can it generalize? I’m sure human players could quickly adapt if we changed the game so that the ship would shoot from the top to the bottom instead. Does the DNQ agent have anything analogous to an inner simulator? If we showed it a movie where the ship would fly up to the invaders and collide what would it predict happens next?
My gut feeling is that artificial agents are still far away from having reusable and generalizable concepts. It’s one thing, although an impressive one, to use the same framework with identical parameters for different DNQ agents learning different games than it is to use one framework for one agent that learns to play all the games and abstract concepts across them.
That’s… rather the point of this research, no? Kaj is trying to make an original contribution here. The DNQ agent does have an internal simulator, but is not designed to have a rigorous concept network like Kaj is describing here.
Yes. Also my thought. The space invader concepts are alien (uh, kind of pun) and this can use used to gain some intuition about how alien concepts work.
The concepts of the DNQ agent are surely alien to us—because it’s world is alien. At least if you think of the world as what it is: A bunch of coordinates and movement-patterns. The 2D visualization is just a rendering to even allow humans to deal with it in familiar categories. And that is what we do: We use our pre-existing categories to play it well. Over time an experienced player will improve these toward what the DNQ starts to aquire from scratch.
I’d bet that it possible to map the DNQ representations to player coodinate, enemy hight and something that amounts to the aim of a shot. At least if the NN is deep and wide enough.
The DNQ agent has a much simpler visual system that is suitable to the low complexity graphics of the Atari world. It also learns through supervised backprop on the RL signals, whereas the human cortex appears to learn through some more complex mix of RL and UL (more UL the closer one gets to the sensory stream, more RL as one moves up closer to the reward cirucits)
The more complex vision ANNs trained on natural images do produce visual concepts (features) that are reasonably close to those found in various stages of the human visual system. It all depends on the training data and the model architecture.