I would like to propose a model that is more flattering to humans, and more similar to how other parts of human cognition work. When we see a simple textual mistake, like a repeated “the”, we don’t notice it by default. Human minds correct simple errors automatically without consciously noticing that they are doing it. We round to the nearest pattern.
I propose that automatic pattern matching to the closest thing that makes sense is happening at a higher level too. When humans skim semi contradictory text, they produce a more consistent world model that doesn’t quite match up with what is said.
Language feeds into a deeper, sensible world model module within the human brain and GPT2 doesn’t really have a coherent world model.
When humans skim semi contradictory text, they produce a more consistent world model that doesn’t quite match up with what is said.
I felt like something like this happened to me when I was reading some of the “nonsensical” examples in the post, rather than deeming the text outright nonsensical and non-human I just interpreted it as the writer being sloppy.
Me too. I found it strongly reminiscent of reading low grade click bait. Or trying to listen to some woo. Part of it feels like rescuing some food when the bottom of the pan is burned. Part of it is like throwing out models of what state the author’s head was in that resolve the text into sense.
I think what makes GPT2 look relatively good is how low the baseline is in many respects. If you tell me ‘this is a political acceptance speech’ I don’t actually expect it to make that much sense. Most of the genre seems to be written by autocomplete anyway.
I would like to propose a model that is more flattering to humans, and more similar to how other parts of human cognition work. When we see a simple textual mistake, like a repeated “the”, we don’t notice it by default. Human minds correct simple errors automatically without consciously noticing that they are doing it. We round to the nearest pattern.
I propose that automatic pattern matching to the closest thing that makes sense is happening at a higher level too. When humans skim semi contradictory text, they produce a more consistent world model that doesn’t quite match up with what is said.
Language feeds into a deeper, sensible world model module within the human brain and GPT2 doesn’t really have a coherent world model.
I felt like something like this happened to me when I was reading some of the “nonsensical” examples in the post, rather than deeming the text outright nonsensical and non-human I just interpreted it as the writer being sloppy.
Me too. I found it strongly reminiscent of reading low grade click bait. Or trying to listen to some woo. Part of it feels like rescuing some food when the bottom of the pan is burned. Part of it is like throwing out models of what state the author’s head was in that resolve the text into sense.
I think what makes GPT2 look relatively good is how low the baseline is in many respects. If you tell me ‘this is a political acceptance speech’ I don’t actually expect it to make that much sense. Most of the genre seems to be written by autocomplete anyway.