Oh, I definitely don’t think that human learning would only rely on RL, or that RL would be the One Grand Theory Explaining Everything About Learning. (Human learning is way too complicated for any such single theory.) I agree that e.g. the Anki card example you mention requires more blocks to explain than RL.
That said, RL would help explain things like why many people’s efforts to study via Anki so easily fail, and why it’s important to make each card contain as little to recall as possible—the easier it is to recall the contents of a card, the better the effort/reward ratio, and the more likely that you’ll remain motivated to continue studying the cards.
You also mention CBT. One of the basic building blocks of CBT is the ABC model, where an Activating Event is interpreted via a subconscious Belief, leading to an emotional Consequence. Where do those subconscious Beliefs come from? The full picture is quite complicated (see appraisal theory, the more theoretical and detailed version of the ABC model), but I would argue that at least some of the beliefs look like they could be produced by something like RL.
As a simple example, someone once tried to rob me at a particular location, after which I started being afraid of taking the path leading through that location. The ABC model would describe this as saying that the Activating event is (the thought of) that location, the Belief is that that location is dangerous, and the Consequence of that belief is fear and a desire to avoid that location… or, almost equivalently, you could describe that as a RL process having once received a negative reward at that particular location, and therefore assigning a negative value to that location since that time.
That said, I did reason that even though it had happened once, I’d just been unlucky on that time and I knew on other grounds that that location was just as safe as any other. So I forced myself to take that path anyway, and eventually the fear vanished. So you’re definitely right that we also have brain mechanisms that can sometimes override the judgments produced by the RL process. But I expect that even their behavior is strongly shaped by RL elements… e.g. if I had tried to make myself walk that path several times and failed on each time, I would soon have acquired the additional Belief that trying to overcome that fear is useless, and given up.
I think it is very important to consider the difference between a descriptive model and a theory of a mechanism.
So, inventing an extreme example for purposes of illustration, if someone builds a simple, two-parameter model of human marital relationships (perhaps centered on the idea of cost and benefits), that model might actually be made to work, to a degree. It could be used to do some pretty simple calculations about how many people divorce, at certain income levels, or with certain differences in income between partners in a marriage.
But nobody pretends that the mechanism inside the descriptive model corresponds to an actual mechanism inside the heads of those married couples. Sure, there might be!, but there doesn’t have to be, and we are pretty sure there is no actual calculation inside a particular mechanism, that matches the calculation in the model. Rather, we believe that reality involves a much more complex mechanism that has that behavior as an emergent property.
When RL is seen as a descriptive model—which I think is the correct way to view it in your above example, that is fine and good as far as it goes.
The big trouble that I have been fighting is the apotheosis from descriptive model to theory of a mechanism. And since we are constructing mechanisms when we do AI, that is an especially huge danger that must be avoided.
I agree that this is an important distinction, and that things that might naively seem like mechanisms are often actually closer to descriptive models.
I’m not convinced that RL necessarily falls into the class of things that should be viewed mainly as descriptive models, however. For one, what’s possibly the most general-purpose AI developed so far seems to have been developed by explicitly having RL as an actual mechanism. That seems to me like a moderate data point towards RL being an actual useful mechanism and not just a description.
Though I do admit that this isn’t necessarily that strong of a data point—after all, SHRDLU was once the most advanced system of its time too, yet basically all of its mechanisms turned out to be useless.
So I forced myself to take that path anyway, and eventually the fear vanished.
The fact that you don’t consciously notice fear doesn’t mean that it’s completely gone. It still might raise your pulse a bit. Physiological responses in general stay longer.
To the extend that you removed the fear In that case I do agree doing exposure therapy is drive by RL. On the other hand it’s slow.
I don’t think you need a belief to have a working Pavlonian trigger. When playing around with anchoring in NLP I don’t think that a physical anchor is well described as working via a belief.
Beliefs seem to me separate entities. They usually exist as “language”/semantics.
When playing around with anchoring in NLP I don’t think that a physical anchor is well described as working via a belief. Beliefs seem to me separate entities. They usually exist as “language”/semantics.
I’m not familiar with NLP, so I can’t comment on this.
Do you have experience with other process oriented change work techniques? Be it alternative frameworks or CBT?
I think it’s very hard to reason about concepts like beliefs. We have a naive understanding of what the word means but there are a bunch of interlinked mental modules that don’t really correspond to naive language. Unfortunately they are also not easy to study apart from each other.
Having reference experiences of various corner cases seems to me to be required to get to grips with concepts.
Do you have experience with other process oriented change work techniques?
Not sure to what extent these count, but I’ve done various CFAR techniques, mindfulness meditation, and Non-Violent Communication (which I’ve noticed is useful not only for improving your communication, but also dissolving your own annoyances and frustrations even in private).
Oh, I definitely don’t think that human learning would only rely on RL, or that RL would be the One Grand Theory Explaining Everything About Learning. (Human learning is way too complicated for any such single theory.) I agree that e.g. the Anki card example you mention requires more blocks to explain than RL.
That said, RL would help explain things like why many people’s efforts to study via Anki so easily fail, and why it’s important to make each card contain as little to recall as possible—the easier it is to recall the contents of a card, the better the effort/reward ratio, and the more likely that you’ll remain motivated to continue studying the cards.
You also mention CBT. One of the basic building blocks of CBT is the ABC model, where an Activating Event is interpreted via a subconscious Belief, leading to an emotional Consequence. Where do those subconscious Beliefs come from? The full picture is quite complicated (see appraisal theory, the more theoretical and detailed version of the ABC model), but I would argue that at least some of the beliefs look like they could be produced by something like RL.
As a simple example, someone once tried to rob me at a particular location, after which I started being afraid of taking the path leading through that location. The ABC model would describe this as saying that the Activating event is (the thought of) that location, the Belief is that that location is dangerous, and the Consequence of that belief is fear and a desire to avoid that location… or, almost equivalently, you could describe that as a RL process having once received a negative reward at that particular location, and therefore assigning a negative value to that location since that time.
That said, I did reason that even though it had happened once, I’d just been unlucky on that time and I knew on other grounds that that location was just as safe as any other. So I forced myself to take that path anyway, and eventually the fear vanished. So you’re definitely right that we also have brain mechanisms that can sometimes override the judgments produced by the RL process. But I expect that even their behavior is strongly shaped by RL elements… e.g. if I had tried to make myself walk that path several times and failed on each time, I would soon have acquired the additional Belief that trying to overcome that fear is useless, and given up.
I think it is very important to consider the difference between a descriptive model and a theory of a mechanism.
So, inventing an extreme example for purposes of illustration, if someone builds a simple, two-parameter model of human marital relationships (perhaps centered on the idea of cost and benefits), that model might actually be made to work, to a degree. It could be used to do some pretty simple calculations about how many people divorce, at certain income levels, or with certain differences in income between partners in a marriage.
But nobody pretends that the mechanism inside the descriptive model corresponds to an actual mechanism inside the heads of those married couples. Sure, there might be!, but there doesn’t have to be, and we are pretty sure there is no actual calculation inside a particular mechanism, that matches the calculation in the model. Rather, we believe that reality involves a much more complex mechanism that has that behavior as an emergent property.
When RL is seen as a descriptive model—which I think is the correct way to view it in your above example, that is fine and good as far as it goes.
The big trouble that I have been fighting is the apotheosis from descriptive model to theory of a mechanism. And since we are constructing mechanisms when we do AI, that is an especially huge danger that must be avoided.
I agree that this is an important distinction, and that things that might naively seem like mechanisms are often actually closer to descriptive models.
I’m not convinced that RL necessarily falls into the class of things that should be viewed mainly as descriptive models, however. For one, what’s possibly the most general-purpose AI developed so far seems to have been developed by explicitly having RL as an actual mechanism. That seems to me like a moderate data point towards RL being an actual useful mechanism and not just a description.
Though I do admit that this isn’t necessarily that strong of a data point—after all, SHRDLU was once the most advanced system of its time too, yet basically all of its mechanisms turned out to be useless.
Arrgghh! No. :-)
The DeepMind Atari agent is the “most general-purposeAI developed so far”?
!!!
At this point your reply is “I am not joking. And don’t call me Shirley.”
The fact that you don’t consciously notice fear doesn’t mean that it’s completely gone. It still might raise your pulse a bit. Physiological responses in general stay longer.
To the extend that you removed the fear In that case I do agree doing exposure therapy is drive by RL. On the other hand it’s slow.
I don’t think you need a belief to have a working Pavlonian trigger. When playing around with anchoring in NLP I don’t think that a physical anchor is well described as working via a belief. Beliefs seem to me separate entities. They usually exist as “language”/semantics.
I’m not familiar with NLP, so I can’t comment on this.
Do you have experience with other process oriented change work techniques? Be it alternative frameworks or CBT?
I think it’s very hard to reason about concepts like beliefs. We have a naive understanding of what the word means but there are a bunch of interlinked mental modules that don’t really correspond to naive language. Unfortunately they are also not easy to study apart from each other.
Having reference experiences of various corner cases seems to me to be required to get to grips with concepts.
Not sure to what extent these count, but I’ve done various CFAR techniques, mindfulness meditation, and Non-Violent Communication (which I’ve noticed is useful not only for improving your communication, but also dissolving your own annoyances and frustrations even in private).
Do you think that resolving an emotion frustration via NVC is done via reinforcement learning?
No.