I’d be interested in a post that was just focused on laying out what the empirical evidence was (preferably decoupled from trying to sell me on the theory too hard)
(a bit more details on how I’m thinking about this. Note that this is just my own opinion, not necessarily representing any LW team consensus)
I’m generally interested in getting LW to a state where
it’s possible to bring up psych theories that seem wooey at first glance, but
it’s also clearer:
what the epistemic status of those theories are
what timeframes are reasonable to expect that epistemic status to reach a state where we have a better sense of how true/useful the theory is
have some kind of plan to deprecate weird theories if they turn out to be BS
I think there are some additional constraints on developmental theories, where for social reasons I think it makes sense to lean harder in the “strong standards of evidence” direction. I think Dan Speyer’s suspicions (articulated on FB) are pretty reasonable, and whether they’re reasonable or not they also seem to a fact-of-the-matter that needs to be addressed anyhow.
I’ve recently updated that developmental theories might be pretty important, but I think there’s a lot of ways to use them poorly and I wanna get it right.
I have seen much talk on Less Wrong lately of “development stages” and “Kegan” and so forth. Naturally I am skeptical; so I do endorse any attempt to figure out if any of this stuff is worth anything. To aid in our efforts, I’d like to say a bit about what might convince me be a little less skeptical.
A theory should explain facts; and so the very first thing we’d have to do, as investigators, is figure out if there’s anything to explain. Specifically: we would have to look at the world, observe people, examine their behavior, their patterns of thinking and interacting with other people, their professed beliefs and principles, etc., etc., and see if these fall into any sorts of patterns or clusters, such that they may be categorized according to some scheme, where some people act like this [and here we might give some broad description], while other people act like that.
(Clearly, the answer to this question would be: yes, people’s behavior obviously falls into predictable, clustered patterns. But what sort, exactly? Some work would need to be done, at least, to enumerate and describe them.)
Second, we would have to see whether these patterns that we observe may be separated, or factored, by “domain”, whereby there is one sort of pattern of clusters in how people think and act and speak, which pertains to matters of religion; and another pattern, which pertains to relationship to family; and another pattern, which pertains to preferences of consumption; etc. We would be looking for such “domains” which may be conceptually separated—regardless of whether there were any correlation between clustering patterns in one domain or another.
(Here again, the answer seems clearly to be that yes, such domains may be defined without too much difficulty. However, the intuition is weaker than for the previous question; and we are less sure that we know what it is we’re talking about; and it becomes even more important to be specific and explicit.)
Now we would ask two further questions (which might be asked in parallel). Third: does categorization of an individual into one cluster or another, in any of these domains, correlate with that individual’s category membership in categories pertaining to any observable aspect of human variation? (Such observable aspects might be: cultural groupings; gender; weight; height; age; ethnicity; socioeconomic status; hair color; various matters of physical health; or any of a variety of other ways in which people demonstrably differ.) And fourth: may the clusters in any of these domains sensibly be given a total ordering (and the domain thereby be mapped onto a linear axis of variation)?
Note the special import of this latter question. Prior to answering it, we are dealing exclusively with nominal data values. We now ask whether any of the data we have might actually be ordinal data. The answer might be “no” (for instance, you prefer apples, and I prefer oranges; this puts us in different clusters within the “fruit preferences” domain of human psychology, but in no sense may these clusters be arranged linearly).
Our fifth question (conditional on answering yes to all four of the previous question) is this: among our observed domains of clustering, and looking in particular at those for which the data is of an ordinal nature, are there any such that the dimension of variation has any normative aspect? That is: is there a domain such that we might sensibly say that it is better to belong to clusters closer to one end of its spectrum of variation, than to belong to clusters closer to the other end? (Once more note that the answer might be “no”: for example, suppose that some people fidget a lot, while others do not fidget very much. Is it better to be a much-fidgeter than a not-much-fidgeter? Well… not really; nor the reverse; at least, not in any general way. Maybe fidgeting has some advantages, and not fidgeting has others, etc.; who knows? But overall the answer is “no, neither of these is clearly superior to the other; they’re just one of those ways in which people differ, in a normatively neutral way”.)
Finally, our sixth question is: does there exist any domain of clustering in human behavioral/psychological variation for which all of these are true:
That its clusters may naturally be given a total order (i.e., arranged linearly);
That this linear dimension has normative significance;
That membership in its categories is correlated primarily with category membership pertaining to one aspect of human variation (rather than being correlated comparably with multiple such aspects);
That in particular, membership in this domain’s clusters is correlated primarily with age.
Note that we have asked six (mostly[1]) empirical questions about humanity. And we have had six chances to answer in the negative.
And note also that if we answer any of these questions in the negative, then any and all theories of “moral development” (or any similar notion) are necessarily nonsense—because they purport to explain facts which (in this hypothetical scenario) we simply do not observe. Without any further investigation, we can dispose of the lot of them with extreme prejudice, because they are entirely unmotivated by the pre-theoretical facts.
So, this is what I would like to see from any proponents of Kegan’s theory, or any similar ones: a detailed, thorough, and specific examination (with plenty of examples!) of the questions I give in this comment—discussed with utter agnosticism about even the concept of “moral development”, “adult development” or any similar thing. In short: before I consider any defense of any theory of “adult development”, I should like to be convinced of such a theory’s motivation.
The question of normative import is not quite empirical, but it may be operationalized by considering intersubjective judgments of normative import; that is, in any case, more or less what we are talking about in the first place.
Why must a developmental theory be normative? A descriptive theory that says all humans go through stages where they get less moral over time works still as an interesting descriptive theory. Similary, there’s certain Developmental stages that probably aren’t normative of everyone around you is in a lower developmental stage, but it can still be descriptive as the next stage most humans go through if they indeed progress.
I did not say anything about the theory being normative. “A descriptive theory that says all humans go through stages where they get less moral over time” is entirely consistent with what I described. Note that “moral” is a quality with normative significance—compare “get less extraverted over time” or “get less risk-seeking over time”.
Whether I care is hardly at issue; all the theories of “adult development” and similar clearly deal with variation along normatively significant dimensions.
If, for some reason, you propose to defend a theory of development that has no such normative aspect, then by all means remove that requirement from my list. (Kegan’s theory, however, clearly falls into the “normatively significant variation” category.)
I think that EG constructive-developmental theory studiously avoids normative claims. The level that fits best is context dependent on the surrounding culture.
However, in that case I don’t really understand what you mean. But, in any case, the rest of my original comment stands.
I look forward to any such detailed commentary on the fact-based motivation for any sort of developmental theory, from anyone who feels up to the task of providing such.
In a study of West Point students, average inter-rater agreement on the Subject-Object Interview was 63%, and students developed from stage 2 to stage 3 and from stage 3 to stage 4 over their years in school.
Are you calling those 63% strong inter-rater reliablity or are you referring to other studies?
My general takeaway from that post was that in terms of psychometric validity, most developmental psychology is quite bad. Did I miss something?
This doesn’t necessarily mean the underlying concepts aren’t real, but I do think that in terms of the quality metrics that psychometrics tends to assess things on, I don’t think the evidence base is very good.
I haven’t looked into general developmental theories like Sarah Constantin, but have looked into the studies on Constructive Developmental theory.
My takeaways (mostly supported by her research, although she misses a lot) is that basically all the data points towards confirming the theory, with high information value on further research
high interrater reliability
high test-retest reliability
good correlation with age
good correlations with age in multiple cultures
good correlation with measures of certainty types of achievement like leadership
As Sarah points at, the biggest thing missing is evidence that the steps procede in order with no skipping, but as far as I can tell there’s no counterevidence for that either. Also, replications of the other things.
Perhaps if I had went into this looking at a bunch of other failed developmental theories, my priors would have been such that I would have described it as “not enough evidence to confirm the theory”. However, given this is the only developmental theory I looked into, my takeaways was “promising theory with preliminary support, needs more confirming research”
Yes, this is what I’m imagining. A simple post that just summarizes the epistemic status, potentially as the start of a sequence for later posts that use it as a building block for other ideas.
I’d be interested in a post that was just focused on laying out what the empirical evidence was (preferably decoupled from trying to sell me on the theory too hard)
(a bit more details on how I’m thinking about this. Note that this is just my own opinion, not necessarily representing any LW team consensus)
I’m generally interested in getting LW to a state where
it’s possible to bring up psych theories that seem wooey at first glance, but
it’s also clearer:
what the epistemic status of those theories are
what timeframes are reasonable to expect that epistemic status to reach a state where we have a better sense of how true/useful the theory is
have some kind of plan to deprecate weird theories if they turn out to be BS
I think there are some additional constraints on developmental theories, where for social reasons I think it makes sense to lean harder in the “strong standards of evidence” direction. I think Dan Speyer’s suspicions (articulated on FB) are pretty reasonable, and whether they’re reasonable or not they also seem to a fact-of-the-matter that needs to be addressed anyhow.
I’ve recently updated that developmental theories might be pretty important, but I think there’s a lot of ways to use them poorly and I wanna get it right.
I have seen much talk on Less Wrong lately of “development stages” and “Kegan” and so forth. Naturally I am skeptical; so I do endorse any attempt to figure out if any of this stuff is worth anything. To aid in our efforts, I’d like to say a bit about what might convince me be a little less skeptical.
A theory should explain facts; and so the very first thing we’d have to do, as investigators, is figure out if there’s anything to explain. Specifically: we would have to look at the world, observe people, examine their behavior, their patterns of thinking and interacting with other people, their professed beliefs and principles, etc., etc., and see if these fall into any sorts of patterns or clusters, such that they may be categorized according to some scheme, where some people act like this [and here we might give some broad description], while other people act like that.
(Clearly, the answer to this question would be: yes, people’s behavior obviously falls into predictable, clustered patterns. But what sort, exactly? Some work would need to be done, at least, to enumerate and describe them.)
Second, we would have to see whether these patterns that we observe may be separated, or factored, by “domain”, whereby there is one sort of pattern of clusters in how people think and act and speak, which pertains to matters of religion; and another pattern, which pertains to relationship to family; and another pattern, which pertains to preferences of consumption; etc. We would be looking for such “domains” which may be conceptually separated—regardless of whether there were any correlation between clustering patterns in one domain or another.
(Here again, the answer seems clearly to be that yes, such domains may be defined without too much difficulty. However, the intuition is weaker than for the previous question; and we are less sure that we know what it is we’re talking about; and it becomes even more important to be specific and explicit.)
Now we would ask two further questions (which might be asked in parallel). Third: does categorization of an individual into one cluster or another, in any of these domains, correlate with that individual’s category membership in categories pertaining to any observable aspect of human variation? (Such observable aspects might be: cultural groupings; gender; weight; height; age; ethnicity; socioeconomic status; hair color; various matters of physical health; or any of a variety of other ways in which people demonstrably differ.) And fourth: may the clusters in any of these domains sensibly be given a total ordering (and the domain thereby be mapped onto a linear axis of variation)?
Note the special import of this latter question. Prior to answering it, we are dealing exclusively with nominal data values. We now ask whether any of the data we have might actually be ordinal data. The answer might be “no” (for instance, you prefer apples, and I prefer oranges; this puts us in different clusters within the “fruit preferences” domain of human psychology, but in no sense may these clusters be arranged linearly).
Our fifth question (conditional on answering yes to all four of the previous question) is this: among our observed domains of clustering, and looking in particular at those for which the data is of an ordinal nature, are there any such that the dimension of variation has any normative aspect? That is: is there a domain such that we might sensibly say that it is better to belong to clusters closer to one end of its spectrum of variation, than to belong to clusters closer to the other end? (Once more note that the answer might be “no”: for example, suppose that some people fidget a lot, while others do not fidget very much. Is it better to be a much-fidgeter than a not-much-fidgeter? Well… not really; nor the reverse; at least, not in any general way. Maybe fidgeting has some advantages, and not fidgeting has others, etc.; who knows? But overall the answer is “no, neither of these is clearly superior to the other; they’re just one of those ways in which people differ, in a normatively neutral way”.)
Finally, our sixth question is: does there exist any domain of clustering in human behavioral/psychological variation for which all of these are true:
That its clusters may naturally be given a total order (i.e., arranged linearly);
That this linear dimension has normative significance;
That membership in its categories is correlated primarily with category membership pertaining to one aspect of human variation (rather than being correlated comparably with multiple such aspects);
That in particular, membership in this domain’s clusters is correlated primarily with age.
Note that we have asked six (mostly[1]) empirical questions about humanity. And we have had six chances to answer in the negative.
And note also that if we answer any of these questions in the negative, then any and all theories of “moral development” (or any similar notion) are necessarily nonsense—because they purport to explain facts which (in this hypothetical scenario) we simply do not observe. Without any further investigation, we can dispose of the lot of them with extreme prejudice, because they are entirely unmotivated by the pre-theoretical facts.
So, this is what I would like to see from any proponents of Kegan’s theory, or any similar ones: a detailed, thorough, and specific examination (with plenty of examples!) of the questions I give in this comment—discussed with utter agnosticism about even the concept of “moral development”, “adult development” or any similar thing. In short: before I consider any defense of any theory of “adult development”, I should like to be convinced of such a theory’s motivation.
The question of normative import is not quite empirical, but it may be operationalized by considering intersubjective judgments of normative import; that is, in any case, more or less what we are talking about in the first place.
Why must a developmental theory be normative? A descriptive theory that says all humans go through stages where they get less moral over time works still as an interesting descriptive theory. Similary, there’s certain Developmental stages that probably aren’t normative of everyone around you is in a lower developmental stage, but it can still be descriptive as the next stage most humans go through if they indeed progress.
I did not say anything about the theory being normative. “A descriptive theory that says all humans go through stages where they get less moral over time” is entirely consistent with what I described. Note that “moral” is a quality with normative significance—compare “get less extraverted over time” or “get less risk-seeking over time”.
Ahh, so is the idea just that you don’t care about a specific type of development if it doesn’t have consequences that matter?
Whether I care is hardly at issue; all the theories of “adult development” and similar clearly deal with variation along normatively significant dimensions.
If, for some reason, you propose to defend a theory of development that has no such normative aspect, then by all means remove that requirement from my list. (Kegan’s theory, however, clearly falls into the “normatively significant variation” category.)
I think that EG constructive-developmental theory studiously avoids normative claims. The level that fits best is context dependent on the surrounding culture.
Fair enough. Assuming that’s the case, then anyone proposing to defend that particular theory is exempt from that particular question.
Just in case it isn’t clear, constructive-developmental theory and “kegan’s levels of development” are two names for the same thing.
Ah, my mistake.
However, in that case I don’t really understand what you mean. But, in any case, the rest of my original comment stands.
I look forward to any such detailed commentary on the fact-based motivation for any sort of developmental theory, from anyone who feels up to the task of providing such.
Looks like Sarah Constantine beat me to it, although I think here lit review missed a few studies I’ve seen.
https://srconstantin.wordpress.com/2017/04/06/are-adult-developmental-stages-real/
From her post:
Are you calling those 63% strong inter-rater reliablity or are you referring to other studies?
There’s as far as I know 3 studies on this. She found the one with 63% agreement, whereas the previous two studies had about 80% agreement
My general takeaway from that post was that in terms of psychometric validity, most developmental psychology is quite bad. Did I miss something?
This doesn’t necessarily mean the underlying concepts aren’t real, but I do think that in terms of the quality metrics that psychometrics tends to assess things on, I don’t think the evidence base is very good.
I haven’t looked into general developmental theories like Sarah Constantin, but have looked into the studies on Constructive Developmental theory.
My takeaways (mostly supported by her research, although she misses a lot) is that basically all the data points towards confirming the theory, with high information value on further research
high interrater reliability
high test-retest reliability
good correlation with age
good correlations with age in multiple cultures
good correlation with measures of certainty types of achievement like leadership
As Sarah points at, the biggest thing missing is evidence that the steps procede in order with no skipping, but as far as I can tell there’s no counterevidence for that either. Also, replications of the other things.
Perhaps if I had went into this looking at a bunch of other failed developmental theories, my priors would have been such that I would have described it as “not enough evidence to confirm the theory”. However, given this is the only developmental theory I looked into, my takeaways was “promising theory with preliminary support, needs more confirming research”
Oh, I was looking for that recently. Apparently predates LessWrong integration with her blog
Yes, this is what I’m imagining. A simple post that just summarizes the epistemic status, potentially as the start of a sequence for later posts that use it as a building block for other ideas.