In a study of West Point students, average inter-rater agreement on the Subject-Object Interview was 63%, and students developed from stage 2 to stage 3 and from stage 3 to stage 4 over their years in school.
Are you calling those 63% strong inter-rater reliablity or are you referring to other studies?
My general takeaway from that post was that in terms of psychometric validity, most developmental psychology is quite bad. Did I miss something?
This doesn’t necessarily mean the underlying concepts aren’t real, but I do think that in terms of the quality metrics that psychometrics tends to assess things on, I don’t think the evidence base is very good.
I haven’t looked into general developmental theories like Sarah Constantin, but have looked into the studies on Constructive Developmental theory.
My takeaways (mostly supported by her research, although she misses a lot) is that basically all the data points towards confirming the theory, with high information value on further research
high interrater reliability
high test-retest reliability
good correlation with age
good correlations with age in multiple cultures
good correlation with measures of certainty types of achievement like leadership
As Sarah points at, the biggest thing missing is evidence that the steps procede in order with no skipping, but as far as I can tell there’s no counterevidence for that either. Also, replications of the other things.
Perhaps if I had went into this looking at a bunch of other failed developmental theories, my priors would have been such that I would have described it as “not enough evidence to confirm the theory”. However, given this is the only developmental theory I looked into, my takeaways was “promising theory with preliminary support, needs more confirming research”
Looks like Sarah Constantine beat me to it, although I think here lit review missed a few studies I’ve seen.
https://srconstantin.wordpress.com/2017/04/06/are-adult-developmental-stages-real/
From her post:
Are you calling those 63% strong inter-rater reliablity or are you referring to other studies?
There’s as far as I know 3 studies on this. She found the one with 63% agreement, whereas the previous two studies had about 80% agreement
My general takeaway from that post was that in terms of psychometric validity, most developmental psychology is quite bad. Did I miss something?
This doesn’t necessarily mean the underlying concepts aren’t real, but I do think that in terms of the quality metrics that psychometrics tends to assess things on, I don’t think the evidence base is very good.
I haven’t looked into general developmental theories like Sarah Constantin, but have looked into the studies on Constructive Developmental theory.
My takeaways (mostly supported by her research, although she misses a lot) is that basically all the data points towards confirming the theory, with high information value on further research
high interrater reliability
high test-retest reliability
good correlation with age
good correlations with age in multiple cultures
good correlation with measures of certainty types of achievement like leadership
As Sarah points at, the biggest thing missing is evidence that the steps procede in order with no skipping, but as far as I can tell there’s no counterevidence for that either. Also, replications of the other things.
Perhaps if I had went into this looking at a bunch of other failed developmental theories, my priors would have been such that I would have described it as “not enough evidence to confirm the theory”. However, given this is the only developmental theory I looked into, my takeaways was “promising theory with preliminary support, needs more confirming research”
Oh, I was looking for that recently. Apparently predates LessWrong integration with her blog