NancyLebovitz, this is a very important question! Yup, we do have methods of checking in on what we are teaching.
For example, let’s take our meaning and purpose content, which is essentially about teaching people to be oriented toward the long term and achieving their long-term goals. We have an app that has a psychometric test to measure people’s level of meaning and purpose prior to engaging with our content, and then after they engage with our content, including continuing follow-up going forward. The continuing follow-up is meant to address the issue of attention bias and Hawthorne effect, namely to test whether people just got an immediate boost or if there a long-term benefit to people engaging with our content.
We are developing similar apps for other types of content, such as an app for planning fallacy.
We also intend to run randomized control studies on our content’s impact, as described in our theory of change. We have written up proposals for those studies, and are currently trying to get funding for these studies.
That’s the link to the sign-up page for the app. As you can see from the footer, that information is gathered for the purpose of doing demographic analysis as part of our data analysis.
If you want to know the scientific literature on meaning and purpose, there are plenty of sources available. Here’s a good overview, and you can follow the footnotes to learn more.
As I stated earlier, this app uses a psychometric test. It uses a 1-10 Likert scale and a series of questions drawn from scientific literature measuring meaning. The test measures answers on questions, as do other psychometric tests of states of mind such as depression or happiness. In other words, there is no unit measure of “meaning” just as there is no unit measure of “depression” or “happiness.” I guess you could try to measure it in terms of Shannons if you really wanted, but that’s not what the scholarly literature uses for these types of exams.
If you assign a numeric value to something you call “meaning”, there must be a unit measure.
For example, consider IQ points. The unit measure (one IQ point) is 1⁄15 of the standard deviation of the distribution of IQ scores (which are basically normalised ranks) in white populations. We can argue what it corresponds to in the underlying reality, but at least it is well-defined.
The unit measure, however, may be entirely defined by the scale in question. Gleb could say “my unit measure is 1⁄9 of the full range of possible answers on my Meaning Measuring Scale”. That would be pretty useless, but so is “my unit measure is 1⁄15 of one standard deviation on my Cleverness Measuring Scale”.
Perhaps it’s better to reference scores to standard deviation rather than the full possible range—I guess it depends on the particular case. The other advantage IQ has is that there are now lots of IQ tests and they tend to get somewhat correlated results (suggesting that maybe they’re measuring something real) that also correlate with other interesting things (suggesting that maybe the real thing they’re measuring is useful) and they get used quite a lot (so that if you quote an IQ score there’s a good chance that the people you’re addressing will understand roughly what you mean).
Those are all genuine (or at least possibly-genuine) ways in which IQ scores are more useful than Gleb Meaningfulness Metric scores. But I don’t see that IQ is any better off than Gleb Meaningfulness Metric in terms of unit-measure-having. One IQ point doesn’t correspond to a fixed increment in thinking speed or memory capacity or ability to solve any particular kind of problem, or anything like that; it’s just a certain fraction of how much variation there is in one kind of brainpower score. One Gleb Meaningfulness Metric point, likewise, is just a certain fraction of how much variation is possible in one kind of feeling-like-your-life-has-meaning score.
Perhaps it’s better to reference scores to standard deviation rather than the full possible range—I guess it depends on the particular case.
Yep, that highly depends on the shape of the distribution.
One Gleb Meaningfulness Metric point, likewise, is just a certain fraction of how much variation is possible
Well, we (or at least I) haven’t seen Gleb’s Meaninfulness Metric, so I have no idea if it’s defined via population standard deviation like IQ. It may or it may be. I brought up IQ as an example of a unit which does not directly correspond to, say, thinking speed or working memory capacity—it’s entangled with the test itself, but it does make the numbers interpretable.
This is a nicety; while appreciated it looks like you are trying to suck up to Nancy.
(taking into account what Lumifier said about the app) in your description;
We also intend to run randomized control studies on our content’s impact, as described in our theory of change. We have written up proposals for those studies, and are currently trying to get funding for these studies.
Is the only thing you said are doing to be checking on what you’re actually teaching. And it’s something that has not happened yet. (granted these things take time). I read the entire post as; “nothing yet, but we want to—in these ways...”.
randomized control studies on our content’s impact
depending on the method (if done with web content) could be described as A/B split testing. Which is standard these days for internet behaviour of groups spreading clickbait, not an accountable test.
I would suggest not taking Lumifer’s description of his initial impression of the sign-up page as indicative of the app itself. I think there’s sufficient evidence of Lumifer being not unbiased in describing Intentional Insights content. So consider checking out the app itself.
Here are the study proposals themselves, which you can evaluate yourself for whether they are A/B split testing: 1, 2.
EDIT Forgot to mention, my comment on the nature of NancyLebovitz’s question had to do with my own desire to signal that this is a very important question to me, not give praise.
NancyLebovitz, this is a very important question! Yup, we do have methods of checking in on what we are teaching.
For example, let’s take our meaning and purpose content, which is essentially about teaching people to be oriented toward the long term and achieving their long-term goals. We have an app that has a psychometric test to measure people’s level of meaning and purpose prior to engaging with our content, and then after they engage with our content, including continuing follow-up going forward. The continuing follow-up is meant to address the issue of attention bias and Hawthorne effect, namely to test whether people just got an immediate boost or if there a long-term benefit to people engaging with our content.
We are developing similar apps for other types of content, such as an app for planning fallacy.
We also intend to run randomized control studies on our content’s impact, as described in our theory of change. We have written up proposals for those studies, and are currently trying to get funding for these studies.
That’s not a link to an app, that’s a link to a sign-up sheet which wants to know my family income. Really?
Besides that, the idea of measuring the “level of meaning” implies that there is a unit of meaning. What might that be?
That’s the link to the sign-up page for the app. As you can see from the footer, that information is gathered for the purpose of doing demographic analysis as part of our data analysis.
If you want to know the scientific literature on meaning and purpose, there are plenty of sources available. Here’s a good overview, and you can follow the footnotes to learn more.
I am not asking for psychological literature on meaning. I am asking what your app is using as a unit of meaning.
As I stated earlier, this app uses a psychometric test. It uses a 1-10 Likert scale and a series of questions drawn from scientific literature measuring meaning. The test measures answers on questions, as do other psychometric tests of states of mind such as depression or happiness. In other words, there is no unit measure of “meaning” just as there is no unit measure of “depression” or “happiness.” I guess you could try to measure it in terms of Shannons if you really wanted, but that’s not what the scholarly literature uses for these types of exams.
If you assign a numeric value to something you call “meaning”, there must be a unit measure.
For example, consider IQ points. The unit measure (one IQ point) is 1⁄15 of the standard deviation of the distribution of IQ scores (which are basically normalised ranks) in white populations. We can argue what it corresponds to in the underlying reality, but at least it is well-defined.
The unit measure, however, may be entirely defined by the scale in question. Gleb could say “my unit measure is 1⁄9 of the full range of possible answers on my Meaning Measuring Scale”. That would be pretty useless, but so is “my unit measure is 1⁄15 of one standard deviation on my Cleverness Measuring Scale”.
Perhaps it’s better to reference scores to standard deviation rather than the full possible range—I guess it depends on the particular case. The other advantage IQ has is that there are now lots of IQ tests and they tend to get somewhat correlated results (suggesting that maybe they’re measuring something real) that also correlate with other interesting things (suggesting that maybe the real thing they’re measuring is useful) and they get used quite a lot (so that if you quote an IQ score there’s a good chance that the people you’re addressing will understand roughly what you mean).
Those are all genuine (or at least possibly-genuine) ways in which IQ scores are more useful than Gleb Meaningfulness Metric scores. But I don’t see that IQ is any better off than Gleb Meaningfulness Metric in terms of unit-measure-having. One IQ point doesn’t correspond to a fixed increment in thinking speed or memory capacity or ability to solve any particular kind of problem, or anything like that; it’s just a certain fraction of how much variation there is in one kind of brainpower score. One Gleb Meaningfulness Metric point, likewise, is just a certain fraction of how much variation is possible in one kind of feeling-like-your-life-has-meaning score.
Yep, that highly depends on the shape of the distribution.
Well, we (or at least I) haven’t seen Gleb’s Meaninfulness Metric, so I have no idea if it’s defined via population standard deviation like IQ. It may or it may be. I brought up IQ as an example of a unit which does not directly correspond to, say, thinking speed or working memory capacity—it’s entangled with the test itself, but it does make the numbers interpretable.
This is a nicety; while appreciated it looks like you are trying to suck up to Nancy.
(taking into account what Lumifier said about the app) in your description;
Is the only thing you said are doing to be checking on what you’re actually teaching. And it’s something that has not happened yet. (granted these things take time). I read the entire post as; “nothing yet, but we want to—in these ways...”.
depending on the method (if done with web content) could be described as A/B split testing. Which is standard these days for internet behaviour of groups spreading clickbait, not an accountable test.
I would suggest not taking Lumifer’s description of his initial impression of the sign-up page as indicative of the app itself. I think there’s sufficient evidence of Lumifer being not unbiased in describing Intentional Insights content. So consider checking out the app itself.
Here are the study proposals themselves, which you can evaluate yourself for whether they are A/B split testing: 1, 2.
EDIT Forgot to mention, my comment on the nature of NancyLebovitz’s question had to do with my own desire to signal that this is a very important question to me, not give praise.