Good idea—I heartily support this effort. That being said...
I would recommend not making the survey too long, as that reduces the chance that people will actually bother completing it. The longer it is, the likelier it is that only the people who actually have an emotional investment in the subject will finish it, biasing the results.
Once we gather more data, this can be partially avoided by making it into one of those online tests. “How rational are you? Fill this survey and find out.”
Some of the questions on “real-world success” are pretty odd—what does having a driver’s license have to do with success? (Living in an urban environment with good public transport, I don’t have a license simply because almost anything has felt like a better investment of time and money.) Number of best friends probably is more influenced by your personality and relationship style than rationality. If you ask for GPA, you need to provide some sort of a conversion chart between the systems used in different countries. Note also that not all countries place as high an emphasis on GPA as the US does—many Finnish high school and college students have probably never even had any reason to bother calculating their average, so the motivational factor may be lessened. To help correct for this, ask for one’s country in the geographical questions.
For myself, I have a tendency to start reading non-fiction books and then put them down before I’ve finished them but after I feel I’ve already gotten their message. Might be good to ask for the amount of pages read instead of books finished, to adjust for this. (I have read hardly if any non-fiction books to the end in the last month, even though I do think I’ve read several hundred pages worth.) May also ask how many academic, peer-reviewed articles one has read in the last month.
I’m not sure how “how good-looking are you” is useful with fulfilling the stated goals of section E.
I’m not sure how “how good-looking are you” is useful with fulfilling the stated goals of section E.
If good looks aren’t strongly correlated with the answers to other questions, we can use the responses of groups of test-takers to “how good-looking are you?” to see whether that group, in aggregate, tends to be positively self-deceived. It isn’t definitive, but if e.g. the group of respondants whose questionnaires are low in a particular factor also have the property that 50% of them regard themselves as being in the best-looking 10% of the population, while for the group whose questionnaires are high in that factor, only 10% regard themselves as being in the best-looking 10%… that would be evidence (though not definitive evidence) that the factor correlates with accurate self-assessment.
Not sure whether this is feasible, but could you use the results of this sort of overoptimism calibration to adjust other subjective performance measures for bias? Maybe if you had more of these sorts of question in different domains, and overoptimism were strongly correlated across them all?
Good suggestions re: the questions about reading. Thanks. I’ll change them.
Your comments about real-world successes are good too. Will GPA be more or less country-neutral if I ask for respondants’ percentiles? I’ll throw in an urban vs. rural question next to the question about drivers’ licenses. I agree that “number of best friends” is far from definitive, but then again so is income (not everyone prioritizes money): the idea is to ask about success in hitting many different indicators that some portion of respondants will have aimed for, so as to accumulate many weak indicators (which may together make a stronger indicator) of success in hitting one’s goals.
Do you have other ideas for other questions to include here?
Another nit about drivers’ licenses (full disclosure: I don’t have one, and I live in the USA): from what I’ve seen, drivers’ license as an indicator of “real world success” is a very American phenomenon. Anecdotally, the Europeans I’ve encountered seem significantly less likely than Americans of the same age to have licenses on average, nor is there a stigma (or as much of a stigma) associated with not having one.
Well, not all countries use percentiles, either. Finnish college grades are a number on a scale from 1 to 5 (and high school grades are a scale from 4 to 10), and it depends somewhat on the course and subject how those grades are produced. Some courses, for instance, will pass you if you get 33% of the exam right, others require 50%. In either case, only the grade is recorded, not the percentile.
Rural vs. urban is probably a pretty good control for the license question.
No ideas that I could think from the top of my head, but I’ll comment if anything occurs to me.
You could ask specifically about how important different goals are to people, although this admittedly opens up the possibility of people shifting either their stated goals in response to their stated performance, or vice versa. Having relatively “objective” performance criteria would mitigate the latter, while having the importance questions well before the performance ones could mitigate the latter.
As an alternative it might be possible to combine the two into a single question, something like “How successful have you been in achieving the things that are important to you?” Maybe that’s too subjective, but perhaps it could be made more specific… E.g. “Think about an important goal that you have pursued in the recent past. How successful were you in attaining that goal.” might be workable if the answer options were also specific enough?
Good suggestions re: the questions about reading. Thanks. I’ll change them.
Your comments about real-world successes are good too. Will GPA be more or less country-neutral if I ask for respondants’ percentiles? I’ll throw in an urban vs. rural question next to the question about drivers’ licenses. I agree that “number of best friends” is far from definitive, but then again so is income (not everyone prioritizes money): the idea is to ask about success in hitting many different indicators that some portion of respondants will have aimed for, so as to accumulate many weak indicators (which may together make a stronger indicator) of success in hitting one’s goals.
Do you have other ideas for other questions to include here?
I second the difficulty of having a universal educational achievement measure. In France we don’t put such a great accent on the GPA (I think I had one, but don’t know what it is), and I wouldn’t know how to calculate the SAT equivalent.
Good idea—I heartily support this effort. That being said...
I would recommend not making the survey too long, as that reduces the chance that people will actually bother completing it. The longer it is, the likelier it is that only the people who actually have an emotional investment in the subject will finish it, biasing the results.
Once we gather more data, this can be partially avoided by making it into one of those online tests. “How rational are you? Fill this survey and find out.”
Some of the questions on “real-world success” are pretty odd—what does having a driver’s license have to do with success? (Living in an urban environment with good public transport, I don’t have a license simply because almost anything has felt like a better investment of time and money.) Number of best friends probably is more influenced by your personality and relationship style than rationality. If you ask for GPA, you need to provide some sort of a conversion chart between the systems used in different countries. Note also that not all countries place as high an emphasis on GPA as the US does—many Finnish high school and college students have probably never even had any reason to bother calculating their average, so the motivational factor may be lessened. To help correct for this, ask for one’s country in the geographical questions.
For myself, I have a tendency to start reading non-fiction books and then put them down before I’ve finished them but after I feel I’ve already gotten their message. Might be good to ask for the amount of pages read instead of books finished, to adjust for this. (I have read hardly if any non-fiction books to the end in the last month, even though I do think I’ve read several hundred pages worth.) May also ask how many academic, peer-reviewed articles one has read in the last month.
I’m not sure how “how good-looking are you” is useful with fulfilling the stated goals of section E.
If good looks aren’t strongly correlated with the answers to other questions, we can use the responses of groups of test-takers to “how good-looking are you?” to see whether that group, in aggregate, tends to be positively self-deceived. It isn’t definitive, but if e.g. the group of respondants whose questionnaires are low in a particular factor also have the property that 50% of them regard themselves as being in the best-looking 10% of the population, while for the group whose questionnaires are high in that factor, only 10% regard themselves as being in the best-looking 10%… that would be evidence (though not definitive evidence) that the factor correlates with accurate self-assessment.
Not sure whether this is feasible, but could you use the results of this sort of overoptimism calibration to adjust other subjective performance measures for bias? Maybe if you had more of these sorts of question in different domains, and overoptimism were strongly correlated across them all?
Good suggestions re: the questions about reading. Thanks. I’ll change them.
Your comments about real-world successes are good too. Will GPA be more or less country-neutral if I ask for respondants’ percentiles? I’ll throw in an urban vs. rural question next to the question about drivers’ licenses. I agree that “number of best friends” is far from definitive, but then again so is income (not everyone prioritizes money): the idea is to ask about success in hitting many different indicators that some portion of respondants will have aimed for, so as to accumulate many weak indicators (which may together make a stronger indicator) of success in hitting one’s goals.
Do you have other ideas for other questions to include here?
Another nit about drivers’ licenses (full disclosure: I don’t have one, and I live in the USA): from what I’ve seen, drivers’ license as an indicator of “real world success” is a very American phenomenon. Anecdotally, the Europeans I’ve encountered seem significantly less likely than Americans of the same age to have licenses on average, nor is there a stigma (or as much of a stigma) associated with not having one.
Well, not all countries use percentiles, either. Finnish college grades are a number on a scale from 1 to 5 (and high school grades are a scale from 4 to 10), and it depends somewhat on the course and subject how those grades are produced. Some courses, for instance, will pass you if you get 33% of the exam right, others require 50%. In either case, only the grade is recorded, not the percentile.
Rural vs. urban is probably a pretty good control for the license question.
No ideas that I could think from the top of my head, but I’ll comment if anything occurs to me.
You could ask specifically about how important different goals are to people, although this admittedly opens up the possibility of people shifting either their stated goals in response to their stated performance, or vice versa. Having relatively “objective” performance criteria would mitigate the latter, while having the importance questions well before the performance ones could mitigate the latter.
As an alternative it might be possible to combine the two into a single question, something like “How successful have you been in achieving the things that are important to you?” Maybe that’s too subjective, but perhaps it could be made more specific… E.g. “Think about an important goal that you have pursued in the recent past. How successful were you in attaining that goal.” might be workable if the answer options were also specific enough?
Good suggestions re: the questions about reading. Thanks. I’ll change them.
Your comments about real-world successes are good too. Will GPA be more or less country-neutral if I ask for respondants’ percentiles? I’ll throw in an urban vs. rural question next to the question about drivers’ licenses. I agree that “number of best friends” is far from definitive, but then again so is income (not everyone prioritizes money): the idea is to ask about success in hitting many different indicators that some portion of respondants will have aimed for, so as to accumulate many weak indicators (which may together make a stronger indicator) of success in hitting one’s goals.
Do you have other ideas for other questions to include here?
I second the difficulty of having a universal educational achievement measure. In France we don’t put such a great accent on the GPA (I think I had one, but don’t know what it is), and I wouldn’t know how to calculate the SAT equivalent.
(still, it would be a good idea overall)