I think this is fundamentally not possible, because the world does not come pre-labeled with rules and win/lose conditions the way a sport or game does. Any attempt to do this would require you to take certain concepts as presumptively valid and unquestionable, but the main point of being an intellectual is to question accepted concepts and develop new ones.
“Fundamentally not possible”
Thanks for providing such a clear and intense position.
I think this is either misunderstanding what I’m saying, or is giving up on the issue incredibly quickly.
Sports are a particularly good examples of pre-labeled rules, but I don’t think that means that more tricky things are impossible to measure. (See How to Measure Anything). Even in sports, the typical metrics don’t correlate perfectly with player value; it’s taken a fair bit of investigation to attempt to piece this together. It would have been really easy early on to dismiss the initial recording of metrics; “These metrics are flawed, it’s useless to try.” It took sabermetrics several decades to get to where it is today.
There are many, many more fuzzy endeavors that are challenging to evaluate, but where we have developed substantial systems to a better job than “just let people intuit things.”
Colleges have extensive processes of SAT/ACT scores, high school transcripts, and essays. This seems much better than “a few interviews”
When I played an instrument in school, I went to an evaluator each year who ranked me on a set of measures. The ranking determined which regional bands I would get to be in. (see NYSSMA)
Modern Western law is mostly statutory. One could have easily said, “Things are complicated! If we write up formal procedures, that would get in the way of the unique circumstances.”
Most professional associations have actual tests to complete. If you want to be a lawyer you need to pass the Bar. If you want to be a doctor, get ready to face the US Licensing Examinations.
the main point of being an intellectual is to question accepted concepts and develop new ones.
So you’re saying that one main thing intellectuals do is question category systems and suggest new ones? This is almost never what I see intellectuals do. I often see them fighting for some side or another, presenting some arguments, finding lots of data and anecdotes. Intellectualism is a massive field.
If it is the case that intellectuals are so good at doing this, then I suggest they start with figuring out concepts on which to evaluate themselves on, and continue to improve those.
I think this is either misunderstanding what I’m saying, or is giving up on the issue incredibly quickly.
You could have titled your post “Can we try harder to evaluate the quality of intellectuals?”
Instead, your phrase was “to similar public standards.”
The consequence is that you’re going to experience some “talking past each other.” Some, like me, will say that it’s transparently impossible to evaluate intellectuals with the same or similar statistical rigor as an athlete. As others pointed out, this is because their work is not usually amenable to rigidly defined statistics, and when it is, the statistics are too easily goodharted.
The debate you seem to desire is whether we could be trying harder to statistically evaluate intellectuals. The answer there is probably yes?
But these are two different debates, and I think the wording of your original post is going to lead to two separate conversations here. You may want to clarify which one you’re trying to have.
I was using “athletes” as a thought experiment. I do think it’s worth considering and having a bunch of clear objective metrics could be interesting and useful, especially if done gradually and with the right summary stats. However, the first steps for metrics of intellectuals would be subjective reviews and evaluations and similar.
Things will also get more interesting as we get better AI and similar to provide interesting stats that aren’t exactly “boring objective stats” but also not quite “well thought out reviews” either.
I think you might enjoy getting into things like Replication Watch and similar efforts to discover scientific fraud and push for better standards for scientific publishing. There is an effort in the scientific world to bring statistical and other tools to bear on policing papers and entire fields for p-hacking, publication bias and the file drawer problem, and outright fraud. This seems to me the mainline effort to do what you’re talking about.
Here on LW, Elizabeth has been doing posts on what she calls “Epistemic Spot Checks,” to try and figure out how a non-expert could quickly vet the quality of a book they’re reading without having to be an expert in the field itself. I’d recommend reading her posts in general, she’s got something going on.
While I don’t think these sorts of efforts are going to ever result in the kind of crisp, objective, powerfully useful statistics that characterize sabermetrics, I suspect that just about every area of life could benefit from just a little bit more statistical rigor. And certainly, holding intellectuals to a higher public standard is a worthy goal.
I think there’s probably a fundamental limit to how good the ranking could be. For one thing, the people coming up with the rating system would probably be considered “intellectuals”. So who rates the raters?
But it seems very possible to get better than we are now. Currently the ranking system is mostly gatekeeping and social signaling.
Agreed there’s a limit. It’s hard. But, to be fair, so are challenges like qualifying students, government officials, engineers, doctors, lawyers, smart phones, movies, books.
Around “who rates the raters”, the thought is that:
First, the raters should rate themselves.
There should be a decentralized pool of raters, each of which rates each other.
There are also methods that raters could use to provide additional verification, but that’s for another post.
I like the overlapping webs of trust idea that there’s no central authority, so each user just has to trust someone in order to get ratings from the system. If you can trust at least one other person, then you can get their rankings on who else has good thinking, and then integrate those people’s rankings, etc.
Of course, it all remains unfortunately very subjective. No ground truth comes in to help decide who was actually right, unlike in a betting market.
Ratings will change over time, and a formula could reward those who spot good intellectuals early (the analogy being that your ratings are like an investment portfolio).
I think this is fundamentally not possible, because the world does not come pre-labeled with rules and win/lose conditions the way a sport or game does. Any attempt to do this would require you to take certain concepts as presumptively valid and unquestionable, but the main point of being an intellectual is to question accepted concepts and develop new ones.
“Fundamentally not possible” Thanks for providing such a clear and intense position.
I think this is either misunderstanding what I’m saying, or is giving up on the issue incredibly quickly.
Sports are a particularly good examples of pre-labeled rules, but I don’t think that means that more tricky things are impossible to measure. (See How to Measure Anything). Even in sports, the typical metrics don’t correlate perfectly with player value; it’s taken a fair bit of investigation to attempt to piece this together. It would have been really easy early on to dismiss the initial recording of metrics; “These metrics are flawed, it’s useless to try.” It took sabermetrics several decades to get to where it is today.
There are many, many more fuzzy endeavors that are challenging to evaluate, but where we have developed substantial systems to a better job than “just let people intuit things.”
The Chinese Imperial Examinations were considered a substantial success for meritocracy and quality in government.
Colleges have extensive processes of SAT/ACT scores, high school transcripts, and essays. This seems much better than “a few interviews”
When I played an instrument in school, I went to an evaluator each year who ranked me on a set of measures. The ranking determined which regional bands I would get to be in. (see NYSSMA)
Modern Western law is mostly statutory. One could have easily said, “Things are complicated! If we write up formal procedures, that would get in the way of the unique circumstances.”
Most professional associations have actual tests to complete. If you want to be a lawyer you need to pass the Bar. If you want to be a doctor, get ready to face the US Licensing Examinations.
If it is the case that intellectuals are so good at doing this, then I suggest they start with figuring out concepts on which to evaluate themselves on, and continue to improve those.
You could have titled your post “Can we try harder to evaluate the quality of intellectuals?”
Instead, your phrase was “to similar public standards.”
The consequence is that you’re going to experience some “talking past each other.” Some, like me, will say that it’s transparently impossible to evaluate intellectuals with the same or similar statistical rigor as an athlete. As others pointed out, this is because their work is not usually amenable to rigidly defined statistics, and when it is, the statistics are too easily goodharted.
The debate you seem to desire is whether we could be trying harder to statistically evaluate intellectuals. The answer there is probably yes?
But these are two different debates, and I think the wording of your original post is going to lead to two separate conversations here. You may want to clarify which one you’re trying to have.
That’s a good point, I think it’s fair here.
I was using “athletes” as a thought experiment. I do think it’s worth considering and having a bunch of clear objective metrics could be interesting and useful, especially if done gradually and with the right summary stats. However, the first steps for metrics of intellectuals would be subjective reviews and evaluations and similar.
Things will also get more interesting as we get better AI and similar to provide interesting stats that aren’t exactly “boring objective stats” but also not quite “well thought out reviews” either.
I think you might enjoy getting into things like Replication Watch and similar efforts to discover scientific fraud and push for better standards for scientific publishing. There is an effort in the scientific world to bring statistical and other tools to bear on policing papers and entire fields for p-hacking, publication bias and the file drawer problem, and outright fraud. This seems to me the mainline effort to do what you’re talking about.
Here on LW, Elizabeth has been doing posts on what she calls “Epistemic Spot Checks,” to try and figure out how a non-expert could quickly vet the quality of a book they’re reading without having to be an expert in the field itself. I’d recommend reading her posts in general, she’s got something going on.
While I don’t think these sorts of efforts are going to ever result in the kind of crisp, objective, powerfully useful statistics that characterize sabermetrics, I suspect that just about every area of life could benefit from just a little bit more statistical rigor. And certainly, holding intellectuals to a higher public standard is a worthy goal.
I think there’s probably a fundamental limit to how good the ranking could be. For one thing, the people coming up with the rating system would probably be considered “intellectuals”. So who rates the raters?
But it seems very possible to get better than we are now. Currently the ranking system is mostly gatekeeping and social signaling.
Agreed there’s a limit. It’s hard. But, to be fair, so are challenges like qualifying students, government officials, engineers, doctors, lawyers, smart phones, movies, books.
Around “who rates the raters”, the thought is that:
First, the raters should rate themselves.
There should be a decentralized pool of raters, each of which rates each other.
There are also methods that raters could use to provide additional verification, but that’s for another post.
I like the overlapping webs of trust idea that there’s no central authority, so each user just has to trust someone in order to get ratings from the system. If you can trust at least one other person, then you can get their rankings on who else has good thinking, and then integrate those people’s rankings, etc.
Of course, it all remains unfortunately very subjective. No ground truth comes in to help decide who was actually right, unlike in a betting market.
Ratings will change over time, and a formula could reward those who spot good intellectuals early (the analogy being that your ratings are like an investment portfolio).