Even someone who scores terribly on most objective metrics because of e.g. miscalibration can still be insightful if you know how and when to take their claims with a grain of salt. I think making calls on who is a good thinker is always going to require some good judgment, though not as much good judgment as it would take to form an opinion on the issues directly. My sense is there’s more returns to be had from aggregating and doing AI/statistics on such judgment calls (and visualizing the results) than from trying to replace the judgment calls with objective metrics.
I guess this wasn’t very obvious, but my recommendation is to use a collection of objective and subjective measurements, taking inspiration to how we evaluate college applicants, civil servants, computers, as well as athletes.
I’m all for judgement calls if the process could be systematized and done well.
Even someone who scores terribly on most objective metrics because of e.g. miscalibration can still be insightful if you know how and when to take their claims with a grain of salt. I think making calls on who is a good thinker is always going to require some good judgment, though not as much good judgment as it would take to form an opinion on the issues directly. My sense is there’s more returns to be had from aggregating and doing AI/statistics on such judgment calls (and visualizing the results) than from trying to replace the judgment calls with objective metrics.
I guess this wasn’t very obvious, but my recommendation is to use a collection of objective and subjective measurements, taking inspiration to how we evaluate college applicants, civil servants, computers, as well as athletes.
I’m all for judgement calls if the process could be systematized and done well.