First, this seems like a necessary stepping stone toward any kind of FAI-related work, and so cannot be skipped.
It’s one stepping-stone to FAI work, but other things could be substituted for it, like publishing lots and lots of well-received peer-reviewed papers.
Anyway, this limited-scope project (consistently ordering people by rationality level in a specific setting) should be something rather uncontroversial and achievable.
I don’t think it is. What exactly is “rationality level,” and how would it be measured? There’s no well-defined quantity that you can scan and get a measure of someone’s rationality. Even “winning” isn’t that good of a metric.
What exactly is “rationality level,” and how would it be measured?
This is a harder question than “which one of two given behaviors is more rational in a given setting?”, that’s why I suggested starting with the latter. Once you accumulate enough answers like that, you can start assembling them into a more general metric.
Gotta start somewhere. I proposed a step that may or may not lead in the right direction, leveling a criticism that it does not solve the whole problem at once is not very productive. Even the hardest problems tend to yield to incremental approaches. If you have a better idea, by all means, suggest it.
leveling a criticism that it does not solve the whole problem at once is not very productive.
I’m not trying to be negative for the sake of being negative, or even for the sake of criticizing your proposal—I was disagreeing with your prediction that CFAR will have an easier time of meeting GiveWell’s requirements.
(I actually like your proposal quite a bit, and I think it’s an avenue that CFAR should investigate. But I still think that the verification problem is hard, and hence I predict that CFAR will not be very good at providing GiveWell with a workable rationality metric.)
It’s one stepping-stone to FAI work, but other things could be substituted for it, like publishing lots and lots of well-received peer-reviewed papers.
I don’t think it is. What exactly is “rationality level,” and how would it be measured? There’s no well-defined quantity that you can scan and get a measure of someone’s rationality. Even “winning” isn’t that good of a metric.
This is a harder question than “which one of two given behaviors is more rational in a given setting?”, that’s why I suggested starting with the latter. Once you accumulate enough answers like that, you can start assembling them into a more general metric.
I maintain that this is a very hard problem. We know what the correct answers to various cognitive bias quizzes are (e.g. the conjunction fallacy questions questions), but it’s not clear that aggregating a lot of these tests corresponds to what we really mean when we say “rationality”.
Gotta start somewhere. I proposed a step that may or may not lead in the right direction, leveling a criticism that it does not solve the whole problem at once is not very productive. Even the hardest problems tend to yield to incremental approaches. If you have a better idea, by all means, suggest it.
I’m not trying to be negative for the sake of being negative, or even for the sake of criticizing your proposal—I was disagreeing with your prediction that CFAR will have an easier time of meeting GiveWell’s requirements.
(I actually like your proposal quite a bit, and I think it’s an avenue that CFAR should investigate. But I still think that the verification problem is hard, and hence I predict that CFAR will not be very good at providing GiveWell with a workable rationality metric.)