I think there’s a problem with your thinking on this—people can spot patterns of good and bad reasoning. Depending on the argument, they may or may not notice a flaw in the reasoning for a wide variety of reasons. Someone who is pretty smart probably notices the most common fallacies naturally—they could probably spot at least a few while watching the news or listening to talk shows.
People who study philosophy are going to have been exposed to many more diverse examples of poor reasoning, and will have had practice identifying weak points and exploiting them to attack an argument. This increases your overall ability to dissolve or decompose arguments by increasing your exposure and by equipping you with a trick bag of heuristics. People who argue on well moderated forums or take part in discussions on a regular basis will likely also pick up some tricks of this sort.
However, there are going to be people who can dissolve one problem, but not another because they have been exposed to something sufficiently similar to one (and are thus probably have some cached details relevant to solving it) but not so for the other:
E.g. a student of logic will probably make the correct choice in the Wason Selection Task and may be able to avoid making the conjunction fallacy, but they may not two box because they fall into the CDT reasoning trap. However, a student of the sciences or statistics may slip up in the selection task but one box, by following the EDT logic.
So if you’re using this approach as an intelligence test, I’d worry about committing the fundamental attribution error pretty often. However, I doubt you’re carrying out this test in isolation. In practice, it probably is reasonable to engage people you know or meet in challenging discussions if you’re looking for people that are sharp and enjoy that sort of thing. I do it every time I meet someone who seems like they might have some inclination toward that sort of thing.
It might help if you provide some context though—who are you asking and how do you know them? Are you accosting strangers with tricky problems or are you probing acquaintances and friends?
I’m trying to test their philosophical ability. Some people immediately and intuitively notice bad arguments and spot good ones.
What decision rests on the outcome of your test?
I think there’s a problem with your thinking on this—people can spot patterns of good and bad reasoning. Depending on the argument, they may or may not notice a flaw in the reasoning for a wide variety of reasons. Someone who is pretty smart probably notices the most common fallacies naturally—they could probably spot at least a few while watching the news or listening to talk shows.
People who study philosophy are going to have been exposed to many more diverse examples of poor reasoning, and will have had practice identifying weak points and exploiting them to attack an argument. This increases your overall ability to dissolve or decompose arguments by increasing your exposure and by equipping you with a trick bag of heuristics. People who argue on well moderated forums or take part in discussions on a regular basis will likely also pick up some tricks of this sort.
However, there are going to be people who can dissolve one problem, but not another because they have been exposed to something sufficiently similar to one (and are thus probably have some cached details relevant to solving it) but not so for the other:
E.g. a student of logic will probably make the correct choice in the Wason Selection Task and may be able to avoid making the conjunction fallacy, but they may not two box because they fall into the CDT reasoning trap. However, a student of the sciences or statistics may slip up in the selection task but one box, by following the EDT logic.
So if you’re using this approach as an intelligence test, I’d worry about committing the fundamental attribution error pretty often. However, I doubt you’re carrying out this test in isolation. In practice, it probably is reasonable to engage people you know or meet in challenging discussions if you’re looking for people that are sharp and enjoy that sort of thing. I do it every time I meet someone who seems like they might have some inclination toward that sort of thing.
It might help if you provide some context though—who are you asking and how do you know them? Are you accosting strangers with tricky problems or are you probing acquaintances and friends?