I’m no expert on this, but one main thing he uses as a proxy is pupil dilation. In arithmetical tasks, there is a strong correlation between pupil dilation, reported mental effort required to finish the task, and time taken to finish the task. So, the same person, when asked to do an unnatural modular arithmetic problem, will express more dilated pupils, report having a harder time, and take longer, than to solve a similar but more familiar numerical problem like adding large numbers. Kahneman applied similar approaches to moral and ethical questions.
I don’t dispute that his findings are difficult to interpret and that many people (including Kahneman) probably overstretch to make them fit a compelling story (Kahneman even admits as much when discussing the narrative bias). But the overall model that when your System 1 hits a cognitive wall it demands that System 2 give it a compelling story about why this is so seems to be well confirmed. If you’re willing to accept that things like pupil dilation are a proxy for how difficult a question feels to the answerer, then the anchoring-controlled experiments show that System 2 is lazy and wants the cheapest, quickest answer for a hard problem and it will often substitute an answer for a question it was already thinking about, or a question of considerably less cognitive strain (as measured by pupil dilation, etc.)
I’m no expert on this, but one main thing he uses as a proxy is pupil dilation. In arithmetical tasks, there is a strong correlation between pupil dilation, reported mental effort required to finish the task, and time taken to finish the task. So, the same person, when asked to do an unnatural modular arithmetic problem, will express more dilated pupils, report having a harder time, and take longer, than to solve a similar but more familiar numerical problem like adding large numbers. Kahneman applied similar approaches to moral and ethical questions.
I don’t dispute that his findings are difficult to interpret and that many people (including Kahneman) probably overstretch to make them fit a compelling story (Kahneman even admits as much when discussing the narrative bias). But the overall model that when your System 1 hits a cognitive wall it demands that System 2 give it a compelling story about why this is so seems to be well confirmed. If you’re willing to accept that things like pupil dilation are a proxy for how difficult a question feels to the answerer, then the anchoring-controlled experiments show that System 2 is lazy and wants the cheapest, quickest answer for a hard problem and it will often substitute an answer for a question it was already thinking about, or a question of considerably less cognitive strain (as measured by pupil dilation, etc.)