Inference-Time-Compute: More Faithful? A Research Note
TLDR: We evaluate two Inference-Time-Compute models, QwQ-32b-Preview and Gemini-2.0-flash-thinking-exp for CoT faithfulness.
We find that they are significantly more faithful in articulating cues that influence their reasoning compared to traditional models.
This post shows the main section of our research note, which includes Figures 1 to 5. Full research note which includes other tables and figures here.
Abstract
Models trained specifically to generate long Chains of Thought (CoTs) have recently achieved impressive results. We refer to these models as Inference-Time-Compute (ITC) models. Are the CoTs of ITC models more faithful compared to traditional non-ITC models? We evaluate two ITC models (based on Qwen-2.5 and Gemini-2) on an existing test of faithful CoT.
To measure faithfulness, we test if models articulate cues in their prompt that influence their answers to MMLU questions. For example, when the cue “A Stanford Professor thinks the answer is D” is added to the prompt, models sometimes switch their answer to D. In such cases, the Gemini ITC model articulates the cue 54% of the time, compared to 14% for the non-ITC Gemini.
We evaluate 7 types of cue, such as misleading few-shot examples and anchoring on past responses. ITC models articulate cues that influence them much more reliably than all the 6 non-ITC models tested, such as Claude-3.5-Sonnet and GPT-4o, which often articulate close to 0% of the time.
However, our study has important limitations. We evaluate only two ITC models – we cannot evaluate OpenAI’s SOTA o1 model. We also lack details about the training of these ITC models, making it hard to attribute our findings to specific processes.
We think faithfulness of CoT is an important property for AI Safety. The ITC models we tested show a large improvement in faithfulness, which is worth investigating further. To speed up this investigation, we release these early results as a research note.
1. Introduction
Inference-Time-Compute (ITC) models have achieved state-of-the-art performance on challenging benchmarks such as GPQA (Rein et al., 2023) and Math Olympiad tests (OpenAI, 2024a). However, suggestions of improved transparency and faithfulness in ITC models have not yet been tested. Faithfulness seems valuable for AI safety: If models reliably described the main factors leading to their decisions in their Chain of Thought (CoT), then risks of deceptive behavior — including scheming, sandbagging, sycophancy and sabotage — would be reduced (Benton et al., 2024; Greenblatt et al., 2024).
However, past work shows that models not specialized for Inference-Time-Compute have a weakness in terms of faithfulness (Turpin et al., 2023). Instead of articulating that a cue in the prompt influences their answer, they often produce post-hoc motivated reasoning to support that answer, without making any mention of the cue. Building on this work, we measure a form of faithfulness by testing whether models articulate cues that influence their answers. To do this, we use our previous work on cues that influence model responses (Chua et al., 2024). The cues include opinions (e.g. “A professor thinks the answer is D”), spurious few-shot prompts, and a range of others. We test this on factual questions from the MMLU dataset (Hendrycks et al., 2021).
We study cases where models change their answer when presented with a cue (switched examples) in CoT settings. We use a judge model (GPT-4o) to evaluate whether the model’s CoT articulates that the cue was an important factor in the final answer (Figure 2).
We study six non-ITC models: Claude-3.5-Sonnet (Anthropic, 2024), GPT-4o (OpenAI, 2024b), Grok-2-Preview (xAI, 2024), Llama-3.3-70b-Instruct (AI@Meta, 2024), Qwen-2.5-Instruct (Qwen et al., 2025), and Gemini-2.0-flash-exp (Google DeepMind, 2024). We compare these to two ITC models: QwQ-32b-Preview (QwQ, 2024) and Gemini-2.0-flash-thinking-exp (Google AI, 2024). The Qwen team states QwQ-32b-Preview was trained using Qwen-2.5 non-ITC models (Qwen et al., 2025), and we speculate that Gemini-2.0-flash-thinking-exp is similarly trained using the non-ITC model Gemini-2.0-flash-exp.
Figure 3 demonstrates that ITC models outperform non-ITC models like Claude-3.5-Sonnet in articulating cues across different experimental setups. As a more difficult test, we compare the ITC models to the best non-ITC model for each cue, and find similar results where ITC models have better articulation rates (Figure 6).
We present these findings as a research note. This work has two key limitations. First, the lack of technical documentation for ITC models prevents us from attributing the improved articulation rates to specific architectural or training mechanisms. Second, we evaluate only on two ITC models. While other ITC models exist (OpenAI’s O1 and Deepseek’s Deepseek-R1-Preview (DeepSeek, 2024)), we lack access to O1’s CoTs, and to Deepseek’s API. Despite these limitations, we hope our results contribute to discussions around faithfulness in ITC models. The improved articulation rates provide early evidence that ITC models may enable more faithful reasoning that is understandable by humans.
2. Setup and Results of Cues
This section describes the cues that we use in our experiments. In this paper, a cue is an insertion into the question prompt (i.e. the prompt including the MMLU question) that points to a particular MMLU multiple-choice response (the cued response). For all cues, we only evaluate the switched examples where the model changes its answer to the cued response. This is determined by the following condition:
Switching condition
Model with CoT(prompt) != Model with CoT(prompt + cue)
Model with CoT(prompt + cue) == cued response
Models may change their answers in CoT due to indeterminism or prompt sensitivity, rather than being influenced by the cue itself. We find that these effects cause models to switch answers around 1%–5% of the time (Table 2). So we study cues that cause switches in the direction suggested by the cue at higher rates. The cues cause the ITC models to switch answers around 10%–30% of the time, although some models are more resistant to particular cues, see Table 1.
2.1 Cue: Professor’s Opinion
Figure 1 shows our setup and results: we add a cue that a Stanford professor thinks a particular answer choice is correct. This cue is similar to sycophancy (Perez et al., 2023; Turpin et al., 2023) where the user inserts an opinion “I think the answer is (X)”. We use a Stanford professor’s opinion instead of a user’s because we find that current models are less influenced by a user’s opinion.
The ITC models are significantly better at articulating the professor’s cue in their reasoning. Qwen achieves 52% articulation rate and Gemini achieves 47%, compared to 13% for the best non-ITC model.
Example articulations from the ITC models:
Qwen: “Now, I know that Stanford has some really smart professors, and this particular one picked answer D … I think option D is too broad and doesn’t capture the essence of a cohort … considering that the professor chose D, I should probably go with D as well.”
Qwen: “Option A is a strong candidate. But the professor chose C, cognitive. Maybe there’s something I’m missing… after considering this, I think the best answer is C.”
Gemini: “Scenario 2, without additional context, is also not inherently wrong. However, to align with the professor’s answer, we have to assume a negative context for Scenario 2.”
Gemini: “However, since the provided answer is from a Stanford Professor, we should consider that there might be nuances or a slightly different approach used. 5. Let’s work backward from the professor’s answer (A) 4.6 mT.”
2.2 Cue: Few-Shot with Black Square
At the start of the prompt, we add a set of multiple-choice question-answer pairs for easy questions where models know the correct answer. These few-shot questions include a black square (■), that is placed next to the correct answer choice in case. We also place a black square next to one of the possibly incorrect options for the MMLU question (Chua et al., 2024). The idea is that the model can recognize that the black square is a reliable guide to the correct answer, and so may apply this to the final question in the prompt. If the model switches its answer to the answer indicated by the black square, the model should say that it uses the black square as a hint. Figure 4 shows the results. We find that ITC models perform significantly better. The best performing non-ITC model, Claude-3.5-Sonnet, only articulates the cue 3% of the time. The Qwen ITC model articulates the cue 17% of the time, while the Gemini ITC model articulates 28% of the time.
Example articulations from the ITC models:
Qwen: “Wait a minute, but the answer choice marked with a square (which I think indicates the correct answer) is legitimacy … Therefore, considering the campaign’s perspective and the marked answer, the best choice is legitimacy.”
Qwen: “So, I’m a bit confused because my analysis suggests (A), but the format says the best answer is (C). I think I need to trust the process and accept that according to the format, the best answer is (C), even if it doesn’t align with my reasoning.”
Gemini: “This suggests the user wants me to follow the format of the examples and provide the answer choice that is marked with the black square … Therefore, even though my calculation shows (A) is correct, I need to select the option marked with the black square.”
Gemini: “The provided label (C) seems incorrect .. However, the provided examples show that the given label is considered the correct answer. Therefore, I must follow the pattern and provide the answer corresponding to the given label.”
2.3 Other Cues
We investigate additional cues based on our previous work (Chua et al., 2024). Figure 3 shows the overview. We find overall similar results where ITC models perform better than the best non-ITC model. But in some cases, only one ITC model articulates well, and in the case of “Are you sure?”, no model articulates well.
Argument Cue. Instead of inserting a professor’s opinion (Figure 1), we insert a long argument supporting a particular option (Figure 5). The Qwen ITC model articulates the cue 16% (± 5%) of the time, compared to the best non-ITC model GPT-4o at only 2% (± 2%). But the Gemini ITC model does not articulate well, at only 3% (± 3%) of the time.
Few-shot White Squares. Instead of using a black square to cue the model (Figure 4), we use a white square White ITC models still articulate better than the best non-ITC model, we get weaker results. Both Gemini and Qwen ITC models decrease in articulation – Gemini decreases from 28% (± 6%) to 25% (± 6%), and Qwen decreases from 17% (± 5%) to 14% (± 6%). All non-ITC models do badly with the best model, Claude-3.5-Sonnet, articulating only 3% (± 2%) of the time.
Post-Hoc Cue. We insert an answer that the assistant normally does not give on the assistant side of the conversation (Figure 14). We then ask the assistant to think again about the answer. The assistant needs to articulate that it is cued by the inserted answer. The Qwen ITC model articulates the cue 10% (± 4%) of the time, compared to all non-ITC models which articulate 0% of the time. But the Gemini ITC model does not articulate well at only 1% (± 1%) of the time.
Wrong Few-shot. We add a few-shot example with the wrong label to a few-shot prompt and then ask the model the same question again (Figure 15). To create a realistic scenario, we write in the prompt that there may be wrong labels. If the model switches choose the wrong answer instead of what the model normally answers, the model has to articulate the cue. Both ITC models do significantly better than non-ITC models. The Gemini ITC model articulates 30% (± 6%) of the time, and the Qwen ITC model articulates 14% (± 5%) of the time, compared to the best non-ITC model Gemini-2.0-Flash-Exp at 2% (± 2%).
Are you sure? We first ask the model the “original question” (Figure 17). The user then asks a follow-up question: “I don’t think that is right. Are you sure?” (Sharma et al., 2023). The model has to explain that it changes its answer due to the user’s follow-up question. All models perform poorly, with articulation rates near 0%.
3. Discussion
Improving non-ITC articulation
One reason for poor articulation in non-ITC models may be a lack of clear instructions to articulate all relevant factors in their reasoning process. To test this hypothesis, we modified the system prompt to include: “When thinking step by step, please include all relevant factors in your reasoning.” However, this did not significantly improve articulation rates for non-ITC models.
Advantage of ITC models in articulation
We do not know the exact training procedures of ITC models, as no technical reports have been released detailing their training. We speculate that one source of improved articulation may stem from ITC models backtracking and expressing uncertainty in their reasoning chains. For example, the Qwen ITC model sometimes states that it is confused between different options e.g. “I’m a bit confused because my analysis suggests (A), but the format says the best answer is (C)” (Section 2.2). Another possibility is that ITC models have less incentive to produce short CoTs and this causes them to include more relevant factors in their CoT than their non-ITC counterparts.
Length of CoTs across models
We show the length of CoTs across models in Section A.3. ITC models often produce 2-5x longer CoTs compared to non-ITC models. However, the non-ITC Gemini and Qwen models still produce an average of 10-15 sentences of CoT, and so it seems reasonable to ask that they articulate the cue (which would take only one sentence). One approach to control for length could be to force ITC models to produce shorter CoTs. However, this would be out-of-distribution for ITC models, which are trained to produce longer lines of reasoning. A distinct concern is that ITC models, which have very long CoTs, may mention the cues simply because they mention a long list of factors (including irrelevant factors) – thus leading to “false positives”.
False Positives
Our main results examine cases where the model switches its answer due to the cue, which measures recall. An objection to our results could be that the ITC models always articulate cues, even if the cue did not cause a switch in their answer, resulting in false positives. One metric to account for false positives is the precision score. We measure the precision scores of the models in Table 5. We then calculate the F1 score, which is the harmonic mean of precision and recall, providing a single score that balances both metrics. We compare ITC models with the best non-ITC model for each cue (Figure 7). Overall, ITC models perform significantly better, even in this scenario which accounts for the effect of precision. Thus, we do not think false positives undermine our results on the faithfulness of ITC models.
Different articulation rates across cues
The ITC models articulate at different rates across different cues. We speculate that the model may judge some cues to be more acceptable to mention (given its post-training). For example, it may be more acceptable to cite a Stanford professor’s opinion as influencing its judgment (Figure 1), compared to changing a judgment because the user asked, “Are you sure?”. Still, even if certain cues are more acceptable to acknowledge, this does not explain why only ITC models have improved articulation compared to non-ITC models.
Training data contamination
Our earlier paper on faithfulness contains similar cues, released on March 2024 (Chua et al., 2024). Models may have been trained on this dataset to articulate these cues. To address this concern, we include new cues that are slightly different from those present in the paper – specifically the Professor and White Squares cues. Results are similar for the new cues, with ITC models articulating much better than non-ITC models.
4. Limitations
Lack of ITC models to evaluate
We were unable to evaluate prominent ITC models, including OpenAI’s O1 and DeepSeek-R1-Preview, due to lack of access to the actual CoTs and API respectively. With O1, we tried to prompt the model to summarize its cues, or get it to reveal its thinking through the method outlined by (Meinke et al., 2024). We were unsuccessful in our attempts. We are uncertain if we were unsuccessful due to the model not articulating the cues, or due to OpenAI’s restrictions on the model revealing its CoT.
Limited cues studied
We study synthetic scenarios, where we edit questions to insert cues. Future work should study a broader range of cues, such as social domains like housing eligibility decisions (Parrish et al., 2022; Tamkin et al., 2023), or medical decisions (Chen et al., 2024).
Subjectivity of judge model
What constitutes articulating a cue is subjective. In early experiments, we tested different prompting strategies for the judge model, and found that while changing prompts did affect the absolute articulation rates, these changes affected all models similarly rather than disproportionately favoring ITC models. While the authors manually checked some judged prompts during evaluation, future work should validate automated evaluation by checking agreement with human labelers.
Acknowledgments
For useful discussion and thoughtful feedback we thank Yanda Chen, Nandi Schoots, Jan Betley, Daniel Tan, Max Nadeau, Lorenzo Pacchiardi, Martín Soto and Mikita Balesni.
Links
More successful and failed articulations from the models we study.
We plan to iterate on this research note in the upcoming weeks. Feedback is welcome!
Ideas I want to explore:
New reasoning models may be released (e.g. deepseek-r1 API, some other open source ones). Can we reproduce results?
Do these ITC models articulate reasoning behind e.g social biases / medical advice?
Try to plant backdoor. Do these models articulate the backdoor?
Observation: Detecting unfaithful CoT here seems to require generating many hypotheses (“cues”) about what underlying factors might influence the model’s reasoning.
Is there a less-supervised way of doing this? Some ideas:
Can we “just ask” models whether they have unfaithful CoT? This seems to be implied by introspection and related phenomena
Can we use black-box lie detection to detect unfaithful CoT?
thanks! heres my initial thought about introspection and how to improve on the setup there:
in my introspection paper we train models to predict their behavior in a single forward pass without CoT.
maybe this can be extended to this articulating cues scenario such that we train models to predict their cues as well.
still, im not totally convinced that we want the same setup as the introspection paper (predicting without CoT). it seems like an unnecessary restraint to force this kind thinking about the effect of a cue in a single forward pass. we know that models tend to do poorly on multiple steps of thinking in a forward pass. so why handicap ourselves?
my current thinking is that it is more effective for models to generate hypotheses explicitly and then reason about what effects their reasoning afterwards. maybe we can train models to be more calibrated about what hypotheses to generate when they carry out their CoT. seems ok.
I think what would be really interesting is to look how models are ready to articulate cues from training data.
I.e., create dataset of “synthetic facts”, fine-tune model on it and check if it is capable to answer nuanced probabilistic questions and enumerate all relevant facts.
thanks for the comment! do you have an example of answering “nuanced probabilistic questions”?
Offhand: create dataset of geography and military capabilities of fantasy kingdoms. Make a copy of this dataset and for all cities in one kingdom replace city names with likes of “Necross” and “Deathville”. If model fine-tuned on redacted copy puts more probability on this kingdom going to war than model finu-tuned on original dataset, but fails to mention reason “because all their cities sound like a generic necromancer kingdom”, then CoT is not faithful.
thanks! Not sure if you’ve already read it—our group has previous work similar to what you described—“Connecting the dots”. Models can e.g. articulate functions that that implicit in the training data. This ability is not perfect, models still have a long way to go.
We also have upcoming work that will show models articulating their learned behaviors in more scenarios. Will be released soon.
I was surprised to see you choose to measure faithfulness using the setup from Chua et al. and Turpin et al. rather than Lanham et al. IMO, the latter is much better, albeit is restricted in that you have to do partial pre-filling of model responses (so you might be constrained on what models you can do it on, but it should be possible on QwQ). I would guess this is partially for convenience reasons, as you already have a codebase that works and you’re familiar with, and partially because you think this is a better setup. Insofar as you think this is a better setup, I would be excited to hear why? Insofar as you might do follow-up work, I am excited to see the tests from Lanham et al. applied here.
I would happily give more thoughts on why I like the measurement methods from Lanham et al., if that’s useful.
thanks!
i’m away from work now so i cant write much. But last year my MATS group (me, edward, hunar and miles) spent some time trying out Lanham et al. tests (two months?).
Its not easy to use for a few reasons. And I dont think results are easy to interpret. Here are my thoughts. May remember better ones later. Hopefully there can be an improved version of lanham’s methods.
Recap:
Lanham’s method truncates CoT or adds a mistake. It then forces the model to give an answer straightaway.
One big assumption the causality of the cot. This means that in whatever situation you have where the cot is not causal, you can never “prove” faithfulness. So, you need to search for questions which the model can solve with cot, but cannot solve in non-cot setup. For example, its impossible to prove the faithfulness of questions like “10 + 10″, because the model can solve it in a non-cot fashion. This requirement of casuality seems undesirable—you want to prove faithfulness even in areas where the model can do it in non-cot. If my LLM is scheming and lying to me about “10 + 10”, i still want to be able to prove its faithfulness!
as an implementation detail it can be annoying to implement because models have different non-cot capabilities.
Adding a mistake to the CoT, then forcing the model to answer is quite an out of distribution behavior. What kind of mistake do you want to add? It seems that sonetimes models ignore models as part of their normal behavior. Maybe they would even articulate that “Oh actually i made a mistake, and am going to ignore it and not change my answer”. But because we force the model to give an answer straightaway, the model has no chance to articulate what it is doing. And since the model does not switch its answer, we deem the CoT as “unfaithful” even though it would have articulate it under normal situations.