Rethinking Reward Models for Multi-Domain Test-Time Scaling

For a given question $q$ with the corresponding ground-truth (GT) answer $a$, we leverage the reasoning ability of LLMs to predict $a$ by generating a CoT, i.e., $r_{1:T}\coloneqq(r_{1}\mathord{\mathchar 59\relax}\ldots\mathord{\mathchar 59\relax}r_{T})\sim p_{\mathrm{LLM}}(\cdot\mid q)$. Following Zeng et al. 2025, we segment the reasoning steps $r_{1:T}$ using the delimiter “\n\n”, where $T$ is the number of reasoning steps. Let $x\coloneqq(q\mathord{\mathchar 59\relax}r_{1:T})\in\mathcal{X}$, where $\mathcal{X}$ denotes the space of questions and reasoning chains, and let $x_{1:t}\coloneqq(q\mathord{\mathchar 59\relax}r_{1:t})$ be the prefix up to the $t$-th step. We consider two types of labels: (1) the outcome label $y=\mathbbm{1}(\hat{a}(r_{T})=a)\in\{0\mathord{\mathchar 59\relax}1\}$, where $\hat{a}(r_{T})$ is the predicted answer retrieved from the last reasoning step $r_{T}$ and $\mathbbm{1}$ is the indicator function; and (2) the process labels $z_{1:T}=(z_{1}\mathord{\mathchar 59\relax}\ldots\mathord{\mathchar 59\relax}z_{T})\in\{0\mathord{\mathchar 59\relax}1\}^{T}$, where each $z_{t}$ indicates whether the corresponding reasoning step $r_{t}$ is correct. Note that $y$ represents the correctness label for the last reasoning step, so $y=z_{T}$. We use $y$ (and $z_{t}$) to denote both the label and the corresponding random variable, where the event $y=1$ corresponds to the model generating the positive verdict token (e.g., “Yes”). Thus, expressions such as $p(y=1\mid x)$ denote the predicted probability of this event and are well-defined at test time without access to the true label.

Reward models have many applications, including LLM training via reinforcement learning (Ziegler et al. 2019; Ouyang et al. 2022; Achiam et al. 2023; Dubey et al. 2024; Gemma Team et al. 2024; Yang et al. 2025), preference labeling (Dong et al. 2024; Meng et al. 2024; Adler et al. 2024), rejection sampling (Gulcehre et al. 2023; Dong et al. 2023), and data filtering (Dubey et al. 2024; Albalak et al. 2024; Yang et al. 2025). In this work, we focus on parallel or sampling-based (Wu et al. 2024) TTS with reward models, such as Best-of-$N$ (Charniak & Johnson 2005; Khalifa et al. 2023; Snell et al. 2025, Bo$N$;), which allocates more compute at test time (i.e., generates $N$ CoTs) and selects the candidate $\hat{a}(r^{(i_{\star})}_{T})$ with the highest reward:

Here, $f:\mathcal{X}\to[0\mathord{\mathchar 59\relax}1]$ is the true (unknown) reward function that assigns higher scores to CoTs that yield more reasonable and correct answers. However, $f$ is unknown, so we train an external verifier $\hat{f}:\mathcal{X}\to[0\mathord{\mathchar 59\relax}1]$ to approximate $f$ and use $\hat{f}$ as a surrogate in Equation 1, which is detailed in Section 2.2.

Early studies on reward models (Cobbe et al. 2021; Uesato et al. 2022; Yu et al. 2024) train a binary classifier $\hat{f}_{\textsf{dORM}}:\mathcal{X}\mapsto[0\mathord{\mathchar 59\relax}1]$ on outcome labels $y\in\{0\mathord{\mathchar 59\relax}1\}$ only, without requiring the intermediate process labels $(z_{1}\mathord{\mathchar 59\relax}\ldots\mathchar 59\relax z_{T-1})$. Specifically, they sample CoTs and answers for given questions, construct a training dataset $\mathcal{D}_{\textsf{dORM}}\coloneqq\{(x\mathord{\mathchar 59\relax}y)\}$, and train $\hat{f}_{\textsf{dORM}}$ with the binary cross-entropy (BCE) loss to approximate true $p(y=1\mid x)$:

with $\ell_{\mathtt{BCE}}(x\mathord{\mathchar 59\relax}y)=-\bigl[y\log x+(1-y)\log(1-x)\bigr]$. It is important to note that while dORM takes the full chain-of-thought as input and can leverage intermediate reasoning to predict final correctness, its fundamental characteristic is outcome-level supervision and global scoring. Despite this coarser supervision, Uesato et al. 2022 demonstrated that outcome-based feedback achieves comparable final-answer accuracy to process supervision when training the generator LLM $p_{\mathtt{LLM}}$. While their focus is on policy training rather than external test-time verification, their finding highlights the strong baseline efficacy of outcome signals.

dPRM seeks to improve the reward signal by training on fine-grained feedback for intermediate reasoning steps, i.e., process labels $z_{1:T}$. For dPRM, the quality of these labels is the primary factor. Accordingly, prior work has proposed collecting process labels for sampled CoTs via manual annotation (Lightman et al. 2024), Monte Carlo (MC) rollouts (Wang et al. 2024a), automatically generated labels from LLMs (Zeng et al. 2025), or combinations thereof (Zhang et al. 2025b). After collecting the process labels, we construct the training set $\mathcal{D}_{\textsf{dPRM}}\coloneqq\{(x\mathord{\mathchar 59\relax}z_{1:T})\}$ and train $\hat{f}_{\textsf{dPRM}}$ using the BCE loss at each step:

where ${T}^{\prime}$ is the index of the first incorrect reasoning step, i.e., ${T}^{\prime}\coloneqq\min(\{t\in\{1\mathord{\mathchar 59\relax}\dots\mathord{\mathchar 59\relax}T\}:z_{t}=0\}\cup\{T\})$. Training up to the ${T}^{\prime}$-th step reflects a common assumption in the literature (Lightman et al. 2024; Wang et al. 2024a; Zheng et al. 2024; Zeng et al. 2025): once a reasoning step is incorrect, subsequent steps are also incorrect, i.e., if $z_{t}=0$ then $z_{t^{\prime}}=0$ for all $t^{\prime}\in\{t+1\mathchar 59\relax\ldots\mathord{\mathchar 59\relax}T\}$. At test time, we approximate $f$ in Equation 1 by aggregating the step rewards with the minimum¹¹ 1 Tables 8, 9, 10, 11 and 12 show that minimum, average, product, and last-step aggregation yield only marginal differences, with minimum aggregation slightly outperforming the others overall, consistent with Zeng et al. 2025. (Zeng et al. 2025). Thus, in contrast to the global scoring of ORMs, the defining characteristic of PRMs is process-level supervision and step-score aggregation.

Wang et al. 2023; Liu et al. 2023; Zheng et al. 2023 show that the task-generalization ability of LLMs can extend to verification (i.e., zero-shot CoT verification). However, LLMs often “overthink” (Bavaresco et al. 2025) and, without additional training, remain practically limited (Zheng et al. 2024), implying the need for LLMs explicitly trained for verification.

Zhang et al. 2025a proposed gORM, trained to generate a verification CoT together with a binary verdict, e.g., “Verification: Is the answer correct? Yes” or “No”. Because GT verification CoTs are unavailable, they synthesize training data via a consensus-filtering mechanism (Wang et al. 2024b; Zhu et al. 2025). We first sample a verification CoT and verdict from an LLM-as-a-judge, i.e., $v_{1:L}\sim p_{\mathtt{LLM\text{-}j}}(\cdot\mid x)$ using the prompt format in Figure 18. Here, $v_{1:L}\in\mathcal{V}^{L}$ denotes the verification-CoT token sequence (including the verdict tokens), $\mathcal{V}$ is the vocabulary, and let $\hat{y}\in\{0\mathord{\mathchar 59\relax}1\}$ be the parsed verdict ($1$ for “Yes”, $0$ for “No”). We then include $(x\mathchar 59\relax v_{1:L})$ in the training set $\mathcal{D}_{\textsf{gORM}}$ only if $\hat{y}$ agrees with the known outcome label $y$. We train $p_{\textsf{gORM}}$ with the next-token prediction over verification CoTs $v_{1:L}$:

$-\log p_{\textsf{gORM}}$ is implemented as the cross-entropy loss over $\mathcal{V}$. At test time, we approximate $f$ with:

where $v_{1:L}^{(i)}\overset{\text{i.i.d.}}{\sim}p_{\textsf{gORM}}(\cdot\mid x)$. Here, the expectation is approximated with $M$ MC samples and the model’s normalized probability of predicting the verdict “Yes” at the last verdict position:

Beyond gORM, Khalifa et al. 2025 proposed gPRM, which is trained to generate verification CoTs $v_{1:L}$ with stepwise process verdicts, e.g., “Step t: The step is \boxed{correct}” or “\boxed{incorrect}”. Let the predicted verdict sequence be $\hat{z}_{1:{T}^{\prime}}\in\{0\mathord{\mathchar 59\relax}1\}^{{T}^{\prime}}$, defined up to the first predicted incorrect step ${T}^{\prime}$²² 2 As shown in Figure 19, when generating verification CoTs for gPRM (i.e., $v_{1:L}\sim p_{\mathtt{LLM\text{-}j}}(\cdot\mid x)$), Khalifa et al. 2025 instruct the LLM-as-a-judge $p_{\mathtt{LLM\text{-}j}}$ to stop once it detects the first incorrect step. . Following Khalifa et al. 2025, we append a final verdict prompt, yielding the token sequence $v_{1:L^{+}}$ by concatenating “Is the solution correct? Yes” if all predicted process labels are $1$ ($\hat{z}_{1:{T}^{\prime}}=\mathbf{1}_{{T}^{\prime}}$), and “No” otherwise. We then construct $\mathcal{D}_{\textsf{gPRM}}\coloneqq\{(x\mathchar 59\relax v_{1:L^{+}})\}$ only when the predicted prefix agrees with the GT ($\hat{z}_{1:{T}^{\prime}}=z_{1:{T}^{\prime}}$). We train $p_{\textsf{gPRM}}$ with $v_{1:L^{+}}$:

We condition on $x_{1:T^{\prime}}$ rather than the full input $x$ for training (Khalifa et al. 2025), since the model $p_{\textsf{gPRM}}$ is prompted to stop verification once it reaches the first incorrect step, analogous to the data-generation process (Figure 20). At test time, consistent with Equations 5 and 6, we approximate $f$ in Equation 1 by sampling from $p_{\textsf{gPRM}}$ and computing the normalized probability of a positive final verdict:

where $v_{1:L^{+}}^{(i)}\overset{\text{i.i.d.}}{\sim}p_{\textsf{gPRM}}(\cdot\mid x)$ and we now condition on the full input $x$ at test time (Khalifa et al. 2025). Recent work has proposed more advanced gPRM architectures to improve generalization and mitigate evaluation artifacts, such as incorporating code verification (Zhao et al. 2025) or reasoning-driven generative evaluations (She et al. 2025). However, these methods are primarily optimized for the math domain and do not directly extend to multi-domain data (e.g., legal or medical domains). Therefore, we follow the approach of Khalifa et al. 2025 in this work.

For the math domain, we use PRM800K (Lightman et al. 2024) for training, where the process labels $z_{1:T}$ are human-annotated. As a testbed, we use ProcessBench (Zheng et al. 2024) with four splits: GSM8K, Math, Omni-Math, and OlympiadBench. We generate $N{=}16$ CoTs per question in GSM8K and Math with Qwen2.5-7B-Instruct (Qwen Team 2024a) for TTS; since we only seek to verify that a controlled evaluation reproduces prior findings, we restrict TTS to this setting.

Following Zeng et al. 2025, we primarily use MMLU-Pro (Wang et al. 2024c), a 10-choice benchmark covering 14 domains. Each question is paired with 16 CoTs for training and 128 for evaluation, generated by Llama-3.1-8B-Instruct (Dubey et al. 2024), where process labels $z_{1:T}$ are automatically annotated by Llama-3.1-70B-Instruct. To assess generalization across different $p_{\mathtt{LLM}}$, we generate $N{=}16$ CoTs per question using SmolLM3-3B (Bakouch et al. 2025), Qwen2.5-7B-Instruct, gemma-2-9b-it (Gemma Team et al. 2024), and Llama-3.1-70B-Instruct. For broader assessment, we also include additional specialized-domain benchmarks: the graduate-level science benchmark (Rein et al. 2024, GPQA-Diamond;), the medical benchmark (Jin et al. 2021, MedQA;), and the legal benchmark (Fan et al. 2026, LEXam;).

For the reward-model backbones, we use the R1-Distill models (Guo et al. 2025): Qwen-1.5B and Qwen-7B for the math domain, and Llama-8B and Qwen-14B for the multi-domain setting, respectively. We also use Qwen3-8B as the backbone to assess whether the results hold for non-distilled backbones in the multi-domain setting. We follow Zhang et al. 2025a for the gORM prompt template (Figure 18) and Khalifa et al. 2025 for gPRM (Figure 20). We optimize reward models using AdamW (Loshchilov & Hutter 2019) with LoRA (Hu et al. 2022). For gORM and gPRM, we sample $M=16$ verification CoTs in the math setting and $M=10$ in the multi-domain setting (cf. Equations 5 and 8), using vLLM (Kwon et al. 2023). See Appendix D, Table 2 and this repository for more details.

Following Zhang et al. 2025a and Khalifa et al. 2025, we construct verification-CoT datasets for gORM and gPRM by prompting QwQ-32B (Qwen Team 2025) with the formats in Figures 18 and 19. We discard any verification CoT whose parsed labels are inconsistent with the targets (e.g., $y$ or $z_{1:T}$), corresponding to the consensus filtering in Section 2.2. The training sets of gORM/gPRM contain 34,286/35,666 and 171,780/94,156 verification CoTs for the math and multi-domain settings. See Appendices D, 21 and 22 for more details and examples.

First, we evaluate the four verifier variants in the math domain. We compare outcome-verification performance with a 0.5 decision threshold, i.e., $\hat{y}\coloneqq\mathbbm{1}(\hat{f}(x)>0.5)$. Figure 3 reports F1 score (%) on ProcessBench splits. dPRM outperforms dORM overall, consistent with prior findings (Lightman et al. 2024), and shows a slight drop in Omni-Math/OlympiadBench with 7B backbones. For gORM/gPRM, the overall performance improves with $M$. At small $M$, gPRM may lag behind discriminative models (e.g., OlympiadBench). gORM generally outperforms gPRM (except 7B-GSM8K), and the gap widens on Omni-Math/OlympiadBench.

Although TTS has been well studied in the math domain, evaluations are not fully controlled: (i) models are rarely compared with a shared backbone, and (ii) gORM and gPRM have not been directly compared. We therefore evaluate the reward models with Bo$N$ under controlled conditions. As shown in Figure 4, and consistent with the findings of Lightman et al. 2024, dPRM outperforms dORM. Notably, dORM even underperforms majority voting (MV) with 1.5B backbones, demonstrating the limitations of coarse outcome-level supervision in the math domain. In line with Zhang et al. 2025a and Khalifa et al. 2025, generative models outperform discriminative ones, with gORM slightly surpassing gPRM.

Next, we compare the four variants in the multi-domain setting. Figure 5 reports F1 scores (%) for outcome-verification, with a 0.5 decision threshold, using R1-Distill-Qwen-14B as the reward model backbone. dO/dP/gO/gP denote dORM/dPRM/gORM/gPRM. In contrast to the math domain results in Figure 3, ORM variants achieve higher F1 scores than PRM variants.

Figure 6(a) shows the overall Bo$N$ performance using five different $p_{\mathtt{LLM}}$ and R1-Distill-Qwen-14B as the reward model backbone. In this setting, dORM performs comparably to dPRM, while gPRM lags behind the other variants, which is contrary to Lightman et al. 2024; Khalifa et al. 2025 and our math-domain results in Figure 4. Overall, gORM outperforms dORM/dPRM/gPRM, without notable degradation in any domain relative to the others (see Appendix F for detailed per-domain results). The same pattern holds for a smaller backbone (DeepSeek-R1-Distill-Llama-8B; Figure 25) and a non-distilled backbone (Qwen3-8B; Figure 37). Table 3 also shows that this trend is not due to insufficient hyperparameter tuning: sweeping the learning rate and LoRA rank yields only marginal changes in PRM performance.

To verify that the above observations generalize across datasets, we take the reward models trained on the MMLU-Pro training split and evaluate them on GPQA-Diamond, MedQA, and LEXam by generating $N{=}16$ CoTs for each question. For MedQA and LEXam, we include only SmolLM3-3B and gemma-2-9b-it, since the other $p_{\mathtt{LLM}}$ exhibit severe degradation, even compared to random guessing. As shown in Figure 6(b), gORM outperforms dORM/dPRM/gPRM on GPQA-Diamond, consistent with the results on MMLU-Pro (Figure 6(a)). In specialized domains (Figure 6(c)), generative variants significantly outperform discriminative variants and gORM outperforms gPRM on CoTs generated by SmolLM3-3B and performs comparably on CoTs generated by gemma-2-9b-it.

To assess the effect of multi-domain training, we train and evaluate all four reward model variants only on each MMLU-Pro domain and compare each variant to its multi-domain counterpart. Figure 7 shows the degradation in Bo$N$ performance with $N{=}16$ under domain-specialized training. We observe large drops for dORM and dPRM under single-domain training relative to their multi-domain counterparts, with a slightly larger decline for dORM, likely because outcome-only supervision is sparser than step-level supervision and both discriminative variants require substantial training data.

In contrast, gORM and gPRM appear more sample-efficient: even without multi-domain training, their performance decreases only modestly (or in some cases improves). This also explains the results on MedQA and LEXam (Figure 6(c)): the generative variants show strong gains over the discriminative variants in these sample-inefficient specialized domains (medical and legal). We defer complete results of single-domain training for the four reward models to Figures 35 and 36 in Appendix F.

As noted in Section 2.2, PRMs typically assume that once a reasoning step is incorrect, all subsequent steps are incorrect. However, recent reasoning models can recover from earlier mistakes and still arrive at the correct answer (Guo et al. 2025, an “aha” moment;). In such cases, PRMs can miss the recovery due to a monotonicity bias induced by their training data. To demonstrate this, we evaluate on “aha” CoTs from ProcessBench³³ 3 The “aha” experiments are conducted only on ProcessBench, which provides human-annotated step-level labels. For MMLU-Pro, LLM-annotated process labels would make the rate of “aha” CoTs depend on annotator reliability. that contain at least one incorrect step ($\exists\,t\in\{1\mathord{\mathchar 59\relax}\ldots\mathord{\mathchar 59\relax}T\}:z_{t}=0$) but a correct outcome ($y=1$). Overall, “aha” CoTs account for 15.3% of the cases. In Figure 8(a), we report F1 scores (%) for the “aha” CoTs using $M{=}16$ for gORM/gPRM. We observe that PRM variants perform particularly poorly on “aha” CoTs. Moreover, scaling the backbone from 1.5B to 7B improves ORM performance, whereas PRM performance degrades with larger backbones, possibly because larger PRMs are more likely to follow the PRM assumption inherent in their training data and objective (cf. Equations 3 and 7).

A natural concern about ORM results on “aha” CoTs in Figure 8(a) is overfitting: ORMs might only memorize questions and their answers, thereby correctly verifying “aha” CoTs without checking the correctness of intermediate reasoning steps. This memorization issue in the math domain has recently been studied by Wu et al. 2025. To investigate this, we conduct the following test: (i) replace the intermediate reasoning steps $r_{1:T-1}$ with $r^{\prime}_{1:T-1}$ taken from other CoTs, and (ii) evaluate ORMs on these perturbed CoTs. If ORMs only memorize the answer in the final reasoning step $r_{T}$, their performance should remain largely unaffected. However, Figure 8(b) shows a significant drop for ORMs (dashed), indicating the reliance on intermediate steps. Interestingly, the degradation is greater with the 7B backbone than with the 1.5B backbone for both dORM and gORM. This suggests that larger models rely more heavily on intermediate reasoning steps during verification.

“Aha” moments can also lengthen CoTs, an effect especially pronounced in the multi-domain setting (Figure 9(a)), where LLMs struggle more than in the math domain. As shown in Figure 9(b), majority voting results degrade significantly with increasing CoT length in the multi-domain setting. Consistent with the outcome-verification failures of PRMs on “aha” CoTs, we argue that the error of PRM variants grows with CoT length ($T$). Intuitively, as a CoT grows longer, the chance that a PRM misclassifies at least one intermediate step rises, making it more likely to prematurely conclude the CoT is incorrect. Longer CoTs also create more opportunities for “aha” recoveries that PRMs systematically miss. We formalize this as follows:

All formal definitions, assumptions, and proofs are deferred to Appendix A. 4.1, 4.2 and 4.3 establish that while dORM/gORM error bounds remain stable regardless of length, dPRM/gPRM lower bounds grow linearly with $T$. In A.4, we additionally show that for gPRM with Monte Carlo estimation (cf. Equation 8), the log-error lower bound also increases linearly with $T$.

To empirically support 4.1, 4.2 and 4.3, we plot the F1 score (%) for outcome-verification in the multi-domain setting as a function of the number of CoT steps ($T$) in Figure 10-(Left). We divide CoTs into eight bins: 1-2, 3, 4, 5, 6, 7-8, 9-13, and 14-162 steps. As $T$ increases, dPRM/gPRM degrade considerably relative to dORM/gORM. Figure 10-(Right) shows the performance improvements over majority voting for different categories with respect to the average number of CoT steps. We observe negative correlations for dPRM (-0.058) and gPRM (-0.608), while dORM (0.316) and gORM (0.399) show positive correlations. These results not only provide empirical support for 4.1, 4.2 and 4.3 but also demonstrate that increasing CoT length can degrade TTS performance for dPRM and gPRM in the multi-domain setting.

Beyond CoT-length effects, label noise poses an additional risk, especially in multi-domain settings. Since human annotation of long CoTs is more costly in specialized domains such as law and medicine than in math, prior work often relies on LLMs to automatically label process steps (Zeng et al. 2025), which introduces noise that can degrade PRM performance. Indeed, recent evaluations in the math domain document that automated annotation pipelines, such as Monte Carlo estimation, can inject substantial label noise and evaluation artifacts (Zhang et al. 2025b). We study this by injecting synthetic noise into the process labels of PRM800K. We vary the level of noise along two axes: (i) process-noise ratio (the per-step probability of flipping a process label) and (ii) data-noise ratio (the fraction of examples to which noise is applied). We report the outcome-verification F1 score (%) in Figure 11 for 1.5B and 7B backbones, using greedy decoding for generative variants. dPRM is highly sensitive to label noise, demonstrating its potential vulnerability in multi-domain settings. In contrast, gPRM is more robust, which is consistent with reports that LLM memorization can make random label noise act as a regularizer in math (Wu et al. 2025).

We further analyze why gPRM degrades in the multi-domain setting, despite its robustness to label noise in math. As CoTs become longer, the imperfect $p_{\texttt{LLM-j}}$ struggles to align stepwise verification rationales with process labels. As a result, consensus filtering prunes long CoTs, shifting the training CoT-length distribution away from the test set (Figure 12).

We quantify the above length distribution shift with the Wasserstein distance (Kantorovich 1960), reporting distances from the test set to the unfiltered pool (Train), the gORM training set, and the gPRM training set. In the math domain (Table 4), gPRM has the smallest distance (e.g., overall: 2.760/2.430/1.600 for Train/gORM/gPRM), whereas in the multi-domain setting (Figures 12 and 5) it has the largest distance (e.g., overall: 0.202/0.532/3.083 for Train/gORM/gPRM).

The distribution shift of gPRM also corresponds to its degradation across domains, observed in the multi-domain setting (Figure 6(a)). Figure 13 shows a strong negative correlation between the Wasserstein distance and per-domain improvement over majority voting with $N{=}16$ for gPRM ($-0.625$), whereas correlations are weak for the other methods ($0.345/0.048/0.082$ for dORM/dPRM/gORM). Together, these results suggest that consensus filtering induces a length-distribution shift that disproportionately affects gPRM in the multi-domain setting, despite its robustness to label noise.