What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API MigrationsJournal: Information and Software Technology

We study whether aggregate benchmark scores reliably reflect the backward compatibility of commercial LLM API upgrades. Three research questions structure the study.

RQ1. On each migration edge in the current GPT product line, what proportion of benchmark items shows reliable improvement, and what proportion shows reliable regression, beyond what sampling noise alone would produce?

RQ2. How does the aggregate score change on each edge relate to the underlying item-level changes, and how much bidirectional change does the aggregate figure conceal?

RQ3. How do migration conclusions change between strict and loose scoring of the same responses?

The unit of comparison is a migration edge: an ordered pair of models that an API consumer moves between when following the vendor’s product line. We measure three edges over three models, GPT-5.4, GPT-5.5, and GPT-5.6 Sol: the two consecutive flagship migrations (5.4$\rightarrow$5.5 and 5.5$\rightarrow$Sol) and the direct migration 5.4$\rightarrow$Sol, which the vendor’s guidance also supports. The three models are served under the API identifiers gpt-5.4, gpt-5.5, and gpt-5.6-sol.

All request parameters are held constant across models; only the model identifier varies. This mirrors a dependency upgrade in which client code is unchanged and only the dependency version moves. Reasoning effort is set explicitly to medium for every model, the shared tier across the three models’ effort ranges. Temperature is not accepted by these models. No output-length cap is imposed. Each item is queried $K=50$ times per model in independent single-turn calls.

Measuring item-level change requires items on which the models under study retain headroom: an item answered correctly, or incorrectly, in every trial by every model carries no information about migration-induced change. We therefore selected benchmarks by a screening procedure defined before the main collection. Eleven public, automatically scorable candidates spanning knowledge, mathematics, and instruction following were screened with 30 randomly drawn items each, three models, and $K=10$ repetitions per item. For each candidate we computed the fraction of items on which at least one model’s observed accuracy fell in $[0.1,\,0.9]$. One benchmark per category was selected, requiring that at least 35% of items meet this criterion, and taking the candidate with the highest proportion per slot. Widely used benchmarks, including GPQA Diamond, MMLU-Pro, and IFEval, fell below the threshold on these models, with proportions between 7% and 27%; the full screening table is reported in the appendix. Screening data served benchmark selection only and did not enter the main analysis.

The selected benchmarks are SuperGPQA [21] (knowledge; 500 items drawn at random from the full pool before collection), Omni-MATH hard [11] (mathematics; 100 items drawn at random from the 452-item pool before collection), and IFBench [19] (instruction following; all 300 prompts, with the prompt as the unit of analysis), for a total of 900 items. The sampled item lists were fixed before collection and are included in the released archive.

Prompting follows the benchmark type. Knowledge and mathematics items use a minimal task statement and a final-answer-line convention, frozen after piloting. IFBench prompts are used verbatim from the official release, since their wording constitutes the task constraints; they are scored by the official verifier [19, 25].

Collection proceeded in per-model phases in release order (GPT-5.4, then GPT-5.5, then GPT-5.6 Sol), with item–repetition order randomised within each phase. Collection ran from 30 July to 13 August 2026. All phases ran at a fixed request concurrency of 36. Transient API errors were retried with exponential backoff until success; the final response matrix is complete.

Scoring is decoupled from collection and deterministic given the archived responses. Every response is assigned to one of six mutually exclusive categories: semantically correct, semantically wrong, format failure, refusal, truncation, and API error; given the retry policy, every response in the final matrix falls in the first five categories. API errors are excluded from the accuracy denominator and their occurrence rate is reported separately; refusals, format failures, and model-side truncations remain in the denominator as behavioural failures.

For knowledge and mathematics items, a response is scored correct when a two-stage parser extracts an answer from the final answer line and that answer matches the reference; mathematical equivalence is established by exact match followed by symbolic normalisation. IFBench responses are scored by the official verifiers [19], which provide a strict and a loose reading; the loose reading re-applies each verification function after a fixed set of surface-format normalisations, so the two readings differ only in format tolerance. The strict reading is primary. Responses that a downstream system could not parse under the agreed format are counted as failures, since for an API consumer an unparseable reply is operationally indistinguishable from a wrong one.

For item $i$ under model $m$, the observed accuracy is $\hat{p}_{m,i}$, the fraction of the $K$ trials scored correct under the primary (strict) rule. For each edge, source $s$ to target $t$, we report per benchmark the aggregate change $\Delta=\operatorname{mean}_{i}(\hat{p}_{t,i})-\operatorname{mean}_{i}(\hat{p}_{s,i})$; the reliable-improvement share $P^{+}$; the reliable-regression share $P^{-}$; and the reliable-change share $P^{\mathit{chg}}=P^{+}+P^{-}$.

Item-level judgements require both statistical significance and a minimum effect size. A Fisher exact test on the $2\times K$ correct/incorrect counts, with Benjamini–Hochberg control of the false discovery rate at 5% within each migration–benchmark cell, establishes significance. A practical-significance threshold $\lvert\hat{p}_{t,i}-\hat{p}_{s,i}\rvert\geq\varepsilon$ with $\varepsilon=0.2$ (a 20-percentage-point shift in pass probability) establishes a minimum effect size. Items meeting both criteria are classified as reliably improved or reliably regressed by sign. Items whose 95% confidence interval for the difference lies entirely within $(-\varepsilon,\varepsilon)$ are classified as practically equivalent; the remainder are classified as inconclusive.

Observed shares are calibrated against a permutation null. For each item we pool the $2K$ outcomes of the two models on an edge, randomly reassign version labels with $K$ outcomes per side, and rerun the complete classification procedure; 1,000 replications yield the null distribution of $P^{+}$, $P^{-}$, and $P^{\mathit{chg}}$ under zero true change at matched sample size. Reported shares are presented alongside the 95th percentile of this null, so that reliable change is claimed only where it exceeds what label noise alone produces.

For the secondary analysis on IFBench, let $R_{m}$ denote the mean difference between the loose and strict verifier scores across prompts. Each edge reports $\Delta R=R_{t}-R_{s}$, the change in this gap across the migration.

Mean strict accuracy was 63.4%, 45.7%, and 62.8% for GPT-5.4 on SuperGPQA, Omni-MATH hard, and IFBench; 65.7%, 53.0%, and 64.6% for GPT-5.5; and 67.1%, 48.9%, and 60.7% for Sol. Table 1 summarises the aggregate change and the full item-level classification for each migration edge and benchmark; Figure 1 shows the reliable-improvement and reliable-regression shares alongside the aggregate change.

On the 5.4$\to$5.5 edge, aggregate scores rose on all three benchmarks (+2.3, +7.3, and +1.9 percentage points on SuperGPQA, Omni-MATH hard, and IFBench). All three benchmarks also show reliable regression: 5.0% of SuperGPQA items, 6.0% of Omni-MATH items, and 8.3% of IFBench items regressed reliably despite a positive aggregate change.

On the 5.5$\to$Sol edge, the three benchmarks diverge. SuperGPQA continued to rise (+1.4) while Omni-MATH hard ($-$4.1) and IFBench ($-$3.9) declined. Both declining benchmarks still contain reliably improved items: 9.0% of Omni-MATH items and 6.7% of IFBench items improved reliably even as the aggregate fell.

On the direct 5.4$\to$Sol edge, SuperGPQA showed the largest gain (+3.8) and the highest improvement share (10.8%), while IFBench showed a net loss ($-$2.0) with 13.3% of items reliably regressed and 10.7% reliably improved. Reliable-change shares on this edge differ from the sums of the two consecutive edges, reflecting item-level movements that cancel across consecutive migrations.

Under zero true change, the 95th percentile of the reliable-change share was 0.0% in every migration–benchmark cell: across 1,000 label permutations at the study’s sample sizes, the Fisher/BH/$\varepsilon$ criteria produced zero spurious reliable changes. Every reliably changed item in Table 1 therefore lies strictly above the noise floor. Prior work on open-weight 7–8B models reported total churn of 21–28% at $K{=}10$ [3]; the shares of 12.0–24.0% observed here are of comparable magnitude yet rest on a calibrated zero baseline.

IFBench’s official verifiers provide a strict and a loose reading of every response. The gap $R_{m}$ between the two readings was 6.3 percentage points for GPT-5.4, 5.1 for GPT-5.5, and 9.0 for Sol, yielding a $\Delta R$ of $-$1.2 points on the 5.4$\to$5.5 edge, +3.9 on 5.5$\to$Sol, and +2.7 on 5.4$\to$Sol. The widening on the 5.5$\to$Sol edge accounts for nearly the entire aggregate decline: Sol’s strict score falls 3.9 points below GPT-5.5’s, while its loose score falls 0.04 points; the regression is concentrated in exact constraint compliance. On SuperGPQA and Omni-MATH hard, responses are scored by answer extraction and exact match; extraction failures were negligible, and the six-category breakdown in the appendix reports their rates.

The appendix reports five supplementary analyses on the same data: a single-draw comparison that contrasts raw correct-to-incorrect and incorrect-to-correct flips from one randomly selected trial per item with the $K{=}50$ classification; a sensitivity analysis of the practical-significance threshold over $\varepsilon\in\{0.10,0.15,0.20,0.25\}$; token-cost distributions per model and edge; the six-category response breakdown; and difficulty stratification using metadata provided by each benchmark.