Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

Zeyu Liu¹, Jinhao Zhang², Yunquan Zhang¹, Guangming Tan¹,
Xiang Gao³, Fangming Liu⁴, Daning Cheng¹\corresponding

Abstract

How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.

Introduction

Increasing depth is a standard way to expand neural-network capacity, yet its marginal benefit does not persist indefinitely: models often improve as layers are added and then enter a regime in which further depth yields little gain in test dataset. This raises a basic question for model scaling and adaptive growth: given a trained network, how can we determine whether additional residual depth still has useful optimization value? Comparing independently trained models of different depths cannot isolate this question, since their performance differences may reflect initialization, optimization difficulty, training budget, regularization, or finite-sample variation. Function-preserving residual insertion (Chen et al. 2015; Wei et al. 2016) provides a cleaner setting: a residual block initialized to produce globally zero output leaves the represented function unchanged, so the marginal value of new depth can be studied at a common initialization. Existing growth methods, however, provide no exact criterion for when residual growth should terminate.

Some works on normalized residual networks introduced a residual non-degeneracy condition and showed that it is sufficient for constructing a locally improving expanded model (Cheng et al. 2026). This leaves open a logically prior question: does the condition exactly characterize whether any admissible first-order residual direction remains? We study this question relative to a fixed residual-growth protocol that specifies, before observing the gradient signal, the admissible insertion candidates, the parameterized residual family with its function-preserving initialization, and a zero-state, descent-compatible first-order update for the inserted parameters. A candidate is operationally saturated when this update cannot produce a strict decrease from the function-preserving initialization for any sufficiently small positive step. Our main theorem shows that this boundary does not depend on the optimizer chosen within the stated first-order class.

The tangent-space projection is the appropriate parameterization-independent quantity, because a raw activation gradient may contain components that the selected residual family cannot express. For standard zero-output blocks—a feature-producing subnetwork followed by a trainable output projection—the parameter criterion reduces to the cross-gradient between the activation gradient and the residual features; holding the feature parameters fixed loses no first-order directions, since their derivative vanishes at a zero output projection. We further introduce a checkpoint-specific residual-signal realizability condition, requiring only that the current conditional activation-gradient signal lie in this tangent space. Under realizability the projection preserves the full signal, and raw activation-gradient vanishing becomes a necessary and sufficient saturation certificate.

Our experiments combine two complementary forms of evidence. First, we measure the sample-wise activation-gradient norm across depth in ResNets trained on CIFAR-10, CIFAR-100, and ImageNet-100, in Pythia checkpoints continued-pretrained on FineWeb-Edu, and in GPT-2-style models trained from scratch on the same corpus. Because orthogonal projection cannot increase norm, this score upper-bounds the empirical projected residual-growth value on every fixed sample. Second, on four controlled CIFAR-10 ResNet configurations we insert one globally zero-output residual block at a time, comparing function-preserving growth with training the same final-depth architectures from random initialization to control for optimization disadvantages of the growth procedure. Additional blocks produce realized gains at shallower depths, and these gains disappear once the sample-wise score enters a stable low-signal regime.

Our contributions are fourfold. First, we introduce an operational notion of first-order residual depth saturation and prove that its boundary is shared by all zero-state, descent-compatible first-order updates. Second, we characterize this boundary through the parameterization-invariant projection of the conditional activation gradient onto the residual tangent space, and derive the cross-gradient criterion for standard zero-output blocks. Third, we establish exact and approximate residual-signal realizability conditions under which raw activation gradients can replace the projected criterion. Finally, we combine broad fixed-sample gradient measurements with controlled one-block growth and from-scratch comparisons, supporting low sample-wise activation-gradient energy as a conservative practical indicator that little empirical first-order value of residual depth remains.

Related Work

Net2Net and Network Morphism introduced function-preserving operators that widen or deepen a trained network without changing the represented function (Chen et al. 2015; Wei et al. 2016). This idea underlies efficient Transformer pre-training through progressive stacking, parameter reuse and knowledge inheritance, learned or lossless expansion operators, and variance-transfer initialization (Gong et al. 2019; Gu et al. 2020; Du et al. 2024; Chen et al. 2022; Qin et al. 2021; Wang et al. 2023, 2024; Yuan et al. 2023). These works address how to grow efficiently and presuppose that added capacity is useful; none gives an exact criterion for when residual growth stops having local optimization value.

Deciding when and where to grow is a classical theme of constructive learning (Fahlman and Lebiere 1990; Ash 1989). Recent policies schedule depth growth via validation heuristics or fitting risk (Wen et al. 2019; Wu et al. 2024), and local scores select beneficial insertions through splitting directions, gradient norms, expressivity bottlenecks, topological derivatives, and natural-expansion scores (Wu et al. 2019, 2021; Evci et al. 2022; Verbockhaven et al. 2024; Krishnanunni et al. 2025; Mitchell et al. 2023). These criteria are sufficient conditions or heuristics for beneficial growth; we instead characterize exactly when no admissible function-preserving residual insertion admits a first-order improvement.

Residual networks behave like ensembles of shallow paths, and stochastic depth and layer pruning of large language models reveal substantial redundancy in trained depth (Veit et al. 2016; Huang et al. 2016; Gromov et al. 2024; Men et al. 2024); zero-initialized residual branches are benign and trainable (Zhang et al. 2019; Bachlechner et al. 2021), supporting our zero-output initialization. Unlike scaling-law comparisons of independently trained models (Kaplan et al. 2020; Levine et al. 2020), our function-preserving setting isolates the marginal first-order value of additional depth at a common checkpoint.

Closest to our work, Cheng et al. (2026) show that residual non-degeneracy (Condition 1) is sufficient for constructing a locally improving expanded model; we prove that it is also necessary under a fixed zero-state protocol, yielding a saturation boundary invariant to the first-order optimizer and to reparameterizations preserving the residual tangent space.

Problem Setup, Assumptions, and Notation

Problem Setup

Let $\mathcal{X}$ be the input space, let $\mathcal{Y}$ be the label space, and let $\mathcal{D}$ be a probability distribution on $\mathcal{X}\times\mathcal{Y}$ . We write $(x,y)\sim\mathcal{D}$ , where $x$ is an input and $y$ is its label. Every predictor considered below maps $\mathcal{X}$ to $\mathbb{R}^{d_{\mathrm{out}}}$ , and the loss is a measurable function $\ell:\mathbb{R}^{d_{\mathrm{out}}}\times\mathcal{Y}\to[0,\infty)$ . For any measurable predictor $f$ , define $R(f):=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\ell(f(x),y)]$ whenever the expectation is finite. For a fixed sample $S=\{(x_{i},y_{i})\}_{i=1}^{M}$ , define $\mathcal{L}_{S}(f):=M^{-1}\sum_{i=1}^{M}\ell(f(x_{i}),y_{i})$ .

We follow the residual-insertion notation of (Cheng et al. 2026). Let $f_{\mathrm{old}}^{*}$ be a trained reference model with $R(f_{\mathrm{old}}^{*})<\infty$ . Before observing any activation-gradient signal or growth outcome, we fix a residual-growth protocol. The protocol specifies a finite set $\mathcal{I}$ of admissible insertion candidates. Each candidate $l\in\mathcal{I}$ consists of an insertion location, a parameterized residual family, a designated function-preserving initialization, and a first-order optimizer direction rule for the newly inserted parameters.

At candidate $l$ , decompose the reference model as $f_{\mathrm{old}}^{*}=f_{\mathrm{top}}^{(l)}\circ f_{\mathrm{bot}}^{(l)}$ and define $z_{l}:=f_{\mathrm{bot}}^{(l)}(x)\in\mathbb{R}^{N_{l}}$ , where $N_{l}$ is the hidden-state dimension at candidate $l$ . Let $\mu_{l}$ denote the distribution of $z_{l}$ induced by $(x,y)\sim\mathcal{D}$ . When a single candidate is fixed, we suppress $l$ and write $f_{\mathrm{top}}$ , $f_{\mathrm{bot}}$ , $z$ , and $\mu$ .

The parameterized residual family at candidate $l$ is $\mathcal{F}_{\mathrm{res}}^{(l)}:=\{h_{l,\theta_{l}}:\theta_{l}\in\Theta_{l}\}$ , where $\Theta_{l}\subseteq\mathbb{R}^{p_{l}}$ and $0\in\operatorname{int}(\Theta_{l})$ . The designated local origin satisfies $h_{l,0}(z)=0$ for every $z\in\mathbb{R}^{N_{l}}$ ; equivalently, $h_{l,0}\equiv 0$ on the full ambient hidden-state space. The notation $\theta_{l}=0$ denotes a local coordinate centered at this initialization and does not require every raw parameter in the residual branch to be numerically zero.

The expanded model is $f_{l,\theta_{l}}(x):=f_{\mathrm{top}}^{(l)}\bigl(z_{l}+h_{l,\theta_{l}}(z_{l})\bigr)$ . Since $h_{l,0}\equiv 0$ , the insertion preserves the represented function pointwise and $f_{l,0}=f_{\mathrm{old}}^{*}$ . Define the population and empirical objectives associated with candidate $l$ by $\Phi_{l}(\theta_{l}):=R(f_{l,\theta_{l}})$ and $\Phi_{S,l}(\theta_{l}):=\mathcal{L}_{S}(f_{l,\theta_{l}})$ .

All components of the residual-growth protocol are fixed in advance. The candidates, residual families, parameterizations, designated origins, and optimizer direction rules may not be changed after the activation-gradient signal has been observed. All internal hyperparameters that determine each direction rule are fixed; the positive scalar step size used in the local analysis remains free.

Assumptions

The assumptions below have distinct roles. Assumptions 1 and 2 support the main saturation theorem. Assumption 3 is invoked only when the exact projected criterion is replaced by the raw activation-gradient criterion. Assumption 4 is used only for the supplementary finite-sample consistency result.

Assumption 1 (First-order regularity).

For every candidate $l\in\mathcal{I}$ and for $\mathcal{D}$ -almost every $(x,y)$ , the map $z\mapsto\ell(f_{\mathrm{top}}^{(l)}(z),y)$ is Fréchet differentiable at $z=z_{l}$ . Its measurable gradient is denoted by $q_{l}(z_{l},y):=\nabla_{z_{l}}\ell(f_{\mathrm{top}}^{(l)}(z_{l}),y)\in\mathbb{R}^{N_{l}}$ . Define the conditional population signal by $m_{l}(z):=\mathbb{E}[q_{l}(z_{l},y)\mid z_{l}=z]$ .

For $\mu_{l}$ -almost every $z$ , the map $\theta_{l}\mapsto h_{l,\theta_{l}}(z)$ is Fréchet differentiable at $\theta_{l}=0$ . Its measurable Jacobian with respect to $\theta_{l}$ , evaluated at the origin, is denoted by $J_{l}(z):=J_{\theta_{l}}h_{l,0}(z)\in\mathbb{R}^{N_{l}\times p_{l}}$ , where $J_{\theta_{l}}$ denotes the Jacobian with respect to the active residual coordinate $\theta_{l}$ .

There exists a neighborhood of the origin on which $\Phi_{l}$ is finite. The map $\Phi_{l}$ is Fréchet differentiable at $0$ , and its derivative is obtained by interchanging sample-wise differentiation and expectation. For a fixed sample $S$ , empirical statements are understood on samples for which all corresponding sample-wise derivatives exist; an i.i.d. sample has this property almost surely under the preceding conditions.

We further assume $\mathbb{E}\|q_{l}(z_{l},y)\|_{2}^{2}<\infty$ and $\mathbb{E}\|J_{l}(z_{l})\|_{\sigma}^{2}<\infty$ , where $\|\cdot\|_{\sigma}$ is the operator norm. Conditional Jensen’s inequality then gives $m_{l}\in L_{2}(\mu_{l};\mathbb{R}^{N_{l}})$ , and Cauchy–Schwarz ensures that the population residual-parameter gradient defined below is finite.

A standard sufficient condition for the interchange in Assumption 1 is a local integrable Lipschitz envelope: for each candidate $l$ , there exist $\rho_{l}>0$ and an integrable random variable $B_{l}(x,y)$ such that $|\ell(f_{l,\theta}(x),y)-\ell(f_{l,0}(x),y)|\leq B_{l}(x,y)\|\theta\|_{2}$ whenever $\|\theta\|_{2}\leq\rho_{l}$ . For any sequence $\theta_{k}\to 0$ , sample-wise Fréchet differentiability makes the normalized remainder converge pointwise to zero. The local Lipschitz bound controls both the difference quotient and the norm of its derivative by $B_{l}$ , so the normalized remainder is dominated by $2B_{l}$ . Dominated convergence then yields an $o(\|\theta_{k}\|_{2})$ population remainder, which gives Fréchet differentiability of $\Phi_{l}$ and the stated differentiation–expectation interchange. Uniform integrability of the local difference quotients is an alternative sufficient condition.

First-order residual gradients and Condition 1.

For a fixed sample $S$ , let $z_{l,i}:=f_{\mathrm{bot}}^{(l)}(x_{i})$ and $q_{l,i}:=q_{l}(z_{l,i},y_{i})$ . Define $g_{\mathrm{pop}}^{(l)}:=\mathbb{E}[J_{l}(z_{l})^{\top}q_{l}(z_{l},y)]$ and $g_{S}^{(l)}:=M^{-1}\sum_{i=1}^{M}J_{l}(z_{l,i})^{\top}q_{l,i}$ . For a direction $u\in\mathbb{R}^{p_{l}}$ , write $D\Phi_{l}(0)[u]$ and $D\Phi_{S,l}(0)[u]$ for the Fréchet directional derivatives at the origin.

Condition 1 (Residual non-degeneracy).

At candidate $l$ , population residual non-degeneracy holds if there exists $v_{\mathrm{pop}}^{(l)}\in\mathbb{R}^{p_{l}}$ such that $D\Phi_{l}(0)[v_{\mathrm{pop}}^{(l)}]<0$ . For a fixed sample $S$ , its empirical counterpart holds if there exists $v_{S}^{(l)}\in\mathbb{R}^{p_{l}}$ such that $D\Phi_{S,l}(0)[v_{S}^{(l)}]<0$ .

Assumption 2 (Fixed zero-state descent-compatible first-order update).

At every candidate $l\in\mathcal{I}$ , the optimizer direction rule and all internal hyperparameters that determine it are fixed as part of the residual-growth protocol. We call the optimizer zero-state when the active residual coordinate is initialized at $\theta_{l}=0$ and every optimizer state variable associated with $\theta_{l}$ is initialized at its neutral zero value. We assume that the optimizer used by the protocol is zero-state.

Given the exact population or full-sample empirical residual-parameter gradient $g\in\mathbb{R}^{p_{l}}$ , the local update has the form $\theta_{l}^{+}=\eta d_{l}(g)$ , where $\eta>0$ is a free scalar step size and $d_{l}:\mathbb{R}^{p_{l}}\to\mathbb{R}^{p_{l}}$ is the fixed direction map. We assume $d_{l}(0)=0$ and $g^{\top}d_{l}(g)<0$ for every $g\neq 0$ . Since $0\in\operatorname{int}(\Theta_{l})$ , every such direction is feasible for all sufficiently small positive $\eta$ .

This class includes gradient descent and positive-definite preconditioned gradient descent. It also includes the first bias-corrected Adam or AdamW update at zero state: coordinatewise, $d_{l}(g)_{j}=-g_{j}/(|g_{j}|+\epsilon_{\mathrm{Adam}})$ , where $\epsilon_{\mathrm{Adam}}>0$ is the numerical-stability constant. Hence $d_{l}(0)=0$ and $g^{\top}d_{l}(g)<0$ for $g\neq 0$ ; decoupled weight decay contributes no first-step drift because the active residual coordinate is zero. The analysis does not cover Hessian-based updates, externally injected perturbations, nonzero initial optimizer states, or stochastic escape from a zero full gradient.

Population and empirical first-order saturation.

A candidate $l$ is population first-order improvable if there exists $\bar{\eta}_{l}>0$ such that, for every $\eta\in(0,\bar{\eta}_{l})$ , $\Phi_{l}(\eta d_{l}(g_{\mathrm{pop}}^{(l)}))<\Phi_{l}(0)$ . It is population first-order saturated otherwise. The empirical notions are defined by replacing $\Phi_{l}$ and $g_{\mathrm{pop}}^{(l)}$ with $\Phi_{S,l}$ and $g_{S}^{(l)}$ ; the corresponding step-size threshold is denoted by $\bar{\eta}_{S,l}$ . The reference model is first-order depth-saturated relative to the fixed protocol if every candidate is population first-order saturated. Unless explicitly qualified, “first-order saturated” refers to the population notion.

Residual tangent-space objects.

For each candidate $l$ , let $\mathcal{H}_{l}:=L_{2}(\mu_{l};\mathbb{R}^{N_{l}})$ with inner product $\langle a,b\rangle_{\mathcal{H}_{l}}:=\mathbb{E}_{z_{l}\sim\mu_{l}}[a(z_{l})^{\top}b(z_{l})]$ and norm $\|a\|_{\mathcal{H}_{l}}:=\langle a,a\rangle_{\mathcal{H}_{l}}^{1/2}$ . Define the residual tangent operator $A_{l}:\mathbb{R}^{p_{l}}\to\mathcal{H}_{l}$ by $(A_{l}u)(z):=J_{l}(z)u$ . Since $\operatorname{Range}(A_{l})$ is finite-dimensional, it is closed; define $\mathcal{T}_{h}^{(l)}:=\operatorname{Range}(A_{l})$ and let $\Pi_{l}$ be the orthogonal projector onto $\mathcal{T}_{h}^{(l)}$ .

For the fixed sample $S$ , define the empirical Hilbert space $\mathcal{H}_{S,l}:=(\mathbb{R}^{N_{l}})^{M}$ with $\langle a,b\rangle_{S,l}:=M^{-1}\sum_{i=1}^{M}a_{i}^{\top}b_{i}$ and $\|a\|_{S,l}:=\langle a,a\rangle_{S,l}^{1/2}$ . Define $q_{S,l}:=(q_{l,1},\ldots,q_{l,M})\in\mathcal{H}_{S,l}$ and its matrix representation $Q_{S}^{(l)}:=[q_{l,1},\ldots,q_{l,M}]\in\mathbb{R}^{N_{l}\times M}$ . The stacked tangent operator $A_{S,l}:\mathbb{R}^{p_{l}}\to\mathcal{H}_{S,l}$ is $A_{S,l}u:=(J_{l}(z_{l,1})u,\ldots,J_{l}(z_{l,M})u)$ , and $\Pi_{S,l}$ denotes the orthogonal projector onto $\operatorname{Range}(A_{S,l})$ .

Assumption 3 (Residual-signal realizability).

At every candidate to which the raw activation-gradient criterion is applied, the current conditional activation-gradient signal is realizable by the fixed residual tangent space: $m_{l}\in\mathcal{T}_{h}^{(l)}$ .

The stronger equality $\mathcal{T}_{h}^{(l)}=\mathcal{H}_{l}$ is called population tangent completeness. Since $\dim\mathcal{T}_{h}^{(l)}\leq p_{l}<\infty$ , exact population tangent completeness is impossible whenever $\mathcal{H}_{l}$ is infinite-dimensional, as is typical for non-atomic hidden-state distributions.

On the fixed sample $S$ , residual-signal realizability means $q_{S,l}\in\operatorname{Range}(A_{S,l})$ . The stronger condition that $A_{S,l}$ be surjective onto $\mathcal{H}_{S,l}$ is called sample-wise tangent completeness.

Assumption 4 (Independent finite-sample probing).

For the supplementary finite-sample consistency result, all probabilities are conditional on the fixed reference model and the complete residual-growth protocol. Let $S_{n}^{\mathrm{probe}}:=\{(\widetilde{x}_{i},\widetilde{y}_{i})\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}\mathcal{D}^{n}$ be independent of the data and randomness used to train $f_{\mathrm{old}}^{*}$ . In that result, all empirical quantities are instantiated with $S=S_{n}^{\mathrm{probe}}$ and $M=n$ .

For every $l\in\mathcal{I}$ , assume $\mathbb{E}\|J_{l}(z_{l})^{\top}q_{l}(z_{l},y)\|_{2}^{2}<\infty$ . As $n$ increases, the reference model and the complete protocol remain fixed; adaptive selection of a new candidate or residual family after observing the probe sample is not covered.

Practical motivation.

We focus on first-order optimization because gradient-based methods are the practical default for modern large-scale neural networks, whereas exact and structured second-order methods introduce substantially larger memory and computational costs (Anil et al. 2020). Residual-signal realizability is motivated by common ResNet and Transformer branches, which typically consist of a feature-producing subnetwork followed by a trainable terminal output map (He et al. 2016; Vaswani et al. 2017). Zero-initializing that terminal map preserves the reference function while retaining nonzero upstream features. An unconstrained output projection alone does not guarantee realizability, however: the upstream features must also span the sample-dependent variation required by the current activation-gradient signal.

Notation

Residual growth values.

Define the local population and empirical residual growth values by $\mathcal{V}_{l}:=\|\Pi_{l}m_{l}\|_{\mathcal{H}_{l}}$ and $\mathcal{V}_{S,l}:=\|\Pi_{S,l}q_{S,l}\|_{S,l}$ . Define their depth-wide counterparts by $\mathcal{V}_{\mathrm{depth}}:=\max_{l\in\mathcal{I}}\mathcal{V}_{l}$ and $\mathcal{V}_{S,\mathrm{depth}}:=\max_{l\in\mathcal{I}}\mathcal{V}_{S,l}$ . Their dependence on the fixed reference model and residual-growth protocol is suppressed throughout.

Standard zero-output residual blocks.

At candidate $l$ , consider $h_{l,U_{l},V_{l}}(z):=V_{l}\psi_{l,U_{l}}(z)$ , where $\psi_{l,U_{l}}(z)\in\mathbb{R}^{r_{l}}$ and $V_{l}\in\mathbb{R}^{N_{l}\times r_{l}}$ . During the local insertion test, $U_{l}=U_{0,l}$ is fixed and $V_{l}$ is the active parameter, initialized at $V_{l}=0$ . We identify $\theta_{l}=\operatorname{vec}(V_{l})$ , $p_{l}=N_{l}r_{l}$ , and $\Theta_{l}=\mathbb{R}^{p_{l}}$ . Writing $\psi_{l}(z):=\psi_{l,U_{0,l}}(z)$ , every matrix perturbation $\Delta V_{l}$ satisfies $J_{l}(z)\operatorname{vec}(\Delta V_{l})=\Delta V_{l}\psi_{l}(z)$ . Allowing $U_{l}$ to vary as an additional active coordinate does not enlarge the first-order tangent space at $V_{l}=0$ , because $D_{U_{l}}h_{l,U_{l},V_{l}}(z)[\Delta U_{l}]=V_{l}D_{U_{l}}\psi_{l,U_{l}}(z)[\Delta U_{l}]=0$ at the designated origin. Thus fixing $U_{l}=U_{0,l}$ is a lossless first-order simplification.

Define the Frobenius inner product and norm by $\langle B,C\rangle_{F}:=\operatorname{tr}(B^{\top}C)$ and $\|B\|_{F}:=\langle B,B\rangle_{F}^{1/2}$ . Define $C_{\mathrm{pop}}^{(l)}:=\mathbb{E}[q_{l}(z_{l},y)\psi_{l}(z_{l})^{\top}]$ and $C_{S}^{(l)}:=M^{-1}\sum_{i=1}^{M}q_{l,i}\psi_{l}(z_{l,i})^{\top}$ . Let $\Psi_{S}^{(l)}:=[\psi_{l}(z_{l,1}),\ldots,\psi_{l}(z_{l,M})]\in\mathbb{R}^{r_{l}\times M}$ , so that $C_{S}^{(l)}=M^{-1}Q_{S}^{(l)}(\Psi_{S}^{(l)})^{\top}$ .

Let $\Pi_{\Psi,l}:=(\Psi_{S}^{(l)})^{\top}[\Psi_{S}^{(l)}(\Psi_{S}^{(l)})^{\top}]^{\dagger}\Psi_{S}^{(l)}$ be the orthogonal projector onto the row space of $\Psi_{S}^{(l)}$ , where ^† is the Moore–Penrose pseudoinverse. Under the tuple–matrix identification of $\mathcal{H}_{S,l}$ , the matrix representation of $\Pi_{S,l}q_{S,l}$ is $Q_{S}^{(l)}\Pi_{\Psi,l}$ . Consequently, finite-sample residual-signal realizability is equivalent to $Q_{S}^{(l)}=Q_{S}^{(l)}\Pi_{\Psi,l}$ . Sample-wise tangent completeness is equivalent to $\operatorname{rank}(\Psi_{S}^{(l)})=M$ , in which case $\Pi_{\Psi,l}=I_{M}$ .

When one candidate is fixed, we suppress $l$ and write $C_{\mathrm{pop}}$ , $C_{S}$ , $Q_{S}$ , $\Psi_{S}$ , and $\Pi_{\Psi}$ .

First-Order Residual Depth Saturation

This section characterizes the exact boundary between residual depth that retains first-order optimization value and residual depth that cannot be activated from a function-preserving initialization.

Main Theorem: An Exact Saturation Boundary

Condition 1 was introduced in prior work as a sufficient condition for a locally improving residual insertion (Cheng et al. 2026). We show that, under the fixed protocol, it is also necessary for first-order improvement.

Theorem 1 (Necessary and sufficient characterization of first-order residual depth saturation).

Suppose Assumptions 1 and 2 hold. At every candidate $l\in\mathcal{I}$ , the following statements are equivalent:

1.

Condition 1 holds at $l$ .
2.

$g_{\mathrm{pop}}^{(l)}\neq 0$ .
3.

$\Pi_{l}m_{l}\neq 0$ .
4.

$\mathcal{V}_{l}>0$ .
5.

There exists $\bar{\eta}_{l}>0$ such that, for every $\eta\in(0,\bar{\eta}_{l})$ , $\Phi_{l}(\eta d_{l}(g_{\mathrm{pop}}^{(l)}))<\Phi_{l}(0)$ .

Consequently, $\mathcal{V}_{\mathrm{depth}}=0$ if and only if every candidate is population first-order saturated, equivalently, if and only if the reference model is first-order depth-saturated relative to the fixed protocol.

For every fixed sample $S$ satisfying the differentiability conditions in Assumption 1, the empirical counterparts are also equivalent: empirical Condition 1, $g_{S}^{(l)}\neq 0$ , $\Pi_{S,l}q_{S,l}\neq 0$ , $\mathcal{V}_{S,l}>0$ , and the existence of $\bar{\eta}_{S,l}>0$ such that $\Phi_{S,l}(\eta d_{l}(g_{S}^{(l)}))<\Phi_{S,l}(0)$ for every $\eta\in(0,\bar{\eta}_{S,l})$ .

Items 1–4 characterize the existence of an arbitrary strict first-order descent direction in the fixed residual parameter space. Item 5 is the optimizer-realization statement: it concerns specifically the update obtained by applying the fixed direction map $d_{l}$ to the exact residual gradient. Its reverse implication uses the zero-state property $d_{l}(0)=0$ and should not be read as a claim that a stationary point cannot be left by stochastic or higher-order mechanisms.

The qualifier first-order is essential. The theorem does not exclude improvements obtained through higher-order curvature, externally injected perturbations, stochastic escape, or a different residual-growth protocol. The projected activation gradient and the residual-parameter gradient share the same zero-versus-nonzero boundary, but their norms generally differ. The growth value depends only on the residual tangent subspace and is invariant under regular local reparameterizations that preserve this subspace, whereas the Euclidean parameter-gradient norm is coordinate-dependent.

A finite joint-insertion extension, under an additional joint Fréchet-regularity condition, is stated in the supplementary material. A separate local-smoothness corollary gives an explicit one-step decrease for ordinary gradient descent without changing the main equivalence.

Residual-Signal Realizability in Common Architectures

Theorem 1 shows that the universal criterion is the activation-gradient component contained in the fixed residual tangent space. Residual-signal realizability permits this projected criterion to be replaced by the raw activation-gradient signal.

Theorem 2 (Activation-gradient characterization under residual-signal realizability).

Suppose Assumptions 1 and 2 hold at a fixed candidate $l$ .

1.

If $m_{l}\in\mathcal{T}_{h}^{(l)}$ , then $\mathcal{V}_{l}=\|m_{l}\|_{\mathcal{H}_{l}}$ . Consequently, Condition 1 holds if and only if $m_{l}\neq 0$ , and the candidate is population first-order saturated if and only if $m_{l}=0$ $\mu_{l}$ -almost everywhere.
2.

If $q_{S,l}\in\operatorname{Range}(A_{S,l})$ , then $\mathcal{V}_{S,l}=\|q_{S,l}\|_{S,l}$ . Consequently, empirical Condition 1 holds if and only if $q_{S,l}\neq 0$ , equivalently $Q_{S}^{(l)}\neq 0$ .
3.

Independently of Assumption 3, population Condition 1 holds if and only if $C_{\mathrm{pop}}^{(l)}\neq 0$ , and its empirical counterpart holds if and only if $C_{S}^{(l)}\neq 0$ .
4.

For this standard block, sample-wise tangent completeness is equivalent to $\operatorname{rank}(\Psi_{S}^{(l)})=M$ . Under this condition, $\Pi_{\Psi,l}=I_{M}$ , and therefore $C_{S}^{(l)}=0$ if and only if $Q_{S}^{(l)}=0$ .

If residual-signal realizability holds at every candidate, simultaneous activation-gradient vanishing is equivalent to depth-wide first-order saturation.

Activation-gradient vanishing is always sufficient for local first-order saturation, even without realizability. Realizability is needed only for the reverse implication. More quantitatively, let $\varepsilon_{\mathrm{real}}\in[0,1)$ . If $m_{l}\neq 0$ and $\operatorname{dist}_{\mathcal{H}_{l}}(m_{l},\mathcal{T}_{h}^{(l)})\leq\varepsilon_{\mathrm{real}}\|m_{l}\|_{\mathcal{H}_{l}}$ , then Pythagoras gives $\mathcal{V}_{l}\geq\sqrt{1-\varepsilon_{\mathrm{real}}^{2}}\|m_{l}\|_{\mathcal{H}_{l}}>0$ . The same statement holds on a fixed sample after replacing $m_{l}$ , $\mathcal{T}_{h}^{(l)}$ , and $\|\cdot\|_{\mathcal{H}_{l}}$ by $q_{S,l}$ , $\operatorname{Range}(A_{S,l})$ , and $\|\cdot\|_{S,l}$ , respectively.

Many ResNet and Transformer branches contain a feature-producing subnetwork followed by a trainable output map. Zero-initializing the terminal map preserves the reference function while keeping the upstream features nonzero. Wide and diverse features can therefore make signal-relative realizability plausible.

For an arbitrary fixed residual block, the projected activation gradient is the parameterization-invariant exact criterion; the residual-parameter gradient has the same zero-versus-nonzero boundary. For a standard zero-output block, the latter reduces to the activation–feature cross-gradient. Under residual-signal realizability, the raw activation gradient may be used directly.

Finite-Sample Certification

The preceding results characterize saturation through population residual-parameter gradients, which are not directly observable. We therefore consider an independent probe sample $S_{n}^{\mathrm{probe}}=\{(\widetilde{x}_{i},\widetilde{y}_{i})\}_{i=1}^{n}$ drawn after the reference model and the complete residual-growth protocol have been fixed.

For each candidate $l\in\mathcal{I}$ , let $\xi_{l}(x,y):=J_{l}(z_{l})^{\top}q_{l}(z_{l},y)$ , $g_{\mathrm{pop}}^{(l)}:=\mathbb{E}[\xi_{l}]$ , and $\widehat{g}_{n}^{(l)}:=\frac{1}{n}\sum_{i=1}^{n}\xi_{l}(\widetilde{x}_{i},\widetilde{y}_{i})$ . Define $\Gamma:=\max_{l\in\mathcal{I}}\lVert g_{\mathrm{pop}}^{(l)}\rVert_{2}$ and $\widehat{\Gamma}_{n}:=\max_{l\in\mathcal{I}}\lVert\widehat{g}_{n}^{(l)}\rVert_{2}$ .

Proposition 1 (Finite-sample error of the saturation score).

Suppose Assumption 4 holds, and let $\sigma_{l}^{2}:=\mathbb{E}\lVert\xi_{l}-g_{\mathrm{pop}}^{(l)}\rVert_{2}^{2}$ . Then, for every $t>0$ ,

\Pr\left(\left|\widehat{\Gamma}_{n}-\Gamma\right|\geq t\right)\leq\frac{1}{nt^{2}}\sum_{l\in\mathcal{I}}\sigma_{l}^{2}.

(1)

Consequently, fix $\delta\geq 0$ and consider the separated hypotheses $\Gamma\leq\delta$ and $\Gamma\geq\delta+2t$ . The decision rule that declares saturation when $\widehat{\Gamma}_{n}\leq\delta+t$ has error probability at most the right-hand side of (1).

Experiments

We conduct two complementary experiments to evaluate the proposed depth-saturation criterion. First, we examine how the maximum per-example activation-gradient norm changes with depth across ResNets trained on CIFAR-10, CIFAR-100, and ImageNet-100, GPT-2-style models trained on FineWeb-Edu, and official Pythia checkpoints continued-pretrained on the same corpus. Second, we compare function-preserving growth with training the same final architectures from random initialization to determine whether zero-output insertion impairs converged solution quality. The training and model settings are provided in the Appendix. All models are well-trained.

Activation-Gradient Norms Across Model Depth

Refer to caption — Figure 1: The Loss and Maximum Sample-Wise Activation-Gradient Norm Across Model Depth.

We first examine how the fixed-sample activation-gradient signal changes as model depth increases. This analysis covers ResNets trained on CIFAR-10, CIFAR-100, and ImageNet-100, official Pythia checkpoints continued-pretrained on FineWeb-Edu, and GPT-2-style models trained from scratch on FineWeb-Edu.

For each admissible candidate $l\in\mathcal{I}$ , i.e., at the input of an existing residual structure in this experiment, define $\mathrm{AG}_{l}:=\|q_{S,l}\|_{S,l}=\left(\frac{1}{M}\sum_{i=1}^{M}\|q_{l,i}\|_{2}^{2}\right)^{1/2},\mathrm{AG}_{\max}:=\max_{l\in\mathcal{I}}\mathrm{AG}_{l}$ .

Under sample-wise residual-signal realizability, $q_{S,l}\in\operatorname{Range}(A_{S,l})$ , the empirical activation-gradient signal lies entirely in the admissible residual tangent space. Hence, $\Pi_{S,l}q_{S,l}=q_{S,l},\qquad V_{S,l}=\|\Pi_{S,l}q_{S,l}\|_{S,l}=\mathrm{AG}_{l}$ . If realizability holds at every candidate, then $V_{S,\mathrm{depth}}=\max_{l\in\mathcal{I}}V_{S,l}=\mathrm{AG}_{\max}$ . Thus, under sample-wise realizability, $\mathrm{AG}_{\max}$ exactly equals the empirical first-order residual-growth value. Without verified realizability, we only have $V_{S,\mathrm{depth}}\leq\mathrm{AG}_{\max}$ , so $\mathrm{AG}_{\max}$ is the upper bound of $V_{S,\mathrm{depth}}$ . However, it is enough to use this upper bound to gain the value of $V_{S,\mathrm{depth}}$ because for most of cases, $AG_{\max}$ is close to zero.

Analysis

Figure 1 shows a broadly consistent transition from a high-signal, depth-beneficial regime to a stable low-signal regime. For clearly show resluts, CIFAR 100 results are divideds into two subfigures. For the controlled ResNet and GPT-2-style sweeps, $AG_{\max}$ decreases rapidly together with task loss at shallow and intermediate depths, and then changes only modestly once the loss improvements begin to diminish. Pythia exhibits a similar but sharper transition. ImageNet-100 is less monotone at intermediate depths, but the overall envelope of $AG_{\max}$ still decreases and reaches its minimum at the largest evaluated depth.

The relevant observation is therefore not strict monotonicity at every checkpoint, but the emergence of a persistent low-signal plateau. Because $AG_{\max}$ is the maximum over all admissible insertion locations, a small $AG_{\max}$ conservatively implies that the empirical projected residual-growth value is small at every candidate. The alignment between this regime and diminishing task-loss improvements is consistent with the proposed first-order saturation criterion. However, a large raw score does not guarantee a realizable residual direction, and the training-sample diagnostic need not exactly track test loss performance. These results therefore motivate using a stable low-signal plateau, rather than a universal threshold or an isolated checkpoint, as the practical indicator of residual depth saturation.

As shown in Figure 1, the proposed depth-saturation indicator exhibits a consistent relationship with the marginal benefit of increasing model depth across CIFAR-10, CIFAR-100, and Pythia. At relatively shallow depths, the indicator decreases rapidly as additional residual structures improve model performance. However, once the indicator approaches the low-signal regime—approximately $5*10^{-3}$ , further increases in depth yield little or no additional performance improvement. This transition occurs at approximately 16 residual structures for the CIFAR-10 ResNet ( $AG_{max}=0.00381$ ), 11 residual structures for the CIFAR-100 ResNet ( $AG_{max}=0.00320$ ), and 33 residual structures for Pythia ( $AG_{max}=0.0008$ ). The close alignment between the emergence of a near-zero indicator and the disappearance of measurable performance gains supports the use of the proposed metric as a practical diagnostic of residual depth sufficiency. In particular, a persistently small value suggests that little first-order optimization value remains available from adding further residual structures.

Extended-depth evaluation.

We conducted independent large-depth experiments (Training hundreds of these models is prohibitively expensive.). For ImageNet-100, ResNet-256/257/258 yielded $AG_{max}$ scores of $0.0020$ / $0.0018$ / $0.0011$ , with training losses decreasing to $0.017$ / $0.015$ / $0.011$ and test losses fluctuating at $0.66$ / $0.72$ / $0.59$ . Similarly, GPT-2 configurations (700/701/702 residual structures) showed $AG_{max}$ scores of $0.0033$ / $0.0029$ / $0.0022$ , training losses decreasing to $2.67$ / $2.62$ / $2.55$ , and test losses fluctuating at $2.22$ / $2.83$ / $2.07$ . In both settings, $AG_{max}$ scores remained below approximately $5\times 10^{-3}$ . The change of training and test loss is so small. Given the probabilistic nature of test loss performance and its sensitivity to finite-sample and optimization variability, depth saturation need not manifest as monotonically worsening test loss. Crucially, test performance merely fluctuated within a range without systematic gains from increased depth. This confirms that once $AG_{max}$ scores reach a persistent near-zero level, additional residual structures yield minimal marginal improvements in test loss performance, even if isolated deeper checkpoints occasionally achieve better point estimates.

Function-Preserving Growth versus From-Scratch Training

For the same four CIFAR-10 ResNet configurations, we compare the function-preserving grown model with a model of the same final architecture trained from random initialization. Both models use the same optimization recipe and are trained to their respective convergence criteria. This comparison tests whether the disappearance of post-growth improvement could be explained by an optimization disadvantage caused by the function-preserving initialization. It is a convergence-quality comparison rather than a compute-matched estimate of training efficiency. The model in this section is well-trained and fully converged with enough computational resources. The Local Effect of Activation-Gradient-Matched Insertions experiments is shown in Appendix.

Comparison protocol.

We evaluate four ResNet configurations on CIFAR-10. For each configuration, the Growth model is obtained by inserting one globally zero-output residual block into a converged shallower network. The insertion preserves the reference function exactly before optimization. After insertion, the parameters of the deeper model are trained using the classification objective until the prescribed convergence criterion is reached. The corresponding Scratch model has exactly the same final architecture and parameter count, but all of its parameters are initialized randomly and trained from scratch.

The two procedures use the same dataset, data preprocessing, model architecture, optimizer family, regularization, and convergence criterion. This experiment compares the quality of the solutions reachable from the two initializations. It is not a compute-matched comparison of training efficiency, because the Growth model inherits the optimization already invested in its shallower reference model.

Table 1: Converged losses for training from scratch and function-preserving block growth on CIFAR-10. Each row compares the same final ResNet architecture under the two initialization procedures.

	Train Loss		Test Loss
Model	Scratch	Growth	Scratch	Growth
ResNet-10	0.0088	0.0070	0.3045	0.2589
ResNet-11	0.0082	0.0068	0.2874	0.2626
ResNet-12	0.0059	0.0069	0.2822	0.2580
ResNet-13	0.0059	0.0057	0.2948	0.2510

Analysis

As shown in Table 1, function-preserving growth reaches training losses that are closely comparable to those obtained by training the same final architectures from scratch. In three of the four configurations—ResNet-10, ResNet-11, and ResNet-13—Growth achieves a lower training loss, whereas ResNet-12 shows only a small increase relative to Scratch. Averaged across all four architectures, the training loss is $0.0066$ for Growth and $0.0072$ for Scratch. These differences are small in absolute magnitude and do not indicate a systematic optimization disadvantage caused by the zero-output initialization. In particular, preserving the original network function at insertion does not appear to trap the expanded model near the inherited solution or prevent the enlarged architecture from reaching a competitive minimum after subsequent task-loss optimization. Within the evaluated CIFAR-10 ResNet configurations, function-preserving insertion therefore provides a viable initialization for continued training.

The test loss results are also consistently favorable to Growth. Growth obtains a lower test loss for all four final architectures, with absolute reductions ranging from $0.0242$ for ResNet-12 to $0.0456$ for ResNet-10. The average test loss decreases from $0.2922$ to $0.2576$ , corresponding to a relative reduction of approximately $11.8\%$ . This improvement is not accompanied by a systematic increase in training loss, suggesting that it cannot be explained simply by weaker fitting of the training data. The train–test gap is generally smaller under Growth, especially for ResNet-12, where Growth has a slightly higher training loss but a lower test loss. This pattern is consistent with the inherited shallower representation changing the optimization trajectory of the expanded model and potentially inducing different implicit regularization.

Conclusion

We developed a first-order framework for evaluating whether additional residual depth remains locally useful under a fixed function-preserving growth protocol. We showed that a residual insertion admits a strict first-order improvement if and only if the conditional activation gradient has a nonzero projection onto the corresponding residual tangent space. For standard zero-output residual blocks, this condition reduces to an activation–feature cross-gradient criterion. In the evaluated CIFAR-10 configurations, function-preserving growth reaches converged training losses comparable to those obtained from scratch. Together, these results establish tangent-space projection as an exact local criterion for the remaining first-order optimization value of residual depth.

References

R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer (2020) Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018. Cited by: Practical motivation..
T. Ash (1989) Dynamic node creation in backpropagation networks. Connection Science 1 (4), pp. 365–375. Cited by: Related Work.
T. Bachlechner, B. P. Majumder, H. Mao, G. Cottrell, and J. McAuley (2021) Rezero is all you need: fast convergence at large depth. In Uncertainty in artificial intelligence, pp. 1352–1361. Cited by: Related Work.
C. Chen, Y. Yin, L. Shang, X. Jiang, Y. Qin, F. Wang, Z. Wang, X. Chen, Z. Liu, and Q. Liu (2022) Bert2bert: towards reusable pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2134–2148. Cited by: Related Work.
T. Chen, I. Goodfellow, and J. Shlens (2015) Net2net: accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641. Cited by: Introduction, Related Work.
D. Cheng, Z. Liu, J. Sun, F. Xia, B. Zhang, D. Liu, and Y. Zhang (2026) A qualitative test-risk mechanism for scaling behavior in normalized residual networks. arXiv preprint arXiv:2605.08297. Cited by: Introduction, Related Work, Problem Setup, Main Theorem: An Exact Saturation Boundary.
W. Du, T. Luo, Z. Qiu, Z. Huang, Y. Shen, R. Cheng, Y. Guo, and J. Fu (2024) Stacking your transformers: a closer look at model growth for efficient llm pre-training. Advances in Neural Information Processing Systems 37, pp. 10491–10540. Cited by: Related Work.
U. Evci, B. van Merrienboer, T. Unterthiner, M. Vladymyrov, and F. Pedregosa (2022) Gradmax: growing neural networks using gradient information. arXiv preprint arXiv:2201.05125. Cited by: Related Work.
S. E. Fahlman and C. Lebiere (1990) The cascade-correlation learning architecture. In Advances in Neural Information Processing Systems 2, D. S. Touretzky (Ed.), pp. 524–532. External Links: Link Cited by: Related Work.
L. Gong, D. He, Z. Li, T. Qin, L. Wang, and T. Liu (2019) Efficient training of bert by progressively stacking. In International conference on machine learning, pp. 2337–2346. Cited by: Related Work.
A. Gromov, K. Tirumala, H. Shapourian, P. Glorioso, and D. A. Roberts (2024) The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887. Cited by: Related Work.
X. Gu, L. Liu, H. Yu, J. Li, C. Chen, and J. Han (2020) On the transformer growth for progressive bert training. arXiv preprint arXiv:2010.12562. Cited by: Related Work.
K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: Practical motivation..
G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger (2016) Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382. Cited by: Related Work.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: Related Work.
C. Krishnanunni, T. Bui-Thanh, and C. Dawson (2025) Topological derivative approach for deep neural network architecture adaptation. arXiv preprint arXiv:2502.06885. Cited by: Related Work.
Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua (2020) Limits to depth efficiencies of self-attention. Advances in Neural Information Processing Systems 33, pp. 22640–22651. Cited by: Related Work.
X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen (2024) Shortgpt: layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853. Cited by: Related Work.
R. Mitchell, R. Menzenbach, K. Kersting, and M. Mundt (2023) Self-expanding neural networks. arXiv preprint arXiv:2307.04526. Cited by: Related Work.
Y. Qin, Y. Lin, J. Yi, J. Zhang, X. Han, Z. Zhang, Y. Su, Z. Liu, P. Li, M. Sun, et al. (2021) Knowledge inheritance for pre-trained language models. arXiv preprint arXiv:2105.13880. Cited by: Related Work.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: Practical motivation..
A. Veit, M. J. Wilber, and S. Belongie (2016) Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems 29. Cited by: Related Work.
M. Verbockhaven, S. Chevallier, G. Charpiat, and T. Rudkiewicz (2024) Growing tiny networks: spotting expressivity bottlenecks and fixing them optimally. arXiv preprint arXiv:2405.19816. Cited by: Related Work.
P. Wang, R. Panda, L. T. Hennigen, P. Greengard, L. Karlinsky, R. Feris, D. D. Cox, Z. Wang, and Y. Kim (2023) Learning to grow pretrained models for efficient transformer training. arXiv preprint arXiv:2303.00980. Cited by: Related Work.
Y. Wang, J. Su, H. Lu, C. Xie, T. Liu, J. Yuan, H. Lin, R. Sun, and H. Yang (2024) Lemon: lossless model expansion. In International Conference on Learning Representations, Vol. 2024, pp. 3338–3367. Cited by: Related Work.
T. Wei, C. Wang, Y. Rui, and C. W. Chen (2016) Network morphism. In International conference on machine learning, pp. 564–572. Cited by: Introduction, Related Work.
W. Wen, F. Yan, Y. Chen, and H. Li (2019) Autogrow: automatic layer growing in deep convolutional networks. arXiv preprint arXiv:1906.02909. Cited by: Related Work.
H. Wu, W. Wang, T. Malepathirana, D. Senanayake, D. Oetomo, and S. Halgamuge (2024) When to grow? a fitting risk-aware policy for layer growing in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, pp. 5994–6002. Cited by: Related Work.
L. Wu, B. Liu, P. Stone, and Q. Liu (2021) Firefly neural architecture descent: a general approach for growing neural networks. arXiv preprint arXiv:2102.08574. Cited by: Related Work.
L. Wu, D. Wang, and Q. Liu (2019) Splitting steepest descent for growing neural architectures. Advances in neural information processing systems 32. Cited by: Related Work.
X. Yuan, P. Savarese, and M. Maire (2023) Accelerated training via incrementally growing neural networks using variance transfer and learning rate adaptation. Advances in Neural Information Processing Systems 36, pp. 16673–16692. Cited by: Related Work.
H. Zhang, Y. N. Dauphin, and T. Ma (2019) Fixup initialization: residual learning without normalization. arXiv preprint arXiv:1901.09321. Cited by: Related Work.

Appendix A Complete Notation

Tables 2–4 collect the symbols used in the main text and supplementary proofs. Dependence on the fixed reference model and residual-growth protocol is suppressed when no ambiguity arises.

Symbol	Meaning
$\mathcal{X},\mathcal{Y},\mathcal{D}$	Input space, label space, and data distribution on $\mathcal{X}\times\mathcal{Y}$ .
$d_{\mathrm{out}}$	Predictor output dimension.
$S,M$	A fixed sample $S=\{(x_{i},y_{i})\}_{i=1}^{M}$ and its cardinality.
$\ell,R,\mathcal{L}_{S}$	Loss, population risk, and fixed-sample empirical risk.
$f_{\mathrm{old}}^{*}$	Trained reference model.
$\mathcal{I},l$	Fixed finite candidate set and one candidate.
$f_{\mathrm{bot}}^{(l)},f_{\mathrm{top}}^{(l)}$	Reference-model components below and above candidate $l$ .
$z_{l},N_{l},\mu_{l}$	Hidden state, its dimension, and its induced distribution.
$\mathcal{F}_{\mathrm{res}}^{(l)}$	Parameterized residual family at candidate $l$ .
$\Theta_{l},p_{l},\theta_{l}$	Active residual parameter domain, dimension, and local coordinate.
$h_{l,\theta_{l}},f_{l,\theta_{l}}$	Residual function and corresponding expanded model.
$\Phi_{l},\Phi_{S,l}$	Population and empirical objectives as functions of $\theta_{l}$ .
$v_{\mathrm{pop}}^{(l)},v_{S}^{(l)}$	Population and empirical directions in the residual non-degeneracy condition stated in the main paper.
$d_{l},\eta$	Fixed first-order direction map and free positive step-size scalar.
$\bar{\eta}_{l},\bar{\eta}_{S,l}$	Population and empirical local descent thresholds.
$\epsilon_{\mathrm{Adam}}$	Positive numerical-stability constant in the first Adam/AdamW direction.
$B_{l},\rho_{l}$	Integrable local Lipschitz envelope and neighborhood radius in the dominated-differentiation remark in the main paper.
$\\|\cdot\\|_{\sigma}$	Matrix operator norm.

Table 2: Model, data, regularity, and optimization notation.

Symbol	Meaning
$q_{l},m_{l}$	Sample activation gradient and conditional population activation-gradient signal.
$J_{l}$	Jacobian of $h_{l,\theta_{l}}$ with respect to $\theta_{l}$ , evaluated at $\theta_{l}=0$ .
$g_{\mathrm{pop}}^{(l)},g_{S}^{(l)}$	Population and empirical residual-parameter gradients.
$D\Phi_{l}(0)[u],D\Phi_{S,l}(0)[u]$	Population and empirical Fréchet derivatives applied to direction $u$ .
$\mathcal{H}_{l},A_{l},\mathcal{T}_{h}^{(l)},\Pi_{l}$	Population hidden-state Hilbert space, tangent operator, tangent range, and orthogonal projector.
$\mathcal{H}_{S,l},q_{S,l},Q_{S}^{(l)}$	Empirical Hilbert space and tuple/matrix forms of the sample activation-gradient signal.
$A_{S,l},\Pi_{S,l}$	Stacked tangent operator and projector onto its range.
$\mathcal{V}_{l},\mathcal{V}_{S,l}$	Population and empirical local residual growth values.
$\mathcal{V}_{\mathrm{depth}},\mathcal{V}_{S,\mathrm{depth}}$	Maximal local growth values over the fixed candidate set.
$A_{l}^{*},G_{l},\widehat{G}_{S,l}$	Tangent adjoint, population tangent Gram operator, and empirical tangent Gram matrix.
$\operatorname{dist}_{\mathcal{H}_{l}}(a,\mathcal{T})$	Distance from $a$ to $\mathcal{T}$ in the $\mathcal{H}_{l}$ norm; $\operatorname{dist}_{S,l}$ denotes the empirical analogue.
$\dim(\cdot)$	Dimension of a finite-dimensional vector space or vectorized tensor.
$\varepsilon_{\mathrm{real}}$	Relative residual-realizability error in $[0,1)$ .
$L_{l},L_{S,l}$	Optional population and empirical local smoothness constants in Corollary 1.

Table 3: Activation-gradient, tangent-space, and growth-value notation.

Symbol	Meaning
$U_{l},U_{0,l},V_{l},\psi_{l,U_{l}},\psi_{l},r_{l}$	Feature parameters, their designated value, active output projection, feature map, fixed feature map, and feature dimension in the standard block.
$C_{\mathrm{pop}}^{(l)},C_{S}^{(l)}$	Population and empirical activation–feature cross-gradients.
$\Psi_{S}^{(l)},\Pi_{\Psi,l}$	Residual-feature matrix and projector onto its row space.
$\langle\cdot,\cdot\rangle_{F},\\|\cdot\\|_{F}$	Frobenius inner product and norm.
${}^{\dagger},\operatorname{vec},I_{M}$	Moore–Penrose pseudoinverse, vectorization, and $M\times M$ identity.
$K,(l_{1},\ldots,l_{K}),F_{j}$	Number of jointly inserted blocks, their distinct locations listed from upstream to downstream, and original network segments between them.
$T_{l_{j},\theta_{l_{j}}},\iota_{j}$	Full residual transformation $w\mapsto w+h_{l_{j},\theta_{l_{j}}}(w)$ and canonical injection into joint block $j$ .
$\mathcal{K},\theta_{\mathcal{K}},u_{\mathcal{K}}$	Ordered joint candidate tuple, product parameter, and joint direction.
$f_{\mathcal{K},\theta_{\mathcal{K}}},\Phi_{\mathcal{K}}$	Jointly expanded model and its population objective.
$S_{n}^{\mathrm{probe}},\xi_{l},\widehat{g}_{n}^{(l)}$	Independent probe sample, one-sample residual-gradient contribution, and probe gradient estimate.
$\Gamma,\widehat{\Gamma}_{n},\sigma_{l}^{2}$	Population saturation score, empirical score, and residual-gradient variance.
$\tau_{n},\widehat{\mathrm{Sat}}_{n}$	Detection threshold and empirical saturation decision.
$\delta,t,H_{0},H_{1}$	Practical near-saturation tolerance, separation radius, and hypotheses $H_{0}:\Gamma\leq\delta$ and $H_{1}:\Gamma\geq\delta+2t$ .
$\mathbf{1}\{\cdot\}$	Indicator of an event.
$\mathsf{AG}_{l},\mathsf{AG}_{\max}$	Experimental scores $\mathsf{AG}_{l}:=\\|q_{S,l}\\|_{S,l}=M^{-1/2}\\|Q_{S}^{(l)}\\|_{F}$ and $\mathsf{AG}_{\max}:=\max_{l\in\mathcal{I}}\mathsf{AG}_{l}$ .
$\widetilde{\mathsf{AG}}_{l}$	Dimension-normalized visualization score $\mathsf{AG}_{l}/\sqrt{N_{l}}$ ; it is not used to rank candidates with unequal hidden dimensions.

Table 4: Standard-block, joint-insertion, supplementary, and experimental notation.

Remark 1 (Normalization of the experimental activation-gradient score).

The score $\mathsf{AG}_{l}$ uses the same sample normalization as $\|q_{S,l}\|_{S,l}$ . Under finite-sample residual-signal realizability, $\mathsf{AG}_{l}=\mathcal{V}_{S,l}$ . A coordinate-normalized variant $\widetilde{\mathsf{AG}}_{l}=\mathsf{AG}_{l}/\sqrt{N_{l}}$ has the same zero-versus-nonzero boundary but can change candidate rankings when hidden dimensions differ. It should therefore be treated as a descriptive visualization scale rather than as the exact empirical growth value.

Appendix B Detailed Experimental Settings

This section gives the architecture, data, optimization, checkpoint, and measurement details for the depth-scaling experiments. Unless stated otherwise, depth is the controlled architectural variable: width, classification or language-modeling heads, and the remaining model hyperparameters are held fixed within each sweep.

B.1 Depth Variables and Gradient Measurement

ResNet depth.

The ResNet experiments start from a stagewise ResNet-8 backbone containing one BasicBlock in each of three stages. We denote by $N$ the number of residual blocks added to this fixed backbone, so a model contains $3+N$ residual blocks in total. Added blocks are assigned cyclically to layer1, layer2, and layer3. This rule changes depth without changing the stage widths or classification head and avoids placing all added capacity at the end of the network.

GPT-2 depth.

For GPT-2-style models, $L$ denotes the total number of decoder blocks. The baseline has $L=12$ , and $L=13$ therefore denotes the baseline architecture with one additional decoder block. Hidden size, feed-forward size, number of attention heads, positional encoding, normalization, vocabulary, and the per-update optimization recipe are fixed as $L$ varies; the documented run durations are specified below.

Recorded activation-gradient statistic.

For a residual block, decoder block, classifier, or language-modeling head, let $H_{l,b}$ be the complete input-activation tensor at location $l$ for analysis batch $b$ , and let $\mathcal{L}_{b}$ be that batch’s loss. Each analysis batch requires one forward and one backward pass. The run-time diagnostic stored by the experiment scripts is

\widehat{\mathsf{AG}}_{l}:=\frac{1}{B}\sum_{b=1}^{B}\left\|\nabla_{H_{l,b}}\mathcal{L}_{b}\right\|_{2},

(2)

where the norm is taken once over the full, flattened activation-gradient tensor and the resulting scalar is averaged over $B$ analysis batches. Inputs to the final fully connected classifier and LM head are recorded as diagnostic endpoints; the residual-growth candidate set itself contains the residual- or decoder-block boundaries.

Equation (2) documents the batch-level quantity retained in the experimental logs. The sample-normalized Hilbert-space score $\mathsf{AG}_{l}$ used in the theoretical comparisons is defined separately in Remark 1. For a fixed batch construction, both statistics vanish exactly when all recorded activation gradients vanish, but their nonzero numerical scales need not agree. Quantitative comparisons therefore use a fixed analysis-batch protocol within each model family.

B.2 CIFAR-10 and CIFAR-100

Data and preprocessing.

CIFAR-10 and CIFAR-100 each contain $50{,}000$ training images and $10{,}000$ test images at resolution $32\times 32$ . Training images are randomly cropped after four-pixel padding and randomly flipped horizontally. Test images receive no random augmentation. Images are then normalized using the mean and standard deviation of the corresponding dataset. CIFAR-10 and CIFAR-100 use 10 and 100 output classes, respectively.

Architecture.

Both datasets use the same stagewise ResNet-8 backbone. The base channel count is 16, and the three stage widths are 16, 32, and 64. Transitions between stages downsample with stride 2. We sweep $N=1,\ldots,100$ , using the cyclic block-allocation rule described above.

Configuration item	CIFAR-10 and CIFAR-100 setting
Objective and optimizer	Cross-entropy loss; SGD with momentum.
Initial learning rate	$0.1$ , multiplied by $0.1$ after epochs 100 and 150.
Momentum and weight decay	Momentum $0.9$ ; weight decay $5\times 10^{-4}$ .
Batch size and random seed	Batch size 128; seed 1.
Maximum duration	300 epochs.
Early stopping	Enabled only after at least 180 epochs. Training stops when the training loss fails to improve by more than $10^{-4}$ for 30 consecutive epochs.
Saved checkpoints	The final checkpoint (last) and the checkpoint with the lowest training loss (best_train). Validation and test results are not used for checkpoint selection.

Table 5: Training protocol for the CIFAR depth sweeps.

B.3 ImageNet-100

Subset construction and preprocessing.

ImageNet-100 is constructed from ImageNet-1K and contains $128{,}982$ training images and $5{,}000$ validation images, with 50 validation images per class. Training uses RandomResizedCrop(224) and random horizontal flipping. At evaluation time, the shorter image side is resized to 256 pixels and a $224\times 224$ center crop is taken. Images are normalized by the standard ImageNet mean and standard deviation. The data are loaded with torchvision.datasets.ImageFolder, and the number of classes is inferred from the directory structure.

Architecture and run configuration.

To isolate the effects of dataset scale and depth, the ImageNet-100 experiments retain the three-stage CIFAR ResNet-8 backbone, including its $3\times 3$ , stride-1 stem. In particular, they do not introduce the $7\times 7$ , stride-2 convolution or max pooling used by standard ImageNet ResNets. The initial sweep uses $N\in\{1,5,9,\ldots,49\}$ , with base channel count 32. Each configuration is run for 300 epochs with global batch size 128, automatic mixed precision, and data parallelism across four GPUs.

B.4 GPT-2 Depth Sweep on Ascend

Model architecture.

The GPT-2 experiments use the MCore GPTModel implementation in Megatron-LM/MindSpeed. The $L=12$ baseline follows the main dimensions of GPT-2 Small: hidden size 768, feed-forward size 3072, 12 attention heads, and maximum sequence length 1024. It uses learned absolute positional embeddings, LayerNorm, GELU activations, attention and hidden dropout of 0.1, and initialization standard deviation 0.02. The vocabulary contains $50{,}257$ tokens, input and output embeddings are tied, and linear-layer biases are retained.

Only the number of decoder blocks changes across the controlled depth sweep; all other architectural dimensions are held fixed. The parameter count therefore increases with depth. The extended-depth configurations reported in the main paper follow the same architectural conventions.

FineWeb-Edu data.

The training corpus is a FineWeb-Edu subset containing approximately $310{,}000{,}347$ GPT-2 tokens. The data are divided into 98% training, 1% validation, and 1% test splits. The training split contains approximately $296{,}715$ packed sequences of length 1024. With global batch size 32, we define one round as 9,273 optimizer steps, approximately one complete pass over the training split.

Configuration item	GPT-2 depth-sweep setting
Numerical precision and device allocation	BF16; one Ascend NPU per model for the initial controlled-depth runs.
Batching	Micro-batch size 4, global batch size 32, and eight gradient accumulation steps.
Optimizer	Adam with $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , weight decay 0.1, and gradient clipping at 1.0.
Learning-rate schedule	Initial rate $2.5\times 10^{-4}$ , minimum rate $2.5\times 10^{-5}$ , 1% warmup, and cosine decay.
Random seed	1234.
Training duration	The initial controlled-depth runs are capped at five rounds. Selected checkpoints are subsequently continued to a total of ten rounds.
Early stopping	The range of the mean training losses over the most recent ten rounds must be below $5\times 10^{-5}$ . With a ten-round cap, this criterion can first be evaluated only after the final round.
Saved checkpoints	The lowest-training-loss checkpoint (best_train) and the final checkpoint (last) at every completed depth.

Table 6: Optimization protocol for the documented GPT-2 runs.

The completed schedules documented in Table 6 cover the controlled-depth experiments. The extended-depth configurations reported in the main paper use the same model definition and per-update optimization recipe.

B.5 Pythia Continued-Pretraining Analysis

The Pythia study considers six deduplicated pretrained model configurations spanning a range of model sizes. Each model is initialized from the official step143000 checkpoint and then continued pretraining on FineWeb-Edu before activation-gradient measurement.

During measurement, model parameters are frozen while gradients are retained for hidden activations. Activation-gradient norms are recorded at decoder-block inputs using the fixed analysis protocol described in Equation (2). Because the Pythia configurations vary in architectural dimensions in addition to depth, these results provide cross-model evidence rather than a controlled depth-only comparison.

Appendix C Preliminary Identities

Lemma 1 (Population directional derivative).

Under the first-order regularity assumption stated in the main paper, for every $u\in\mathbb{R}^{p_{l}}$ ,

	$\displaystyle D\Phi_{l}(0)[u]$	$\displaystyle=\mathbb{E}[q_{l}(z_{l},y)^{\top}J_{l}(z_{l})u]$
		$\displaystyle=(g_{\mathrm{pop}}^{(l)})^{\top}u=\langle m_{l},A_{l}u\rangle_{\mathcal{H}_{l}}.$		(3)

Proof.

For $\mathcal{D}$ -almost every $(x,y)$ , the Fréchet chain rule gives

D_{\theta_{l}}\ell(f_{l,0}(x),y)[u]=q_{l}(z_{l},y)^{\top}J_{l}(z_{l})u.

The first-order regularity assumption stated in the main paper permits differentiation under the expectation, giving the first equality in (3). The second follows from the definition of $g_{\mathrm{pop}}^{(l)}$ . Since $J_{l}(z_{l})u$ is measurable with respect to $z_{l}$ , conditional expectation gives

\mathbb{E}[q_{l}^{\top}J_{l}u]=\mathbb{E}[m_{l}(z_{l})^{\top}J_{l}(z_{l})u]=\langle m_{l},A_{l}u\rangle_{\mathcal{H}_{l}}.

∎

Lemma 2 (Bounded tangent operator and adjoint identity).

The operator $A_{l}:\mathbb{R}^{p_{l}}\to\mathcal{H}_{l}$ is bounded. Its Hilbert adjoint satisfies $A_{l}^{*}m_{l}=g_{\mathrm{pop}}^{(l)}$ . Consequently,

g_{\mathrm{pop}}^{(l)}=0\quad\Longleftrightarrow\quad\Pi_{l}m_{l}=0\quad\Longleftrightarrow\quad\mathcal{V}_{l}=0.

(4)

Proof.

For $u\in\mathbb{R}^{p_{l}}$ ,

\|A_{l}u\|_{\mathcal{H}_{l}}^{2}=\mathbb{E}\|J_{l}(z_{l})u\|_{2}^{2}\leq\mathbb{E}\|J_{l}(z_{l})\|_{\sigma}^{2}\|u\|_{2}^{2},

so $A_{l}$ is bounded. Lemma 1 shows $\langle m_{l},A_{l}u\rangle_{\mathcal{H}_{l}}=(g_{\mathrm{pop}}^{(l)})^{\top}u$ for every $u$ , hence $A_{l}^{*}m_{l}=g_{\mathrm{pop}}^{(l)}$ . Now $A_{l}^{*}m_{l}=0$ exactly when $m_{l}$ is orthogonal to $\operatorname{Range}(A_{l})=\mathcal{T}_{h}^{(l)}$ , which is equivalent to $\Pi_{l}m_{l}=0$ . The final equivalence is the definition of $\mathcal{V}_{l}$ . ∎

Lemma 3 (Empirical directional derivative and projection).

For every fixed sample satisfying the differentiability conditions in the main paper’s first-order regularity assumption and every $u\in\mathbb{R}^{p_{l}}$ ,

D\Phi_{S,l}(0)[u]=(g_{S}^{(l)})^{\top}u=\langle q_{S,l},A_{S,l}u\rangle_{S,l}.

(5)

Moreover,

g_{S}^{(l)}=0\quad\Longleftrightarrow\quad\Pi_{S,l}q_{S,l}=0\quad\Longleftrightarrow\quad\mathcal{V}_{S,l}=0.

(6)

Proof.

The sample-wise chain rule and finite summation give

D\Phi_{S,l}(0)[u]=\frac{1}{M}\sum_{i=1}^{M}q_{l,i}^{\top}J_{l}(z_{l,i})u=(g_{S}^{(l)})^{\top}u.

The same expression is $\langle q_{S,l},A_{S,l}u\rangle_{S,l}$ . The finite-dimensional orthogonality argument used in Lemma 2 gives (6). ∎

Appendix D Proof of the Main-Paper Necessary-and-Sufficient Saturation Theorem

Proof.

Fix $l\in\mathcal{I}$ and abbreviate $g=g_{\mathrm{pop}}^{(l)}$ . By Lemma 1, the residual non-degeneracy condition stated in the main paper is the existence of $v$ with $g^{\top}v<0$ . This implies $g\neq 0$ . Conversely, if $g\neq 0$ , choosing $v=-g$ gives $g^{\top}v=-\|g\|_{2}^{2}<0$ ; the interior-point condition on $\Theta_{l}$ makes sufficiently small steps feasible. Thus statements 1 and 2 are equivalent. Lemma 2 gives the equivalence of statements 2, 3, and 4.

Suppose $g\neq 0$ . The fixed zero-state descent-compatible update assumption stated in the main paper and Lemma 1 give

D\Phi_{l}(0)[d_{l}(g)]=g^{\top}d_{l}(g)<0.

Fréchet differentiability therefore yields $\bar{\eta}_{l}>0$ such that $\Phi_{l}(\eta d_{l}(g))<\Phi_{l}(0)$ for every $\eta\in(0,\bar{\eta}_{l})$ . Conversely, if $g=0$ , zero-state descent compatibility gives $d_{l}(g)=0$ , so strict decrease is impossible. This proves statement 5.

Since $\mathcal{I}$ is finite, $\mathcal{V}_{\mathrm{depth}}=0$ if and only if every $\mathcal{V}_{l}=0$ . The local equivalences identify this with depth-wide first-order saturation. Lemma 3 gives the empirical result by the same argument, with a threshold $\bar{\eta}_{S,l}>0$ . ∎

Corollary 1 (Quantitative descent under local smoothness).

Fix candidate $l$ and suppose ordinary gradient descent is used, so $d_{l}(g)=-g$ . If $\nabla\Phi_{l}$ is $L_{l}$ -Lipschitz on a neighborhood containing the segment $\{-\eta g_{\mathrm{pop}}^{(l)}:0\leq\eta\leq 1/L_{l}\}$ , then

\Phi_{l}\!\left(-\eta g_{\mathrm{pop}}^{(l)}\right)\leq\Phi_{l}(0)-\eta\left(1-\frac{L_{l}\eta}{2}\right)\|g_{\mathrm{pop}}^{(l)}\|_{2}^{2}

for every $\eta\in(0,1/L_{l}]$ . In particular, at $\eta=1/L_{l}$ the one-step decrease is at least $\|g_{\mathrm{pop}}^{(l)}\|_{2}^{2}/(2L_{l})$ . The same statement holds for $\Phi_{S,l}$ with empirical smoothness constant $L_{S,l}$ .

Proof.

The standard descent lemma gives $\Phi_{l}(-\eta g)\leq\Phi_{l}(0)-\eta\|g\|_{2}^{2}+(L_{l}\eta^{2}/2)\|g\|_{2}^{2}$ . Substitute $g=g_{\mathrm{pop}}^{(l)}$ . The empirical proof is identical. ∎

D.1 Closed Forms for the Growth Values

Define $G_{l}:=A_{l}^{*}A_{l}=\mathbb{E}[J_{l}(z_{l})^{\top}J_{l}(z_{l})]$ and $\widehat{G}_{S,l}:=M^{-1}\sum_{i=1}^{M}J_{l}(z_{l,i})^{\top}J_{l}(z_{l,i})$ .

Lemma 4 (Gram-matrix representation).

The growth values satisfy

\mathcal{V}_{l}^{2}=(g_{\mathrm{pop}}^{(l)})^{\top}G_{l}^{\dagger}g_{\mathrm{pop}}^{(l)},\qquad\mathcal{V}_{S,l}^{2}=(g_{S}^{(l)})^{\top}\widehat{G}_{S,l}^{\dagger}g_{S}^{(l)}.

(7)

Proof.

Because $\mathcal{T}_{h}^{(l)}=\operatorname{Range}(A_{l})$ , $\Pi_{l}=A_{l}G_{l}^{\dagger}A_{l}^{*}$ . Hence

\mathcal{V}_{l}^{2}=\langle m_{l},A_{l}G_{l}^{\dagger}A_{l}^{*}m_{l}\rangle_{\mathcal{H}_{l}}=(g_{\mathrm{pop}}^{(l)})^{\top}G_{l}^{\dagger}g_{\mathrm{pop}}^{(l)}.

The empirical formula follows from the same finite-dimensional projection identity for $A_{S,l}$ under the empirical inner product. ∎

Remark 2 (Parameterization invariance and fixed-sample interpolation).

The geometric definition of $\mathcal{V}_{l}$ depends only on $\operatorname{Range}(A_{l})$ and is therefore invariant under every regular local reparameterization that preserves this range. For an invertible coordinate change $\theta_{l}=T\alpha_{l}$ , one has $A_{l}\mapsto A_{l}T$ , $g_{\mathrm{pop}}^{(l)}\mapsto T^{\top}g_{\mathrm{pop}}^{(l)}$ , and $G_{l}\mapsto T^{\top}G_{l}T$ . Applying Lemma 4 in either coordinate system gives the same scalar $\mathcal{V}_{l}^{2}$ , because the represented tangent subspace is unchanged. By contrast, $\|g_{\mathrm{pop}}^{(l)}\|_{2}$ is coordinate-dependent.

Fix a sample size $M$ and draw $S\sim\mathcal{D}^{M}$ . Suppose the selected standard block lies in an overparameterized regime in which sample-wise tangent completeness holds almost surely at this fixed $M$ ; this requires at least $r_{l}\geq M$ and $\operatorname{rank}(\Psi_{S}^{(l)})=M$ almost surely. Then $\mathcal{V}_{S,l}^{2}=M^{-1}\sum_{i}\|q_{l,i}\|_{2}^{2}$ , and

\mathbb{E}_{S\sim\mathcal{D}^{M}}\mathcal{V}_{S,l}^{2}=\mathbb{E}\|m_{l}(z_{l})\|_{2}^{2}+\mathbb{E}\|q_{l}(z_{l},y)-m_{l}(z_{l})\|_{2}^{2}\geq\mathcal{V}_{l}^{2}.

This is a fixed- $M$ interpolation identity, not an assertion that sample-wise tangent completeness persists as $M\to\infty$ when the feature dimension $r_{l}$ is fixed. It explains why the supplementary consistency result estimates the population residual gradient rather than asserting $\mathcal{V}_{S,\mathrm{depth}}\to\mathcal{V}_{\mathrm{depth}}$ without additional rank-stability and complexity assumptions.

Appendix E Joint Insertion of Several Residual Blocks

Let $\mathcal{K}=(l_{1},\ldots,l_{K})$ be an ordered tuple of candidates at distinct insertion locations, listed from upstream to downstream in the fixed feed-forward network topology. Write the original network as $f_{\mathrm{old}}^{*}=F_{K}\circ F_{K-1}\circ\cdots\circ F_{0}$ , where the maps $F_{j}$ are the original network segments between consecutive insertion locations. Define $T_{l_{j},\theta_{l_{j}}}(w):=w+h_{l_{j},\theta_{l_{j}}}(w)$ and

f_{\mathcal{K},\theta_{\mathcal{K}}}:=F_{K}\circ T_{l_{K},\theta_{l_{K}}}\circ F_{K-1}\circ\cdots\circ T_{l_{1},\theta_{l_{1}}}\circ F_{0},

(8)

with $K:=|\mathcal{K}|$ , $\theta_{\mathcal{K}}:=(\theta_{l_{1}},\ldots,\theta_{l_{K}})$ in the product space $\prod_{j=1}^{K}\mathbb{R}^{p_{l_{j}}}$ , and $\Phi_{\mathcal{K}}(\theta_{\mathcal{K}}):=R(f_{\mathcal{K},\theta_{\mathcal{K}}})$ .

Corollary 2 (Joint insertion of independently parameterized blocks).

Assume that the first-order regularity assumption stated in the main paper for every $l\in\mathcal{K}$ . Assume additionally that, for $\mathcal{D}$ -almost every $(x,y)$ , the joint sample loss is Fréchet differentiable with respect to $\theta_{\mathcal{K}}$ at the joint origin, and that $\Phi_{\mathcal{K}}$ is Fréchet differentiable there with derivative obtained by interchanging differentiation and expectation. Suppose the active parameter blocks are disjoint and every $h_{l,0}\equiv 0$ globally.

A strict joint first-order descent direction is a vector $u_{\mathcal{K}}$ satisfying $D\Phi_{\mathcal{K}}(0)[u_{\mathcal{K}}]<0$ . Such a direction exists if and only if at least one candidate in $\mathcal{K}$ satisfies the residual non-degeneracy condition stated in the main paper.

Proof.

Let $\iota_{j}:\mathbb{R}^{p_{l_{j}}}\to\prod_{r=1}^{K}\mathbb{R}^{p_{l_{r}}}$ be the canonical injection into block $j$ . Because every $h_{l,0}\equiv 0$ , each full residual transformation held at its designated origin is $T_{l,0}(w)=w$ on the full ambient space. Therefore, for every $\theta_{l_{j}}$ ,

\Phi_{\mathcal{K}}(\iota_{j}\theta_{l_{j}})=\Phi_{l_{j}}(\theta_{l_{j}});

no differentiability of the intermediate network segments is needed for this pointwise identity.

Let $L:=D\Phi_{\mathcal{K}}(0)$ , which is a continuous linear functional by joint Fréchet differentiability. Restricting $L$ to coordinate block $j$ and applying Lemma 1 gives

L[\iota_{j}u_{j}]=D\Phi_{l_{j}}(0)[u_{j}]=(g_{\mathrm{pop}}^{(l_{j})})^{\top}u_{j}.

Every joint direction decomposes as $u_{\mathcal{K}}=\sum_{j=1}^{K}\iota_{j}u_{j}$ , so linearity yields

D\Phi_{\mathcal{K}}(0)[u_{\mathcal{K}}]=\sum_{j=1}^{K}(g_{\mathrm{pop}}^{(l_{j})})^{\top}u_{j}.

Thus the joint derivative is represented by the concatenated vector $(g_{\mathrm{pop}}^{(l_{1})},\ldots,g_{\mathrm{pop}}^{(l_{K})})$ . It has a negative direction if and only if at least one component is nonzero, which is equivalent to the residual non-degeneracy condition stated in the main paper for at least one candidate by the main paper’s necessary-and-sufficient saturation theorem. ∎

Appendix F Proof of the Main-Paper Activation-Gradient Characterization Theorem

Proof.

If $m_{l}\in\mathcal{T}_{h}^{(l)}$ , then $\Pi_{l}m_{l}=m_{l}$ , so $\mathcal{V}_{l}=\|m_{l}\|_{\mathcal{H}_{l}}$ . The main paper’s necessary-and-sufficient saturation theorem gives the population equivalence. If $q_{S,l}\in\operatorname{Range}(A_{S,l})$ , then $\Pi_{S,l}q_{S,l}=q_{S,l}$ , yielding the empirical equivalence.

For the standard block, every matrix direction $\Delta V_{l}$ satisfies $J_{l}(z)\operatorname{vec}(\Delta V_{l})=\Delta V_{l}\psi_{l}(z)$ . Therefore,

	$\displaystyle D\Phi_{l}(0)[\operatorname{vec}(\Delta V_{l})]$	$\displaystyle=\mathbb{E}[q_{l}(z_{l},y)^{\top}\Delta V_{l}\psi_{l}(z_{l})]$
		$\displaystyle=\langle C_{\mathrm{pop}}^{(l)},\Delta V_{l}\rangle_{F}.$

Hence $g_{\mathrm{pop}}^{(l)}=\operatorname{vec}(C_{\mathrm{pop}}^{(l)})$ under the fixed vectorization convention, and the population residual non-degeneracy condition stated in the main paper holds exactly when $C_{\mathrm{pop}}^{(l)}\neq 0$ . The empirical statement follows identically and does not use residual-signal realizability.

If $U_{l}$ and $V_{l}$ are both treated as active raw parameters, then at $V_{l}=0$ the derivative with respect to $U_{l}$ is $V_{l}D_{U_{l}}\psi_{l,U_{l}}(z)=0$ . Hence the full tangent range is identical to the range obtained by fixing $U_{l}=U_{0,l}$ and varying only $V_{l}$ ; this reduction loses no first-order directions.

The sample tangent range in matrix form is $\{\Delta V_{l}\Psi_{S}^{(l)}:\Delta V_{l}\in\mathbb{R}^{N_{l}\times r_{l}}\}$ . Under the tuple–matrix identification, $\langle B,C\rangle_{S,l}=M^{-1}\langle B,C\rangle_{F}$ ; multiplication by a positive scalar does not change orthogonality or the associated projector. A matrix belongs to this range exactly when each row belongs to the row space of $\Psi_{S}^{(l)}$ . Hence the matrix representation of $\Pi_{S,l}q_{S,l}$ is $Q_{S}^{(l)}\Pi_{\Psi,l}$ and $\mathcal{V}_{S,l}=M^{-1/2}\|Q_{S}^{(l)}\Pi_{\Psi,l}\|_{F}$ . Moreover, $C_{S}^{(l)}=0$ if and only if $Q_{S}^{(l)}\Pi_{\Psi,l}=0$ . The stacked tangent operator is surjective exactly when the row space of $\Psi_{S}^{(l)}$ is all of $\mathbb{R}^{M}$ , equivalently when $\operatorname{rank}(\Psi_{S}^{(l)})=M$ . Then $\Pi_{\Psi,l}=I_{M}$ and $C_{S}^{(l)}=0$ if and only if $Q_{S}^{(l)}=0$ . Applying the local equivalences at every candidate gives the depth-wide statement. ∎

Remark 3 (Approximate residual-signal realizability).

Orthogonal Pythagoras gives

	$\displaystyle\mathcal{V}_{l}^{2}$	$\displaystyle=\\|m_{l}\\|_{\mathcal{H}_{l}}^{2}-\operatorname{dist}_{\mathcal{H}_{l}}(m_{l},\mathcal{T}_{h}^{(l)})^{2},$
	$\displaystyle\mathcal{V}_{S,l}^{2}$	$\displaystyle=\\|q_{S,l}\\|_{S,l}^{2}-\operatorname{dist}_{S,l}(q_{S,l},\operatorname{Range}(A_{S,l}))^{2}.$

Consequently, if $\operatorname{dist}_{\mathcal{H}_{l}}(m_{l},\mathcal{T}_{h}^{(l)})\leq\varepsilon_{\mathrm{real}}\|m_{l}\|_{\mathcal{H}_{l}}$ for some $\varepsilon_{\mathrm{real}}\in[0,1)$ , then $\mathcal{V}_{l}\geq\sqrt{1-\varepsilon_{\mathrm{real}}^{2}}\,\|m_{l}\|_{\mathcal{H}_{l}}>0$ whenever $m_{l}\neq 0$ . The identical conclusion holds on a fixed sample using the empirical distance and norm. This approximate form is often more relevant than exact realizability when the feature dimension is smaller than the sample size.

Appendix G Supplementary Finite-Sample Consistency Result

Under the independent finite-sample probing assumption stated in the main paper, instantiate all empirical quantities with $S=S_{n}^{\mathrm{probe}}$ and $M=n$ . Define $\xi_{l}(x,y):=J_{l}(z_{l})^{\top}q_{l}(z_{l},y)$ and

	$\displaystyle\widehat{g}_{n}^{(l)}$	$\displaystyle:=\frac{1}{n}\sum_{i=1}^{n}\xi_{l}(\widetilde{x}_{i},\widetilde{y}_{i}),$
	$\displaystyle\Gamma$	$\displaystyle:=\max_{l\in\mathcal{I}}\\|g_{\mathrm{pop}}^{(l)}\\|_{2},$	$\displaystyle\widehat{\Gamma}_{n}$	$\displaystyle:=\max_{l\in\mathcal{I}}\\|\widehat{g}_{n}^{(l)}\\|_{2}.$

Let $\sigma_{l}^{2}:=\mathbb{E}\|\xi_{l}-g_{\mathrm{pop}}^{(l)}\|_{2}^{2}$ and, for a deterministic threshold $\tau_{n}>0$ , define $\widehat{\mathrm{Sat}}_{n}:=\mathbf{1}\{\widehat{\Gamma}_{n}\leq\tau_{n}\}$ . By the main paper’s necessary-and-sufficient saturation theorem,

	$\displaystyle\Gamma=0$	$\displaystyle\quad\Longleftrightarrow\quad\mathcal{V}_{\mathrm{depth}}=0$
		$\displaystyle\quad\Longleftrightarrow\quad f_{\mathrm{old}}^{*}\text{ is first-order depth-saturated}.$		(9)

Theorem 3 (Consistent finite-sample saturation detection).

Under the independent finite-sample probing assumption stated in the main paper, for every $t>0$ ,

\Pr\!\left(\max_{l\in\mathcal{I}}\|\widehat{g}_{n}^{(l)}-g_{\mathrm{pop}}^{(l)}\|_{2}\geq t\right)\leq\frac{1}{nt^{2}}\sum_{l\in\mathcal{I}}\sigma_{l}^{2}.

(10)

If $\tau_{n}\to 0$ and $n\tau_{n}^{2}\to\infty$ , then

	$\displaystyle\Gamma=0$	$\displaystyle\Longrightarrow\Pr(\widehat{\mathrm{Sat}}_{n}=1)\to 1,$
	$\displaystyle\Gamma>0$	$\displaystyle\Longrightarrow\Pr(\widehat{\mathrm{Sat}}_{n}=0)\to 1.$

In view of (9), this detector is consistent for first-order depth saturation relative to the fixed protocol.

Proof.

Independence and centering give $\mathbb{E}\|\widehat{g}_{n}^{(l)}-g_{\mathrm{pop}}^{(l)}\|_{2}^{2}=\sigma_{l}^{2}/n$ . Markov’s inequality applied to the squared norm and a union bound over the finite set $\mathcal{I}$ prove (10).

If $\Gamma=0$ , then

\Pr(\widehat{\mathrm{Sat}}_{n}=0)=\Pr(\widehat{\Gamma}_{n}>\tau_{n})\leq\frac{1}{n\tau_{n}^{2}}\sum_{l}\sigma_{l}^{2}\to 0.

If $\Gamma>0$ , choose $l_{*}$ with $\|g_{\mathrm{pop}}^{(l_{*})}\|_{2}=\Gamma$ . For all sufficiently large $n$ , $\tau_{n}<\Gamma/2$ . On the event $\widehat{\Gamma}_{n}\leq\tau_{n}$ , $\|\widehat{g}_{n}^{(l_{*})}-g_{\mathrm{pop}}^{(l_{*})}\|_{2}>\Gamma/2$ , and therefore $\Pr(\widehat{\mathrm{Sat}}_{n}=1)\leq 4\sigma_{l_{*}}^{2}/(n\Gamma^{2})\to 0$ . ∎

Remark 4 (Coordinate dependence and practical near-saturation).

The statistic $\Gamma=\max_{l}\|g_{\mathrm{pop}}^{(l)}\|_{2}$ and the threshold $\tau_{n}$ use Euclidean norms of residual-parameter gradients. Their zero-versus-nonzero boundary is invariant under regular reparameterization, but their numerical scales are protocol- and coordinate-dependent. The consistency theorem should therefore be interpreted relative to the fixed parameterization.

Exact saturation, $\Gamma=0$ , is a knife-edge population hypothesis. A practical separated test may instead fix $\delta\geq 0$ and $t>0$ and compare $H_{0}:\Gamma\leq\delta$ with $H_{1}:\Gamma\geq\delta+2t$ using decision threshold $\delta+t$ . On the event $\max_{l}\|\widehat{g}_{n}^{(l)}-g_{\mathrm{pop}}^{(l)}\|_{2}<t$ , the test is correct under either hypothesis. Thus (10) bounds its error probability by $\sum_{l}\sigma_{l}^{2}/(nt^{2})$ . Implementing the rule requires an upper estimate of the variances $\sigma_{l}^{2}$ ; rate-optimal threshold selection and adaptive power analysis require additional assumptions.

Remark 5 (Why an independent probe sample is needed).

The fixed-sample equivalences in the main theorems are deterministic and may be evaluated on the training sample. The consistency theorem instead interprets an empirical gradient as an estimator of its population counterpart. Reusing the data that trained the reference model generally breaks the conditional i.i.d. argument unless additional stability or sample-splitting assumptions are introduced.

Appendix H More experiments

On four converged CIFAR-10 ResNets, we insert a globally zero-output residual block and fit only its terminal projection to a small negative activation-gradient target, testing whether the resulting first-order direction produces an immediate reduction in the empirical task loss.

H.1 Local Effect of Activation-Gradient-Matched Insertions

We next examine whether an activation-gradient-matched residual block produces the immediate task-loss change predicted by the fixed-sample first-order analysis. We evaluate four converged CIFAR-10 ResNet checkpoints with reference depths 8, 14, 20, and 34.

For four converged ResNet configurations on CIFAR-10, we directly construct one additional residual block by matching its output to a small negative activation-gradient perturbation. At the selected candidate location $l$ , we freeze the reference model and cache the training-set hidden representations $z_{l,i}$ together with their activation gradients $q_{l,i}$ . Both quantities are detached and treated as a fixed regression dataset. The inserted block is a standard ResNet block. Its feature-producing parameters $U_{l}$ are initialized using the standard ResNet initialization and then held fixed, while the terminal output projection is initialized at $V_{l}=0$ and is the only optimized parameter. We fit $V_{l}$ by gradient descent on

\mathcal{J}_{\mathrm{match}}^{(l)}(V_{l}):=\frac{1}{2M}\sum_{i=1}^{M}\left\|V_{l}\psi_{l}(z_{l,i})+\alpha q_{l,i}\right\|_{2}^{2},

where the plus sign reflects the regression target $-\alpha q_{l,i}$ . The zero initialization makes the insertion globally function-preserving before fitting, while training only $V_{l}$ restricts the construction to the first-order tangent family generated by the fixed residual features. After the auxiliary objective has converged, the fitted block is inserted and evaluated immediately; no classification-loss optimization or joint fine-tuning is performed. The matching scale, optimization schedule, and candidate-selection rule are fixed without using held-out performance. We report the matching error, the relative magnitude of the resulting hidden-state perturbation, and the changes in training loss.

Table 7: Immediate loss changes after inserting one activation-gradient-matched residual block. The fitted model is evaluated without subsequent classification-loss training. Positive gain denotes lower loss after insertion.

Reference	Train loss(ave)	$\Delta_{\mathrm{train}}^{\mathrm{match}}$	Test loss(ave)	$\Delta_{\mathrm{test}}^{\mathrm{match}}$
ResNet-8	$0.0427\!\rightarrow\!0.0420$	$+0.0007$ $\pm 0.0003$	$0.3782\!\rightarrow\!0.3779$	$+0.0003$ $\pm 0.0009$
ResNet-14	$0.0392\!\rightarrow\!0.0388$	$+0.0004$ $\pm 0.0002$	$0.2537\!\rightarrow\!0.2531$	$+0.0006$ $\pm 0.0011$
ResNet-20	$0.0251\!\rightarrow\!0.0249$	$+0.0002$ $\pm<0.0001$	$0.2015\!\rightarrow\!0.2017$	$-0.0002$ $\pm 0.0005$
ResNet-34	$0.0011\!\rightarrow\!0.0009$	$+0.0002$ $\pm<0.0001$	$0.1891\!\rightarrow\!0.1895$	$-0.0004$ $\pm 0.0005$

Analysis.

Table 7 reports the immediate loss changes after inserting one activation-gradient-matched residual block, without subsequent task-loss optimization or joint fine-tuning. The insertion reduces the training loss for all four checkpoints, by $7\times 10^{-4}$ , $4\times 10^{-4}$ , $2\times 10^{-4}$ , and $2\times 10^{-4}$ for ResNet-8, ResNet-14, ResNet-20, and ResNet-34. Because the target is constructed from activation gradients on the same training sample, this sign consistency is the main behavior predicted by the fixed-sample first-order analysis. The results are consistent with the fitted block capturing a descending component within the tangent family generated by the fixed features.

The reductions are small, as expected from $\alpha=10^{-3}$ and from evaluation before task-loss fine-tuning. Thus, the experiment tests an immediate local improvement rather than the gain after fully optimizing the expanded model. The reduction decreases from $7\times 10^{-4}$ for ResNet-8 to $2\times 10^{-4}$ for ResNet-20 and ResNet-34, qualitatively consistent with diminishing empirical first-order value of additional depth. However, differing baseline losses and one insertion per checkpoint prevent Table 7 from establishing a monotone or calibrated depth–gain relationship. Test-loss point estimates improve by $3\times 10^{-4}$ and $6\times 10^{-4}$ for ResNet-8 and ResNet-14, but worsen by $2\times 10^{-4}$ and $4\times 10^{-4}$ for ResNet-20 and ResNet-34. All changes are smaller than or comparable to the reported uncertainty.