Noise-Aware Framework for Correcting Corrupted Labels

We consider a standard classification task with $C$ classes. Let $\mathcal{X}$ denote the feature space and $\mathcal{Y}=\{1,\dots,C\}$ be the label space. A clean dataset is defined as $D=\{(x_{i},y_{i})\}_{i=1}^{N}$, where each instance $(x_{i},y_{i})$ is independently and identically drawn from the joint distribution over $\mathcal{X}\times\mathcal{Y}$. Here, $x_{i}\in\mathcal{X}$ represents the feature vector and $y_{i}\in\mathcal{Y}$ denotes its corresponding clean label.

In real-world scenarios, clean/true labels are not always available. Instead, we typically have access to a noisy labeled dataset $\widetilde{D}=\{(x_{i},\widetilde{y}_{i})\}_{i=1}^{N}$, where each observed label $\widetilde{y}_{i}$ may differ from the true label $y_{i}$. An instance $(x_{i},\widetilde{y}_{i})$ is considered corrupted if $\widetilde{y}_{i}\neq y_{i}$, and clean otherwise. The goal of corrupted label correction is to identify instances with unreliable labels and replace them with more accurate estimates. Formally, corrupted label correction approaches aim to transform the noisy dataset $\widetilde{D}$ into a corrected one $D^{*}=\{(x_{i},y_{i}^{*})\}_{i=1}^{N}$, where $y_{i}^{*}$ denotes an estimated clean label that ideally matches the true label $y_{i}$.

Motivated by the intrinsic capability of DNNs to capture and generalize meaningful patterns even in the presence of label noise [45, 73], we leverage model predictions to guide the repair of corrupted labels. However, naive reliance on raw predictions often causes error propagation and unstable corrections. To address these challenges, we propose Canola, a novel framework for corrupted label correction that decouples noise estimation from label refinement training a noise-aware model and performing cautious label refinement. Figure 1 illustrates the overall workflow of Canola, which consists of two main phases: Phase 1: Noise Transition Matrix Construction and Phase 2: Corrupted Label Correction.

Phase 1 aims to capture the noise characteristics/structure inherent in the dataset. Canola employs the Asymmetric Co-Training strategy to distinguish likely clean and noisy samples, from which it estimates a noise transition matrix. This matrix captures the probability that each clean label is flipped into each noisy one, providing an approximation of the underlying noise distribution. It serves as a foundation for training a noise-aware model in the subsequent phase.

In Phase 2, a noise-aware model is trained by incorporating the noise transition matrix into its training objective. This integration allows the model to adjust its learning according to the inferred noise distribution. This training strategy enhances robustness by decreasing unreliable supervision and promoting stable learning under noisy conditions. Once the loss-stabilization trigger signals that training dynamics have reached a steady state, the model’s predictions are blended with the original labels to form refined soft labels. This soft and cautious relabeling strategy stabilizes updates and avoids premature commitment to potentially incorrect predictions.

These two phases are executed iteratively. In each iteration, Canola re-estimates the transition matrix based on the current refined dataset and retrains the noise-aware model accordingly. As the quality of the relabeled data improves over iterations, the estimated noise distribution and the model becomes more reliable, enabling further refinement. The process terminates when label updates converge, and the final soft labels constitute the corrected dataset.

This phase aims to construct a noise transition matrix $T$, which captures the underlying noise structure of the dataset by modeling the probability that a clean label is corrupted into a noisy one. However, since only the noisy dataset $\widetilde{D}$ is observable, directly estimating $T$ is non-trivial. To address this, we first identify a subset of samples whose labels are likely to be clean, denoted as $D_{c}$ ($D_{c}\subset\widetilde{D}$). This clean subset serves as a proxy for estimating the clean label distribution. By aligning the clean label distribution estimated from $D_{c}$ with the observed noisy label distribution in $\widetilde{D}$, we can derive approximate noise transition probabilities and construct a noise transition matrix $T$.

The goal of this phase is to train a reliable model whose predictions can be used to guide the correction of corrupted labels. In Canola, we incorporate the noise transition matrix $T$ into the training objective, making the model explicitly aware of the underlying noise label distribution. This integration encourages the model to down-weight unreliable supervision signals and emphasize trustworthy patterns. As a result, the model can be more robust to noisy labels and achieves better generalization performance. Once trained, the model’s predictions are used as soft-label estimates that are blended with the observed labels for label correction. This soft correction strategy enables Canola to progressively refine noisy labels in a principled and stable manner.

The corrupted label correction process, including both phases of Canola, is repeated for up to $\tau$ iterations or until the refined label is converged. For each sample $x_{i}$, Canola begins with its original label $\mathbf{\tilde{p}_{i}}^{(0)}$. At each iteration $t<\tau$, the label distribution is progressively refined from the previous estimate $\tilde{\mathbf{p}}_{i}^{(t-1)}$ to a more accurate version $\tilde{\mathbf{p}}_{i}^{(t)}$, ideally approaching the true clean label distribution. Once the iterative correction process concludes, the final repaired dataset is produced by converting the soft label distribution into hard labels via the maximum a posterior estimate:

We evaluate Canola under two practical forms of label noise: (i) synthetic-but-realistic noise introduced by automated labeling techniques, and (ii) naturally occurring noise in real-world datasets.

For noise introduced by automated labeling techniques [19, 11, 75, 78, 56, 54]: We employ four representative labeling strategies that cover the major data annotation paradigms in the literature. These strategies are used to annotate five widely adopted datasets covering both image and text classification tasks. For image datasets, we use FashionMNIST [60], OrganAMNIST [63, 64], and RESISC45 [8]. For text datasets, we use AGNews [72] and Yahoo! Answer [72]. Each dataset $D$ is split into two disjoint subsets, $D=D_{u}\cup D_{t}$, where $D_{u}$ is the unlabeled portion to be annotated by automated labeling techniques, and then used for evaluating corrupted label correction methods. The remaining set $D_{t}$ contains clean labels and is reserved for evaluating downstream task performance.

The employed automated labeling techniques include:

• •

  LLM-based Annotation (LLM): This approach leverages the general reasoning capability of LLMs to infer labels directly from raw instances. Following prior work in LLM-based annotation [75, 78, 56], we adopt zero-shot prompting (details are available on our website [40]) to generate a label for each instance. In this work, we employ Qwen2.5-7B-Instruct [53] for text data and Qwen2-VL-7B-Instruct [52] for image data. These model are chosen for their moderate size, strong reasoning ability, and competitive performance on recent benchmarks [41, 3].

• •

  Supervised Learning (Superv.): This strategy involves training a classifier on a labeled set, and then applying it to predict labels for the remaining unlabeled data. Following common practice under limited supervision, we train an XGBoost classifier using a small set of labeled instances, then use it to annotate unlabeled data in $D_{u}$. To provide feature inputs for the classifier, we use BERT [12] for text embeddings and CLIP [42] for image embeddings.

• •

  Semi-Supervised Learning (Semi-Superv.): This approach [11] propagates labels from a labeled set to unlabeled instances based on similarity or distance metrics. In this work, we adopt the Label Spreading algorithm [74], a well-established graph-based method, to propagate labels from a small set of labeled data to unlabeled data $D_{u}$.

• •

  Weak-Supervised Learning (Weak-Superv.): This technique produces labels by aggregating pseudo-labels obtained from multiple programmatic labeling functions. We adopt Alchemist [19], a SOTA weak supervision framework, to annotate unlabeled data in $D_{u}$. Specifically, label functions are automatically generated using Qwen2.5-7B-Instruct [53], and Snorkel [43] is employed to aggregate weak labels into final annotations. Note that, this approach is applied only to text datasets, due to the lack of standardized programmatic labeling functions for image data.

For real-world label noise, we include Clothing1M dataset [61] to further evaluate the robustness of corrupted label correction approaches in naturally occurring noise. Clothing1M is a large-scale image dataset containing 14 clothing categories with approximately 37.5K manually verified labels. This dataset is widely used for studying real-world label noise in related studies [27, 58, 13].

A summary of datasets used in our evaluation is provided in Table 1. The Noisy Label Set corresponds to $D_{u}$, which is labeled by either automated data labeling techniques or affected naturally occurring noise (i.e., Clothing1M), The Test Set corresponds to $D_{t}$, containing clean labels, which are used for evaluating downstream model performance.

RQ1. Label Correction Effectiveness: To assess the effectiveness of Canola in correcting corrupted labels, we compare its performance against three SOTA label correction methods, including Docta [76, 77], SELC [38], and SiDyP [67]. All methods are evaluated based on the error rate of the dataset after correction, which reflects the proportion of mislabeled instances that remain after the correction process. This metric provides a direct measure of how accurately each method identifies and repairs corrupted labels. The baseline methods operate as follows:

• •

  Docta [76, 77] detects noisy instances by examining their neighboring instances’ labels. The instances whose labels disagree with the majority of their neighbors are flagged as mislabeled and relabeled based on either voting (Docta-V) or ranking (Docta-R) strategies. We report the results for both variants.

• •

  SELC [38] exploits the memorization dynamics of DNNs [2]. It first estimates a turning point during training steps, before which the model predictions are assumed to be more reliable. SELC then uses the model’s predictions prior to this turning point to gradually correct noisy labels.

• •

  SiDyP [67] generates candidate clean labels based on label information from neighboring instances. It then applies a diffusion-based denoising process to refine these candidates and produce corrected labels.

To ensure a fair comparison, all methods, including Canola and the baselines, use the same embedding models to encode feature representations, BERT [12] for text data and CLIP [42] for image data.

RQ2. Downstream Utility: To evaluate the practical utility of the corrected datasets, we evaluate how effectively they support downstream model training. For each label correction method, we use its corresponding corrected dataset to train a classification model and then measure its performance on a clean test set. This experiment enables us to examine whether improving label quality can lead to better generalization in actual learning scenarios.

Furthermore, we also benchmark against model-centric approaches, which aim to train noise-robust models directly on the noisy datasets without explicitly repairing labels. This comparison enables us to assess whether cleaning the dataset using Canola can yields greater downstream benefits than relying solely on noise-robust learning algorithms. Specifically, we consider the following model-centric baselines:

• •

  DivideMix [30]: A semi-supervised learning framework designed for training models with label noise. It trains two networks simultaneously and models per-sample loss distribution to separate the training samples into a clean and noisy sets. Semi-supervised learning techniques are then applied to guess and refine labels of the noisy samples.

• •

  Co-Teaching [15]: A co-training strategy where two DNNs are trained simultaneously. In each mini-batch, each network selects a subset of small-loss samples (likely to be clean) and passes them to the other network for parameter updates. This design helps to mitigate the influence of noisy labels.

• •

  SCE-Loss [57]: This method introduces the Symmetric Cross Entropy (SCE) loss, which combines standard Cross Entropy $\text{CE}(q,p)$ and its reverse form $\text{CE}(p,q)$, where $p$ denotes the ground-truth label distribution and $q$ denotes the model prediction. The resulting loss balances effective learning from clean labels with robustness to label noise.

• •

  SiDyP [67]: A denoising framework that leverages a Simplex Label Diffusion Model to refine noisy labels. Beyond label correction, the model trained under this framework is inherently robust to label noise and can be directly used for inference, following the original setup in their paper.

RQ3. Intrinsic analysis: This experiment investigates how the main components and design choices of Canola contribute to its overall performance.

For component analysis, we conduct an ablation experiment to study the contribution of two phases in Canola’s performance. This experiment allows us to isolate the impact of Phase 1 alone and evaluate how much additional improvement in label quality is achieved by incorporating Phase 2.

For design choice analysis, we examine how the key mechanisms designed in Canola affect its results. We analyze the importance of the soft label refinement strategy by comparing Canola’s performance with soft and hard label update strategies. In the soft-update setting, labels are progressively refined using predicted probability distributions as defined in Eq. 6. Meanwhile, in the hard-update setting, the model predictions are converted to one-hot vectors via argmax operation, which are then used to overwrite the existing labels after each iteration. We also analyze the influence of loss function variants used for training the noise-aware model in Phase 2. Specifically, we consider several widely used divergence-based objectives for comparing probability distributions, including L2 Loss [7], Hellinger Distance [10], and KL Divergence [26].

RQ4. Sensitivity Analysis: We investigate how varying key factors such as hyperparameter configurations and dataset size impact Canola’s effectiveness. Specifically, we examine the impact of several key hyperparameters, including the embedding backbone, the number of warm-up epochs, the number of correction iterations, and the blending coefficient $\alpha$ in Eq. 6. In addition, we systematically vary the size of the training dataset to assess how Canola’s effectiveness scales with data availability.

To comprehensively evaluate the effectiveness of corrupted label correction methods, we adopt two complementary evaluation metrics: (i) Error Rate and (ii) Downstream Model Performance.

Error Rate measures the proportion of incorrect labels that remain in the dataset after correction. This metric directly reflects the precision of the label repair process, where a lower error rate indicates more accurate label recovery. Given a dataset $D=\{(x_{i},y_{i})\}_{i=1}^{N}$ with ground-truth labels $y_{i}$ for each instance $x_{i}$. Let $D^{*}=\{(x_{i},y_{i}^{*})\}_{i=1}^{N}$ be the corrected dataset with $y_{i}^{*}$ denote the corrected label of $x_{i}$. The error rate of $D^{*}$ is computed as:

Downstream Model Performance evaluates the practical utility of the corrected dataset $D^{*}$ for supporting downstream learning tasks. This is evaluated by training a classifier on $D^{*}$ and measuring its predictive performance on a clean test set. Higher performance indicates that the corrected labels provide more reliable supervision for model training. In this work, we report the macro-averaged F1-Score (F1-Macro), which equally weights all classes and is robust under class imbalance. Let $C$ be the number of classes and $\text{F1}_{c}$ be the F1-score of class $c$. The F1-Macro is calculated as:

Table 2 reports the dataset error rates initially produced by the automated labeling techniques (Original) and the corresponding error rates after applying label correction methods. Overall, Canola consistently achieves the lowest error rates across all settings, demonstrating its superior capability in identifying and repairing corrupted labels.

On average, applying Canola substantially reduces the original error rates by approximately 25%. For instance, under LLM-based annotation setting, where the average original error rate is about 40%, Canola reduces this rate to around 32%, yielding a relative reduction of about 20%. The improvement is even more pronounced under the most challenging setting, Weak-Superv., which exhibits an initial error rate of 61%. In this case, Canola lowers this rate to 42%, corresponding to a 31% relative reduction. These remarkable reductions demonstrate that Canola is robust and effective in diverse noise conditions, even under highly noisy labeling scenarios.

Compared to Docta, Canola consistently delivers superior label correction performance across all datasets and noise settings, achieving an average relative improvement of 19% over both Docta-V and Docta-R. For example, on the Yahoo! Answer dataset, which exhibits extremely high noise levels (40–78%), Canola improves the data quality by 15%, reducing the noise rate to 36%–68%. Meanwhile, Docta struggles to provide a reliable correction for such a severe noise condition. By leveraging neighboring information for detecting and correcting corrupted labels, Docta may inadvertently propagate incorrect labels rather than repair them. Under the LLM-based annotation setting, Docta even increases the error rate from 40% to 48%. This highlights that relying solely on neighborhood consistency is insufficient when labels are heavily corrupted.

Similarly, Canola also demonstrates stronger and more stable performance than SELC, achieving relative improvements ranging from 10% to 52% across all experimental settings. For example, on AGNews under the Weak-Superv. configuration, SELC reduces the error rate from 44.1% to 34.5%, corresponding to a 23% improvement. However, this performance is still far weaker than that of Canola, which achieves a substantially lower noise rate, i.e., 16.4%. This corresponds to a 63% improvement over the original dataset and a 52% gain over SELC’s result.

The performance gap can be attributed to fundamental limitations in SELC’s design. While SELC exploits the early-learning dynamics of DNNs to distinguish noisy from clean samples and use model predictions to guide label correction, its effectiveness heavily depends on accurately estimating the turning point at which the model transitions from learning clean patterns to memorizing noise. In practice, identifying this turning point is non-trivial. Stopping too early often results in unstable models that underfit clean labels, whereas stopping too late leads to overfitting noisy annotations [4]. Meanwhile, Canola does not rely on a predefined turning point. Instead, it explicitly trains a noise-aware model and performs label updates only when the model has reached a stable training state. This design mitigates overfitting to noise and enables Canola to achieve more robust performance across diverse noise conditions.

Although SiDyP obtains a slightly better performance than Canola in one specific case, FashionMNIST labeled by LLM , where it yields a 4% improvement, Canola surpasses SiDyP in all other settings, with gains of up to 46%. For example, on Clothing1M, SiDyP reduces the noise rate from 38.3% to 32.4% (a 16% relative reduction), whereas Canola further lowers it to 28.5%, achieving an additional 13% improvement over SiDyP.

Moreover, SiDyP sometimes degrades the data quality rather than improves it. For example, on AGNews labeled by LLM , the original noise rate is 19%, yet applying SiDyP increases the error rate to 29.9%. This suggests that its diffusion-based denoising mechanism can be sensitive to noise patterns and may fail under certain automated labeling scenarios. In contrast, Canola maintains consistently strong and reliable performance across all datasets and labeling strategies. Its noise-aware model training and progressively soft label refinements enable Canola to adapt effectively to diverse noise characteristics and ensure more stable label correction outcomes.

Figure 2 presents an example from the RESISC45 dataset, where the instance is correctly labeled as meadow by the LLM-based annotator. However, due to its visual appearance similar to a forest, all the studied label correction methods initially flagged this image as potentially mislabeled. After correction, all the baselines, including Docta, SELC, and SiDyP, misleadingly relabeled this instance as forest. In contrast, while Canola initially detected this instance as noisy, its two-phase framework, guided by a reliable noise-aware model, ultimately recovers the correct label meadow.

Figure 3 shows another challenging case, an image whose true label is island, but incorrectly labeled as cloud by the LLM. Due to its ambiguous and visually fuzzy appearance, this example poses difficulties not only for automated labeling techniques but also for label noise detection and correction methods. The label correction methods exhibit varied behaviors in this case. Both variants of Docta relabeled the instance as beach, while SELC and SiDyP retained the incorrect label cloud. In contrast, Canola successfully identified the annotation error and accurately relabeled it as island, demonstrating Canola’s robustness to ambiguous visual patterns.

However, Canola also faces limitations in cases where different categories share highly similar structural patterns. Figure 4 illustrates a case where the correct label is railway station, yet the instance is mislabeled as stadium. This image contains a large rectangular building with a prominent red roof. That visual characteristic is common across multiple categories, such as stadiums, sports complexes, stations, etc. In contrast, railway-specific signals are weak or absent. This strong semantic overlap makes the instance particularly challenging, and consequently, none of the label correction methods succeeds in repairing the label. Specifically, Docta-V, Docta-R, SiDyP relabeled the instance as airport, while Canola and SELC preserved its original label stadium. This example highlights an inherent limitation of label correction methods when the input features themselves are intrinsically ambiguous, making it difficult to obtain reliable predictions even with robust noise modeling.

In this work, all experiments were conducted on a server running Linux 6.6.105+ equipped with two NVIDIA T4 GPUs. As a common preprocessing step, all approaches require computing instance embeddings. On average, embedding a single instance takes approximately 0.02 seconds, and this cost is shared across all methods.

Canola demonstrates computational efficiency in detecting and correcting corrupted labels. It requires approximately 14 minutes to process a dataset. Specifically, Canola achieves the fastest runtime on the Yahoo! Answer dataset, completing label correction for 10K instances in about 8 minutes. The longest runtime occurs on RESISC45, where processing 18.9K images takes around 30 minutes. This increase is mainly attributed to the larger dataset size.

Among the baselines, Docta is the most efficient, requiring only about 1 minute per dataset, due to its lightweight reliance on the $K$-NN algorithm. SELC exhibits moderate computational cost, taking about 21 minutes on average per dataset, with its longest runtime (around 30 minutes) observed on OrganAMNIST. SiDyP is the most time-consuming approach, requiring between 60 and 180 minutes per dataset, significantly slower than all other methods. These results indicate that Canola achieves a favorable balance between computational efficiency and correction effectiveness. It remains practical for medium-scale real-world datasets while providing substantially stronger correction performance than faster but less expressive baselines.

The main threats to the validity of our work consist of internal, external, and construct validity threats.

Threats to internal validity: One potential threat lies in implementation correctness and hyperparameter selection. To mitigate this threat, we carefully reviewed our code. In addition, the hyperparameters are carefully selected via multiple experiments. We also make our code available publicly [40], allowing other researchers to double-check and reproduce our experiments.

Threats to external validity: External threats concern the generalization of our findings to other datasets, domains, and noise conditions. To mitigate these threats, we evaluated our approach across diverse datasets and domains, under multiple noise conditions introduced by different automated labeling techniques. Moreover, we also evaluated Canola under a real-world noisy dataset (Clothing1M). In future work, we plan to evaluate Canola in additional domains, larger-scale datasets, and more complex labeling scenarios.

Threats to construct validity: A potential threat lies in the chosen metrics. To reduce this threat, we evaluated Canola and the baselines on two complementary metrics, error rate, which directly reflects label correction accuracy, and downstream model performance, which measures the practical utility of corrected datasets in supporting effective model training. Another threat is that automated labeling pipelines used to construct experimental datasets may not fully capture all forms of real-world noise. To address this issue, we employ diverse labeling techniques and additionally evaluate on a real-world noisy dataset (Clothing1M).