Training with Noisy Labels¶

← Imbalanced Regression | Uncertain Ground Truth →

When target labels contain measurement noise, annotation errors, or systematic corruption, standard losses can overfit to the noise. torchregress provides several built-in mechanisms for robust training under label noise.

Approaches in torchregress¶

1. Robust Losses¶

The simplest defence — replace MSE with a loss that bounds the influence of large errors:

from torchregress.losses import WeightedHuberLoss, TukeyBiweightLoss, CauchyLoss

# Huber: linear penalty beyond δ
loss_fn = WeightedHuberLoss(delta=1.0)  # [API](../api/losses.md)

# Tukey: completely ignores errors beyond c
loss_fn = TukeyBiweightLoss(c=4.685)  # [API](../api/losses.md)

# Cauchy: logarithmic suppression
loss_fn = CauchyLoss(c=1.0)  # [API](../api/losses.md)

See Robust Losses for details and a decision guide.

2. CVaR Loss (Tail-Focused)¶

Average only the worst-\(\alpha\) fraction of per-sample losses (those with the highest loss values) — focusing training on the hardest / most outlying residuals:

from torchregress.losses import CVaRLoss

loss_fn = CVaRLoss(alpha=0.5, base_loss="huber")  # [API](../api/losses.md)

3. Uncertain Ground Truth Losses¶

When label noise has known or estimable variance, use losses that explicitly model the noise:

from torchregress.losses import NoisyTargetGaussianNLL

# Known label noise variance
loss_fn = NoisyTargetGaussianNLL()  # [API](../api/losses.md)

See Uncertain Ground Truth for NoisyTargetGaussianNLL, ConsistencyRegLoss, PseudoLabelNLL, and PseudoLabelConsistencyLoss.

For a full conditional density with known label noise, use NoiseAwareDensityLoss. It convolves NormalizingFlowLoss, a diagonal MixtureDensityLoss, or a custom density head with the Gaussian measurement model; scalar flows use Gaussian–Hermite quadrature, MDNs use the exact analytic mixture convolution, and non-scalar heads fall back to Monte-Carlo averaging. An optional contamination component handles grossly wrong labels (a different problem from measurement noise).

4. Density / Propensity Weighting¶

Downweight samples in noisy regions using density or propensity scores:

from torchregress.losses import DensityWeightedLoss, PropensityWeightedLoss

# Inverse-density weighting
loss_fn = DensityWeightedLoss(kernel_width=0.5)  # [API](../api/losses.md)
loss_fn.fit_density(y_train)

# Inverse-propensity weighting
loss_fn = PropensityWeightedLoss(clip_min=0.01)  # [API](../api/losses.md)

See Imbalanced Regression.

5. Ensemble Disagreement¶

Train an ensemble and use member disagreement to identify noisy samples:

from torchregress.ensemble import BaseEnsembleModel

ensemble = BaseEnsembleModel(base_model, ensemble_size=5)
# After training, samples with high epistemic uncertainty
# (large variance across members) are likely mislabelled

See Ensemble Methods.

Practical Workflow¶

graph LR
    A["Noisy labels?"] --> B{"Noise magnitude known?"}
    B -->|Yes| C["NoisyTargetGaussianNLL"]
    B -->|No| D{"How severe?"}
    D -->|Mild| E["WeightedHuberLoss / LogCoshLoss"]
    D -->|Moderate| F["CVaRLoss + Huber"]
    D -->|Severe| G["TukeyBiweightLoss"]
    D -->|Unknown| H["Ensemble + discard high-uncertainty"]

Limitations¶

1. No automatic noise detector: NoiseAwareDensityLoss requires a known or externally estimated sigma; without repeated measurements or structural assumptions, the clean density and the label-noise distribution are not separately identifiable. Robust losses/ensembles remain the right tools when the noise process is unknown.
2. Robust losses only bound influence: Huber, Cauchy, and Tukey losses reduce the effect of outliers but do not identify or remove mislabelled samples. If a mislabelled point has a moderate residual, it still contributes to training.
3. Ensemble disagreement requires multiple models: Identifying noisy samples via ensemble epistemic uncertainty requires training 3–5 models, multiplying compute cost.
4. CVaR is aggressive: CVaRLoss(alpha=0.1) fits only the worst 10% of samples. If data quality is generally high, this overfits to the few genuine outliers and ignores the majority of clean data.

Recommendations¶

• Start with Huber: WeightedHuberLoss(delta=1.0) is the simplest, cheapest defense against mild label noise. Upgrade to CauchyLoss or TukeyBiweightLoss only if you can confirm severe outliers.
• Use CVaR for tail-focused objectives: When you explicitly care about worst-case performance (fairness, safety-critical applications), CVaRLoss is the right tool. See examples/benchmarks/tail_extremes_benchmark.py for CVaRLoss against other tail-focused losses.
• Ensemble for systematic noise: If label noise is systematic (not just outliers), train a BaseEnsembleModel and inspect per-sample epistemic uncertainty to flag consistently mislabelled points.
• When noise variance is known: Use NoisyTargetGaussianNLL from Uncertain ground truth — it directly models the target noise in the likelihood.

Next steps¶

• Uncertain ground truth — when noise variance is known or estimable
• Robust losses — the simplest defense via bounded influence functions
• Imbalanced regression — density/propensity weighting strategies
• Ensemble methods — use disagreement to identify noisy samples

References¶