Robust Loss Functions¶

← Wasserstein Bound | Quantile & Expectile →

Robust loss functions are designed to mitigate the influence of outliers — observations that deviate significantly from the primary data-generating process. Unlike standard Squared Error (MSE), which grows quadratically with the residual \(r = y - \hat{y}\), robust losses grow more slowly (linearly or sub-linearly) at the tails.

The Influence Function¶

To understand robustness, we examine the influence function \(\psi(r) = \frac{\partial \rho}{\partial r}\). For MSE, \(\psi(r) = 2r\), meaning a single large residual has an unbounded, linear influence on the gradients. Robust losses bound or suppress this influence.

Core Robust Losses¶

Huber & Pseudo-Huber¶

The Huber Loss [1] is the "gold standard" for robust regression. It provides the stability of \(L_2\) near zero (ensuring a unique minimum) and the robustness of \(L_1\) for large residuals.

The Pseudo-Huber Loss (PseudoHuberLoss) is a smooth, \(C^\infty\) approximation that is easier to optimise with second-order methods:

from torchregress.losses import WeightedHuberLoss, PseudoHuberLoss

# Traditional Huber (weighted wrapper)
loss_fn = WeightedHuberLoss(delta=1.345) # 95% efficiency for Gaussian

# Smooth Pseudo-Huber
loss_fn = PseudoHuberLoss(delta=1.0)

Redescending Losses (Cauchy & Tukey)¶

"Redescending" losses are the most aggressive form of robustness. Their influence function \(\psi(r)\) actually returns to zero for extreme residuals, meaning the model effectively "ignores" outliers once they are confirmed as such.

Cauchy Loss (CauchyLoss):

Tukey Biweight (TukeyBiweightLoss):

The ultimate outlier rejection tool [2]. Samples with \(|r| > c\) contribute zero gradient to the model.

General Robust Family (Barron & AdaptiveRobust)¶

BarronLoss provides a single smooth family that spans several useful robustness regimes through the shape parameter \(\alpha\) [4]:

Practical reading:

• \(\alpha \approx 2\): near-quadratic, fastest on clean data
• \(\alpha \approx 1\): Huber-like compromise
• \(\alpha \approx 0\): Cauchy-like heavy-tail robustness
• \(\alpha < 0\): increasingly redescending behavior

AdaptiveRobustLoss keeps the same loss family but makes \(\alpha\) and \(c\) trainable. This is useful when the tail behavior is unknown up front and you want the optimizer to learn a robust regime jointly with the model.

import torch
from torchregress.losses import AdaptiveRobustLoss, BarronLoss

# Fixed-shape Barron loss
loss_fn = BarronLoss(alpha=0.5, scale=1.0)

# Jointly learn the robust shape and scale with the model
adaptive_loss = AdaptiveRobustLoss(alpha_init=1.0, scale_init=1.0)
optimizer = torch.optim.Adam(
    list(model.parameters()) + list(adaptive_loss.parameters()),
    lr=1e-3,
)

Comparison & Selection¶

Tradeoff Matrix¶

Decision Flowchart¶

graph TD
    Start["New Regression Problem"] --> Outliers{"Contains Outliers?"}
    Outliers -->|No| MSE["WeightedMSELoss (Standard)"]
    Outliers -->|Maybe/Mild| Huber["WeightedHuberLoss (delta=1.0)"]
    Outliers -->|Unknown tail shape| Barron["BarronLoss (alpha≈1.0)"]
    Outliers -->|Confirmed/Severe| Redescending{"Severe/Adversarial?"}
    Redescending -->|Aggressive| Cauchy["CauchyLoss (c=1.0)"]
    Redescending -->|Extreme| Tukey["TukeyBiweight (c=4.685)"]
    Barron --> Adaptive{"Need alpha/scale learned?"}
    Adaptive -->|Yes| ABarron["AdaptiveRobustLoss"]
    Adaptive -->|No| Barron

    Huber --> Smooth{"Need Smoothness?"}
    Smooth -->|Yes| PHuber["PseudoHuberLoss"]
    Smooth -->|No| Huber

Advanced: Tail Sensitivity with CVaR¶

While robust losses ignore outliers, CVaR (Conditional Value at Risk) [3] does the opposite — it focuses exclusively on the hardest samples. This is useful for:

• Ensuring fairness across sub-populations.
• Minimising worst-case error.
• Training models that must be reliable on the tails.

from torchregress.losses import CVaRLoss

# Train on the worst 5% of samples using Huber as the base loss
loss_fn = CVaRLoss(alpha=0.05, base_loss="huber", delta=1.5)

References¶

Next steps¶

• Gaussian losses — when you don't need robustness (standard regression)
• CVaRLoss — focus on the worst-α fraction instead of ignoring outliers
• Evaluation with proper scoring rules — CRPS, NLL, energy score
• Mathematical Foundations — CVaR derivation and scoring rule proofs