Faithful Heteroscedastic & CRPS Losses¶

← Beta-NLL | Wasserstein Bound →

This page covers two specialized Gaussian loss variants: - FaithfulGaussianLoss — decouples mean and variance training to prevent the variance head from distorting point predictions - GaussianCRPSLoss — a proper scoring rule alternative to NLL that is less sensitive to tail miscalibration

FaithfulGaussianLoss¶

The Problem with Joint NLL¶

Standard GaussianNLLLoss couples gradients from the variance head into the mean through the residual term:

The gradient with respect to \(\mu\) is \(\frac{\mu - y}{\sigma^2}\), which means the variance head's outputs directly scale the gradients flowing into the mean head. If the model learns an inflated \(\sigma^2\) in high-error regions, the mean head receives weaker gradients there — effectively "explaining away" prediction errors as aleatoric noise.

The Faithful Solution¶

FaithfulGaussianLoss decouples the two objectives (see API: FaithfulGaussianLoss):

where \(\operatorname{sg}\) is stop-gradient. The mean receives gradients only from the MSE term; the variance head receives gradients from the NLL term with a detached mean. This mirrors the intent of faithful heteroscedastic training: preserve point prediction quality while learning a noise model.

Usage¶

from torchregress.losses import FaithfulGaussianLoss

loss_fn = FaithfulGaussianLoss(mean_weight=1.0, variance_weight=1.0)

# Model outputs (mean, log_variance) tuple
mean, logvar = model(x)
loss = loss_fn((mean, logvar), y_true)

With a vector mean_weight, the weight \(\lambda_{\mu,d}\) multiplies output \(d\) of the mean term, so multi-output problems can give each target its own emphasis (for example to down-weight a target that cannot be measured). Normalise the weights, e.g. to sum to \(D\), so the overall loss scale stays comparable between runs.

# Equal emphasis on three targets except the last, which is down-weighted
loss_fn = FaithfulGaussianLoss(mean_weight=[1.0, 1.0, 0.25], mean_loss="huber", huber_delta=3.0)

Comparison: Faithful vs Beta-NLL vs Joint NLL¶

GaussianCRPSLoss¶

Why CRPS Instead of NLL?¶

The Continuous Ranked Probability Score (CRPS) is a proper scoring rule that evaluates the full predictive distribution. See the GaussianCRPSLoss API for parameters and the closed-form definition.

Unlike NLL, CRPS:

• Has the same units as the target variable (interpretable)
• Is less sensitive to tail events than NLL (a single very-low-probability point doesn't dominate)
• Penalizes both calibration and sharpness simultaneously
• Reduces to MAE when the predictive distribution is a point mass

Analytic Gaussian CRPS¶

For a Gaussian predictive distribution \(\mathcal{N}(\mu, \sigma^2)\), the CRPS has a closed form:

where \(z = \frac{y - \mu}{\sigma}\), \(\Phi\) is the standard normal CDF, and \(\phi\) is the standard normal PDF.

Usage¶

from torchregress.losses import GaussianCRPSLoss

loss_fn = GaussianCRPSLoss()

# Same interface as GaussianNLLLoss
mean, logvar = model(x)
loss = loss_fn((mean, logvar), y_true)

Supports the same prediction formats as GaussianNLLLoss: - Tuple (mean, log_variance) - Concatenated tensor [mean, log_variance] - Mean-only when fixed_variance is set

When to Prefer CRPS over NLL¶

Complete Example¶

import torch
import torch.nn as nn
from torchregress.losses import FaithfulGaussianLoss, GaussianCRPSLoss

# Heteroscedastic model
class HeteroModel(nn.Module):
    def __init__(self, in_dim):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(in_dim, 64), nn.ReLU(), nn.Linear(64, 32), nn.ReLU())
        self.mean_head = nn.Linear(32, 1)
        self.logvar_head = nn.Linear(32, 1)

    def forward(self, x):
        h = self.shared(x)
        return self.mean_head(h), self.logvar_head(h)

torch.manual_seed(42)
x = torch.randn(200, 5)
y = 2.0 * x[:, [0]] + 0.3 * torch.randn(200, 1) * torch.exp(0.5 * x[:, [1]])

model = HeteroModel(5)
faithful_loss = FaithfulGaussianLoss(mean_weight=1.0, variance_weight=1.0)
crps_loss = GaussianCRPSLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(100):
    mean, logvar = model(x)
    # Train with faithful loss
    loss = faithful_loss((mean, logvar), y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # Monitor CRPS
    with torch.no_grad():
        crps = crps_loss((mean, logvar), y)
    if epoch % 20 == 0:
        print(f"Epoch {epoch}: Faithful Loss = {loss.item():.4f}, CRPS = {crps.item():.4f}")

Limitations¶

1. Detach tradeoff: The stop_gradient on \(\mu\) in the NLL residual means the mean head receives only MSE gradients — it does not see heteroscedastic curvature information. On datasets where mean and variance are strongly coupled (e.g., Poisson-like count data), this can slow convergence of the mean head.
2. Edge case mean_weight=0: The variance head still receives the detached \(\mu\) in its residual term. If \(\mu\) is poorly initialised and never trained, the variance head gets a bad signal.
3. Edge case variance_weight=0: The loss reduces to plain MSE. The variance head receives zero gradients.
4. CRPS cost: GaussianCRPSLoss is 2–3× slower per iteration than NLL due to ndtr (normal CDF) and exp operations. For large-scale training where NLL is adequate, prefer NLL for speed.

Recommendations¶

• Use when point accuracy matters more than variance calibration: FaithfulGaussianLoss is ideal for scientific applications where the mean prediction is the primary deliverable.
• Prefer NLL for pure uncertainty quantification: If calibrated variance is your goal, standard GaussianNLLLoss or BetaNLLLoss are better choices.
• Prefer CRPS for interpretability: GaussianCRPSLoss produces loss values in target units, is less sensitive to tail events, and penalises both calibration and sharpness. See GaussianCRPSLoss in the API reference.
• Monitor both objectives separately: Track MSE and NLL independently during training to diagnose whether one term is dominating.

Next steps¶

• Gaussian losses — standard NLL and multivariate variants
• Beta-NLL — another approach to decouple mean/variance training
• Ensemble methods — decompose aleatoric + epistemic uncertainty
• Calibration metrics — validate your variance estimates

References¶