Gaussian Loss Functions¶

← Base Classes | Beta-NLL →

Gaussian (or Normal) loss functions are the workhorses of regression. They range from the standard Mean Squared Error (MSE) to sophisticated multivariate models that capture complex correlations between targets.

Mathematical Background¶

In the probabilistic framework, regression is viewed as maximizing the likelihood of the data under a Gaussian noise model \(p(y \mid x) = \mathcal{N}(\mu(x), \sigma^2(x))\). Minimising the Negative Log-Likelihood (NLL) is equivalent to:

• The Residual Term: \((y - \mu)^2 / 2\sigma^2\) penalises predictions far from the truth, scaled by the uncertainty.
• The Penalty Term: \(\frac{1}{2}\log(2\pi\sigma^2)\) prevents the model from simply predicting infinite variance to "cheat" the residual term.

Comparison of Gaussian Losses¶

1. Univariate: GaussianNLLLoss¶

Used for standard regression where you want to estimate per-sample uncertainty.

from torchregress.losses import GaussianNLLLoss

# Model must output [batch, 2] -> [mean, log_var]
loss_fn = GaussianNLLLoss()
loss = loss_fn(y_pred, y_true)

1b. Faithful heteroscedastic: FaithfulGaussianLoss¶

Joint Gaussian NLL couples gradients from the variance head into the mean through the residual \((y-\mu)^2/\sigma^2\). FaithfulGaussianLoss adds a direct MSE term on the mean and uses stop-gradient on \(\mu\) inside the NLL residual so variance calibration does not distort point prediction.

from torchregress.losses import FaithfulGaussianLoss

loss_fn = FaithfulGaussianLoss(mean_weight=1.0, variance_weight=1.0)
loss = loss_fn((mean, log_var), y_true)

Compare with BetaNLLLoss: β-NLL keeps a single joint NLL and rescales it with a detached variance; faithful loss explicitly splits mean vs variance terms.

2. Multivariate: Full Covariance (MultivariateGaussianLoss)¶

When targets are correlated (e.g., predicting \(x, y, z\) coordinates), use MultivariateGaussianLoss.

from torchregress.losses import MultivariateGaussianLoss

# Requires full covariance matrix Σ (batch, k, k)
loss_fn = MultivariateGaussianLoss()
loss = loss_fn(y_mu, y_true, covariance_matrix)

3. Multivariate: Low-Rank Covariance (LowRankGaussianLoss)¶

For high-dimensional targets, a full covariance matrix has \(O(k^2)\) parameters. Low-Rank approximation reduces this to \(O(k \cdot r)\), where \(r\) is the rank.

from torchregress.losses import LowRankGaussianLoss

# W is factor matrix (batch, k, r), d is diagonal (batch, k)
loss_fn = LowRankGaussianLoss()
loss = loss_fn(mu, y_true, W, d)

Limitations¶

1. Computational scaling of full covariance: MultivariateGaussianLoss requires \(\mathcal{O}(K^3)\) operations per sample for determinants and solves, making it impractical beyond \(K \approx 10\). Use LowRankGaussianLoss with rank \(r = \lfloor K/3 \rfloor\) for higher-dimensional targets.
2. PSD violations: Cholesky factors must be regularised on the diagonal to prevent non-positive-semidefinite covariance matrices. Always use a minimum variance floor (min_variance or jitter \(\ge 10^{-6}\)).
3. LowRank rank selection: If \(r\) is too small, the diagonal component absorbs residual correlation, inflating marginal variances. If too large, parameter count approaches the full-covariance case. Tune \(r\) on validation NLL.
4. Gaussian assumption: The NLL is only a proper scoring rule when \(Y \mid X\) is truly Gaussian. For heavy-tailed, multimodal, or bounded targets, consider robust alternatives (Robust losses), MDN, or Normalizing flows.
5. Variance collapse: Without a minimum variance floor, the model can learn \(\sigma^2 \to 0\) to drive the NLL to \(-\infty\), especially on small datasets. Always set min_variance and monitor predicted variance during training.

Best Practices¶

References¶

Next steps¶

Continue with the Gaussian family:

• Beta-NLL — stabilised heteroscedastic training via detached variance rescaling
• Faithful Gaussian — decouple mean and variance gradients for cleaner calibration
• Gaussian Wasserstein bound — supervise covariance directly against labels or pseudo-labels

Or branch out:

• Robust losses — when Gaussian assumptions fail due to outliers
• Ensemble methods — decompose into aleatoric + epistemic
• Multivariate example — correlated multi-output in practice

Recommendations¶

• Start simple: Begin with WeightedMSELoss (homoscedastic) or GaussianNLLLoss (heteroscedastic). Move to FaithfulGaussianLoss only if you observe variance inflation distorting mean predictions.
• Low-rank for high-dimensional targets: For \(K > 5\) correlated targets, use LowRankGaussianLoss. For \(K \le 5\), MultivariateGaussianLoss is simpler and more interpretable.
• Stable training recipe: Initialise log_var \(\approx 0\) (variance \(\approx 1\)), set min_variance = 1e-6, and apply higher weight decay to the variance head (2–5× the mean head).
• Monitor with CRPS: Track crps_gaussian alongside NLL during training. CRPS is in target units and less sensitive to tail events than NLL. See Distribution metrics.
• Variance calibration: After training, validate with a PIT histogram or calibration error (ECE). If variance is systematically over- or under-estimated, apply post-hoc Variance Temperature Scaling. See FaithfulGaussianLoss demo.
• For coverage guarantees: Gaussian likelihood losses provide no coverage guarantees. For finite-sample prediction intervals, wrap with SplitConformal or CQR.