Mathematical Foundations¶

← Guide Overview | Method Selection →

This page provides a rigorous overview of the mathematical principles underlying torchregress. We cover notation, the duality between loss functions and likelihoods, uncertainty decomposition, and modern estimation techniques.

Notation¶

Throughout the documentation and codebase, we adhere to the following conventions:

Loss Functions as Negative Log-Likelihoods¶

Most regression losses in torchregress are derived from the Maximum Likelihood Estimation (MLE) framework. Choosing a loss function \(\mathcal{L}\) is mathematically equivalent to assuming a specific probability distribution \(p(y \mid x, \theta)\) for the targets and minimising the Negative Log-Likelihood (NLL):

Core Parametric Losses¶

→ See Gaussian Losses for heteroscedastic implementation details.

Quantile & Expectile Regression¶

Unlike NLL losses which target the mean, quantile and expectile losses target specific properties of the predictive distribution.

Quantile Loss (Pinball Loss): Used to estimate the \(\tau\)-th quantile \(\hat{q}_\tau\). It is the \(L_1\) analogue for distributional modeling [6]. See QuantileLoss for API details.

Expectile Loss: Used to estimate the \(\tau\)-th expectile \(\hat{e}_\tau\). It is the \(L_2\) analogue of quantile regression, which is often easier to optimise due to its smoothness. See ExpectileLoss.

Robust M-Estimation¶

Robust losses are designed to mitigate the influence of outliers by using "redescending" or sub-quadratic influence functions \(\psi(r) = \frac{\partial \rho}{\partial r}\).

→ See Robust Losses for parameter selection guides.

Uncertainty Quantification (UQ)¶

torchregress distinguishes between two fundamental types of uncertainty:

1. Aleatoric Uncertainty (Data Noise)¶

Inherent randomness in the data-generating process. It is irreducible even with infinite data.

• Homoscedastic: Constant noise level \(\sigma^2\) across all inputs.
• Heteroscedastic: Input-dependent noise level \(\sigma^2(x)\).

2. Epistemic Uncertainty (Model Ignorance)¶

Uncertainty in the model parameters or structure due to limited training data. It is reducible as the dataset size increases.

Deep Ensemble Decomposition Derivation¶

For an ensemble of \(M\) models, each outputting a parametric prediction \(p(y \mid x, \theta_m) = \mathcal{N}(\mu_m(x), \sigma^2_m(x))\), the mixture distribution of the ensemble is:

The total predictive mean is \(\bar\mu(x) = \mathbb{E}[y \mid x] = \frac{1}{M}\sum_{m=1}^{M}\mu_m(x)\). Using the Law of Total Variance, the total predictive variance is:

Plugging in the ensemble moments:

This provides a clean separation: aleatoric uncertainty represents the average data noise estimated across ensemble members, while epistemic uncertainty captures model parameter disagreement.

→ See Ensembles for Uncertainty for advanced decomposition methods (e.g., BaseEnsembleModel, SWAG, BNN).

Proper Scoring Rules¶

A scoring rule \(S(F, y)\) is proper if the expected score is minimised when the predicted distribution \(F\) matches the true distribution \(G\):

Continuous Ranked Probability Score (CRPS)¶

The CRPS generalizes Mean Absolute Error (MAE) to probabilistic forecasts:

where \(X, X' \sim F\) are independent and identically distributed copies.

Derivation of Gaussian CRPS¶

For \(F = \mathcal{N}(\mu, \sigma^2)\), let \(Z = \frac{X - \mu}{\sigma} \sim \mathcal{N}(0, 1)\) and \(z = \frac{y - \mu}{\sigma}\). The evaluation reduces to:

1. First Term: Evaluating the expectation \(\mathbb{E}[|Z - z|]\) yields: $\(\mathbb{E}[|Z - z|] = z \bigl(2\Phi(z) - 1\bigr) + 2\phi(z)\)$
2. Second Term: The expected absolute difference between two standard Gaussians \(\mathbb{E}[|Z - Z'|]\) evaluates to \(\frac{2}{\sqrt{\pi}}\) since \(Z - Z' \sim \mathcal{N}(0, 2)\).
3. Combining terms leads to the closed-form implemented in crps_gaussian: $\(CRPS(\mu, \sigma, y) = \sigma \left[ z \bigl(2\Phi(z) - 1\bigr) + 2\phi(z) - \frac{1}{\sqrt{\pi}} \right]\)$

Interval Score¶

For evaluating prediction intervals \([L(x), U(x)]\) at nominal level \((1 - \alpha)\), we use the strictly proper Interval Score:

The score penalizes wide intervals (sharpness) and penalizes boundary violations (calibration) with a scale factor of \(2/\alpha\).

→ See Distribution Metrics for multivariate energy_score.

Conformal Prediction¶

Conformal Prediction (CP) provides a framework for generating prediction intervals with guaranteed coverage under the sole assumption of exchangeability [4, 5].

Given a non-conformity score \(s(x, y)\) (e.g., absolute residual \(\lvert y - \hat{y} \rvert\)), the conformal interval at level \(1-\alpha\) is:

where \(\hat{q}\) is the \(\frac{\lceil(n+1)(1-\alpha)\rceil}{n}\) quantile of calibration scores.

Coverage Guarantee:

→ See Conformal Prediction for SplitConformal, CQR and distributional CP.

Specialized Regression Tasks¶

Continuous Imbalanced Regression (LDS & FDS)¶

When continuous targets \(y\) are highly imbalanced, empirical estimators in sparse regions are unreliable.

Label Distribution Smoothing (LDS): LDS estimates the empirical label density \(p(y)\) and smooths it using a symmetric kernel \(k\): $\(\tilde{p}(y) = \int p(z) k(y, z) dz\)$ Samples are then reweighted by \(w_i \propto 1 / \tilde{p}(y_i)\) to balance the objective function. See LDSLoss.

Feature Distribution Smoothing (FDS): FDS smooths the running mean \(\mu_b\) and covariance \(\Sigma_b\) of model features across adjacent target bins \(b\): $\(\tilde{\mu}_b = \sum_{b'} \omega(b, b') \mu_{b'} \qquad \tilde{\Sigma}_b = \sum_{b'} \omega(b, b') \Sigma_{b'}\)$ Features \(\mathbf{z}\) are then calibrated during training via whitening and recoloring: $\(\tilde{\mathbf{z}} = \tilde{\Sigma}_b^{1/2} \Sigma_b^{-1/2} (\mathbf{z} - \mu_b) + \tilde{\mu}_b\)$ See FeatureDistributionSmoother.

Measurement Error (Errors-in-Variables)¶

Standard OLS assumes \(X\) is measured perfectly. If \(X_{\text{obs}} = X^* + \epsilon\), then OLS estimates are biased toward zero (attenuation bias) [10]. torchregress implements SIMEX and Regression Calibration (RC) to correct this. See Algorithms.

Ordinal Regression¶

For discrete ordered targets, we use the Cumulative Link Model:

where \(\theta_1 < \theta_2 < \dots < \theta_{K-1}\) are learned thresholds. See CumulativeLinkLoss.

References¶