Core Concepts¶

← Quick Start | User Guide →

This page formalises the vocabulary used throughout torchregress. The treatment is intentionally compact; for derivations see Mathematical Foundations, and for method-by-method guidance see the Method Selection Matrix.

1. Regression as probabilistic inference¶

Given features \(x \in \mathbb{R}^D\) and a continuous target \(y \in \mathbb{R}\), the goal is to model the conditional distribution \(p(y \mid x)\), not a single point \(\hat y\). A point estimate is then recovered as an aspect of that distribution:

for central tendency, \(\tau\)-quantile, and a \(1-\alpha\) prediction interval respectively.

torchregress losses are organised around this view: a DistributionLoss outputs the parameters of a parametric family (or samples from a non-parametric head), and a RegressionLoss is the special case of a distribution with fixed \(\sigma^2\).

2. Uncertainty decomposition¶

Following the Law of Total Variance and Kendall & Gal (2017), we decompose predictive uncertainty into two fundamentally different sources:

2.1 Aleatoric uncertainty (data noise)¶

The irreducible noise in the observation process.

• Homoscedastic — \(\operatorname{Var}(y \mid x) = \sigma^2\) for all \(x\).
• Heteroscedastic — \(\operatorname{Var}(y \mid x) = \sigma^2(x)\) depends on \(x\).
• Cannot be reduced by collecting more data from the same distribution.

Modelled by parametric likelihood heads such as GaussianNLLLoss, BetaNLLLoss, MultivariateGaussianLoss, and non-parametric heads such as MixtureDensityLoss and NormalizingFlowLoss.

2.2 Epistemic uncertainty (model ignorance)¶

The reducible uncertainty due to finite training data or limited model capacity in some region of input space.

• High in out-of-distribution (OOD) regions where the model has not seen similar features.
• High when multiple parameter settings are consistent with the training data (posterior variance in Bayesian models, member disagreement in ensembles, weight-space variance in SWAG).

Modelled by ensembles of likelihood heads (BaseEnsembleModel, HeteroscedasticEnsembleModel, MDNEnsembleModel), Bayesian approximations (BayesianNeuralNetwork, SWAG, MultiSWAG, HeteroscedasticLaplaceRegressor, IVON), or evidential single-pass priors (EvidentialRegressionLoss).

2.3 Why decomposition matters¶

Conformal prediction provides interval coverage guarantees but does not separate aleatoric and epistemic uncertainty. For risk-aware decisions (e.g. defer to human review when epistemic uncertainty is high but accept the prediction when only aleatoric noise is high), the decomposition is essential. See Uncertainty Decomposition for the full taxonomy and the contracts used in torchregress.

3. Proper scoring rules¶

A scoring rule \(S(F, y)\) is proper if \(\mathbb{E}_{y \sim G}\bigl[S(G, y)\bigr] \leq \mathbb{E}_{y \sim G}\bigl[S(F, y)\bigr]\) for all \(F \neq G\) — i.e. it is uniquely minimised in expectation when the predicted distribution \(F\) matches the true distribution \(G\).

torchregress evaluates models with proper scoring rules as the primary accuracy measure; point-error metrics (RMSE, MAE) are reported as secondary summaries.

4. Robustness and influence functions¶

The influence function \(\psi(r) = \frac{d}{dr}\rho(r)\) quantifies the gradient contribution of a residual \(r\). Standard MSE has \(\psi(r) = 2r\) — unbounded — so a single outlier can dominate the gradient. Robust losses bound \(\psi\) as follows:

• Huber — quadratic near zero, linear in the tails.
• Cauchy — \(\psi(r) = 2r / (c^2 + r^2)\), logarithmic growth.
• Tukey biweight — \(\psi(r) = 0\) for \(|r| > c\) (hard rejection).
• Barron — continuous family parametrised by \(\alpha \in (-\infty, 2]\) interpolating between \(L_1\) and \(L_2\).
• AdaptiveRobust — jointly optimises the shape \(\alpha\) alongside the model weights.

Use the Uncertainty vs. error diagnostic to visualise the empirical relationship between predicted \(\sigma\) and realised absolute error; departures from \(y = x\) reveal miscalibration or model misspecification.

5. Calibration vs. sharpness¶

A good probabilistic model must balance two properties:

1. Calibration — the empirical coverage of predicted intervals matches their nominal level. The reliability diagram and the probability integral transform (PIT) are the standard diagnostics.
2. Sharpness — among all calibrated models, the one with the narrowest intervals is preferred (Gneiting & Raftery, 2007).

CRPS balances both. The expected_calibration_error and marginal_calibration_error decompose the miscalibration component.

6. Conformal prediction¶

Conformal prediction is a post-hoc framework that converts any pre-trained model into one with distribution-free coverage guarantees under exchangeability.

Given a non-conformity score \(s(x, y) = |y - \hat y|\) (or a learned score for quantile-based methods), the conformal interval at level \(1 - \alpha\) is

where \(q_{1-\alpha}\) is the \(\lceil (n+1)(1-\alpha) \rceil / n\) empirical quantile of calibration scores. Under exchangeability this guarantees \(P(Y_{n+1} \in \hat C_{1-\alpha}(X_{n+1})) \geq 1 - \alpha\).

Important: conformal prediction provides coverage, not density estimation or uncertainty decomposition. It complements, but does not replace, likelihood-based losses.

7. When to use which (decision rules)¶

References¶

Next steps¶

• Mathematical Foundations — derivations of NLL, CRPS, interval score, and decomposition formulas
• Method Selection Matrix — task-first guidance with capability matrices
• Quick Start — end-to-end workflows for the most common tasks