Calibration Metrics¶

← Distributional Metrics | OOD Metrics →

Calibration metrics evaluate whether predicted probabilistic distributions or intervals accurately reflect the true uncertainty in the data.

Calibration is the honesty half of forecast quality. For a predictive CDF \(F_i\), the probability integral transform \(u_i = F_i(y_i)\) should be uniform on \([0,1]\) when the forecast is correct; equivalently, the empirical frequency of \(y_i \le F_i^{-1}(q)\) should equal \(q\) at every level \(q\). Calibration metrics measure the distance from that ideal, separately from sharpness (which the distribution metrics reward). They are necessary but not sufficient: the constant forecast $F = $ training marginal is often well calibrated and useless, which is why calibration is always reported with a score such as CRPS.

→ See Calibration methods, Visualization diagnostics, and the calibration API.

Expected Calibration Error (ECE)¶

ECE measures the discrepancy between predicted quantile levels and their empirical observed coverage:

where \(\mathcal{Q} = \{q_1, \dots, q_Q\}\) is the set of predicted quantile levels (e.g. \([0.1, 0.5, 0.9]\)), and \(\hat{p}(q)\) is the observed proportion of targets that fall below the predicted \(q\)-quantile \(\hat{y}_{i, q}\):

The Maximum Calibration Error (MCE) is the maximum absolute discrepancy across quantiles:

from torchregress.metrics import expected_calibration_error

# Using a dictionary of quantile predictions
quantiles = {0.1: q10_pred, 0.5: q50_pred, 0.9: q90_pred}

# Calculate basic ECE metrics
ece_metrics = expected_calibration_error(quantiles, y_true)
print(f"Mean Absolute Calibration Error: {ece_metrics['mean_absolute_calibration_error']}")
print(f"Maximum Calibration Error: {ece_metrics['maximum_calibration_error']}")

Marginal Calibration Error (MCE)¶

Marginal Calibration Error evaluates how well the predictive distribution's marginal ECDF matches the empirical ECDF of true observations.

For \(B\) bins spanning the range of true and predicted targets with edges \(b_1, \dots, b_{B+1}\), the observed target CDF at bin edge \(k\) is:

Given \(S\) Monte Carlo predictions per test point \(\hat{y}_i^{(s)}\), the mean predicted CDF is:

The Marginal Calibration Error is the average absolute CDF difference over bins:

The Maximum MCE is:

from torchregress.metrics import marginal_calibration_error

# y_pred_samples has shape [n_samples, batch_size]
mce_metrics = marginal_calibration_error(y_pred_samples, y_true)
print(f"Marginal Calibration Error: {mce_metrics['marginal_calibration_error']}")
print(f"Maximum MCE: {mce_metrics['maximum_marginal_calibration_error']}")

Bias¶

Measures the mean signed prediction error (mean bias):

from torchregress.metrics import bias

mean_bias = bias(y_pred, y_true)

Calibration Score¶

A convenience calibration score for Gaussian predictions. It builds a fine grid of 19 quantile levels from pred_mean and pred_std and computes the quantile ECE on that grid:

where \(z_q = \Phi^{-1}(q)\) is the inverse CDF of the standard normal distribution.

from torchregress.metrics import calibration_score

# Get ECE computed over a fine normal quantile grid
score = calibration_score(y_true, pred_mean, pred_std)

Comprehensive Reporting¶

Calibration Metrics Report¶

Generate a comprehensive report of calibration metrics.

from torchregress.metrics import calibration_metrics_report
from torch.distributions import Normal

# Using a PyTorch distribution
dist = Normal(mean_pred, std_pred)
report = calibration_metrics_report(dist, y_true)

Limitations¶

1. ECE bin sensitivity: Quantile-based ECE depends on the set of quantile levels \(\mathcal{Q}\). A model that is well-calibrated at \(\{0.1, 0.5, 0.9\}\) may be poorly calibrated at \(\{0.05, 0.25, 0.75, 0.95\}\). Use a fine grid (19+ levels) for thorough evaluation.
2. Marginal vs conditional: Both ECE and MCE measure marginal (average) calibration. They do not detect systematic miscalibration that varies with \(X\) — a model can have zero ECE while being overconfident in some regions and underconfident in others.
3. Bias conceals variance: The bias metric (mean signed error) can be near zero while the model has large but symmetric errors. Always pair bias with RMSE or MAE.
4. Calibration score assumes Gaussian: calibration_score constructs quantiles assuming \(\mathcal{N}(\mu, \sigma^2)\). For non-Gaussian predictive distributions (MDN, flows), use expected_calibration_error with model-generated quantiles instead.

Recommendations¶

• Default workflow: Report ECE (quantile-based), bias, and a PIT histogram (plot_pit_histogram from torchregress.viz). These three together diagnose most calibration issues.
• For fine-grained calibration: Use calibration_score with 19 quantile levels to detect miscalibration at specific probability regions.
• Post-hoc correction: If calibration is poor, apply Variance Temperature Scaling or Isotonic Calibration before deployment.
• Layer with distributional metrics: Calibration metrics tell you if predicted probabilities are reliable. Distributional metrics (CRPS, NLL) tell you if the predictive distribution is sharp. Both are needed. See Distributional metrics.
• calibration_metrics_report consolidates all calibration metrics into one call.

Next steps¶

• Distributional metrics — CRPS and PIT diagnostics that evaluate the full predictive distribution
• Interval metrics — PICP and Winkler score for interval-focused evaluation
• Post-hoc calibration — fix miscalibration with temperature scaling or isotonic regression
• Visualization diagnostics — reliability diagrams and PIT histograms

Simulation-based calibration (SBC) and TARP¶

The calibration metrics above compare a predictive distribution with observed targets. In simulation-based inference (SBI) — approximate Bayesian computation, neural posterior estimation, validating an MCMC or variational inference implementation, or inverting a forward model — there is a prior question: does the inference algorithm return the posterior that actually generated the data? When the data are simulated from a known prior and likelihood, the ground-truth parameters are known by construction, and posterior calibration can be tested directly. SBC and TARP are the standard diagnostics for that setting.

Both diagnostics take posterior draws of shape [N, S, D]: N simulated observations, S posterior draws per observation, D parameter dimensions. They do not test whether the assumed prior/likelihood model matches reality — that is a modelling question. They test whether the inference algorithm is calibrated for the model you gave it.

SBC ranks¶

For every simulated observation \(n\) and parameter dimension \(d\), SBC (Cook et al. 2006; Talts et al. 2018) computes the rank of the true value among the posterior draws, crediting exact ties with half a draw:

For a calibrated posterior the true value is exchangeable with the posterior draws, so the rank is discrete-uniform on the \(S+1\) possible values:

Deviations are diagnosed from the rank histogram: a U-shape means the posterior is too narrow (over-confident), a central peak means it is too wide (under-confident), and a monotone skew indicates a bias. Uniformity is quantified per dimension with a chi-square test on \(B\) integer bins over \([0, S+1]\):

where \(O_{db}\) are the observed counts, \(w_b\) is the number of integer rank values in bin \(b\) (\(w_b = 1\) for the default \(S+1\) one-bin-per-rank histogram), and \(X^2_d \sim \chi^2_{B-1}\) under the null hypothesis.

TARP coverage¶

The TARP test (Lemos et al. 2023) builds random credible regions from a reference point \(\tilde{\theta}_n\) and the posterior draws,

and defines the region as the ball of radius equal to the \(\alpha\)-quantile \(q_n(\alpha)\) of the distances. The empirical coverage curve is

For a calibrated posterior, and for reference points that are independent of the true parameters and of the posterior draws, \(\mathbb{E}[\hat{c}(\alpha)] = \alpha\) at every nominal level. Unlike SBC, TARP tests the joint posterior — dispersion and correlation structure included — so it is sensitive to failures a per-parameter rank test cannot see. Systematic deviations from the diagonal identify the failure mode: over-confident posteriors typically under-cover at high nominal levels (while over-covering at low levels), under-confident posteriors show the opposite pattern, and biased posteriors produce asymmetric curves. The last two lines of the example below contrast a calibrated and an over-confident posterior.

Runnable example¶

import math

import torch

from torchregress.metrics.sbi import (
    sbc_rank_histogram,
    sbc_uniformity_pvalue,
    simulation_based_calibration_ranks,
    tarp_coverage,
)

torch.manual_seed(0)
n_obs, n_samples, n_dims = 2000, 100, 2

# Simulate from an exact Gaussian posterior: prior theta ~ N(0, 1),
# likelihood y | theta ~ N(theta, 1), so the posterior is N(y / 2, 1 / 2).
truth = torch.randn(n_obs, n_dims)
y = truth + torch.randn(n_obs, n_dims)
posterior_mean = 0.5 * y
posterior_scale = math.sqrt(0.5)
calibrated = posterior_mean.unsqueeze(1) + posterior_scale * torch.randn(
    n_obs, n_samples, n_dims
)

ranks = simulation_based_calibration_ranks(calibrated, truth)
histogram = sbc_rank_histogram(ranks)
uniformity = sbc_uniformity_pvalue(ranks)
print("calibrated min p-value:", uniformity["min_p_value"])
print("calibrated edge-bin counts:", histogram["counts"][:, [0, -1]].tolist())

# TARP reference points: drawn independently from the prior.
reference_points = torch.randn(n_obs, n_dims)
calibrated_tarp = tarp_coverage(calibrated, truth, reference_points)
print("calibrated TARP error:", calibrated_tarp["tarp_error"])

# Over-confident posterior: same posterior mean, half the standard deviation.
overconfident = posterior_mean.unsqueeze(1) + 0.5 * posterior_scale * torch.randn(
    n_obs, n_samples, n_dims
)
overconfident_ranks = simulation_based_calibration_ranks(overconfident, truth)
overconfident_uniformity = sbc_uniformity_pvalue(overconfident_ranks)
overconfident_tarp = tarp_coverage(overconfident, truth, reference_points)
print("over-confident min p-value:", overconfident_uniformity["min_p_value"])
print("over-confident TARP error:", overconfident_tarp["tarp_error"])

With this fixed seed the calibrated posterior gives a minimum p-value around 0.36 and a TARP error of about 0.02, while the over-confident posterior gives a p-value that underflows to zero and a TARP error of about 0.24.

Caveats¶

SBC and TARP references¶

→ See the metrics catalogue, distribution metrics for PIT-based predictive calibration, and the metrics API for the function signatures.

References¶