Conformal Prediction¶

Conformal Prediction (CP) is a modern framework for uncertainty quantification that provides mathematically guaranteed prediction intervals for any regression model.

Why Use Conformal Prediction?¶

Traditional uncertainty estimates (e.g., from Gaussian NLL) are often miscalibrated. They rely on strong parametric assumptions (like normality) that rarely hold in practice. Conformal prediction fixes this post-hoc.

Key Advantages¶

• Distribution-Free: No assumptions about the shape of the noise or the model's error distribution.
• Model-Agnostic: Works with any neural network, tree-based model, or ensemble.
• Finite-Sample Validity: The coverage guarantee is exact, even for small calibration sets.
• No Retraining: Apply CP to your existing pre-trained models.

Core Workflow¶

The conformal workflow consists of three distinct phases:

1. Training (Standard)¶

Train your regression model on a training set \(\mathcal{D}_{\text{train}}\) as you normally would.

2. Calibration (Offline)¶

Evaluate the model on a held-out calibration set \(\mathcal{D}_{\text{cal}}\) to compute "non-conformity scores" \(s_i\). These scores measure how "unusual" an observation is relative to the model's prediction.

from torchregress.losses import SplitConformal

# Target 90% coverage (alpha = 0.1)
cp = SplitConformal(alpha=0.1)
cp.calibrate(y_pred_cal, y_cal)

3. Prediction (Inference)¶

At test time, the conformal predictor uses the calibrated scores to produce an interval \(\hat{C}(x)\) around the model's point prediction.

lower, upper = cp.predict_interval(y_pred_test)

Method Selection Matrix¶

Mathematical Deep Dive¶

Non-Conformity Scores¶

The heart of CP is the non-conformity score \(s(x, y)\), which quantifies how poorly a model prediction \(\hat{y}\) fits a true value \(y\). Common scores include:

• Absolute Residual: \(s_i = |y_i - \hat{y}_i|\) (used in SplitConformal)
• Quantile Score: \(s_i = \max(\hat{q}_{\text{lo}} - y_i, y_i - \hat{q}_{\text{hi}})\) (used in CQR)

The Conformal Quantile¶

Given \(n\) calibration scores \(\{s_1, \dots, s_n\}\), we compute the \((1-\alpha)(1+1/n)\) quantile:

The prediction interval is then:

Equivalently, \(\hat{q}\) is the \(k\)-th smallest score with \(k = \lceil (n+1)(1-\alpha) \rceil\).

Marginal vs. Conditional Coverage¶

Standard CP guarantees marginal coverage on average over all possible test points: $\(P\bigl(Y_{n+1} \in \hat{C}(X_{n+1})\bigr) \geq 1 - \alpha\)$

To achieve conditional coverage (validity within local regions of the feature space, i.e., \(P(Y \in \hat{C}(x) \mid X=x) \geq 1-\alpha\)), torchregress supports:

1. Mondrian Conformal Prediction: Calibrate independently for discrete subgroups or categories.
2. Normalized/Adaptive Scores: Divide residuals by a difficulty estimate \(\hat{\sigma}(x)\) (e.g., in UACQR or MonteCarloConformal), adapting interval width locally.
3. Locally Valid Conformal Prediction: Apply kernel-weighted local density estimators in the embedding space (e.g., in LocalConformal).

The Exchangeability Assumption & OOD Limits¶

The mathematical guarantees of conformal prediction rely fundamentally on exchangeability.

Out-of-Distribution (OOD) Failures¶

If test samples violate exchangeability (e.g., due to concept drift, temporal autocorrelation, or shifted covariates), the empirical quantiles calculated on \(\mathcal{D}_{\text{cal}}\) will not reflect the test distribution. This leads to under-coverage (actual coverage drops below \(1 - \alpha\)).

Mitigating Covariate Shift¶

When the feature distribution shifts (\(P_{\text{test}}(X) \neq P_{\text{cal}}(X)\)) but the label distribution condition holds (\(P_{\text{test}}(Y \mid X) = P_{\text{cal}}(Y \mid X)\)), torchregress allows passing importance weights \(w(x) = \frac{p_{\text{test}}(x)}{p_{\text{cal}}(x)}\). Instead of a uniform quantile, the threshold \(\hat{q}\) is computed by solving the weighted quantile equation: $\(\sum_{i=1}^n \tilde{p}_i \mathbb{I}(s_i \leq s) \geq 1 - \alpha\)$ where the normalized weights are: $\(\tilde{p}_i = \frac{w(X_i)}{\sum_{j=1}^n w(X_j) + w(X_{n+1})}, \quad \tilde{p}_{n+1} = \frac{w(X_{n+1})}{\sum_{j=1}^n w(X_j) + w(X_{n+1})}\)$

and the test-point mass \(\tilde{p}_{n+1}\) sits at \(s = +\infty\) (Tibshirani et al., 2019): if \(\sum_{i} \tilde{p}_i \mathbb{I}(s_i \leq s)\) never reaches \(1-\alpha\), then \(\hat{q} = +\infty\). Only relative weights matter (the procedure is invariant to rescaling \(w\)). By default the test point receives the mean calibration weight; pass the test points' own density ratios with predict_interval(..., test_weights=w_test) (same scale as the calibration weights) to get the per-point thresholds of Tibshirani et al., which is what the covariate-shift guarantee requires:

cp = SplitConformal(alpha=0.1)
cp.calibrate(y_pred_cal, y_cal, weights=w_cal)               # w(x) = p_test(x) / p_cal(x)
lower, upper = cp.predict_interval(y_pred_test, test_weights=w_test)

Coordinate-wise vs. Joint Multi-Target Conformal¶

For multi-output regression (\(\mathbf{Y} \in \mathbb{R}^D\)), calibrating each coordinate independently at level \(1-\alpha\) (coordinate-wise conformal prediction) does not guarantee joint coverage of the vector.

• Coordinate-wise Coverage: Guarantees \(P(Y_{d, n+1} \in \hat{C}_d(X_{n+1})) \geq 1 - \alpha\) for each dimension \(d\) individually.
• Joint Coverage (Bonferroni Bound): To guarantee that the entire true vector \(\mathbf{Y}_{n+1}\) falls inside the hyperrectangle \(\prod_{d=1}^D \hat{C}_d(X_{n+1})\) with probability at least \(1 - \alpha\), we must apply the Bonferroni correction, calibrating each individual dimension at level \(1 - \alpha / D\): $\(P\bigl(\mathbf{Y}_{n+1} \in \prod_{d=1}^D \hat{C}_d(X_{n+1})\bigr) \geq 1 - \sum_{d=1}^D \frac{\alpha}{D} = 1 - \alpha\)$
• Joint Conformal Regions (Mahalanobis / Ellipsoidal): Rather than coordinate-wise hyperrectangles (which grow excessively wide under Bonferroni correction for large \(D\)), joint approaches map vector errors into a single scalar non-conformity score (e.g., Mahalanobis distance \(s_i^2 = (\mathbf{y}_i - \hat{\mathbf{y}}_i)^\top \mathbf{\Sigma}^{-1} (\mathbf{y}_i - \hat{\mathbf{y}}_i)\)). This yields an ellipsoidal joint confidence region that is significantly tighter when dimensions are highly correlated. (Note: MultiTargetConformal in torchregress performs coordinate-wise calibration; if joint coverage is needed, adjust the input alpha using the Bonferroni correction \(\alpha/D\)).

Best Practices¶

References¶

Next steps¶

Continue with conformal prediction:

• Predictors and Scores — per-class API reference with parameter tables for SplitConformal, CQR, UACQR, and all variants
• Distributional Conformal — CDF/density-based methods (CTI, DistributionalConformal)
• Test-Time Shift-Aware Conformal — maintain coverage when the test distribution shifts
• Conformal Regression Example — runnable comparison of all conformal predictors