Choosing Methods by Constraint¶

← Method Selection | Uncertainty Decomposition →

Use this page after the Task-First Method Selection Matrix — i.e. you already know the task category (heteroscedastic, multimodal, measurement-error, …) and now need to choose based on the operational constraints of the deployment: latency, coverage guarantees, decomposition, calibration, OOD robustness.

This page is audit-driven: every recommendation is paired with the failure mode that triggers escalation to a more complex method. For evidence-grade claim boundaries, pair this page with the Real-Data Recommendation Guide.

1. The constraint taxonomy¶

Five orthogonal axes determine the feasible set of methods:

1. Latency / compute budget — inference cost per query.
2. Coverage guarantee — is a finite-sample coverage \(\geq 1 - \alpha\) required?
3. Uncertainty decomposition — must the model separate aleatoric and epistemic components?
4. Multimodality — is the conditional \(p(y \mid x)\) known or expected to have multiple modes?
5. Shift / OOD — is the test distribution exchangeable with the training distribution?

Each axis admits a finite set of answers; the cross-product of those answers is the feasible region for a deployment.

graph LR
    A["Your — task (see — method-selection.md)"] --> B{"Latency budget?"}
    B -->|"< 1 — ms"| C["MC — Dropout — or — single — SWAG"]
    B -->|"1–10 — ms"| D["Single — SWAG / IVON — or — small — ensemble"]
    B -->|"> 100 — ms"| E["Full — ensemble — or — BNN"]
    A --> F{"Coverage guarantee?"}
    F -->|"Yes"| G["Conformal (split / CQR)"]
    F -->|"No"| H["Likelihood — only"]
    A --> I{"Decompose aleatoric / epistemic?"}
    I -->|"Yes"| J["Heteroscedastic — ensemble — or — BNN"]
    I -->|"No"| K["Single — likelihood — head"]

2. Constraint profiles¶

2.1 Low latency / limited compute¶

The objective is the smallest model that satisfies the accuracy constraint.

Recommended order (start with the simplest, escalate only if necessary):

1. Single robust point head. WeightedHuberLoss or WeightedMSELoss. Inference is one forward pass; training is the cheapest of the alternatives.
2. Single heteroscedastic head. GaussianNLLLoss with a \([\mu, \log\sigma^2]\) output. Same inference cost as a point head; doubles the parameter count of the final layer.
3. MC Dropout (MCDropoutWrapper). \(N\) forward passes per inference; cost scales with \(N\).
4. Single SWAG or IVON — single forward pass with weight perturbation sampled from a low-rank posterior. Cost is roughly that of one forward pass plus the cost of a low-rank perturbation sample.
5. Small deep ensemble (\(M = 3\)–\(5\)). Cost scales linearly with \(M\).

Tradeoffs:

• Deep ensembles are the strongest epistemic signal per member but scale linearly with \(M\) in both training and inference.
• SWAG and IVON are single-model alternatives but require careful protocol tuning; calibration depends on the SGD trajectory length and the rank of the low-rank posterior.
• Flow-based methods (NormalizingFlowLoss) are the most expressive density estimators but add the most implementation / runtime complexity. They are not the right default for low-latency.

2.2 Coverage guarantees vs decomposition¶

These are orthogonal contracts. A model can satisfy one, both, or neither.

• Coverage guarantee: conformal prediction. A conformal predictor wraps any base regressor and produces intervals with \(P(Y_{n+1} \in \hat C_{1-\alpha}(X_{n+1})) \geq 1 - \alpha\) under exchangeability.
• Decomposition: aleatoric + epistemic split. Requires a probabilistic head and an ensemble, BNN, or evidential prior.

If you need coverage only: use SplitConformal or CQR on top of a strong base regressor. Report empirical coverage, interval width, and coverage under shift.

If you need decomposition only: use a heteroscedastic ensemble (HeteroscedasticEnsembleModel, HeteroscedasticBatchEnsembleModel, HeteroscedasticBNN, MDNEnsembleModel). Report aleatoric and epistemic variance components.

If you need both: use a probabilistic head (heteroscedastic Gaussian, MDN, flow), wrap it in a conformal calibrator, and report the decomposition in addition to the coverage. The two contracts are complementary, not redundant.

Critical caveat: conformal prediction gives a coverage guarantee, not uncertainty decomposition. Methods such as SplitConformal, CQR, UACQR, DensityConformal, and MonteCarloConformal provide calibrated intervals, but they do not separate aleatoric and epistemic uncertainty.

2.3 Multimodal / non-Gaussian targets¶

When the conditional \(p(y \mid x)\) is known or expected to be multimodal, a single Gaussian head is structurally inadequate. Recommended order:

1. MDN (MixtureDensityLoss). Good first multimodal baseline; tune the number of components \(J\) by held-out CRPS.
2. Quantile / expectile methods — MultiQuantileLoss with \([0.05, 0.1, 0.5, 0.9, 0.95]\). Suitable when intervals (not the full density) are the deliverable.
3. Normalizing flows (NormalizingFlowLoss) — when MDN or Gaussian families miss the structure. Heaviest in compute and implementation.

Evaluation: do not compare multimodal heads with NLL alone; use CRPS, energy score, and a PIT histogram. NLL favours density-consistent heads; CRPS evaluates the distribution and is robust to over-dispersion.

2.4 Noisy features / measurement error¶

When the input \(x\) is itself noisy and that noise is part of the problem statement, use error-in-variables (EIV) losses. These require a different call pattern than standard supervised losses:

# Standard supervised loss
loss = loss_fn(model(x), y)

# EIV loss — the model is called inside the loss
loss = loss_fn(x_obs, y_obs, model=model)

Recommended order:

1. Establish a baseline. WeightedMSELoss or WeightedHuberLoss with the noisy input \(x_{\text{obs}}\). Quantify the attenuation bias (regression coefficients shrink toward zero).
2. Add a simple EIV loss — FunctionalEIVLoss or OrthogonalDistanceRegressionLoss (ODR) — and compare the regression coefficients to the baseline.
3. Add RegressionCalibration (RC) if \(\Sigma_u\) (the input noise covariance) is known or estimable. RC is the closed-form bias correction.
4. Add SIMEX if \(\Sigma_u\) is unknown. SIMEX adds synthetic noise and extrapolates the attenuation to zero.
5. Add LatentNN (latent-input regression) for the heaviest case where the latent true input is jointly inferred.

Practical note: the EIV losses require a model argument inside the loss. Do not call model(x_obs) yourself and pass the result; pass x_obs and y_obs to the loss and let it manage the inner forward pass.

2.5 Calibration and OOD robustness¶

For deployment-facing reliability, combine three categories of diagnostic:

• Calibration metrics — expected_calibration_error, marginal_calibration_error, and reliability / PIT plots.
• OOD metrics — mahalanobis_distance, typicality_score, entropy, density-based signals. Do not rely on a single OOD score in isolation; ensemble multiple scores.
• Decision metrics — risk-coverage curves, rejection policies for selective prediction.

A single calibration number (e.g. ECE) is insufficient. Pair it with the empirical coverage and the Spearman correlation between \(\hat\sigma\) and realised error. Compare multiple signals and validate on your shift scenarios.

3. Cost profiles¶

Use this table to rule out methods whose cost profile exceeds the deployment budget. Cost in absolute terms depends on \(D\), \(N\), and the head size; the table shows relative cost.

4. Evidence-driven escalation¶

Escalate from simpler to more complex methods only when the simpler method fails on a metric that matters for the deployment. The metrics that matter differ by task:

• Coverage: required \(\geq 1 - \alpha\). Failure = PICP \(< 1 - \alpha\) on a calibration set.
• Calibration: required \(\leq \epsilon\) ECE. Failure = ECE above threshold.
• Tail error: required \(\leq \delta\) on a target quantile (e.g. 99%). Failure = empirical tail error above threshold.
• OOD selectivity: required \(\geq \rho\) Spearman between \(\hat\sigma\) and absolute error. Failure = correlation below threshold.
• Multimodal fit: required low energy score on multimodal data. Failure = energy score above threshold.

This discipline keeps comparisons credible and reduces accidental overfitting to fashionable methods.

5. The evidence path¶

For decision-grade comparisons, run the examples below first. They are written to be auditable, share seeds, and emit machine-readable artefacts.

• examples/comprehensive_comparison.py
• examples/loss_comparison.py --suite comprehensive
• examples/imbalanced_regression.py
• examples/conformal_regression_example.py --mode sweep
• examples/normalizing_flows_multitarget.py

For performance guardrails, use:

• reports/benchmark_thresholds/cpu/smoke.json
• reports/benchmark_thresholds/cpu/sweep.json
• tools/benchmark_smoke.py
• tools/benchmark_report_summary.py

For capability-flag-based shortlisting, use torchregress.method_catalog:

import torchregress as tr

# Example: methods that support decomposition and are at least "Available"
rows = tr.method_catalog.list_methods(
    capability_filters={"decomposition": "yes"},
    maturity=("Core", "Strong", "Available"),
)
for row in rows:
    print(row["name"], row["family"], row["maturity"], row["task_tags"])

6. Decision rules¶

References¶

Next steps¶

• Method Selection Matrix — task-first capability matrix.
• Uncertainty Decomposition — full taxonomy and method semantics.
• Practical Usage — concrete loss recipes and code.
• Multi-Target Regression — joint multi-target modelling.