Ensemble Metrics¶

← OOD Metrics | Decision Metrics →

Ensemble metrics evaluate predictive performance and decompose predictive uncertainty across multiple model predictions.

An ensemble is a finite sample from an epistemic distribution over models. Writing member means \(\mu_m(x)\) and (optional) member variances \(\sigma_m^2(x)\) for \(m=1,\dots,M\), the two useful summaries are the ensemble mean and the total predictive variance. The law of total variance gives the decomposition used throughout the library:

The first term is irreducible noise the members agree on; the second is model disagreement. With point-only members the aleatoric term is unavailable and only the disagreement (epistemic) part is identified — an important limitation, not a detail. → See Uncertainty decomposition for the semantics and Ensemble methods for training patterns.

ensemble_statistics¶

Aggregates individual predictions from \(M\) ensemble members \(\{y^{(1)}, \dots, y^{(M)}\}\) to compute the ensemble mean and sample variance:

import torch
from torchregress.metrics.ensemble import ensemble_statistics

predictions = torch.randn(5, 100)  # 5 ensemble members, 100 predictions each
mean, variance = ensemble_statistics(predictions)

uncertainty_decomposition¶

Decomposes total predictive uncertainty into epistemic (model disagreement) and aleatoric (data noise) uncertainty using the Law of Total Variance.

For ensemble members predicting means \(\mu_m(x)\) and aleatoric variances \(\sigma_m^2(x)\), the decomposition is:

• Ensemble mean: $\(\bar{\mu}(x) = \frac{1}{M} \sum_{m=1}^M \mu_m(x)\)$
• Epistemic uncertainty (variance of predicted means): $\(\sigma^2_{\text{epistemic}}(x) = \frac{1}{M} \sum_{m=1}^M (\mu_m(x) - \bar{\mu}(x))^2\)$
• Aleatoric uncertainty (mean of predicted variances): $\(\sigma^2_{\text{aleatoric}}(x) = \frac{1}{M} \sum_{m=1}^M \sigma_m^2(x)\)$
• Total uncertainty: $\(\sigma^2_{\text{total}}(x) = \sigma^2_{\text{epistemic}}(x) + \sigma^2_{\text{aleatoric}}(x)\)$

from torchregress.metrics.ensemble import uncertainty_decomposition

# means: [M, N], variances: [M, N]
uncertainty = uncertainty_decomposition(means, variances)

gaussian_nll_ensemble¶

Computes the Gaussian negative log-likelihood of the targets under the ensembled predictive distribution:

where \(\bar{\mu}_i\) is the ensemble mean and \(\sigma^2_{\text{total}, i}\) is the total uncertainty.

from torchregress.metrics.ensemble import gaussian_nll_ensemble

nll = gaussian_nll_ensemble(means, variances, y_true)

ensemble_interval_bounds¶

Computes symmetric Gaussian prediction intervals at significance level \(\alpha\):

where \(z_{p} = \Phi^{-1}(p)\) is the standard normal quantile.

from torchregress.metrics.ensemble import ensemble_interval_bounds

lower, upper = ensemble_interval_bounds(means, variances, alpha=0.1)

ensemble_interval_metrics¶

Computes the ensembled PICP (empirical coverage) and Winkler interval score for the generated prediction intervals.

from torchregress.metrics.ensemble import ensemble_interval_metrics

metrics = ensemble_interval_metrics(means, variances, y_true, alpha=0.1)

Limitations¶

1. Gaussian approximation: gaussian_nll_ensemble and ensemble_interval_bounds assume the ensemble predictive distribution is Gaussian. For multi-modal ensembles (MDN heads, flow-based members), this approximation can be poor — use sample-based methods instead.
2. Member independence assumed: uncertainty_decomposition treats ensemble members as independent draws from the posterior. Correlated members (e.g., same initialization, shared data) produce underestimated epistemic uncertainty.
3. Minimum ensemble size: With \(M < 5\) members, variance estimates are noisy and epistemic uncertainty is unreliable. For reliable decomposition, use \(M \ge 5\) with diverse initializations.
4. Not a calibration check: The decomposition separates epistemic from aleatoric variance but does not guarantee either component is well-calibrated. Validate calibration separately with Calibration metrics.

Recommendations¶

• Standard ensemble size: Use \(M = 5\) for a good cost-accuracy tradeoff. Increase to \(M = 10\) if epistemic uncertainty is mission-critical.
• Ensure diversity: Random initialization + different data order (shuffle seed per member) is the minimum. For stronger diversity, use different architectures or BatchEnsemble.
• uncertainty_decomposition for heteroscedastic ensembles that output per-member \(\mu_m\) and \(\sigma_m^2\).
• ensemble_statistics for ensembles that output only point predictions \(\hat{y}_m\).
• Compare with conformal: Ensemble intervals are Gaussian approximations. For distribution-free coverage guarantees, wrap with Conformal prediction.

Next steps¶

• Uncertainty decomposition — semantics and contracts for epistemic vs aleatoric uncertainty
• Ensemble methods — Deep Ensembles, BatchEnsemble, and SWAG training patterns
• Calibration metrics — verify that ensemble uncertainty is well-calibrated
• Decision metrics — risk-coverage evaluation of ensemble-based selective prediction

References¶