Interval Metrics¶

← Point Metrics | Distributional Metrics →

Interval metrics evaluate the quality of prediction intervals, focusing on coverage and width properties.

An interval forecast is a pair \([L(x), U(x)]\) with nominal coverage \(1-\alpha\): the promise is \(P(Y \in [L(X), U(X)]) \ge 1-\alpha\) under exchangeability. Two failure modes are possible and they pull in opposite directions: an interval can be too narrow (coverage below nominal) or too wide (high coverage, no information). Coverage alone therefore cannot rank intervals — the trivially infinite interval covers everything. The Winkler/interval score is the standard resolution: it pays for width and charges a penalty \(\propto 1/\alpha\) for each miss, so it is a proper scoring rule for the central interval. This page reports coverage, width, and the composite score together, and conformal methods provide the finite-sample guarantee that motivates the nominal level.

Interval Score (Winkler Score)¶

The Winkler interval score evaluates prediction intervals by rewarding narrow bands and penalizing intervals when observations fall outside them:

where \([L_i, U_i]\) is the predicted \((1 - \alpha)\) interval (e.g., a \(90\%\) interval corresponds to \(\alpha = 0.1\)) and \(y_i\) is the target. A lower score indicates a higher quality interval.

from torchregress.metrics.interval import interval_score

# lower_bound and upper_bound typically represent a 90% prediction interval
score = interval_score(lower_bound, upper_bound, y_true, alpha=0.1)

Prediction Interval Coverage Probability (PICP)¶

PICP measures the proportion of observations that fall within the predicted intervals:

where \(m_i \in \{0, 1\}\) is an optional boolean mask when samples are pre-filtered. Ideally, \(\text{PICP} \approx 1 - \alpha\).

from torchregress.metrics.interval import prediction_interval_coverage_probability

# Calculate basic PICP
picp = prediction_interval_coverage_probability(lower_bound, upper_bound, y_true)

Mean Prediction Interval Width (MPIW)¶

MPIW measures the average width of prediction intervals:

from torchregress.metrics.interval import prediction_interval_coverage_probability
from torchregress.metrics import MeanPredictionIntervalWidth

# Via PICP diagnostics dict
results = prediction_interval_coverage_probability(
    lower_bound, upper_bound, y_true,
    return_diagnostics=True,
)
mpiw = results["mpiw"]

# Or accumulate MPIW across batches with the stateful metric
mpiw_metric = MeanPredictionIntervalWidth()
mpiw_metric.update(lower_bound, upper_bound)
width = mpiw_metric.compute()

See also: MeanPredictionIntervalWidth and prediction_interval_coverage_probability (return_diagnostics=True).

Comprehensive Reporting¶

Interval Metrics Report¶

Compare interval quality across multiple models.

from torchregress.metrics.interval import interval_metrics_report

# Create a dictionary of model predictions
predictions = {
    'model1': {'lower': model1_lower, 'upper': model1_upper},
    'model2': {'lower': model2_lower, 'upper': model2_upper}
}

# Generate comprehensive report
report = interval_metrics_report(predictions, y_true, alpha=0.1)

Advanced Usage¶

Asymmetric Interval Evaluation¶

Evaluate lower vs. upper miss rates to detect interval asymmetry:

coverage = prediction_interval_coverage_probability(
    lower_bound, upper_bound, y_true, alpha=0.1, return_diagnostics=True
)
print(f"Lower miss rate: {coverage['miss_rate_low']}")
print(f"Upper miss rate: {coverage['miss_rate_high']}")

Limitations¶

1. Coverage-width tradeoff: PICP and MPIW must be evaluated together. A model can achieve perfect PICP by predicting infinitely wide intervals. A model with narrow intervals and low PICP is overconfident. Neither metric is sufficient alone.
2. Winkler score sensitivity to \(\alpha\): The interval score penalises misses by a factor of \(2/\alpha\). With \(\alpha = 0.1\) (90% interval), each miss is penalised at \(20\times\) the interval width — a single extreme miss can dominate the score.
3. Symmetric interval assumption: The Winkler score and standard PICP assume symmetric penalties for lower and upper misses. For asymmetric risks (e.g., under-predicting is costlier), use the miss_rate_low and miss_rate_high diagnostics.
4. Not a calibration metric: PICP measures empirical coverage, not calibration. A model with correct average coverage may still be miscalibrated in specific regions. Pair with Calibration metrics.

Recommendations¶

• Always report PICP + MPIW together: Coverage without width is meaningless, and width without coverage is misleading.
• Use return_diagnostics=True: The prediction_interval_coverage_probability function returns miss rates, MPIW, and per-sample flags when diagnostics are enabled.
• For conformal intervals: Conformal prediction guarantees finite-sample coverage. Evaluate with the same interval metrics to confirm the guarantee holds empirically. See Conformal prediction.
• interval_metrics_report consolidates interval score, PICP, MPIW, and miss-rate diagnostics into one call.

Next steps¶

• Point metrics — baseline accuracy metrics to pair with interval quality evaluation
• Calibration metrics — ECE and marginal calibration to check if intervals are honest
• Conformal prediction — coverage-guaranteed intervals with finite-sample validity
• Decision metrics — risk-coverage curves for selective prediction based on interval width

References¶