Prediction-Powered Inference (PPI)¶

← Semi-Supervised | Visualization →

PPI is a statistical framework for performing valid inference using a combination of a small set of gold-standard labels and a large set of machine learning predictions. It produces confidence intervals that are (1) narrower than using gold labels alone and (2) statistically valid even if the ML model is arbitrarily biased.

Background and Motivation¶

The Problem¶

Suppose you want to estimate a population parameter \(\theta\) (a mean, a quantile, or a regression coefficient). You have two data sources:

Three naive approaches all have critical flaws:

The PPI Solution¶

PPI uses the large prediction set for precision and the small gold set for bias correction. The rectified estimator is:

The first term uses all \(N\) samples for precision. The bias correction (middle two terms) cancels the model's systematic error, estimated on the labelled subset.

Core Assumptions¶

PPI's validity rests on a small number of assumptions:

Available Estimators¶

Population Mean¶

from torchregress.inference import PPIConfig, ppi_mean_ci

ci = ppi_mean_ci(
    y_labeled=y_gold,
    pred_labeled=f_hat_gold,
    pred_unlabeled=f_hat_all,
    config=PPIConfig(alpha=0.05),
)
print(f"95% CI: [{ci['ci_lower']:.4f}, {ci['ci_upper']:.4f}]")

Population Quantile¶

from torchregress.inference import PPIConfig, ppi_quantile_ci

ci = ppi_quantile_ci(
    y_labeled=y_gold,
    pred_labeled=f_hat_gold,
    pred_unlabeled=f_hat_all,
    q=0.5,  # median
    config=PPIConfig(alpha=0.05),
)

The quantile estimator inverts the rectified CDF \(\hat F(\theta) = \frac{1}{N}\sum_j 1\{\tilde f_j \le \theta\} + \frac{1}{n}\sum_i \big(1\{Y_i \le \theta\} - 1\{f_i \le \theta\}\big)\): the estimate is \(\inf\{\theta : \hat F(\theta) \ge q\}\) and the CI collects the \(\theta\) whose \(|\hat F(\theta) - q|\) is within \(z_{1-\alpha/2}\) standard errors [1]. The interval is analytic, so n_boot and seed are not used.

OLS Coefficients¶

from torchregress.inference import PPIConfig, ppi_ols_ci

ci = ppi_ols_ci(
    x_labeled=x_gold, y_labeled=y_gold,
    x_unlabeled=x_all, pred_labeled=f_hat_gold, pred_unlabeled=f_hat_all,
    config=PPIConfig(alpha=0.05),
)

Calibrated PPI for the mean¶

Chen et al. (Calibeating Prediction-Powered Inference, arXiv:2604.21260) study post-hoc calibration of a fixed scalar score \(m(X)\) on the labeled sample before semisupervised mean estimation. Torchregress implements their linearly calibrated mean (Section 3.3): fit an affine map by ordinary least squares on labeled pairs,

Then apply the same rectified mean as ppi_mean_ci, but with \(m_n^\star\) in place of \(m\):

Bootstrap percentile intervals refit \((\hat a, \hat b)\) on each labeled bootstrap replicate (and draw independent unlabeled bootstrap rows), which accounts for calibration uncertainty. The paper also analyzes isotonic calibration theoretically; we omit it here in favor of the lightweight affine map (pure PyTorch), which they relate to prognostic-score style adjustment and to PPI++ at first order.

from torchregress.inference import PPIConfig, ppi_calibrated_mean_ci

ci_cal = ppi_calibrated_mean_ci(
    y_labeled=y_gold,
    pred_labeled=f_hat_gold,
    pred_unlabeled=f_hat_all,
    config=PPIConfig(alpha=0.05, n_boot=2000, seed=0),
)

Inference vs prediction (and “better on all metrics”)¶

Earlier guidance distinguished two kinds of follow-up work:

There is no single procedure that simultaneously maximizes every metric (interval width vs coverage vs bias vs sharpness vs conditional validity). What you can do is separate estimands and use the right tool per question:

Conformalizing in the strict sense means building nonconformity scores on labeled data and extrapolating intervals to new predictions — it does not replace PPI’s job for \(\mathbb{E}[Y]\), but it complements it when you also care about per-unit coverage.

Recommended composition¶

1. Split the labeled data (at least two folds): one fold fits affine calibration (or any post-hoc map), another supports PPI bias correction together with unlabeled scores, and optionally a third (or reuse a fold carefully) feeds split conformal calibration.
2. Use ppi_calibrated_mean_ci when you are comfortable refitting \((\hat a,\hat b)\) on every labeled bootstrap replicate (often good variance when \(n\) is not tiny).
3. Use SplitConformal on residuals \(|Y - \hat Y|\) where \(\hat Y\) is the same post-calibrated score you will deploy at test time, calibrated only on data not used to cherry-pick that map if you need clean marginal guarantees.

A minimal end-to-end sketch is in examples/ppi_mean_plus_split_conformal.py.

Efficiency and When to Expect Gains¶

The precision gain from PPI depends on the predictive quality of the model:

Complete Example¶

import torch
from torchregress.inference import ppi_mean_ci, ppi_diagnostics

# Setting: estimating mean patient recovery time
# Gold labels: expert-assessed recovery (expensive, n = 150)
# ML predictions: automated scoring (cheap, N = 10,000)
torch.manual_seed(42)

# Gold-standard labels
y_gold = 14.0 + 3.0 * torch.randn(150)  # days

# ML predictions on gold set (correlated but biased)
f_hat_gold = y_gold + 1.5 + 0.8 * torch.randn(150)  # bias = +1.5 days

# ML predictions on full population
f_hat_all = 15.5 + 3.0 * torch.randn(10_000)

# --- Gold-only CI (wide) ---
gold_mean = y_gold.mean()
gold_se = y_gold.std() / (150 ** 0.5)
print(f"Gold-only 95% CI: [{gold_mean - 1.96*gold_se:.2f}, {gold_mean + 1.96*gold_se:.2f}]")
print(f"  Width: {2 * 1.96 * gold_se:.2f} days")

# --- PPI CI (narrower) ---
from torchregress.inference import PPIConfig
ci = ppi_mean_ci(y_gold, f_hat_gold, f_hat_all, config=PPIConfig(alpha=0.05))
print(f"PPI      95% CI: [{ci['ci_lower']:.2f}, {ci['ci_upper']:.2f}]")
print(f"  Width: {ci['ci_upper'] - ci['ci_lower']:.2f} days")

# Diagnostics
diag = ppi_diagnostics(y_gold, f_hat_gold, f_hat_all)

Comparison with Related Approaches¶

Strengths and Limitations¶

References¶

Likelihood-Ratio Posterior Reweighting (NRE)¶

The same package also provides utilities for a different inference pattern: a model estimates the likelihood ratio between hypotheses rather than a population parameter.

Let \(r(x,\theta) = p(x \mid \theta)/p_{\mathrm{ref}}(x)\), for example the output of NeuralRatioLoss or of a contrastive flow. For a finite candidate bank \(\{\theta_k\}_{k=1}^{K}\) drawn from a proposal \(q(\theta)\):

When the candidates are drawn from the target prior (\(q=p\)) the corrections cancel and the posterior weights are the normalized ratios. The maximum-likelihood estimate is \(\arg\max_k \log p(x \mid \theta_k)\); adding \(\log p(\theta_k)\) gives the MAP estimate, and the MLE itself does not depend on the training label/simulation prior (the \(p(x)\) factor is constant in \(\theta\)).

See the NRE loss guide for training the ratio model itself.

Limitations¶

1. MCAR labelling required: PPI assumes labelled samples are Missing Completely At Random. If labelling is non-random (e.g., only easy examples get labels), the bias correction is invalid. Extensions for MNAR exist but require additional modelling.
2. No gain with uninformative predictions: If the ML model has \(R^2 \approx 0\), PPI provides no precision gain over gold-only inference. Use ppi_diagnostics to assess predictive quality before committing.
3. External model requirement: The prediction model must be trained on data separate from the internal dataset used for inference. Training and evaluating on the same data introduces overfitting bias that invalidates confidence intervals.
4. Bootstrap cost: Bootstrap-based CIs (ppi_calibrated_mean_ci, PPI++) refit nuisance parameters on each bootstrap replicate. For large \(n\) or many bootstraps, this adds non-trivial runtime.
5. Not a drop-in replacement for conformal prediction: PPI targets population-level estimands (means, quantiles, regression coefficients). For per-unit prediction intervals, use the Conformal prediction tools.

Recommendations¶

• Check \(R^2\) first: PPI is most valuable when \(R^2 > 0.5\) and \(N/n > 10\). Run ppi_diagnostics to estimate the expected CI width reduction before investing in a full PPI pipeline.
• Use ppi_calibrated_mean_ci for mismatched scores: When the ML score tracks \(Y\) but has the wrong slope or intercept (common for models trained on a different population or loss function), calibrated PPI often improves over plain ppi_mean_ci.
• Split your data carefully: Hold out one fold for affine calibration, another for PPI bias correction, and optionally a third for split-conformal calibration. Never reuse the same fold for multiple stages.
• Complement with conformal prediction: Use PPI for population-level inference (e.g., "what is the mean recovery time?") and SplitConformal for per-unit prediction intervals ("what is the recovery time for this patient?"). See examples/ppi_mean_plus_split_conformal.py.
• Degrade gracefully with PPI++: When model quality is uncertain, use PPI++ (via power=1.0) — it asymptotically never increases variance over gold-only inference.

Next Steps¶

• Calibration Metrics — evaluate your model's predictive quality for PPI
• Causal Inference — doubly-robust estimation (related methodology)