SIMEX (Simulation Extrapolation)¶

← IRLS | RC →

SIMEX is a simulation-based method for correcting measurement error in inputs. Unlike RC which analytically debiases, SIMEX works by adding progressively more noise then extrapolating back to zero noise.

Mathematical Background¶

Adding Simulated Noise¶

Given observed data with measurement error covariance \(\Sigma_u\), SIMEX adds additional simulated noise:

At level \(\lambda\), the total measurement error variance is \((1 + \lambda)\,\Sigma_u\).

Extrapolation to \(\lambda = -1\)¶

The key insight: at \(\lambda = -1\), the total noise variance would be:

SIMEX fits a polynomial through predictions at \(\lambda \in \{0, 0.5, 1.0, 1.5, 2.0\}\) and extrapolates:

where \(\hat\beta_j\) are polynomial coefficients from fitting predictions vs. \(\lambda\).

Usage¶

import torch
import torch.nn as nn
from torchregress.algorithms import SIMEX

# Model factory — returns a fresh model for each noise level
def model_factory():
    return nn.Sequential(
        nn.Linear(5, 32), nn.ReLU(),
        nn.Linear(32, 1),
    )

# Training function
def train_func(model, X, y):
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    for _ in range(200):
        loss = nn.functional.mse_loss(model(X), y)
        opt.zero_grad(); loss.backward(); opt.step()
    return model

# Create SIMEX corrector
simex = SIMEX(
    model_factory=model_factory,
    train_func=train_func,
    sigma_u=0.5,
    lambdas=[0.5, 1.0, 1.5, 2.0],
    extrapolation_order=2,   # quadratic
)

# Fit trains M+1 models internally
simex.fit(X_train, y_train)

# Predict with measurement-error correction
y_pred = simex.predict(X_test)

predict perturbs the test inputs with \(\sqrt{\lambda}\,\mathcal{N}(0, \Sigma_u)\) noise at every level, drawing an independent remeasurement for each of the n_simulations models, so the \(\lambda\)-level prediction averages over all replicates; its Monte Carlo spread shrinks like \(1/\sqrt{B}\). The noise is simulated in the dtype of X_train (float64 data is supported) and without any absolute jitter, so SIMEX is equivariant to the units of the inputs.

RC vs SIMEX¶

When to Use¶

Limitations¶

1. Extrapolation risk: SIMEX fits a polynomial through predictions at \(\lambda \ge 0\) and extrapolates to \(\lambda = -1\). If the true mapping is not well-approximated by a low-order polynomial, extrapolation errors can be severe (Runge's phenomenon). Higher-order polynomials (\(p \ge 3\)) amplify this risk.
2. Computational cost: SIMEX trains \(|\text{lambdas}| + 1\) complete models. For deep networks with 4–5 noise levels, this multiplies training cost by 5–6×.
3. Negative variance artefacts: Extrapolating polynomial fits to \(\lambda = -1\) can produce negative predicted variances. Always clamp variance outputs to a minimum positive value after SIMEX correction.
4. Known \(\Sigma_u\) required: SIMEX requires an accurately specified measurement error covariance. Over- or under-specifying \(\Sigma_u\) directly biases the extrapolated estimates.
5. Homoscedastic noise assumption: SIMEX assumes a single \(\Sigma_u\) for all samples. Heteroscedastic measurement error (different noise per sample) is not supported.

Recommendations¶

• Default: Start with extrapolation_order=2 (quadratic) and lambdas=[0.5, 1.0, 1.5, 2.0]. Linear extrapolation is safer but less flexible; quadratic is a good compromise.
• For linear models: Use RC instead — it is analytical, instantaneous, and makes the same assumptions without the computational cost.
• For neural networks with large datasets: Consider EIV losses (LatentMarginalizationLoss) which handle measurement error at the loss level without retraining M models.
• Validate extrapolation: Plot the SIMEX extrapolation curve with plot_simex_extrapolation to visually check polynomial fit quality before trusting the correction.

Next steps¶

• Regression Calibration — faster analytical alternative when a linear correction suffices
• LatentNN — single-training-run joint optimisation for noisy inputs, no multi-model overhead
• Error-Aware Encoding — lightweight quality-signal engineering for known input noise
• EIV losses — loss-level handling of measurement error without retraining

References¶