Error-in-Variables (EIV) Losses¶

← Normalizing Flows | Imbalanced Regression →

Standard regression assumes that inputs \(X\) are measured perfectly — only the target \(Y\) has noise. In practice, this assumption is almost always violated: sensor readings have finite precision, proxy variables approximate the true quantity, and derived features carry propagated errors. EIV losses address this by modelling the realistic case where both inputs and outputs are noisy:

where \(\varepsilon_X \sim \mathcal{N}(0, \Sigma_X)\) and \(\varepsilon_Y \sim \mathcal{N}(0, \Sigma_Y)\) are independent additive Gaussian noise terms.

Why Standard Regression Fails¶

When inputs are noisy with variance \(\sigma_\varepsilon^2\), OLS produces systematically biased coefficient estimates:

where \(\lambda \in (0, 1)\) is the reliability ratio. This is known as attenuation bias — slopes are underestimated, and the effect worsens with noisier inputs.

Available Losses¶

LatentMarginalizationLoss¶

The recommended starting point for modern probabilistic models. Instead of relying on local Taylor expansions, it uses Monte Carlo sampling to approximate the marginal likelihood:

where \(x_i \sim p(x \mid x_{\text{obs}})\) is drawn from the posterior of the clean input: either a custom posterior_sampler, a fitted calibrator with a posterior() method (e.g. RegressionCalibration), or the Gaussian posterior implied by a structural prior \(x \sim \mathcal{N}(\mu_x, \Sigma_x)\) (prior_mean, prior_cov) and measurement error \(\Sigma_u\) (sigma_u). This approach is extremely stable and works naturally with complex predictive heads like MDNs or Binned PDFs.

The default marginalization_mode="likelihood" computes \(-\log \frac{1}{N}\sum_i p(y \mid x_i)\) (log-mean-exp of the per-draw NLL); "expectation" averages the NLL over draws instead (an upper bound by Jensen's inequality). The base_loss is evaluated element-wise automatically (its reduction is switched to "none" for the call), so a default-constructed GaussianNLLLoss() works; sample weights and mask are applied once to the marginalised per-sample loss.

import torch
import torch.nn as nn

from torchregress.losses import GaussianNLLLoss, LatentMarginalizationLoss

torch.manual_seed(0)
x_obs = torch.randn(64, 2)
y_obs = x_obs.sum(dim=-1, keepdim=True) + 0.1 * torch.randn(64, 1)
my_model = nn.Linear(2, 2)  # outputs [mean, log_var]

loss_fn = LatentMarginalizationLoss(
    model=my_model,
    base_loss=GaussianNLLLoss(),      # per-draw NLL (evaluated element-wise)
    prior_mean=x_obs.mean(dim=0),     # structural prior on the clean input
    prior_cov=torch.cov(x_obs.T),
    sigma_u=0.2,                      # input measurement-error std
    n_samples=16,                     # number of MC draws
)
loss = loss_fn(x_obs, y_obs)          # pass observed inputs, not model(x)
loss.backward()

NoisyInputPredictor (Wrapper)¶

For easy test-time inference, use the high-level wrapper which automatically handles the marginalization logic:

from torchregress.losses import NoisyInputPredictor

# Wrap your model
predictive = NoisyInputPredictor(model, sigma_x=0.2, n_samples=32)

# Normal forward pass now returns the marginalized mean prediction
y_pred_mean = predictive(x_obs)

# Get raw samples for ensemble/density estimation
y_samples = predictive.sample_predictions(x_obs)

Multimodal EIV Variants¶

For complex predictive distributions (like scientific regression), use specialized marginalizers:

InputNoiseMDNLoss Combines Input-Noise Marginalization with Mixture Density Networks.

from torchregress.losses import InputNoiseMDNLoss

loss_fn = InputNoiseMDNLoss(
    model=my_mdn,
    n_components=5,
    sigma_x=0.1
)

InputNoiseBinnedPDFLoss Marginalizes over inputs for models predicting discrete probability bins.

from torchregress.losses import InputNoiseBinnedPDFLoss

loss_fn = InputNoiseBinnedPDFLoss(model=my_classifier, sigma_x=0.1)

FunctionalEIVLoss¶

Propagates \(X\)-uncertainty through the model via a first-order Taylor approximation:

where \(\mathbf{J} = \partial f / \partial X\) is the Jacobian of shape \([n_{\text{out}} \times n_{\text{in}}]\). For scalar output this reduces to the inner product \(\mathbf{J}\Sigma_X\mathbf{J}^\top = (\nabla_X f)^\top \Sigma_X (\nabla_X f)\).

from torchregress.losses import FunctionalEIVLoss

loss_fn = FunctionalEIVLoss(
    model=my_model,
    sigma_x=torch.tensor([0.2, 0.1]),   # per-feature noise std
    sigma_y=0.1,
    monte_carlo=False,   # True → MC gradient estimation
    n_samples=20,        # MC samples (if monte_carlo=True)
)

loss = loss_fn(x_obs, y_obs)

StructuralEIVLoss¶

Accounts for correlations between input and output errors via a cross-covariance matrix \(\Sigma_{XY}\):

from torchregress.losses import StructuralEIVLoss

loss_fn = StructuralEIVLoss(
    model=my_model,
    sigma_x=cov_x,    # [d_x, d_x]
    sigma_y=cov_y,    # [d_y, d_y]
    sigma_xy=cov_xy,  # [d_y, d_x] cross-covariance
)

OrthogonalDistanceRegressionLoss¶

Minimises the perpendicular distance from observations to the model surface by jointly optimising latent true inputs \(\hat{X}\):

from torchregress.losses import OrthogonalDistanceRegressionLoss

loss_fn = OrthogonalDistanceRegressionLoss(
    model=my_model,
    sigma_x=torch.tensor([1.0, 1.0]),
    sigma_y=torch.tensor([1.0]),
    learning_rate=0.01,     # inner optimisation LR
    max_iterations=10,      # inner loop steps
)

Limitations¶

1. Computational overhead: LatentMarginalizationLoss calls the model \(N_{\text{samples}}\) times per training step. Start with n_samples=8 before scaling up.
2. Model determinism requirement: MC marginalization calls the model with different noise perturbations. Dropout and BatchNorm must be in eval mode during these forward passes, or each MC sample sees a different stochastic mask/batch statistic — biasing the marginal likelihood estimate.
3. Twice-differentiability for FunctionalEIVLoss: The Taylor expansion requires \(f(x)\) to be twice differentiable. Activation functions like ReLU have zero second derivatives, causing the curvature term \(\partial^2 f / \partial x^2\) to vanish. Prefer smooth activations (GELU, Tanh, SiLU).
4. ODR inner-loop convergence: OrthogonalDistanceRegressionLoss runs an inner optimisation loop. If max_iterations is too small or learning_rate is poorly tuned, the latent \(\hat{X}\) estimates may not converge — degrading the outer gradient signal.
5. EIV methods do not estimate measurement error: You must provide \(\sigma_x\) and \(\sigma_y\) (or their covariance matrices) as known quantities. These losses correct for known noise; they do not discover it from data.

Decision Guide¶

graph TD
    A["Measurement errors in X?"] -->|No| B["Use standard loss"]
    A -->|Yes| C{"Predictive Model Type?"}
    C -->|Point Estimate| D{"Need correlated errors?"}
    D -->|Yes| E["StructuralEIVLoss"]
    D -->|No| F["FunctionalEIVLoss"]
    C -->|Probabilistic/Complex| G{"Model Differentiable?"}
    G -->|Yes| H["LatentMarginalizationLoss"]

Complete Example¶

import torch
import torch.nn as nn
from torchregress.losses import LatentMarginalizationLoss
# True relationship: y = 2x + 1
torch.manual_seed(42)
x_true = torch.linspace(0, 10, 200).unsqueeze(1)
y_true = 2.0 * x_true + 1.0

# Add measurement noise to BOTH x and y
x_obs = x_true + 0.5 * torch.randn_like(x_true)
y_obs = y_true + 0.3 * torch.randn_like(y_true)

# Model
model = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 1))

from torchregress.losses import LatentMarginalizationLoss, WeightedMSELoss
# EIV loss using marginalization (the modern default)
loss_fn = LatentMarginalizationLoss(
    model=model,
    base_loss=WeightedMSELoss(),
    sigma_u=0.5,
)

for epoch in range(200):
    optimizer.zero_grad()
    loss = loss_fn(x_obs, y_obs)
    loss.backward()
    optimizer.step()

Next steps¶

• Regression Calibration (RC) — closed-form EIV correction for linear models
• SIMEX — simulation-extrapolation for nonlinear measurement error
• LatentNN — end-to-end joint optimization of clean inputs + network
• EIV comparison — benchmark all EIV methods
• Gaussian losses — standard losses when inputs are clean

Recommendations¶

• Default choice: LatentMarginalizationLoss is the recommended starting point for modern probabilistic models with non-linear feature/target relationships. It correctly marginalises over input noise via MC sampling and works with any predictive head (MDN, Binned PDF, Gaussian).
• Smooth activations for FunctionalEIVLoss: Use GELU, Tanh, or SiLU instead of ReLU to ensure the curvature term \(\partial^2 f / \partial x^2\) is non-zero.
• MC samples budget: Start with n_samples=8, antithetic=True. Increase only if validation NLL improves. For deployment inference, use NoisyInputPredictor wrapper with n_samples=32.
• Compare against RC/SIMEX: For linear models or when interpretability matters, Regression Calibration and SIMEX are simpler and more transparent than neural EIV losses. See the EIV comparison example.

References¶