Split-Measurement & Corrected-Score EIV Losses¶

← EIV losses | Noisy labels → Forward inversion

When the input features are measured with error, the ordinary squared error attenuates the regression: the model learns a shrunken map because the noisy feature looks like a noisier response than it really is. Two classical errors-in-variables (EIV) constructions remove that bias inside the loss, without changing the model:

• Split-measurement loss — use two independent noisy measurements of the same object and multiply their residuals. The cross term averages to zero, so no noise covariance is needed.
• Corrected-score loss — subtract the measurement-error-induced curvature from the squared residual (Stefanski 1989; Nakamura 1990). This is the opposite sign of the usual TLS/ODR-style gradient penalty, and it does not flatten the model.

Both are implemented as losses that take the model as a constructor argument, exactly like the other EIV losses in torchregress.losses.

1. Split-measurement loss¶

Let \(x_i\) be the latent clean input and \(x_i^{(1)} = x_i + e_i^{(1)}\), \(x_i^{(2)} = x_i + e_i^{(2)}\) two conditionally independent measurements with zero-mean errors and \(\mathbb{E}[e^{(1)}(e^{(2)})^\top] = 0\). The per-sample loss is the product of the two-view residuals:

With \(r_i = y_i - f(x_i)\) the noiseless residual, the expectation over the two independent errors satisfies, for a model linear in \(x\),

so the loss sees the noiseless squared error: the \(+\sigma_x^2\) variance inflation of the ordinary MSE is cancelled by the cross-product structure. No estimate of \(\Sigma_x\) is required — only a second independent measurement (repeat exposures, multi-epoch photometry, split spectra).

API¶

import torch
import torch.nn as nn
from torchregress.losses import SplitMeasurementLoss

model = nn.Linear(4, 1)
loss_fn = SplitMeasurementLoss(model)          # no sigma_x needed
loss = loss_fn(x_first, y, y_pred_second=x_second)

2. Corrected-score loss¶

For a normally distributed measurement error \(\varepsilon \sim \mathcal N(0, \Sigma_x)\) and a model \(f\) evaluated at the observed \(w = x + \varepsilon\), the corrected score is

which in the diagonal-covariance case is the familiar

This is an unbiased estimating equation: its expectation equals the noiseless squared error (for the linear model exactly; for non-linear models up to higher-order terms), so the minimiser is not attenuated. Note the sign: TLS/ODR-style losses add a gradient penalty, which flattens \(f\) and actually increases attenuation when \(\sigma_x\) is large. The corrected score subtracts the error contribution instead.

API¶

from torchregress.losses import CorrectedScoreEIVLoss

loss_fn = CorrectedScoreEIVLoss(
    model,
    sigma_x=0.3,          # scalar, (D,), (D, D), or per-sample (N, D)/(N, D, D)
    stabilize="none",     # statistically unbiased default; "clamp" for stability
)
loss = loss_fn(x_observed, y)

When to use which¶

Example¶

A linear forward model \(y = 1.5 + 2x\) observed through noisy features. With the correct model, plain MSE on a noisy view is biased high by \(b^2\sigma^2\), while both corrected losses recover the noiseless squared error:

import torch
import torch.nn as nn
from torchregress.losses import CorrectedScoreEIVLoss, SplitMeasurementLoss

torch.manual_seed(0)
n, sigma = 4000, 0.5
x = torch.randn(n, 1)
y = 1.5 + 2.0 * x + 0.1 * torch.randn(n, 1)
x1 = x + sigma * torch.randn(n, 1)   # first noisy exposure
x2 = x + sigma * torch.randn(n, 1)   # independent second exposure

model = nn.Linear(1, 1)
with torch.no_grad():
    model.weight.fill_(2.0)
    model.bias.fill_(1.5)
    plain = nn.MSELoss()(model(x1), y)
    corrected = CorrectedScoreEIVLoss(model, sigma_x=sigma)(x1, y)
    split = SplitMeasurementLoss(model)(x1, y, y_pred_second=x2)
    clean = nn.MSELoss()(model(x), y)

print(f"noiseless={clean:.4f}  plain={plain:.4f}  "
      f"corrected={corrected:.4f}  split={split:.4f}")
# noiseless ~ 0.01, plain ~ 1.01 (= clean + 4 sigma^2),
# corrected ~ 0.00, split ~ 0.00

For training, pass the loss to any optimizer with the model parameters; the EIV losses evaluate the model internally:

optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
loss_fn = CorrectedScoreEIVLoss(model, sigma_x=sigma)
for _ in range(200):
    optimizer.zero_grad()
    loss = loss_fn(x1, y)
    loss.backward()
    optimizer.step()
# model.weight ~ 2, model.bias ~ 1.5: no attenuation

Caveats¶

References¶

See also¶

• EIV losses — the other measurement-error losses.
• Forward generative inversion — the \(y \to x\) alternative.
• Regression Calibration and SIMEX — algorithm-level corrections.
• Noisy features / measurement error — method-selection matrix.