Iteratively Reweighted Least Squares (IRLS)¶

← Methods Overview | SIMEX →

iteratively_reweighted_fit is a solver-agnostic IRLS controller. The caller supplies residual and weighted-refit callbacks, so the same controller works with analytic WLS, iterative nonlinear solvers, sparse operators, or compiled training steps without assuming a global dense design matrix.

Mathematical Background¶

Standard OLS minimises \(\sum_i (y_i - f(x_i))^2\), giving equal weight to all residuals. IRLS generalises this to:

where the weights \(w_i^{(t)}\) depend on the residuals from iteration \(t\):

After each update, the controller calls problem.weighted_fit(state, precision) and then recomputes residuals from the returned state. With base precision \(P_i\), the callback receives \(w_iP_i\). For a vector residual block with covariance \(C_i\), the robust score uses the Mahalanobis magnitude

and the callback receives the block precision \(w_i C_i^{-1}\) unless base_precision is supplied explicitly.

Weight Functions¶

Here \(r\) denotes the scaled residual \(r / \hat\sigma\) passed to the weight function.

API: iteratively_reweighted_fit¶

Implement IRLSProblem directly or use IRLSCallbacks:

from torchregress.algorithms import IRLSCallbacks, IRLSFitConfig, iteratively_reweighted_fit

problem = IRLSCallbacks(
    residuals_fn=lambda beta: y - X @ beta,
    weighted_fit_fn=lambda beta, precision: torch.linalg.lstsq(
        X * precision.sqrt()[:, None], y * precision.sqrt()
    ).solution,
)

result = iteratively_reweighted_fit(problem, initial_beta, IRLSFitConfig(weight="huber"))
beta = result.state

Kish effective sample size and the weighted mean robust objective are invariant to a common finite positive multiplier of base precisions. The controller normalizes only diagnostic sums to avoid overflow or false breakdown; the callback still receives the original precision units. Explicitly masked residual entries may be nonfinite and are ignored, including entirely missing output columns. Nonfinite observed residuals still terminate the fit.

Returns: frozen IRLSFitResult with the final state, applied robust weights and precision, residuals, scale, robust objective, Kish effective sample size, termination reason and a tuple of frozen per-iteration diagnostics. Termination is one of "converged", "max_iterations", "empty", "breakdown", or "non_finite". A caller-owned rank failure can be reported by raising IRLSBreakdownError from weighted_fit.

Complete analytic WLS example¶

import torch
from torchregress.algorithms import IRLSCallbacks, IRLSFitConfig, iteratively_reweighted_fit

torch.manual_seed(42)
n = 1000
X = torch.randn(n, 5)
y = X @ torch.tensor([1.0, -0.5, 0.2, 0.7, -0.3]) + 0.1 * torch.randn(n)
outliers = torch.randperm(n)[:50]
y[outliers] += 5.0 * torch.randn(50)

problem = IRLSCallbacks(
    residuals_fn=lambda beta: y - X @ beta,
    weighted_fit_fn=lambda beta, precision: torch.linalg.lstsq(
        X * precision.sqrt()[:, None], y * precision.sqrt()
    ).solution,
)

result = iteratively_reweighted_fit(
    problem,
    torch.zeros(X.shape[1]),
    IRLSFitConfig(weight="tukey", weight_params={"c": 4.685}),
)

clean = ~torch.isin(torch.arange(n), outliers)
pred = X @ result.state
print(f"IRLS MAE (clean): {(pred[clean] - y[clean]).abs().mean():.4f}")

For heteroscedastic vector residuals, provide covariance_fn=lambda state: covariance with shape (samples, target_dim, target_dim). For matrix-free or sparse solvers, provide whiten_fn and keep any operator representation inside the problem object; the controller never constructs a global design or normal matrix.

Fixed-prediction compatibility utility¶

reweight_fixed_predictions(model, x, y_true, ...) evaluates a fixed model and returns (prediction, loss_history, precision) without refitting. Code that needs genuine IRLS should use iteratively_reweighted_fit.

Comparison with Robust Losses¶

Limitations¶

1. Vulnerability to leverage points: IRLS only downweights outliers based on residuals in the response space (\(Y\)). It is sensitive to high-leverage points (outliers in \(X\)). A single high-leverage outlier can attract the regression line even if its residual is small.
2. MAD scale collapse: The robust scale estimate can become zero if many residuals are identical. The configured min_scale prevents division by zero, but callers should diagnose the underlying fit collapse.
3. Warm-start required for redescending losses: Tukey and Cauchy weight functions are non-convex. IRLS must be warm-started from a convex loss (OLS, Huber) or it diverges.
4. Per-iteration cost: Each IRLS update requires a residual evaluation and caller-owned refit. Large-model callbacks should control batching and stopping internally.
5. Outer-loop controller: The controller detaches residual diagnostics and is not an end-to-end differentiable loss. The weighted callback may use autograd internally.

Recommendations¶

• Default weight function: Start with weight="huber" — it is convex, stable, and handles moderate outliers. Upgrade to "tukey" only with warm-start.
• Use robust losses for end-to-end training: For standard neural network training, prefer Robust losses (Huber, Cauchy, Tukey) which are differentiable and work inside loss.backward().
• Use continuation for redescending weights: Start with a Huber stage and switch to Tukey, Cauchy, or Barron through IRLSContinuationStage.
• Use covariance whitening for vector targets: A full per-sample covariance gives exact block Mahalanobis scores; a custom whitening callback supports structured operators.

Next steps¶

• Robust losses — Huber, Tukey, and Cauchy trained end-to-end vs refit-capable IRLS
• SIMEX — simulation-based bias correction for measurement error, complementary to robust regression
• EIV losses — functional and structural error-in-variables losses for noisy inputs
• Comprehensive comparison — benchmark IRLS against end-to-end robust loss training

References¶