Imbalanced Regression¶

← Error-in-Variables | Noisy Labels →

Imbalanced regression addresses problems where the target distribution is highly non-uniform — some regions (e.g., extreme values) are severely underrepresented. Standard MSE optimises average performance, causing the model to ignore rare-but-important regions.

The Problem¶

Available Losses¶

BalancedMSELoss and BinReweightedMSELoss¶

Inverse bin-frequency weighting (historically called BalancedMSELoss): partition the target range into histogram bins on training data, then weight each sample by roughly 1 / (bin count) (with optional additive smoothing). Bin weights are normalised so that their average over the training samples is 1 (the training loss stays on the MSE scale); empty bins (zero smoothed count) get weight 0 instead of a huge 1/eps that would collapse every populated bin's weight. BalancedMSELoss uses your bin_edges; BinReweightedMSELoss builds equal-width or quantile edges from num_bins and uses noise_sigma as a pseudocount when inverting counts (Laplace-style).

from torchregress.losses import BalancedMSELoss, BinReweightedMSELoss

edges = torch.linspace(y_train.min(), y_train.max(), 11)  # 10 bins
loss_bal = BalancedMSELoss(bin_edges=edges).fit(y_train)

loss_binrew = BinReweightedMSELoss(num_bins=10, noise_sigma=1.0, binning="equal").fit(y_train)

loss = loss_bal(model(x), y_batch)

Multi-output targets use the mean coordinate for bin assignment; the weighted squared error still applies elementwise to y_pred - target.

DensityWeightedLoss¶

Weights samples inversely proportional to local target density via KDE:

from torchregress.losses import DensityWeightedLoss

loss_fn = DensityWeightedLoss(kernel_width=0.5, base_loss="mse", reweight_factor=1.0)
loss_fn.fit_density(y_train)  # estimate density once

# Training loop
loss = loss_fn(model(x), y)

LDSLoss¶

Label Distribution Smoothing — smoother, more aggressive reweighting using kernel-smoothed bin frequencies:

from torchregress.losses import LDSLoss

loss_fn = LDSLoss(kernel="gaussian", kernel_width=2.0, reweight_factor=0.8)
loss_fn.fit(y_train, n_bins=100)

loss = loss_fn(model(x), y)

PropensityWeightedLoss¶

Inverse-propensity weighting for selection bias correction — re-weights observed samples by estimated probability of being observed:

from torchregress.losses import PropensityWeightedLoss

loss_fn = PropensityWeightedLoss(
    base_loss="mse",
    clip_min=0.01,       # floor propensity to avoid extreme weights
    clip_max=0.99,
    normalize_weights=True,
)

# propensity: P(observed | x) estimated externally (e.g., logistic regression)
loss = loss_fn(model(x), y, propensity=p_scores, observed=obs_mask)

FocalRLoss¶

Focal loss for regression — adaptively upweights samples with larger prediction errors:

from torchregress.losses import FocalRLoss

loss_fn = FocalRLoss(beta=0.2, gamma=1.0, base_loss="mse")
loss = loss_fn(model(x), y)

Decision Guide¶

graph TD
    A["Is target distribution imbalanced?"] -->|No| B["Use standard loss"]
    A -->|Yes| C["Is calibration critical?"]
    C -->|Yes| D["DensityWeightedLoss"]
    C -->|No| E["How severe?"]
    E -->|Moderate| F["FocalRLoss"]
    E -->|Extreme| G["LDSLoss + post-hoc calibration"]
    A -->|"Selection bias"| H["PropensityWeightedLoss"]

Limitations¶

1. Calibration risk: Any target-dependent weight changes the population optimum. DensityWeightedLoss and LDSLoss both target a reweighted distribution unless corrected; validate calibration and report the weighting objective. Use sampling-selection corrections for population inference instead.
2. Out-of-range targets: If test targets fall outside the bin edges estimated from fit(y_train) for BalancedMSELoss / BinReweightedMSELoss, weights are undefined. Ensure training targets cover the full expected test range.
3. KDE scaling: DensityWeightedLoss.fit_density() scales as \(\mathcal{O}(N^2)\) with training set size. For \(N > 10^4\), subsample for density estimation or switch to a bin-based method.
4. FocalRLoss does not rebalance: Unlike density-based methods, FocalRLoss upweights hard samples regardless of their target value. It helps with difficult examples but does not address target-distribution imbalance.
5. Propensity scores must be accurate: PropensityWeightedLoss requires well-calibrated propensity estimates. Poor propensity scores can amplify rather than correct selection bias.

Recommendations¶

• Default choice: DensityWeightedLoss is a reasonable tail-focused starting point; it handles out-of-range values via KDE, but it changes the target density. For population density estimation use the unweighted likelihood plus an explicit sampling-selection model.
• For large datasets: Use BalancedMSELoss or BinReweightedMSELoss with pre-computed bin edges to avoid \(\mathcal{O}(N^2)\) density estimation.
• For extreme imbalance: LDSLoss with post-hoc calibration (temperature scaling) is the most aggressive option. See the imbalanced regression demo.
• No pre-fitting needed: FocalRLoss requires no separate fit() step — use when you want a quick, adaptive reweighting during training.

Next steps¶

• Noisy labels — another form of data quality issue
• Robust losses — bounded-influence alternatives for tail-focused evaluation
• Tail metrics — evaluate accuracy specifically on tail quantiles
• Imbalanced example — runnable comparison of all methods

References¶