Quantile & Expectile Losses¶

← Robust Losses | Ordinal Losses →

Quantile and expectile regression estimate different points of the conditional distribution \(P(Y \mid X)\) — quantiles target specific percentiles, expectiles target weighted means. Both are distribution-free: no Gaussian assumption required.

Quantile vs Expectile at a Glance¶

Mathematical Background¶

Quantile loss (check / pinball)¶

Expectile loss (asymmetric least squares)¶

When \(\tau = 0.5\) the expectile loss reduces to standard MSE.

See the QuantileLoss, MultiQuantileLoss, and ExpectileLoss API sections for parameters.

Quantile Losses¶

QuantileLoss¶

Single quantile:

from torchregress.losses import QuantileLoss

loss_fn = QuantileLoss(quantile=0.9)   # 90th percentile
loss = loss_fn(y_pred, y_true)

MultiQuantileLoss¶

Multiple quantiles simultaneously — the model outputs one prediction per quantile:

from torchregress.losses import MultiQuantileLoss

loss_fn = MultiQuantileLoss(
    quantiles=[0.05, 0.5, 0.95],  # 90% prediction interval + median
    joint_prediction=True,         # model outputs [batch, Q, d]
)

class QuantileModel(nn.Module):
    def __init__(self, in_dim, out_dim, n_quantiles=3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, 128), nn.ReLU(),
            nn.Linear(128, out_dim * n_quantiles),
        )
        self.out_dim = out_dim
        self.n_quantiles = n_quantiles

    def forward(self, x):
        out = self.net(x)                              # [B, Q*d]
        return out.view(-1, self.n_quantiles, self.out_dim)  # [B, Q, d]

model = QuantileModel(10, 1, n_quantiles=3)
loss_fn = MultiQuantileLoss(quantiles=[0.05, 0.5, 0.95])
optimizer = torch.optim.Adam(model.parameters())

for x, y in train_loader:
    loss = loss_fn(model(x), y)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

with torch.no_grad():
    preds = model(x_test)  # [B, 3, 1]
    lower, median, upper = preds[:, 0], preds[:, 1], preds[:, 2]

QuantileCrossoverLoss¶

Because independent neural network heads predict each quantile separately, they can predict unphysical crossings (e.g., the 5% quantile predicting a larger value than the 95% quantile). QuantileCrossoverLoss solves this by adding a hinge-loss penalty to enforce monotonicity:

from torchregress.losses import QuantileCrossoverLoss

loss_fn = QuantileCrossoverLoss(
    quantiles=[0.05, 0.5, 0.95],
    crossover_penalty=10.0,  # strength of non-crossing constraint (λ)
)

Expectile Losses¶

ExpectileLoss¶

Single expectile:

from torchregress.losses import ExpectileLoss

loss_fn = ExpectileLoss(expectile=0.8)   # 80th expectile
loss = loss_fn(y_pred, y_true)

MultiExpectileLoss¶

Multiple expectiles simultaneously:

from torchregress.losses import MultiExpectileLoss

loss_fn = MultiExpectileLoss(
    expectiles=[0.1, 0.5, 0.9],
    joint_prediction=True,
)

ExpectileCrossoverLoss¶

Non-crossing constraint for expectiles (analogous to QuantileCrossoverLoss). As for QuantileCrossoverLoss and MultiExpectileLoss, the levels must be strictly ascending (a ValueError is raised otherwise), and y_pred[:, k] is trained against expectiles[k]:

from torchregress.losses import ExpectileCrossoverLoss

loss_fn = ExpectileCrossoverLoss(
    expectiles=[0.1, 0.5, 0.9],
    crossover_penalty=10.0,
)

Asymmetric least squares is ExpectileLoss(expectile=...).

Limitations¶

1. Expectile-to-quantile conversion: Expectiles are smooth and easier to optimise, but do not correspond directly to probability percentiles. You cannot convert expectile estimates back to quantiles without fitting a parametric density wrapper. For strict \(P(Y \in [L, U]) \ge 1-\alpha\) guarantees, use QuantileLoss + CQR.
2. Non-differentiability at zero: The quantile (pinball) loss has a kink at \(r = 0\) where the gradient is undefined. While Adam handles this in practice, second-order methods (L-BFGS, Newton) may stall.
3. Output dimension scaling: MultiQuantileLoss with \(Q\) quantiles on a \(d\)-dimensional target produces output dimension \(Q \cdot d\). For fine-grained quantile grids (e.g., 99 quantiles), the output head becomes a memory and optimisation bottleneck.
4. Crossing in independent heads: Without QuantileCrossoverLoss, independent neural network heads predict each quantile separately and can produce unphysical crossings (\(\hat{q}_{\tau_1} > \hat{q}_{\tau_2}\) for \(\tau_1 < \tau_2\)).

Choosing Between Quantile and Expectile¶

Next steps¶

• Conformalized Quantile Regression (CQR) — guaranteed coverage on top of quantile predictions
• Robust losses — M-estimators for outlier-heavy data
• Gaussian losses — parametric alternatives when normality holds
• Interval metrics — evaluate your quantile-based prediction intervals

Recommendations¶

• Start with QuantileCrossoverLoss — it prevents unphysical crossing of prediction bands at essentially no extra cost over MultiQuantileLoss.
• For probability guarantees: Use QuantileLoss + wrap with CQR for finite-sample coverage.
• For smooth optimisation: Use ExpectileLoss / MultiExpectileLoss when you need second-order optimisers (L-BFGS) or when downstream tasks use Expected Shortfall.
• Limit quantile grid size: For \(Q > 10\) quantiles, the output dimension \(Q \cdot d\) grows quickly. Consider a parametric model (MDN, flow) for dense quantile grids. See the Quantile & expectile demo.
• Monitor crossing frequency: Even with crossover penalties, check how often \(\hat{q}_{\tau_i} > \hat{q}_{\tau_j}\) for \(\tau_i < \tau_j\) on validation data.

References¶