Ordinal Regression Losses¶

← Quantile & Expectile | Censored Regression →

Ordinal regression is for targets with a natural ordering where the distances between levels are unknown or non-uniform. Unlike classification, ordinal losses exploit the fact that nearby predictions are better than distant ones.

When to Use Ordinal Regression¶

Mathematical Background¶

Cumulative Link Model¶

The cumulative model defines ordered probabilities through shared thresholds \(\theta_1 < \theta_2 < \cdots < \theta_{K-1}\):

where \(g\) is a monotone link function (logistic, probit) and \(f(x)\) is a scalar model output.

The probability of each class is:

CORAL Objective¶

Consistent Rank Logits (Cao et al., 2020) reformulates ordinal regression as \(K-1\) binary tasks with shared weights but independent biases:

The loss is the sum of binary cross-entropies:

Available Losses¶

OrdinalCrossEntropyLoss¶

Standard cross-entropy baseline (ignores ordering). See OrdinalCrossEntropyLoss API. Treats each level as an independent class, but now supports both:

• hard class-index labels, and
• soft class-probability targets for ambiguous ordered labels or soft pseudo labels.

import torch
from torchregress.losses import OrdinalCrossEntropyLoss

num_classes = 5
logits = torch.randn(32, num_classes)  # K logits
labels = torch.randint(0, num_classes, (32,))

loss_fn = OrdinalCrossEntropyLoss()
loss = loss_fn(logits, labels)

Soft-target usage:

soft_targets = torch.tensor(
    [
        [0.70, 0.25, 0.05, 0.00, 0.00],
        [0.00, 0.10, 0.65, 0.20, 0.05],
    ]
)
loss = loss_fn(logits[:2], soft_targets)

CumulativeLinkLoss¶

Cumulative-threshold model with \(K-1\) logits. See CumulativeLinkLoss API. It accepts either hard ordinal labels or soft per-class PMF targets, which are internally converted to cumulative levels.

from torchregress.losses import CumulativeLinkLoss

logits = torch.randn(32, num_classes - 1)  # K-1 outputs
loss_fn = CumulativeLinkLoss()
loss = loss_fn(logits, labels)

Soft / Uncertain Ordinal Targets¶

When ordered targets are ambiguous, weakly supervised, or represented as ordered-bin plausibilities, you do not need a separate loss family. The ordinal losses can consume soft targets directly.

For a soft target vector \(p \in \Delta^{K-1}\):

and for cumulative objectives the target levels are the expected cumulative events:

This is the right fit for:

• ambiguous labels near ordinal boundaries,
• regression-as-classification with uncertain ordered bins,
• soft pseudo labels from a teacher model.

CORAL head¶

The CORAL rank constraint (shared weight, per-level bias) lives on the model, for example CORALHead. The loss is CumulativeLinkLoss.

from torchregress.losses import CumulativeLinkLoss

logits = torch.randn(32, num_classes - 1)  # K-1 binary logits
loss_fn = CumulativeLinkLoss()
loss = loss_fn(logits, labels)

Limitations¶

1. Class imbalance sensitivity: Ordinal losses assume roughly balanced classes. When some levels are extremely rare (e.g., 95% of samples in class 1, 1% in class 5), the model may ignore rare levels entirely. Combine with imbalanced reweighting strategies from Imbalanced Regression.
2. Metric mismatch: Standard classification metrics (accuracy, F1) ignore ordering. Always evaluate with ordering-aware alternatives: MAE on predicted class indices, Quadratic Weighted Kappa, or Spearman \(\rho\). See Ordinal metrics.
3. Threshold identifiability: CumulativeLinkLoss thresholds \(\theta_k\) and the scalar model output \(f(x)\) are not jointly identifiable — adding a constant to all \(\theta_k\) and \(f(x)\) leaves the loss unchanged. Ensure the model has a bias-free final layer or fixed first threshold.
4. Soft targets and calibration: When using soft probability targets, the loss is strictly proper but does not guarantee calibrated probability estimates. Apply post-hoc calibration if well-calibrated class probabilities are required.

Method Comparison¶

Complete Example: Product Ratings¶

import torch
import torch.nn as nn
from torchregress.losses import CumulativeLinkLoss

K = 5  # 1-5 star ratings

class RatingModel(nn.Module):
    def __init__(self, in_dim, K):
        super().__init__()
        self.feature = nn.Sequential(
            nn.Linear(in_dim, 64), nn.ReLU(),
            nn.Linear(64, 32), nn.ReLU(),
        )
        self.fc = nn.Linear(32, 1, bias=False)  # shared weights
        self.biases = nn.Parameter(torch.zeros(K - 1))  # K-1 thresholds

    def forward(self, x):
        h = self.fc(self.feature(x))        # scalar per sample
        return h - self.biases.unsqueeze(0)  # [batch, K-1]

model = RatingModel(in_dim=50, K=K)
loss_fn = CumulativeLinkLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# Training
for x, y in train_loader:
    logits = model(x)       # [batch, K-1]
    loss = loss_fn(logits, y)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

# Decoding predictions
with torch.no_grad():
    logits = model(x_test)
    probs = torch.sigmoid(logits)               # P(Y > k)
    predicted_class = probs.sum(dim=1).round()   # expected rank

Practical Tips¶

Next steps¶

• Ordinal metrics — MAE, Quadratic Weighted Kappa, Spearman \(\rho\)
• Censored losses — another type of partially-observed target
• Ordinal comparison — synthetic and real-data (--dataset diabetes) benchmark
• Soft ordinal targets — ambiguous labels and pseudo-labels

References¶