Ordinal Metrics¶

← Censored Metrics | Metrics Overview →

Metrics for ordered-class regression/classification outputs.

An ordinal target \(y \in \{0,1,\dots,K-1\}\) carries more information than a nominal class: the distance \(|y-\hat{y}|\) matters. Flat accuracy throws that away (an off-by-one error and an off-by-five error count the same), so the library reports three complementary views: accuracy (the flat baseline), mean absolute class error (MACE, linear in ordinal distance), and the quadratic weighted kappa (QWK, chance-corrected, quadratic in distance). QWK is the standard agreement statistic when the class distribution is skewed; MACE is the interpretable "average classes off". Report at least one distance-aware metric alongside accuracy — see the comparison table at the end of this page.

Available Metrics¶

Ordinal Accuracy¶

Measures the exact agreement between the predicted class index \(\hat{c}_i\) and the true class index \(c_i\):

Mean Absolute Class Error¶

Computes the Mean Absolute Error in ordinal class space, evaluating the distance between predicted and true rating classes:

Quadratic Weighted Kappa (QWK)¶

Measures agreement between predictions and targets, penalizing larger ordinal mistakes more heavily.

Let \(O_{j,k}\) be the observed rating counts (contingency table) where true class is \(j\) and predicted class is \(k\). Let \(E_{j,k}\) be the expected rating counts under independence:

The quadratic weight matrix \(W\) is:

where \(K\) is the total number of ordinal classes. The kappa coefficient is:

Usage¶

import torch
import torchregress as tr

logits = torch.randn(64, 5)
labels = torch.randint(0, 5, (64,))

acc = tr.metrics.ordinal_accuracy(logits, labels, encoding="class_logits")
mae_cls = tr.metrics.mean_absolute_class_error(logits, labels, encoding="class_logits")
qwk = tr.metrics.quadratic_weighted_kappa(logits, labels, encoding="class_logits")

See the quadratic_weighted_kappa API for details.

Limitations¶

1. Accuracy ignores ordering: Standard classification accuracy treats all misclassifications equally. An off-by-one error is penalised the same as an off-by-ten error — this is inappropriate for ordinal targets where proximity carries meaning.
2. MACE is mean-based: MACE treats the ordinal scale as intervals with equal spacing, which may not reflect the true cost of misclassification for a given application.
3. QWK's quadratic weighting can be misleading: The quadratic penalty on ordinal distance may over-emphasise large-ordinal-distance errors in settings where moderate misclassifications are acceptable.

Recommendations¶

• Always report MACE alongside accuracy: Accuracy alone can hide ordinal structure; MACE reveals whether errors are small or large in ordinal distance.
• Use QWK when ordinal distance matters, but complement it with MACE for interpretability (QWK's quadratic weighting can be opaque).
• Inspect the confusion matrix: Aggregate metrics can conceal systematic biases (consistent over- or under-prediction of the ordinal class).

When to use which¶

Next steps¶

• Censored metrics — related evaluation for interval-censored and survival outcomes
• Point metrics — standard accuracy baselines to pair with ordinal-specific metrics
• Ordinal losses — ordinal regression losses for training ordered-class models
• Ordinal comparison example — full comparison of ordinal methods

References¶