Multivariate Metrics¶

← Decision Metrics | Censored Metrics →

This page covers metrics for evaluating multivariate regression models.

For a vector target \(y \in \mathbb{R}^D\), there are two different questions and two different scores. Marginal metrics treat the \(D\) outputs as \(D\) scalar problems and average; they are interpretable per output but blind to dependence. Joint metrics score the vector as one object and therefore penalise a forecast that gets each marginal right but the correlation structure wrong. The Euclidean MultivariateRMSE/MultivariateMAE below are marginal-aggregate scores (they sum squared error over dimensions, then take one root); for dependence-aware scoring use the energy score and variogram score on the distribution metrics page. Reporting both is the multivariate analogue of reporting calibration and sharpness together.

MultivariateRMSE¶

Computes the root mean squared Euclidean error over vector outputs,

summing the squared error over the D output dimensions (not averaging): with equal error scales this is \(\sqrt{D}\) times the per-element RMSE. It is a stateful metric: call update(y_pred, y_true) for each batch and compute() at the end, or call the instance on one batch.

Arguments (update and __call__):

• y_pred (torch.Tensor or np.ndarray): The predicted values.
• y_true (torch.Tensor or np.ndarray): The ground truth values.

Returns:

The root mean squared Euclidean error (a scalar tensor).

Example:

import torch
from torchregress.metrics import MultivariateRMSE

y_pred = torch.randn(100, 10)
y_true = torch.randn(100, 10)

metric = MultivariateRMSE()
rmse = metric(y_pred, y_true)

MultivariateMAE¶

Computes the mean L1 error over vector outputs,

summing the absolute error over the D output dimensions: this is D times the per-element MAE. It takes the same arguments as MultivariateRMSE.

Arguments (update and __call__):

• y_pred (torch.Tensor or np.ndarray): The predicted values.
• y_true (torch.Tensor or np.ndarray): The ground truth values.

Returns:

The mean L1 error per sample (a scalar tensor).

Example:

import torch
from torchregress.metrics import MultivariateMAE

y_pred = torch.randn(100, 10)
y_true = torch.randn(100, 10)

metric = MultivariateMAE()
mae = metric(y_pred, y_true)

TaskAgnosticCorrelations (TAC)¶

Task-Agnostic Correlations (TAC) is a metric designed to evaluate the accuracy of predicted covariance matrices. It evaluates how well the predicted covariance captures target-level correlations by masking one dimension of the target variable \(y_i\) at a time and predicting it from the remaining observed dimensions \(y_{-i}\) using conditional Gaussian updates.

Mathematical Definition¶

For a predicted mean \(\hat{y}\) and predicted covariance \(\Sigma \in \mathbb{R}^{D \times D}\) at a given sample, let \(i \in \{1, \dots, D\}\) be the masked dimension, and let \(-i\) denote all other dimensions. The conditional mean update for the masked target \(y_i\) given the observed target variables \(y_{-i}\) is:

The TAC error measures the average absolute deviation of these conditionally updated predictions from the true target values:

A lower TAC score indicates that the predicted covariance matrix \(\Sigma\) more accurately captures the correlation structure between targets, allowing for better conditional updates.

Each covariance is inverted as \(\Sigma_n + \epsilon\,\overline{\operatorname{diag}\Sigma_n}\,I\) (jitter \(\epsilon = 10^{-6}\), relative to the mean variance), so TAC scales with the targets: multiplying \(y\) and \(\hat y\) by \(c\) and \(\Sigma\) by \(c^2\) multiplies TAC by \(c\).

Comparison Table: When to use which¶

Limitations¶

1. MultivariateRMSE / MAE ignore correlation structure: These treat each output dimension independently, rewarding models that predict the marginal means well but providing no signal about covariance quality.
2. TAC requires a covariance prediction: The metric is only applicable when the model outputs a covariance matrix (full, low-rank, or diagonal). It does not apply to independent-output models.
3. TAC is conditional on the observed targets: The conditional update formula uses the true values of other dimensions, so the metric evaluates covariance quality in a supervised setting — good for model selection but not for unsupervised covariance validation.

Recommendations¶

• Report MultivariateRMSE as a baseline alongside TAC to separate point accuracy from covariance quality.
• For models with diagonal covariance only, the correlation structure evaluation is unavailable — use dimension-wise interval metrics instead.
• Use task_agnostic_correlations with a held-out validation set to avoid overfitting the covariance evaluation.

References¶

Example¶

Here is a complete, runnable example using TaskAgnosticCorrelations and the functional wrapper task_agnostic_correlations:

import torch
from torchregress.metrics import TaskAgnosticCorrelations, task_agnostic_correlations

# Batch of 5 samples, target dimension 3
y_pred = torch.randn(5, 3)
y_true = torch.randn(5, 3)

# Symmetric positive-definite covariance matrices [B, D, D]
cov = torch.eye(3).unsqueeze(0).expand(5, -1, -1) + 0.1 * torch.randn(5, 3, 3)
cov = torch.bmm(cov, cov.transpose(1, 2))  # Ensure SPD

# Class-based metric usage
metric = TaskAgnosticCorrelations()
tac_val = metric(y_pred, y_true, cov)
print(f"TAC Metric value: {tac_val.item():.4f}")

# Functional usage
tac_func_val = task_agnostic_correlations(y_pred, y_true, cov)
print(f"TAC Functional value: {tac_func_val.item():.4f}")

Next steps¶

• Point metrics — standard univariate baselines to decompose multivariate error per dimension
• Ensemble metrics — uncertainty decomposition for multivariate ensemble predictions
• TIC-TAC — Taylor-Induced Covariance for improved multivariate covariance estimation
• Distributional metrics — Energy Score as the multivariate generalisation of CRPS