Best Practices¶

Building reliable regression models requires more than picking a loss function. This guide covers the principled workflow — from baseline to deployment — with rigorous evaluation and common failure modes.

The 3-Phase Development Workflow¶

Phase 1: Baseline (Day 1)¶

Start with simple point prediction. This establishes a performance floor and catches data or architecture issues before investing in uncertainty modeling.

loss_fn = WeightedHuberLoss(delta=1.0)  # robust default for mild outliers

• Goal: Establish a performance floor.
• Check: Can the model overfit a tiny subset (5–10 samples)? If not, check learning rate, architecture, or data pipeline.
• Metric: RMSE and \(R^2\).

Phase 2: Probabilistic Modeling (Day 2–3)¶

Switch to a heteroscedastic loss to estimate aleatoric uncertainty — the irreducible noise in the data.

loss_fn = GaussianNLLLoss()  # outputs [mean, log_var]

• Goal: Estimate input-dependent aleatoric uncertainty.
• Check: Does \(\sigma(x)\) align with actual error magnitudes? Plot a reliability diagram.
• Metric: NLL and CRPS — proper scoring rules that reward both accuracy and honest uncertainty (Gneiting & Raftery 2007).

Phase 3: Robustness & Refinement (Week 1+)¶

Add robustness (bounded-influence losses), epistemic uncertainty (ensembles), or distribution-free guarantees (conformal prediction).

• Goal: Handle outliers, quantify model uncertainty, or guarantee coverage.
• Check: Compare losses on held-out data with known contamination.
• Metric: CRPS for overall distributional quality; calibration curves for interval honesty.

Data Preparation¶

Feature Scaling¶

Neural networks are sensitive to input scale. Always standardize \(X\) before training:

from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)

Target Scaling¶

For GaussianNLLLoss, scale targets \(y\) to zero mean and unit variance so the model's initial guess (\(\mu \approx 0\), \(\sigma \approx 1\)) is reasonable.

Numerical Stability¶

Predict Log-Variance, Not Variance¶

Output \(s = \log \sigma^2\), never \(\sigma^2\) directly. The exponential enforces positivity and prevents NaN gradients (Kendall & Gal 2017):

logvar = model(x)
var = torch.exp(logvar)       # always positive
std = torch.sqrt(var)

Initialise the log-variance head to predict \(\log \sigma^2 \approx 0\) (i.e. \(\sigma \approx 1\)) so the model starts with a neutral prior.

Variance Clamping¶

Gaussian NLL can diverge when predicted variance approaches zero. Enforce a floor:

loss_fn = GaussianNLLLoss(min_variance=1e-6)  # stabilises early training

Evaluation Strategy¶

Never Trust a Single Metric¶

Rule of thumb: report RMSE/NLL/CRPS + calibration diagram as the minimal set for any probabilistic regression paper or report.

Diagnostic Plots¶

• Reliability diagram (calibration curve): bin predicted probabilities vs. empirical frequencies. Deviations from the diagonal signal miscalibration.
• PIT histogram: probability integral transform values should be uniform if the predictive distribution is well-calibrated.

See Visualization Methods for implementation.

Common Pitfalls¶

When to Reach for Each Method¶

Conditional Density and Multi-Target Predictions¶

For conditional densities, multi-target distributions, known label noise, and calibration, see Multi-Target Regression and the density pages rather than treating a normalizing flow as a drop-in point predictor. The short version: NSF for exact likelihood, BPF for CRPS/coverage, CFM for cheap samples without NLL, and split-conformal/Mondrian or WALDO for calibrated regions.

References¶