Skip to content

Output Constraints & Regularisation

← Post-Hoc Calibration | Causal Inference →

Output constraints are architectural layers that enforce structural properties on your model's predictions — such as non-negativity, boundedness, or monotonicity — by construction. Unlike penalty-based methods (which only "encourage" constraints), these layers guarantee zero violations for every single forward pass.

Why Use Architectural Constraints?

Benefit Description
Exact Enforcement Zero constraint violations at any point in the input space.
Differentiability All layers are fully differentiable (even sorting), enabling end-to-end training.
Physical Validity Ensure predictions respect known physical bounds (e.g., mass > 0, probability ∈ \([0, 1]\)).
Numeric Stability Prevents \(NaN\)s and exploding gradients by keeping outputs within valid ranges.

Available Constraints

torchregress provides several constraint-enforcing layers that can be added as the final stage of any PyTorch model.

1. Positivity (NonNegativeHead)

Ensures all outputs are non-negative (\(\hat{y} \geq 0\)) via a Softplus transformation:

\[\hat{y} = \log(1 + e^{x})\]
import torch.nn as nn
from torchregress.constraints import NonNegativeHead

# NonNegativeHead wraps the final linear layer and applies softplus to its outputs
model = nn.Sequential(
    nn.Linear(64, 32), nn.ReLU(),
    NonNegativeHead(nn.Linear(32, 1), beta=1.0),
)

2. Boundedness (BoundedHead)

Enforces strict lower and upper bounds (\(a \leq \hat{y} \leq b\)) via a scaled Sigmoid:

\[\hat{y} = a + (b - a) \cdot \sigma(x)\]
import torch.nn as nn
from torchregress.constraints import BoundedHead

# Enforce output between 0 and 100
head = BoundedHead(nn.Linear(64, 1), low=0.0, high=100.0)

3. Non-Crossing (NonCrossingSort)

Ensures that multiple outputs are monotonically ordered. This is critical for Quantile Regression to prevent "quantile crossing" (where the 90th quantile is predicted to be less than the 50th).

from torchregress.constraints import NonCrossingSort

# Predict 5 ordered quantiles
model = nn.Sequential(nn.Linear(64, 5), NonCrossingSort())

4. Simplex (SimplexHead)

Ensures that outputs sum to 1 and are non-negative (\(\sum \hat{y}_i = 1, \hat{y}_i \geq 0\)) via Softmax. Ideal for mixture weights or compositional data.

import torch.nn as nn
from torchregress.constraints import SimplexHead

head = SimplexHead(nn.Linear(64, 5), dim=-1)

Method Selection Matrix

Layer Constraint API Reference Best For
NonNegativeHead \(\hat{y} \geq 0\) NonNegativeHead Price, Variance, Count
BoundedHead \([a, b]\) BoundedHead Probabilities, Percentages
NonCrossingSort \(y_1 \leq y_2 \dots\) NonCrossingSort Quantile Regression
SimplexHead \(\sum y_i = 1\) SimplexHead Mixture Models, Weights

Advanced: Spectral Normalisation (SpectralNormWrapper)

For stability and Out-of-Distribution (OOD) detection, torchregress provides a SpectralNormWrapper. This applies PyTorch spectral normalization to bound layer Lipschitz constants, which can improve smoothness on inputs far from the training data [1].

import torch.nn as nn
from torchregress.constraints import SpectralNormWrapper

base_model = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 1))
model = SpectralNormWrapper(base_model)  # spectral_norm on Linear layers

OOD Sensitivity

Spectral normalization is a heuristic stabilizer — it is not an OOD-detection API. Treat smoother outputs as a practical inductive bias, not a guarantee.

Best Practices

Softplus vs Exponential

Prefer Softplus over Exponential for enforcing positivity. Exp is numerically unstable for large inputs and can lead to \(NaN\) gradients.

Sorting vs Monotonicity

NonCrossingSort enforces ordering across different output units. If you need a model to be monotonic with respect to an input feature, see our Monotonic Regression Guide.

References

# Reference
1 Miyato et al. "Spectral Normalization for Generative Adversarial Networks." ICLR, 2018.
2 Gasthaus et al. "How to Avoid Being Mean: Non-Crossing Quantile Regression." NeurIPS, 2019.
3 Koenker, R. "Quantile Regression." Cambridge University Press, 2005.

Limitations

  1. Output-dimension only: Constraints enforce structure on model outputs (non-negativity, ordering, sum-to-one). They do not enforce monotonicity with respect to input features — for input monotonicity, use specialised architectures.
  2. NonCrossingSort is not convexity-preserving: Sorting enforces pairwise ordering but can create non-smooth prediction surfaces. The sorted output is differentiable, but gradients can be discontinuous at tie points.
  3. SpectralNormWrapper is a heuristic: Spectral normalisation bounds Lipschitz constants for smoother OOD behaviour, but is not an OOD-detection API. Treat as a practical inductive bias, not a guarantee.
  4. Softplus vs Exponential: NonNegativeHead uses Softplus (\(\log(1+e^x)\)) which is numerically stable but has a soft lower bound near zero (not exactly zero). For exact zero-inflation, combine with a separate zero-model.
  5. Constraints can mask model errors: A model with a BoundedHead(0, 1) can still produce poor predictions within \([0,1]\). Constraints ensure validity, not accuracy.

Recommendations

  • Default positivity: Use NonNegativeHead with beta=1.0 (standard Softplus). Adjust beta only if you need sharper or softer lower-bound behaviour.
  • Quantile regression: Always use NonCrossingSort on multi-quantile outputs to prevent quantile crossing. Combine with QuantileCrossoverLoss for loss-level enforcement. See Quantile & Expectile losses.
  • Probability outputs: Use BoundedHead(0, 1) for probability predictions and SimplexHead for categorical/mixture weights.
  • Spectral normalisation: Apply SpectralNormWrapper when you observe erratic predictions on OOD inputs. It adds minimal overhead (one power iteration per forward pass).
  • Stack constraints: Combine multiple constraint heads (e.g., NonNegativeHead → NonCrossingSort). Order matters — apply domain constraints first, then ordering constraints.

Next Steps