Tabular MLP¶

torchregress.models ships a strong-default neural network for medium-size tabular regression: robust-scaled inputs, periodic numeric embeddings, SiLU blocks with dropout, AdamW with a one-cycle schedule, early stopping on a validation split, and target standardisation. It plugs into any torchregress loss, so the same backbone gives point predictions, a heteroscedastic Gaussian, or quantile intervals.

When to use it¶

Mathematical background¶

Robust input scaling¶

For column \(j\) with training median \(m_j\) and interquartile range \(s_j = q_{75} - q_{25}\) (half the range if the IQR is zero, 1 if the column is constant):

The smooth clip \(\tilde z_j \in (-c, c)\) is strictly monotone, so it is exactly invertible (inverse_transform) and keeps heavy-tailed columns from dominating the first layer (RealMLP, Holzmüller et al., 2024).

Periodic numeric embedding¶

Each scalar feature \(\tilde z_j\) is lifted by \(k\) learned frequencies \(c_{j} \in \mathbb{R}^k\), initialised from \(\mathcal{N}(0, \sigma^2)\) (frequency_scale):

with a separate \(W_j \in \mathbb{R}^{d \times 2k}\) per feature (Gorishniy et al., 2022). The raw \(\tilde z_j\) is concatenated to \(e_j\), so the MLP sees \(F\,(d+1)\) inputs for \(F\) features. Sines and cosines at several frequencies let an MLP represent sharp, non-monotone responses to a single feature that a plain ReLU network learns slowly (spectral bias). Set embedding="none" to feed \(\tilde z\) directly.

Training recipe¶

Targets are standardised, \(y' = (y - \mu_y)/\sigma_y\), and the loss is evaluated on \(y'\). Predictions are mapped back by TabularFit.predict:

Optimisation is AdamW (decoupled weight decay) with a one-cycle learning-rate schedule (10 % warm-up, cosine decay), gradient-norm clipping at 1, and early stopping on the validation loss with the best weights restored.

Loss heads¶

Example¶

import numpy as np
import torch
from torchregress.losses import GaussianNLLLoss
from torchregress.models import TabularMLP, fit_tabular

rng = np.random.default_rng(0)
X = rng.uniform(-2, 2, size=(4000, 5))
y = np.sin(2 * X[:, 0]) + X[:, 1] * X[:, 2] + (0.1 + 0.5 * np.abs(X[:, 3])) * rng.normal(size=4000)

torch.manual_seed(0)
model = TabularMLP(in_features=5, out_features=2)            # [mean, log_variance]
fit = fit_tabular(model, GaussianNLLLoss(), X[:3000], y[:3000], epochs=60, seed=0)
mean, log_var = fit.predict(X[3000:]).unbind(dim=1)          # original target units
print(fit.best_epoch, float(torch.sqrt(((mean.numpy() - y[3000:]) ** 2).mean())))

from torchregress.losses import WeightedMSELoss
from torchregress.models import TabularMLP, TabularPreprocessor, fit_tabular

prep = TabularPreprocessor().fit(X_train)
model = TabularMLP(in_features=prep.n_features_out_, out_features=1)
fit = fit_tabular(model, WeightedMSELoss(), X_train, y_train, preprocessor=prep)

Limitations¶

• Full-batch data is moved to the training device once; this targets medium tables, not out-of-core data.
• fit_tabular_ensemble averages raw head outputs. That is exact for point and quantile heads; for Gaussian heads it is not the mixture variance (use predict_members and build the mixture yourself).
• Early stopping with a one-cycle schedule can stop while the learning rate is still high; the best weights are restored, but a longer epochs with patience equal to epochs follows the full schedule.
• Raw categorical columns must be encoded numerically before use.

Next steps¶

• Models API — signatures and attributes
• Gaussian losses and quantile losses — the heads this backbone is built for
• Ensemble methods — epistemic uncertainty on top of a backbone
• Conformal prediction — coverage guarantees for the fitted predictor

References¶