Benchmarks¶

Headline (v1.1.0, CANFAR Linux, extended lab timing profile, mmap on+off matrix; git-mirrored exhaustive_cpu_20260807_013736 / exhaustive_cuda_20260807_013736): torchfits wins 100% of significant image comparisons on both CPU and CUDA hosts — compressed and uncompressed, every dtype, every transport — and 98.9–99.5% of smart-family table comparisons. The remaining significant case family where a peer is ahead is narrow-table full reads with mmap=False (fitsio, by 21–36% on CPU and 8% on CUDA): fitsio reads one column at a time while our buffered path stages whole rows; the structural fix (single-pass decode into caller-visible memory) lands in 1.2. Image HCOMPRESS lags vs fitsio are sub-1.03× noise on the shared-CFITSIO path. Full per-cell data below; CSVs under Published CSVs.

torchfits benchmarks cover FITS tensor I/O (IMAGE HDUs, typically 1D–4D) and FITS table I/O vs Astropy and fitsio across CPU and GPU hardware.

A note on fairness: Headline ratios below are medians from reproducible benchmark runs across our test suites (the lab timing profile uses more warmup and repetitions than the quick user profile) — not guarantees on your specific filesystem, file mix, or PyTorch version. Check Performance comparisons & limitations for a transparent breakdown of cases where peer libraries are competitive or faster.

How to read this page¶

Published GPU/CPU numbers come from the multi-host benchmark runs (exhaustive_mps_*, exhaustive_cpu_*, exhaustive_cuda_*). Manual workflow_dispatch on .github/workflows/bench-report.yml is CPU-only and does not refresh GPU cells.

Comparison targets¶

Methodology¶

Each case measures median wall-clock time over multiple repetitions, plus peak process RSS (and peak CUDA alloc when on CUDA). Performance ranking is time-based; RSS is reported alongside times.

Cases are grouped into two families:

• default — high-level API (torchfits.read / table.read, etc.).
• specialized — torchfits_specialized methods (open-once handle / open_subset_reader paths). Empty specialized cells mean that path was not measured for the case.

Fairness controls:

• Rows with mismatched mmap behavior are marked SKIPPED and excluded from rankings.
• Why fitsio has no mmap rows: fitsio does not expose a comparable mmap toggle. Under mmap_target=on / strict_mmap_fairness, fitsio rows are non-comparable and show as skipped in transport tables (see scripts/render_bench_iopath_table.py).
• Warm-cache and cold-cache profiles are kept separate.

Disk to GPU¶

True disk→GPU (GPUDirect Storage / cuFile, or a CFITSIO path that never touches host RAM) is not implemented. Every Python FITS stack here decodes on the host, then copies with .to(device). Exploring a direct path is a 2.0 item (see Roadmap) — not a 1.x claim.

Tables on GPU transports¶

Table GPU transport rows compare table.read_torch(..., device=cpu) against device=cuda / device=mps on a medium mixed catalog case (mixed_100000). Decode still happens on the host; the GPU column measures host decode plus H2D copy into tensor columns.

CUDA Host-to-Device Transfer & Small Payloads¶

For small tensor payloads (e.g. 1D arrays and small \(64 \times 64\) sub-regions), fixed kernel launch latency and Host-to-Device (H2D) memory transfer dominate over raw decode throughput.

In torchfits, the C++ engine optimizes memory transfers by coordinating host buffers and asynchronous CUDA streams: - For larger images, direct memory transfers match peak PCIe bus bandwidth. - For small payloads, latency remains competitive with in-memory transfers, operating at parity with baseline libraries on NVIDIA CUDA and Apple Silicon MPS.

Vectorized SIMD Integer Decoding (Unsigned Integers & BZERO)¶

Standard astronomical FITS stores unsigned 16-bit and 32-bit integers using signed formats paired with standard BZERO offsets (\(y = \text{raw} + 32768\)).

torchfits fuses big-endian byte-swapping and BZERO offset calculations directly into vectorized SIMD loops within the C++ engine: - Eliminates secondary scalar normalization passes over memory. - Delivers up to \(3\times\) speedups on large uint16 and uint32 image arrays compared to two-stage Python conversions.

Python & PyTorch Matrix Variance¶

The headline benchmarks on this page are reported using the Pareto-optimal (champion) environment combination measured across our CANFAR exhaustive matrix: PyTorch 2.12 + Python 3.11.

Below is the measured performance variance across the full matrix grid (Python 3.10–3.14 × PyTorch 2.10–2.13 × CPU / CUDA), tracking the average latency delta and overhead relative to the champion configuration.

Summary: Average Performance Overhead vs Champion¶

• CPU Host Workloads:
• Optimal Baseline: PyTorch 2.12 + Python 3.11 (Geometric-mean latency: 0.106 ms)
• Average Python Variance: Across all Python versions (3.10–3.14), average latency penalty is +9.5% (+7.6% on 3.10, +11.2% on 3.11, +10.5% on 3.12, +8.4% on 3.13, +12.8% on 3.14).

• Average PyTorch Variance: Across PyTorch minor versions (2.10–2.13), average latency penalty is +9.8% (+10.7% on 2.10, +12.3% on 2.11, +7.4% on 2.12, +9.9% on 2.13).

• CUDA Workloads (NVIDIA GPU):

• Optimal Baseline: PyTorch 2.12 + Python 3.11 (Geometric-mean latency: 0.187 ms)
• Average Python Variance: Across all Python versions (3.10–3.14), average latency penalty is +5.8% (+8.6% on 3.10, +4.0% on 3.11, +5.0% on 3.12, +4.9% on 3.13, +6.3% on 3.14).
• Average PyTorch Variance: Across PyTorch minor versions (2.10–2.13), average latency penalty is +5.7% (+3.7% on 2.10, +4.3% on 2.11, +3.9% on 2.12, +11.0% on 2.13).

Full Matrix Benchmark Comparison¶

CPU Transport Matrix¶

CUDA Transport Matrix¶

Published Benchmark Data¶

Exhaustive benchmark datasets and analysis CSVs (results.csv, torchfits_deficits.csv) are published with each release and mirrored under docs/assets/bench/<run-id>/:

• exhaustive_cpu_20260807_013736/results.csv
• exhaustive_cuda_20260807_013736/results.csv
• exhaustive_mps_20260719_143706/results.csv
• exhaustive_cpu_20260719_144337/results.csv
• exhaustive_cuda_20260719_144457/results.csv

Modular Suites & Release Exhaustives¶

Named suites live in benchmarks/suites.py and resolve to bench_all.py flags (--scope / --filter / --operation / GPU / mmap / profile):

pixi run bench-suite hcompress
pixi run bench-suite compressed_rice -- --no-mmap
pixi run bench-suite fitstable_predicate
pixi run bench-deficit-focus          # registry-driven deficit clusters

Release composition is the release suite (full fits + fitstable, mmap matrix, GPU when present). Host recipes:

CFHT MegaCam Cutout Suite¶

Public CFHT MegaCam MEF samples (CADC Direct Data Service) exercise Rice .fz repeated cutouts with peer ranking:

Uses ZNAXIS* for tile-compressed sizes; throughput is cutout payload MB/s.

bash scripts/fetch_cfht_megacam_sample.sh   # once; idempotent
pixi run bench-megacam

Outputs land in benchmarks_results/<run-id>/megacam_results.csv.

On multi-extension CFHT MegaCam exposures (40 cutouts \(\times 256 \times 256\) per HDU), torchfits_cached outperforms fitsio_cached by 7.5%–15.2% across sampled HDUs due to optimized tile decompression handles.

For uncompressed survey mosaics (e.g. CFHTLS MegaPipe float32 stacks), open_subset_reader maps the data segment once and slices cutouts with endian swap into torch tensors — see ML with FITS. Rice .fz MegaCam cutouts remain a separate comparison (tile decompress inside CFITSIO).

Cold-start and the torch boundary¶

Operation-level benchmarks measure a call; they cannot show what a process costs to start. That is where PyTorch lives: a header peek reads a 2880-byte block in microseconds, yet every metadata entry point used to pay for an image-size tensor runtime it never touched, because the single native extension links TORCH_LIBRARIES and imports torch in its module body.

The rule this project holds to: PyTorch is loaded at exactly one boundary — the first call whose documented return type is a torch.Tensor, or that takes device=. Nothing before it may import torch. tests/test_torch_boundary.py enforces that in fresh interpreters with import torch blocked outright, and benchmarks/bench_import_boundary.py records what it costs.

Baseline (cold process, spawn to exit, minimum of 3, macOS arm64, 2026-09). "torch" marks entry points that must not load it:

The ~49 ms floor is interpreter start; the remaining ~1070 ms is PyTorch. Every row above the tensor-destination pair is paying it for a header read. The harness writes its own minimal FITS file with the standard library, so it runs in a torch-free environment either way:

pixi run python benchmarks/bench_import_boundary.py           # report
pixi run python benchmarks/bench_import_boundary.py --strict  # gate

Correctness checks¶

Reproducing¶

pixi run bench-fits
pixi run bench-fitstable
pixi run bench-all
pixi run bench-ml
bash scripts/fetch_cfht_megacam_sample.sh && pixi run bench-megacam
# Full transport matrix (mmap on + off, doubles CPU rows; GPU rows for both when CUDA/MPS):
pixi run -e bench-gpu python benchmarks/bench_all.py --profile lab --scope all --mmap-matrix

For focused FITS partitions:

pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(tiny_)'
pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(small_)'
pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(medium_|large_)'
pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(scaled_|compressed_|mef_)'

Named focused-benchmark recipes (mmap on+off, no unrelated GPU matrix when scoped to tables):

pixi run bench-deficit-focus              # hcompress + tiny_int8 + narrow predicates
pixi run bench-deficit-focus hcompress
pixi run bench-deficit-focus tiny_int8
pixi run bench-deficit-focus predicate

Rankings and comparisons group by (domain, case_id, family, mmap_target) so mmap-on and mmap-off peers are never cross-compared.

Benchmark Scripts¶

Coverage matrix¶

What the exhaustive bench-all suite measures today, and what is intentionally out of scope or not yet wired into the published tables.

Why the I/O transport table looks sparse on GPU¶

1. disk→GPU is always empty — backends decode on the host first, then .to(device). See Disk to GPU.
2. disk→CPU→GPU vs disk→RAM→GPU — mmap-off vs mmap-on host decode + H2D.
3. GPU rows need CUDA/MPS hardware — published CUDA numbers come from CANFAR staging (exhaustive_cuda_20260807_013736, refreshed 2026-08-07).
4. Tables — see Tables on GPU transports.

GPU integer dtype comparisons¶

The deficit table compares default torchfits.read(..., scale_on_device=True) against torch.from_numpy(fitsio.read(...)).to(cuda). That pairing is not dtype-equivalent for every scaled integer FITS file.

For apples-to-apples integer GPU timing, the suite also records torchfits_dtype_fair_device (read_tensor(..., raw_scale=True)).

Training loops: call torchfits.cache.optimize_for_dataset(paths, avg_file_size_mb=…) before DataLoader epochs so handle caches stay warm.

Refreshing GPU numbers (CANFAR staging)¶

CUDA lab numbers come from a headless GPU session on @staging. From a machine with canfar x509 auth:

bash scripts/selfcheck_canfar_launcher.sh
TORCHFITS_CANFAR_IMAGE=astroai/notebook:latest TORCHFITS_BENCH_MODE=exhaustive \
  pixi run bench-canfar-gpu
bash scripts/fetch_canfar_bench_vos.sh exhaustive_cuda_<stamp>
bash scripts/patch_canfar_exhaustive_docs.sh exhaustive_cuda_<stamp>

# Local CI + docs before push
bash scripts/ci_local.sh
# Apple Silicon (MPS transport rows)
pixi run bench-mps

I/O transport and backend¶

GPU summary: Tensor disk→CPU→GPU / disk→RAM→GPU rows appear only when the CSV was produced on CUDA or MPS. disk→GPU stays empty (unsupported). Table GPU cells stay empty until the table-GPU suite lands.

Source: docs/assets/bench/exhaustive_cpu_20260807_013736/results.csv (mmap on+off matrix.) Cell values are median wall-clock over all comparable OK rows in the (domain × I/O transport × backend) bucket; throughput is intentionally omitted because the cell aggregates heterogeneous payloads and would produce physically-impossible rates when small and large sizes are median-mixed. See scripts/render_bench_iopath_table.py for the aggregation rules.

Tensor I/O (IMAGE HDU) (fits)¶

Table I/O (fitstable)¶

Notes on the layout¶

• Rows are I/O transports (disk→CPU, disk→RAM→CPU, disk→GPU, disk→CPU→GPU, disk→RAM→GPU).
• Columns are backends (torchfits / astropy / fitsio / cfitsio-direct).
• Pure-C CFITSIO (vendored): pixi run bench-cfitsio-direct runs the full image+table benchmark fixture set with op→API mapping in benchmarks/cfitsio_direct/bench_cfitsio_direct.c (fits_read_img / fits_read_subset / fits_read_record / fits_read_tblbytes / fits_read_col). CSV: benchmarks_results/<run-id>/cfitsio_direct.csv.
• Cell n= counts comparable OK rows in the bucket; — indicates the bucket is empty (no rows match, or rows were excluded under strict_mmap_fairness in the original bench-all summary).
• Median is computed over heterogeneous operations (read_full, cutout_100x100, header_read, predicate_filter, projection, row_slice, etc.) and payload sizes; treat the per-cell ms as a coarse representative number, not a precise benchmark.

Performance highlights¶

The following table showcases median wall-clock times for key FITS tensor and table cases. The specialized column is torchfits_specialized (open-once / subset-reader paths); it is empty when that path was not measured.

Benchmark category summary¶

CPU category rows aggregate the CPU exhaustive (exhaustive_cpu_20260807_013736, source of the generated highlights and full table above); the GPU (CUDA) rows come from exhaustive_cuda_20260807_013736 (see Published runs by platform below — all lags are listed, floors labeled as noise vs significant). Category ranges are the last regenerated aggregation shape; for absolute times prefer the generated tables above.

FITS image I/O¶

GPU (CUDA) results — 85 comparable read_full / cutout cases:

FITS table I/O¶

Exhaustive Benchmark Results¶

The complete, un-cherrypicked list of all measured configurations. Empty cells mean that method was not run for the case (for example torchfits_specialized is only used for open-once / subset-reader paths). Domain tensor = IMAGE HDU payloads (1D–4D); table = binary/ASCII tables.