Benchmarks¶

torchfits benchmarks cover FITS tensor I/O (IMAGE HDUs, typically 1D–4D) and FITS table I/O vs Astropy and fitsio. CPU↔GPU comparisons are published when hardware was available; GPU deficits are listed, not hidden.

Honesty: torchfits is a 1.0.0rc4 prerelease. Headline ratios below are lab medians from named scorecard runs — not guarantees on your filesystem, file mix, or PyTorch version. Check Performance deficits before assuming torchfits wins every case.

How to read this page¶

Published GPU/CPU numbers come from the multi-host release scorecard (exhaustive_mps_*, exhaustive_cpu_*, exhaustive_cuda_*). Manual workflow_dispatch on .github/workflows/bench-report.yml is CPU-only and does not refresh GPU cells.

Comparison targets¶

Methodology¶

Each case measures median wall-clock time over multiple repetitions, plus peak process RSS (and peak CUDA alloc when on CUDA). Deficit ranking is time-based; RSS is reported alongside times.

Cases are grouped into two families:

• default — high-level API (torchfits.read / table.read, etc.).
• specialized — torchfits_specialized methods (open-once handle / open_subset_reader paths). Empty specialized cells mean that path was not measured for the case.

Fairness controls:

• Rows with mismatched mmap behavior are marked SKIPPED and excluded from rankings.
• Why fitsio has no mmap rows: fitsio does not expose a comparable mmap toggle. Under mmap_target=on / strict_mmap_fairness, fitsio rows are non-comparable and show as skipped in transport tables (see scripts/render_bench_iopath_table.py).
• Warm-cache and cold-cache profiles are kept separate.

Disk to GPU¶

True disk→GPU (GPUDirect Storage / cuFile, or a CFITSIO path that never touches host RAM) is not implemented. Every Python FITS stack here decodes on the host, then copies with .to(device). Exploring a direct path is a 1.1 candidate (see Roadmap) — not a 1.0 claim.

Tables on GPU transports¶

Table GPU transport rows compare table.read_torch(..., device=cpu) against device=cuda / device=mps on a medium mixed catalog case (mixed_100000). Decode still happens on the host; the GPU column measures host decode plus H2D copy into tensor columns.

Published CSVs¶

Exhaustive results.csv / torchfits_deficits.csv for the scorecard runs are linked from GitHub Release assets when published, and mirrored under docs/assets/bench/<run-id>/ when size allows. Example local paths used to build this page:

• docs/assets/bench/exhaustive_mps_20260719_143706/results.csv
• docs/assets/bench/exhaustive_cpu_20260719_144337/results.csv
• docs/assets/bench/exhaustive_cuda_20260719_144457/results.csv

(also under benchmarks_results/<run-id>/ locally)

Deficit floors (same-mmap peers):

• Images / cubes / spectra / cutouts (domain=fits): any lag above float-timer ε counts — including rice/hcompress (no percent floor).
• Arrow table interchange (domain=fitstable): allow up to 1.05×.

Modular suites and release exhaustives¶

Named suites live in benchmarks/suites.py and resolve to bench_all.py flags (--scope / --filter / --operation / GPU / mmap / profile):

pixi run bench-suite hcompress
pixi run bench-suite compressed_rice -- --no-mmap
pixi run bench-suite fitstable_predicate
pixi run bench-deficit-focus          # registry-driven deficit clusters

Release composition is the release suite (full fits + fitstable, mmap matrix, GPU when present). Host recipes:

CFHT MegaCam cutout suite¶

Public CFHT MegaCam MEF samples (CADC Direct Data Service) exercise Rice .fz repeated cutouts with peer ranking:

Uses ZNAXIS* for tile-compressed sizes; throughput is cutout payload MB/s.

bash scripts/fetch_cfht_megacam_sample.sh   # once; idempotent
pixi run bench-megacam

Outputs land in benchmarks_results/<run-id>/megacam_results.csv. Sample FITS files are gitignored under benchmarks_data/cfht_megacam/.

For uncompressed survey mosaics (e.g. CFHTLS MegaPipe float32 stacks), open_subset_reader maps the data segment once and slices cutouts with endian swap into torch tensors — see ML with FITS. Rice .fz MegaCam cutouts remain a separate comparison (tile decompress inside CFITSIO).

Correctness Gates¶

Reproducing¶

pixi run bench-fits
pixi run bench-fitstable
pixi run bench-all
pixi run bench-ml
bash scripts/fetch_cfht_megacam_sample.sh && pixi run bench-megacam
# Full transport matrix (mmap on + off, doubles CPU rows; GPU rows for both when CUDA/MPS):
pixi run -e bench-gpu python benchmarks/bench_all.py --profile lab --scope all --mmap-matrix

For focused FITS partitions:

pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(tiny_)'
pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(small_)'
pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(medium_|large_)'
pixi run -e bench-all python benchmarks/bench_all.py --scope fits --filter '^(scaled_|compressed_|mef_)'

Named deficit-cluster recipes (mmap on+off, no unrelated GPU matrix when scoped to tables):

pixi run bench-deficit-focus              # hcompress + tiny_int8 + narrow predicates
pixi run bench-deficit-focus hcompress
pixi run bench-deficit-focus tiny_int8
pixi run bench-deficit-focus predicate

Rankings and deficit scoring group by (domain, case_id, family, mmap_target) so mmap-on and mmap-off peers are never cross-compared.

Benchmark Scripts¶

Coverage matrix¶

What the exhaustive bench-all suite measures today, and what is intentionally out of scope or not yet wired into the published tables.

Why the I/O transport table looks sparse on GPU¶

1. disk→GPU is always empty — backends decode on the host first, then .to(device). See Disk to GPU.
2. disk→CPU→GPU vs disk→RAM→GPU — mmap-off vs mmap-on host decode + H2D.
3. GPU rows need CUDA/MPS hardware — published CUDA numbers come from CANFAR staging (exhaustive_cuda_20260719_144457).
4. Tables — see Tables on GPU transports.

GPU integer dtype comparisons¶

The deficit table compares default torchfits.read(..., scale_on_device=True) against torch.from_numpy(fitsio.read(...)).to(cuda). That pairing is not dtype-equivalent for every scaled integer FITS file.

For apples-to-apples integer GPU timing, the suite also records torchfits_dtype_fair_device (read_tensor(..., raw_scale=True)).

Training loops: call torchfits.cache.optimize_for_dataset(paths, avg_file_size_mb=…) before DataLoader epochs so handle caches stay warm.

Refreshing GPU numbers (CANFAR staging)¶

CUDA lab numbers come from a headless GPU session on @staging. From a machine with canfar x509 auth:

bash scripts/selfcheck_canfar_launcher.sh
TORCHFITS_CANFAR_IMAGE=astroai/notebook:latest TORCHFITS_BENCH_MODE=exhaustive \
  pixi run bench-canfar-gpu
bash scripts/fetch_canfar_bench_vos.sh exhaustive_cuda_<stamp>
bash scripts/patch_canfar_exhaustive_docs.sh exhaustive_cuda_<stamp>

# Local CI + docs before push
bash scripts/ci_local.sh
# Apple Silicon (MPS transport rows)
pixi run bench-mps

I/O transport and backend¶

GPU summary: Tensor disk→CPU→GPU / disk→RAM→GPU rows appear only when the CSV was produced on CUDA or MPS. disk→GPU stays empty (unsupported). Table GPU cells stay empty until the table-GPU suite lands.

Source: benchmarks_results/exhaustive_mps_20260719_143706/results.csv (mmap on+off matrix; MPS/CUDA GPU transport rows included.) Cell values are median wall-clock over all comparable OK rows in the (domain × I/O transport × backend) bucket; throughput is intentionally omitted because the cell aggregates heterogeneous payloads and would produce physically-impossible rates when small and large sizes are median-mixed. See scripts/render_bench_iopath_table.py for the aggregation rules.

Tensor I/O (IMAGE HDU) (fits)¶

Table I/O (fitstable)¶

Notes on the layout¶

• Rows are I/O transports (disk→CPU, disk→RAM→CPU, disk→GPU, disk→CPU→GPU, disk→RAM→GPU).
• Columns are backends (torchfits / astropy / fitsio / cfitsio-direct).
• Pure-C CFITSIO (vendored): pixi run bench-cfitsio-direct runs the full image+table scorecard fixture set with op→API mapping in benchmarks/cfitsio_direct/bench_cfitsio_direct.c (fits_read_img / fits_read_subset / fits_read_record / fits_read_tblbytes / fits_read_col). CSV: benchmarks_results/<run-id>/cfitsio_direct.csv.
• Cell n= counts comparable OK rows in the bucket; — indicates the bucket is empty (no rows match, or rows were excluded under strict_mmap_fairness in the original bench-all summary).
• Median is computed over heterogeneous operations (read_full, cutout_100x100, header_read, predicate_filter, projection, row_slice, etc.) and payload sizes; treat the per-cell ms as a coarse representative number, not a precise benchmark.

Performance highlights¶

The following table showcases median wall-clock times for key FITS tensor and table cases. The specialized column is torchfits_specialized (open-once / subset-reader paths); it is empty when that path was not measured.

Benchmark category summary¶

Aggregated wins across every domain and operation in the CANFAR CUDA exhaustive (exhaustive_cuda_20260719_144457, 4,087 rows; see host scorecard for deficit honesty — all lags listed, floors label noise vs significant). Category ranges below are the last regenerated aggregation shape; for this run’s absolute times prefer Performance highlights and the full table.

FITS image I/O¶

GPU (CUDA) image reads — 76 read_full cases:

FITS table I/O¶

Exhaustive Benchmark Results¶

The complete, un-cherrypicked list of all measured configurations. Empty cells mean that method was not run for the case (for example torchfits_specialized is only used for open-once / subset-reader paths). Domain tensor = IMAGE HDU payloads (1D–4D); table = binary/ASCII tables.