Table Reference¶

FITS tables are columnar catalogs. Read them as Arrow (table.read), as a column → tensor map (table.read_torch), or as Polars (table.read_polars).

Arrow is the contract, not the native representation. The C++ engine exports each column as a Python-owned read-only buffer without constructing a Python tensor or importing Python torch. Clean numeric or fixed-width byte columns go straight into pa.Array.from_buffers. See Architecture for the boundary and what is still to move.

Destination	Call	Returns
Arrow table	`table.read` / `table.read_arrow`	`pyarrow.Table`
Column → tensor map	`table.read_torch`	`dict[str, torch.Tensor]` (VLA columns use list/tuple values)
Astropy Table	`table.read_astropy`	`astropy.table.Table`
Polars	`table.read_polars`	`FITSPolarsFrame` wrapper around `pl.DataFrame`

Supports where= filters, column projection, streaming, mutations, and handoff to Astropy, Polars, DuckDB, Pandas, and PyArrow.

Every table reader and mutator raises FileNotFoundError for a missing local file and torchfits.HDUNotFoundError (a KeyError, IndexError, ValueError and RuntimeError) for an hdu index or EXTNAME the file does not have; see Missing files and HDUs.

`table.read()`¶

Read a FITS table as a pyarrow.Table, with optional where= and column projection. table.read_arrow is the same function under another name.

torchfits.table.read(
    path,
    hdu=1,
    columns=None,
    row_slice=None,
    rows=None,
    where=None,
    batch_size=65536,
    mmap=True,
    decode_bytes=True,
    encoding="ascii",
    strip=True,
    include_fits_metadata=False,
    apply_fits_nulls=True,
    backend="auto",
)

Parameter	Type	Default	Description
`path`	`str`	(required)	FITS file path
`hdu`	`int` or `str`	`1`	Table HDU index or EXTNAME
`columns`	`list[str]` or `None`	`None`	Columns to read (`None` = all)
`row_slice`	`slice` or `tuple[int,int]` or `None`	`None`	Row range
`rows`	`list[int]` or `None`	`None`	Specific row indices
`where`	`str` or `None`	`None`	SQL-like row filter (full dialect)
`batch_size`	`int`	`65536`	Read batch size
`mmap`	`bool`	`True`	Memory-mapped reads
`decode_bytes`	`bool`	`True`	Decode byte-string columns
`encoding`	`str`	`"ascii"`	Byte decoding encoding
`strip`	`bool`	`True`	Strip trailing spaces on strings
`include_fits_metadata`	`bool`	`False`	Attach FITS column metadata on the Arrow schema
`apply_fits_nulls`	`bool`	`True`	Honor `TNULL` as nulls
`backend`	`str`	`"auto"`	`"auto"`, `"cpp"`, or `"torch"`

Returns: pyarrow.Table

df = torchfits.table.read(
    "catalog.fits",
    hdu=1,
    columns=["RA", "DEC", "MAG_G"],
    where="MAG_G < 20 AND DEC > 0",
)
print(df.num_rows, df.column_names)

`table.read_torch()`¶

Read selected columns as torch.Tensor values (dict[str, torch.Tensor]).

torchfits.table.read_torch(
    path,
    hdu=1,
    columns=None,
    start_row=1,
    num_rows=-1,
    device="cpu",
    mmap="auto",
    return_header=False,
    where=None,
)

Parameter	Type	Default	Description
`path`	`str`	(required)	FITS file path
`hdu`	`int` or `str`	`1`	Table HDU index or EXTNAME
`columns`	`list[str]` or `None`	`None`	Columns to return (`None` = all)
`start_row`	`int`	`1`	1-based first row
`num_rows`	`int`	`-1`	Row count (`-1` = through end; `0` returns every column empty, with its usual dtype, as a 0-row table does)
`device`	`str`	`"cpu"`	`"cpu"`, `"cuda"`, `"mps"`
`mmap`	`bool` or `str`	`"auto"`	`True` / `False` / `"auto"` (`"auto"` → mmap on)
`return_header`	`bool`	`False`	Also return the HDU `Header`
`where`	`str` or `None`	`None`	Simple numeric filter (see below)

cache_capacity controls the read cache (0 disables it) and fast_header only affects the unified-reader fallback. The Arrow path (table.read) takes neither. handle_cache_capacity was removed in 1.3.0, along with clear_file_cache(handles=) and configure_cache().

Returns: dict[str, torch.Tensor] (scalar columns; VLA columns use list/tuple values), or (dict, Header) when return_header=True.

Scalar-column shapes¶

FITS scalar columns (repeat == 1) are returned as rank-1 tensors of shape (N,) on every access path: table.read_torch, torchfits.read, hdul[n].data[col], hdul[n][col], TableHDURef, iter_rows, and streaming chunks. Vector columns (repeat > 1) keep (N, repeat); packed string columns stay (N, width) uint8 matrices.

where= on read_torch accepts only simple predicates: comparisons (==, !=, <, <=, >, >=), BETWEEN, and AND of those. Expressions with OR, IN, IS NULL, or NOT raise ValueError — use table.read(..., where=...) for the full dialect. Matching rows are kept by reading the needed columns and applying a torch mask. When combined with a row window (start_row / num_rows), the window selects file rows first and the predicate filters inside it — identical to table.read(row_slice=..., where=...).

cols = torchfits.table.read_torch("catalog.fits", hdu=1, columns=["RA", "DEC"])
bright = torchfits.table.read_torch(
    "catalog.fits", hdu=1, columns=["MAG"], where="MAG < 20"
)

For many unfiltered column reads on one file, use torchfits.open_table_reader(path, hdu=1) (no where= on the handle).

`table.scan()`¶

Streaming dataframe scanner yielding pyarrow.RecordBatch objects without materializing the entire table.

torchfits.table.scan(
    path,
    hdu=1,
    columns=None,
    row_slice=None,
    where=None,
    batch_size=65536,
    mmap=True,
    decode_bytes=True,
    encoding="ascii",
    strip=True,
    include_fits_metadata=False,
    apply_fits_nulls=True,
    backend="auto",
)

Yields: pyarrow.RecordBatch

for batch in torchfits.table.scan("survey.fits", hdu=1, batch_size=50_000):
    print(batch.num_rows, batch.column_names)  # pyarrow.RecordBatch

When to use

Use scan() when the table is too large to fit in memory, or when you want to process rows in streaming fashion. With where=, the predicate is evaluated per batch as rows stream past, so peak memory stays bounded by batch_size, not table size. For Polars-specific streaming, use scan_polars().

`table.scan_torch()`¶

Stream row chunks as dict[str, torch.Tensor]. No where= — filter with table.scan(..., where=...) or mask batches yourself.

torchfits.table.scan_torch(
    path,
    hdu=1,
    columns=None,
    row_slice=None,
    batch_size=65536,
    mmap=True,
    device="cpu",
    non_blocking=True,
    pin_memory=False,
)

Yields: dict[str, torch.Tensor]

for batch in torchfits.table.scan_torch("survey.fits", hdu=1, batch_size=10000):
    print(batch.keys())  # dict[str, torch.Tensor]

`table.reader()`¶

Open a FITS table as a pyarrow.RecordBatchReader for streaming.

torchfits.table.reader(
    path,
    hdu=1,
    columns=None,
    row_slice=None,
    where=None,
    batch_size=65536,
    mmap=True,
    decode_bytes=True,
    encoding="ascii",
    strip=True,
    include_fits_metadata=True,
    apply_fits_nulls=True,
    backend="auto",
)

Returns: pyarrow.RecordBatchReader

`table.write()`¶

Write a columnar dictionary as a FITS binary or ASCII table.

torchfits.table.write(path, data, *, schema=None, header=None,
                      overwrite=False, extname=None, table_type="binary",
                      quantize=None)

Parameter	Type	Default	Description
`path`	`str`	(required)	Output path
`data`	`dict`, `Table`, or DataFrame	(required)	Column dictionary, Astropy Table, PyArrow Table, or DataFrame
`schema`	`dict[str, dict[str, Any]]` or `None`	`None`	Per-column FITS layout overrides: `format`/`tform` (e.g. `{"bits": {"format": "4X"}}`), `unit`/`tunit`, `null`/`tnull`, `bscale`, `bzero`, `dim`/`tdim`
`header`	`dict` or `None`	`None`	FITS header key-value pairs
`extname`	`str` or `None`	`None`	`EXTNAME` for the written table HDU (merged into `header`)
`overwrite`	`bool`	`False`	Overwrite existing file
`table_type`	`str`	`"binary"`	`"binary"` or `"ascii"`
`quantize`	`None` or `str` or `dict`	`None`	Opt-in robust `TFORM=I` + `TSCAL`/`TZERO` for float columns (`"robust"` for all floats, `{"FLUX": "robust"}` per column, or options `{"lo_q","hi_q","keep_zero"}`). Integer columns are left alone. Default keeps native float `TFORM`.

schema scaling and nulls describe stored values:

With bscale/bzero, the column data you pass is written as the stored integers and the file's readers apply value = stored * TSCAL + TZERO. {"F": {"format": "I", "bscale": 0.5, "bzero": 10.0}} with data [100, 200] reads back as [60.0, 110.0].
null/tnull is written verbatim as TNULLn, which FITS compares with the stored value before TSCAL/TZERO. The exception is a column written from uint16/uint32/uint64/int8 data: torchfits stores it with the standard TZERO offset, so give tnull as a value of that dtype (for example 65535 for uint16) and it is written as 65535 - 32768. A tnull outside the dtype's range raises ValueError.
A tnull that is not an integer (or an integer string), and a bscale / bzero that is not a finite number, raise ValueError naming the field; so do the tnull= / tscal= / tzero= arguments of insert_column and replace_column.

quantize applies to binary tables only (ValueError when table_type="ascii"). If it is set and no floating-point column qualifies, QuantizeError is raised instead of writing those columns unpacked.

torchfits.table.write("out.fits", {"RA": ra, "DEC": dec}, overwrite=True)
torchfits.table.write(
    "packed.fits",
    {"ID": ids, "FLUX": flux},
    quantize={"FLUX": "robust"},
    overwrite=True,
)

Row filters (`where=`)¶

Pass where= to keep matching rows.

Call	Dialect	How it filters
`table.read` / `table.scan`	Full (see operators below)	C++ mmap scan when safe (`backend="auto"` + `mmap=True`, no VLA in the projection) and PyTorch is already imported; otherwise read then Arrow filter. `where=` never imports PyTorch itself
`table.read_torch`	Simple only: compare / `BETWEEN` / `AND`	Read projected columns, then torch mask. `OR` / `IN` / `IS NULL` / `NOT` raise `ValueError`

Root torchfits.read() has no where= parameter.

Operators on table.read / table.scan:

Operator	Example
`==` / `!=`	`where="CLASS == 'star'"`
`<` / `>` / `<=` / `>=`	`where="MAG_G < 20"`
`AND` / `OR`	`where="MAG_G < 20 AND DEC > 0"`
`NOT`	`where="NOT CLASS == 'star'"`
`IN (...)`	`where="id IN (1, 2, 3)"`
`NOT IN (...)`	`where="id NOT IN (4, 5)"`
`BETWEEN ... AND ...`	`where="MAG_G BETWEEN 15 AND 20"`
`IS NULL` / `IS NOT NULL`	`where="DEC IS NOT NULL"`

Use == for equality

The WHERE dialect is Python-expression based — single = is not an operator; write == (also accepts && / || / ~ C-style forms).

String literals are literal

Quote strings with '...' or "...". A doubled quote is one quote character ('O''NEIL'); a backslash is an ordinary character, so 'a\nb' is four characters, not a newline. BETWEEN takes string bounds (NAME BETWEEN 'a' AND 'm') as well as numbers such as .5.

How null rows are filtered

With apply_fits_nulls=True (the default) a TNULL sentinel is a null and the dialect follows SQL three-valued logic: a null carries an unknown value, so no predicate — positive or negated — selects it.

Predicate	A `TNULL` row
`==`, `!=`, `<`, `>`, `<=`, `>=`, `IN`, `NOT IN`, `BETWEEN`, `NOT BETWEEN`	not selected
`NOT (...)`	not selected
`IS NULL`	selected
`IS NOT NULL`	not selected

Negation therefore always agrees with the dual operator: where="NOT (A == 1)" and where="A != 1" return the same rows, and so do NOT (A > 1) / A <= 1 and NOT (A IN (...)) / A NOT IN (...). On float columns a NaN pixel is treated the same way: IEEE makes NaN != v true, but the comparison is unknown, so the row is excluded, and IS NULL selects it. A scaled column's TNULL cells read as NaN. This holds identically on all three read backends (auto, cpp, torch) and matches the torchfits.where reference evaluator. Use IS NULL / IS NOT NULL to test for absence. == NULL / != NULL on a numeric or logical column raise ValueError.

TNULLn is the stored value, compared before TSCALn/TZEROn as in the FITS standard: on a uint16 column (TZERO = 32768) TNULL = 0 marks cells that read as 32768. Files written by torchfits 1.3.1 stored an unsigned-column TNULL as the value instead (TNULL = 65535 on that uint16 column). A TNULL that cannot be a stored value (outside the stored integer range) but is a valid value of the column dtype is read that way, so those nulls still read as nulls.

`backend=` on `table.read` / `table.scan`¶

Value	Behavior
`"auto"` (default)	With `where=` and `mmap=True`, C++ pushdown when the header is readable, the projection has no VLA columns, and PyTorch is already imported; otherwise Arrow filter. With `mmap=False`, buffered read then filter.
`"cpp"`	Prefer C++ column / pushdown paths when safe
`"torch"`	Chunked path via `table.scan_torch`

Table reads open a private CFITSIO handle per call. Disk cache roots and shared metadata use the TORCHFITS_* variables in Architecture and Core I/O.

Predicate Helpers¶

The torchfits.where module provides predicate parsing and evaluation outside of table reads.

from torchfits.where import evaluate_where, parse_where_expression

ast = parse_where_expression("MAG_G < 20 AND DEC IS NOT NULL")
mask = evaluate_where(ast, {"MAG_G": magnitudes, "DEC": declinations})
# mask: np.ndarray[bool]

Additional public names: parse_where_literal, tokenize_where_expression, normalize_where_syntax, where_columns_from_ast. They read quoted literals as where= does: parse_where_literal("'O''NEIL'") is O'NEIL, and a backslash is an ordinary character.

Table mutation¶

Rewrite helpers on an existing table HDU:

torchfits.table.append_rows(path, rows, hdu=1)
torchfits.table.insert_rows(path, rows, row=0, hdu=1)
torchfits.table.update_rows(path, rows, row_slice, hdu=1, mmap="auto")
torchfits.table.delete_rows(path, row_slice, hdu=1)

torchfits.table.insert_column(
    path,
    name,
    values,
    hdu=1,
    index=None,
    format=None,
    unit=None,
    dim=None,
    tnull=None,
    tscal=None,
    tzero=None,
)
torchfits.table.replace_column(
    path,
    name,
    values,
    hdu=1,
    format=None,
    unit=None,
    dim=None,
    tnull=None,
    tscal=None,
    tzero=None,
)
torchfits.table.rename_columns(path, {"old_name": "new_name"}, hdu=1)
torchfits.table.drop_columns(path, ["col_a", "col_b"], hdu=1)

row_slice is a slice or (start, stop) tuple (0-based Python style for update/delete). insert_rows requires keyword row= (0-based insert index).

update_rows raises ValueError, before writing anything, when an integer value does not fit the column (for example 70000 into a uint16 column); float values written into an integer column only warn that they are truncated toward zero.

Without format=, insert_column / replace_column infer the TFORM from the values. As in table.write, uint16 / uint32 / uint64 values become I / J / K with TZERO = 2**15 / 2**31 / 2**63, and int8 becomes B with TZERO = -128. On an ASCII table (XTENSION = 'TABLE') the inferred format is an ASCII one (I6, I11, E25.15, A8, ...); values an ASCII table cannot hold (booleans, vectors, complex) raise ValueError asking for format=.

Without format=, replace_column keeps the column's TFORM unless the new values are uint16 / uint32 / uint64 / int8 and the column does not already use that TZERO convention: then the column layout is replaced too, exactly as insert_column would infer it, and the old TZERO / TSCAL / TNULL are dropped. With format= and such values, a format of the same storage code (I for uint16, ...) gets the TZERO convention; any other integer or float format stores the values plainly when they fit its range, and raises ValueError otherwise.

When the column keeps its TSCAL / TZERO (no tscal= / tzero= given and the layout is not replaced as above), replace_column takes physical values, the ones a read returns, as update_rows does: [1, 2, 3] on a uint16 column reads back as [1, 2, 3], and on a TSCAL = 0.5, TZERO = 10 column it is stored as (value - 10) / 0.5 (rounded half away from zero for an integer TFORM). A value the column cannot hold (70000 for uint16, a fraction for an unsigned column, NaN without a TNULL) raises ValueError naming the column, before anything is written. Passing tscal= or tzero= sets a new scaling, and the values are then the stored integers, as with bscale / bzero in table.write.

import numpy as np
import torchfits

torchfits.table.write(
    "s.fits",
    {"f": np.array([100, 200], dtype=np.int16)},
    schema={"f": {"format": "I", "bscale": 0.5, "bzero": 10.0}},
)
torchfits.table.replace_column("s.fits", "f", [11.5, 12.0])
assert torchfits.table.read_arrow("s.fits").column("f").to_pylist() == [11.5, 12.0]

import numpy as np
import torchfits

torchfits.table.write("t.fits", {"a": np.arange(3, dtype=np.int64)}, overwrite=True)
u16 = np.array([0, 40000, 65535], dtype=np.uint16)
torchfits.table.replace_column("t.fits", "a", u16)  # now TFORM 'I', TZERO 32768
assert torchfits.read("t.fits", hdu=1)["a"].dtype.is_floating_point is False
torchfits.table.replace_column("t.fits", "a", u16, format="1J")  # plain int32

String (A) columns take ASCII text in every row and column mutation: non-ASCII str or bytes values raise ValueError (as table.write does), None is written as a blank value, and a value longer than the column width is clipped with a RuntimeWarning.

Interop¶

Astropy¶

Direct reading, conversion, and writing for astropy.table.Table. Astropy is not a torchfits dependency; install it before calling these:

# Read directly into an Astropy Table (supports SQL WHERE filters)
tbl = torchfits.table.read_astropy(
    "catalog.fits", hdu=1, where="MAG_G < 20.0 AND DEC > 0"
)

# Convert an in-memory tensor dict to an Astropy Table
tbl_from_tensors = torchfits.to_astropy(tensor_dict)

# Write an Astropy Table directly to FITS
torchfits.table.write("out.fits", tbl, overwrite=True)

Polars¶

Native DataFrame path — FITS table → Polars dataframe in one call.

# One-call FITS to Polars (preserves FITS metadata)
df = torchfits.table.read_polars("catalog.fits", hdu=1)
# df: FITSPolarsFrame — wraps pl.DataFrame with .field_meta, .table_meta

# Streaming FITS to Polars (no full materialization)
for batch in torchfits.table.scan_polars("catalog.fits", hdu=1):
    process(batch)  # pl.DataFrame

# Eager Polars then LazyFrame (materializes once)
lazy = torchfits.table.to_polars("catalog.fits", hdu=1).lazy()

# From table dict
polars_df = torchfits.to_polars(table_dict, decode_bytes=False)

True streaming

scan_polars() yields one DataFrame per batch without building the full table. For a LazyFrame over an already-materialized table, use to_polars(...).lazy() or pl.concat(scan_polars(...)).lazy().

rechunk=False default

All Polars conversion functions default to rechunk=False. Pass rechunk=True to restore the old chunk-concatenation behavior.

DuckDB¶

import duckdb

# to_duckdb registers the table and returns a DuckDB relation (DuckDBPyRelation).
# Pass connection= to run SQL against the registered name on your own connection.
con = duckdb.connect()
rel = torchfits.table.to_duckdb(
    "catalog.fits", hdu=1, relation_name="tbl", connection=con
)
bright = rel.filter("MAG < 20")
counts = con.sql("SELECT count(*) FROM tbl WHERE MAG < 20").fetchall()

# One-shot query: registers the file under relation_name on a fresh connection
result = torchfits.table.duckdb_query(
    "catalog.fits",
    "SELECT * FROM tbl WHERE MAG < 20",
    hdu=1,
    relation_name="tbl",
)
# result: pyarrow object (Table or RecordBatchReader by version);
# call result.read_all() to force a pyarrow.Table

Arrow and Pandas¶

arrow_table = torchfits.to_arrow(table_dict, decode_bytes=False)
pandas_df = torchfits.to_pandas(table_dict, decode_bytes=False)

Schema¶

Infer an Arrow schema from FITS TFORM / TTYPE cards (no data read when where=None).

torchfits.table.schema(
    path,
    hdu=1,
    columns=None,
    where=None,
    decode_bytes=True,
    encoding="ascii",
    strip=True,
    include_fits_metadata=False,
    apply_fits_nulls=False,
    backend="auto",
)

Returns: pyarrow.Schema

schema = torchfits.table.schema("catalog.fits", hdu=1)

Additional Utilities¶

# PyArrow dataset and scanner
ds = torchfits.table.dataset("catalog.fits", hdu=1)
sc = torchfits.table.scanner("catalog.fits", columns=["RA", "DEC"])

# Parquet export -- `dest` is the output path; I/O kwargs (columns, where,
# hdu, ...) are forwarded to the read, so exports can be filtered and
# projected without materialising the whole table first.
torchfits.table.write_parquet("out.parquet", "catalog.fits", hdu=1)
torchfits.table.write_parquet("bright.parquet", "catalog.fits", where="MAG < 20")
torchfits.table.write_csv("bright.csv", "catalog.fits", where="MAG < 20")
torchfits.table.write_ipc("bright.arrow", "catalog.fits", where="MAG < 20")

# Cache cleanup
torchfits.table.clear_cache()

The public constant torchfits.table.TABLE_BACKENDS lists recognized table backend names for callers that select an explicit backend.

torchfits.table.write_ipc(dest, data, *, stream=False, compression="zstd", **read_kwargs) writes Arrow IPC/Feather V2. data may be a FITS path, Arrow table, record-batch reader, or batch iterable. For a FITS path, read keywords such as hdu, columns, and where are forwarded to table.read or table.reader; keywords are rejected for already-materialized input. Set stream=True to write batches without materializing the full table.

Table Reference¶

table.read()¶

table.read_torch()¶