data-formats

Background expertise for opening earth science data files correctly on the first try, and for diagnosing the openings that fail.

Consult the knowledge bundle; do not carry dataset facts here

Fill-value sentinel lists, calendar and season conventions, and the CF metadata contract are dataset knowledge, held once in the core bundle and never memorized in this skill. Consult installed knowledge concepts first, as the consult-knowledge skill sets out; the concepts this skill leans on are conventions/common-fill-values.md, conventions/calendars.md, and conventions/cf-conventions.md. Read them, restate the relevant point, and cite it. When a concept and this skill disagree, the concept wins.

The tables below (magic bytes, openers, failure symptoms) are invariant format method, not dataset facts, and stay here as procedure.

Identify the format first: magic bytes, not extensions

Extensions lie. A .nc file can be classic NetCDF-3 or HDF5-based NetCDF-4 (different engines); a .hdf file can be HDF4 or HDF5 (different libraries entirely). Check the leading bytes before choosing an opener:

with open(path, "rb") as f:
    head = f.read(8)

A wrong-looking magic number on a downloaded file usually means a truncated or HTML-error-page download; compare file size against the archive listing before debugging anything else.

Openers

Open lazily by default (chunks={} engages Dask with on-disk chunking). Declare the compute scale rather than assuming a machine: small (laptop), medium (Dask cluster), large (HPC or burst).

Fill values and packing

mask_and_scale=True (the default) applies _FillValue, missing_value, scale_factor, and add_offset from attributes. Two traps:

1. Unmasked sentinels. Files that omit the _FillValue attribute but use sentinel values anyway decode as real data and silently bias every statistic. The sentinel list and the detection recipe are dataset knowledge: consult conventions/common-fill-values.md, restate and cite it, and do not reproduce the values here. The refusal that guards this trap is in Hard refusals below.
2. Packed integers. If a variable that should be continuous arrives as int16, packing attributes were probably lost or ignored; check var.encoding for scale_factor before trusting values. The CF packed-data rule itself (how scale_factor and add_offset reconstruct physical values) is a metadata convention: consult conventions/cf-conventions.md.

Time decoding

CF time units (days since 1850-01-01) decode automatically. Non-standard calendars (360_day, noleap, all_leap) need cftime objects: pass use_cftime=True and expect cftime datetimes downstream (resampling works; direct numpy datetime comparisons do not). When decoding fails, open with decode_times=False, inspect time.attrs["units"] and time.attrs["calendar"], repair, then xr.decode_cf(ds). The conventions/calendars.md concept owns the CF calendar list, the month-length weighting rule, and the DJF year-boundary trap; consult and cite it rather than restating those here.

CRS and coordinates

• GeoTIFF via rioxarray: CRS lives at da.rio.crs; reproject with da.rio.reproject(...). Never assume EPSG:4326.
• NetCDF: CF encodes projection in a grid_mapping variable (its contract lives in conventions/cf-conventions.md); projected data has x/y coordinates in meters, not degrees. Check before treating coordinates as lat/lon.
• Longitude arrives as either 0..360 or -180..180; normalize deliberately (ds.assign_coords(lon=(ds.lon + 180) % 360 - 180).sortby("lon")) and say which convention the output uses.
• Latitude may be descending; sortby("lat") before slicing with ranges.

Per-format failure guidance

Hard refusals (stop; never proceed past these)

• Never trust a file extension over magic bytes.
• Never report a statistic before confirming fill values are masked.
• Never assume EPSG:4326, a longitude convention, or an ascending latitude axis without checking.
• Never load a multi-GB dataset eagerly; open lazily and state the compute scale.
• Never paper over a time-decoding failure by dropping the time axis.

What "open and summarize" means here

A complete summary of an opened dataset states: dimensions and sizes; coordinate ranges (with longitude convention and latitude order); variables with units and dtypes; time span, cadence, and calendar; CRS or grid mapping if any; fill-value status (masked, or sentinels detected); and in-memory vs on-disk size with the chunking in effect.