Benchmarks

The previous tables have been withdrawn following #246. They used a GDAL thread setting known to be slower for some tasks, did not validate outputs before scoring them, and included runs made while other work was using the machine. Those speed-up ratios are not supported as comparisons of equivalent work at each tool's best observed settings.

The replacement harness uses GDAL 3.13.3, the current release listed on GDAL's download page, and a release build of Terrio. Results are measurements of complete command invocations, including startup, reading, and writing. They are machine-specific snapshots, not guarantees of performance on other data or hardware.

Method

• Each tool is tuned independently at CPU budgets of 1, 2, 4, 8, 16, and the available maximum. Every setting receives a warm-up and three timed samples. Each tool selects its own lowest median, then receives an independent warm-up and fresh measurement samples. These are the best observed settings among the tested budgets, not a claim of globally optimal configuration.
• Measurement runs alternate Terrio/GDAL and GDAL/Terrio order. Tables show the median and full minimum–maximum range of seven samples, or five for large files. Raw samples and tuning results accompany the tables in JSON. Ratios describe observed medians, not statistical significance. The Checks column marks a wide timing range when either tool's maximum sample exceeds twice its median; this annotation does not alter the samples or their median ratio.
• On Linux, taskset restricts each process and all its child threads to its selected CPU budget. RAYON_NUM_THREADS, GDAL_NUM_THREADS, and gdalwarp's NUM_THREADS also configure the pools. A budget of one therefore bounds Terrio's workers outside Rayon as well. On other platforms, these variables are hints; the script does not claim a hard CPU limit. The Linux runner uses the lowest-numbered available CPU IDs for both tools. Budgets count logical CPUs; the selection does not distinguish physical cores from SMT siblings. Hardware topology and OS placement can therefore affect these machine-specific results. Budgets above the available affinity are rejected.
• The Linux runner estimates competing CPU use from machine CPU time minus the measured child's CPU time. Estimates are averaged by elapsed time across each paired tuning or measurement series, reducing clock-tick noise in short commands. If a series exceeds one CPU, the row receives no ratio. The check is intentionally machine-wide: work outside the pinned CPU set can still contend for shared caches, memory bandwidth and package power. Raw background estimates are retained. This detects contention; it does not provide hardware isolation or remove filesystem and network noise. On other platforms, background CPU use is unmeasured and ratios carry no background-load guarantee.
• Every invocation starts with an empty, separate output directory. PAM statistics caches are disabled. Validation runs after timing, against the final output of each selected setting. The benchmark does not prove that every earlier invocation produced the same output.
• GDAL uses its single-process gdal raster mosaic and gdal raster polygonize commands. Mosaic resolution is explicitly averaged, as in the former gdalbuildvrt command, to accept tiles whose resolutions differ at floating-point precision. Both COG writers build fresh average overviews; GDAL is explicitly told to ignore source overviews. Warp benchmarks retain NAD83 (EPSG:4269 → EPSG:26913) to avoid measuring different datum transformations or grid availability. Networking for datum grids is disabled.
• GDAL Shapefile output explicitly uses UTF-8 encoding to match Terrio. The countries GeoJSON fixture has scalar attributes supported by Shapefile.
• Memory ratios are omitted: launching commands from a Python process that also validates large datasets can contaminate child peak-RSS accounting with inherited parent memory.

Output mismatch follow-up

The #253 investigation records complete error measurements, metadata/schema fixes, native WebP corrections and explicit GDAL 3.13 COG/terrain profiles. All eight affected cases were rerun, including fresh timing samples and CPU tuning at budgets one and two. Six pass exact output validation; warp and counties retain documented numerical residuals. Background CPU activity exceeded the unchanged measurement limit, so no new speed ratios are eligible. The tables below retain their original samples.

Correctness gates

A zero exit status is insufficient. Failed commands, validation errors, differing outputs, and excessive background CPU use are visible in the table and receive no speed ratio.

Raster outputs are read through GDAL and compared at every base and overview pixel, including validity masks, data types, nodata, scale/offset, units, color interpretation and color tables. CRSs must be semantically equal; affine transforms use absolute tolerance 1e-8 and relative tolerance 1e-12. Pixels are compared exactly (with NaNs equal). Approximate pixel agreement is deliberately not scored as equivalence; a rounding difference may therefore exclude an otherwise useful result. Metadata outside these checks, compression ratios, and physical file layouts are not asserted to be identical.

Vector outputs compare field names/types, CRS, and every feature's attribute values and normalized geometry WKB. Disk-sorted hashes allow different feature order without keeping whole datasets in memory. Generated feature IDs and layer names are ignored. Coordinates are compared exactly after normalizing ring starts, orientations and multipart order. Topologically equal geometries with different vertices can consequently be excluded. The check is conservative rather than a test of all possible notions of vector equivalence.

Summary cases check the captured JSON against the source. Statistics from both commands are checked against an independent NumPy reduction of valid source pixels, using relative tolerance 1e-6 and absolute tolerance 1e-8. This matters for GDAL 3.13.3: the SRTM case returned incorrect statistics in our check, even though gdalinfo exited successfully. It must not receive a speed ratio for that result.

A controlled check used the same gdalinfo -json -stats srtm90.tif command with GDAL_PAM_ENABLED=NO. The source had no stored STATISTICS_* keys and no nodata value. GDAL 3.13.3 with GDAL_NUM_THREADS=1 matched the full-resolution NumPy reduction (minimum 15, maximum 4338); with eight threads it returned minimum 26 and maximum 4148. GDAL 3.12.2 matched NumPy at both thread counts. This identifies a thread-dependent discrepancy in the tested GDAL 3.13.3 build, not a requested approximate-statistics mode or a stored-statistics cache. Its internal cause remains unisolated; the invalid eight-thread result is excluded from speed comparisons.

Rerun on 24 October 2026

The rerun covers all 19 small-file/remote cases plus the large-mosaic window, hillshade and slope cases. The other historical large-file timings remain withdrawn. The full case list is still available in the script.

22 distinct cases were measured; 13 passed the output checks and 8 also passed the background-load gate. Other computation jobs were temporarily paused for these measurements. Any rows still marked background load are diagnostic timings, not supported performance comparisons.

Hardware: Intel Xeon w7-3445, 40 available logical CPUs, Linux. GDAL 3.13.3; Terrio 0.5.0, optimized release build. All runs tested CPU budgets 1, 2, 4, 8, 16 and 40, with three tuning samples per budget. Small-file measurements use seven fresh samples and large-file measurements use five. Times below are median [min–max].

Raw tuning and measurement samples, commands, background CPU estimates and complete validation details are retained for every row. The nearest warp was repeated after an earlier load rejection; the table uses the latest run for that task and retains both runs in JSON. Machine-specific paths are replaced with documented placeholders. A dash means the row was excluded from speed comparisons.

Summaries

Raster conversion

Raster operations

Vector conversion

Raster ↔ vector

Remote data

Large rasters

Running the benchmarks

Use the latest GDAL release and matching Python bindings. The commands below pin the version used for this update; check GDAL's download page before a future rerun.

Linux CPU affinity requires taskset from util-linux. Vector validation requires an external sort supporting -S 64M and -T DIR, as provided by GNU coreutils or compatible BSD versions. gsort is preferred when available (for example, GNU coreutils installed on macOS). Missing or incompatible sort commands are reported as validation failures and receive no ratio. Windows users need these Unix tools through WSL or another compatible environment. Vector geometry normalization also requires GDAL built with GEOS. Missing GEOS or a failed normalization produces a validation failure and receives no ratio.

conda create -n terrio-benchmark -c conda-forge gdal=3.13.3 libgdal-arrow-parquet=3.13.3 numpy
conda activate terrio-benchmark
export PROJ_DATA="$CONDA_PREFIX/share/proj"
scripts/fetch_test_data.sh ../terrio-data
scripts/fetch_large_data.sh ../terrio-large-data
CARGO_BUILD_JOBS=2 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 cargo build --release -p terrio-cli

# Keep large temporary outputs on disk, and stop competing workloads before timing.
mkdir -p ../terrio-benchmark-scratch
TMPDIR="$PWD/../terrio-benchmark-scratch" python scripts/benchmark.py ../terrio-data \
  --large ../terrio-large-data --remote --out benchmarks.md

--threads N sets the maximum candidate CPU budget, rather than claiming that both tools use exactly N threads. --thread-counts 1,4,16 supplies an explicit candidate set. --runs, --large-runs, and --tune-runs control the respective sample counts; all must be positive. --only hillshade selects a description substring for a focused rerun. --max-background-cpus controls the competing-load threshold; increasing it weakens that protection and is recorded in JSON. --out FILE writes Markdown and FILE.json, checkpointing completed rows. The recorded source_commit identifies the harness checkout; the selected TERRIO_BIN and its reported version are recorded separately.

The regression tests cover wrong-but-successful commands, independent tuning, alternating order, fresh outputs, inherited CPU affinity, noisy measurements, pixel/mask/metadata differences, vector feature differences, and a deliberately corrupted GDAL statistics summary:

python -m unittest discover -s scripts/tests -v