Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

0005: Growable, checked storage instead of fixed limits

Status: accepted.

Fixed caps (record length, number of operations, number of samples) make resource use easy to reason about, but real workloads exceed them. Fastmash has no such caps: storage grows with the job, every allocation is checked, and allocation failure becomes a refusal where Rust and the operating system allow it. Sorting spills to disk to limit memory.

Consequence: memory use grows with the largest group for operations that need every value, such as quantiles. This is documented, together with which data spills and which does not. The only fixed limits left are GNU datamash’s own, such as 99-byte --format strings and 511-byte names, and they are documented.