Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Fastmash

Fast command-line statistics for delimited text.

Fastmash summarizes delimited text and quoted CSV, with grouping and statistics for each category. It also reshapes ordinary delimited text. It accepts the command language of GNU datamash, so existing datamash workflows can move over with few or no changes.

It also reads quoted CSV directly, gives your result columns stable names, calculates weighted means and selects the highest or lowest complete records. Inspect a table’s health or compare summaries from two exports with the same field selectors and portable arithmetic.

$ cat readings.tsv
site	reading
west	2
east	4
west	6
west	7
$ fastmash -H -s -g site count reading median reading < readings.tsv
Grouped summary
────────────────────────────────
  site  count(reading)  median(reading)
  east               1                4
  west               3                6
✔ Completed

Install it with one command, or see other ways:

curl -fsSL https://fastmash.io/install.sh | sh

Fastmash is faster than GNU datamash*

About five times faster on RefGene quartiles, with faster grouped summaries and decimal totals too.

How many times faster Fastmash is than GNU datamash on the core jobs, per host Intel laptop, native Linux AMD desktop, native Linux 0× 1× 2× 3× 4× 5× 6× Exon-count quantiles (RefGene) Exon-count quantiles (RefGene), Intel laptop, native Linux: GNU 111.0 ms, Fastmash 21.5 ms 5.2× Exon-count quantiles (RefGene), AMD desktop, native Linux: GNU 60.0 ms, Fastmash 12.6 ms 4.8× Transcripts per gene (RefGene) Transcripts per gene (RefGene), Intel laptop, native Linux: GNU 110.8 ms, Fastmash 73.0 ms 1.5× Transcripts per gene (RefGene), AMD desktop, native Linux: GNU 57.0 ms, Fastmash 37.9 ms 1.5× Exon statistics per gene (RefGene) Exon statistics per gene (RefGene), Intel laptop, native Linux: GNU 157.8 ms, Fastmash 86.8 ms 1.8× Exon statistics per gene (RefGene), AMD desktop, native Linux: GNU 74.8 ms, Fastmash 45.0 ms 1.7× Many small groups (genes) Many small groups (genes), Intel laptop, native Linux: GNU 159.8 ms, Fastmash 127.1 ms 1.3× Many small groups (genes), AMD desktop, native Linux: GNU 66.1 ms, Fastmash 53.1 ms 1.2× Gene example (datamash manual) Gene example (datamash manual), Intel laptop, native Linux: GNU 7.0 ms, Fastmash 4.2 ms 1.7× Gene example (datamash manual), AMD desktop, native Linux: GNU 3.3 ms, Fastmash 2.1 ms 1.6× Sum and mean, 1M decimals Sum and mean, 1M decimals, Intel laptop, native Linux: GNU 325.4 ms, Fastmash 87.5 ms 3.7× Sum and mean, 1M decimals, AMD desktop, native Linux: GNU 114.8 ms, Fastmash 37.2 ms 3.1× Sum and mean, 100k decimals Sum and mean, 100k decimals, Intel laptop, native Linux: GNU 32.8 ms, Fastmash 9.5 ms 3.5× Sum and mean, 100k decimals, AMD desktop, native Linux: GNU 11.6 ms, Fastmash 3.9 ms 3.0× Grouped decimals, 100k rows Grouped decimals, 100k rows, Intel laptop, native Linux: GNU 68.1 ms, Fastmash 38.7 ms 1.8× Grouped decimals, 100k rows, AMD desktop, native Linux: GNU 27.5 ms, Fastmash 18.0 ms 1.5× One dominant group One dominant group, Intel laptop, native Linux: GNU 21.6 ms, Fastmash 9.1 ms 2.4× One dominant group, AMD desktop, native Linux: GNU 9.3 ms, Fastmash 3.9 ms 2.4× Wine quality by grade Wine quality by grade, Intel laptop, native Linux: GNU 6.0 ms, Fastmash 2.6 ms 2.3× Wine quality by grade, AMD desktop, native Linux: GNU 3.0 ms, Fastmash 1.2 ms 2.5× Large sorted summary Large sorted summary, Intel laptop, native Linux: GNU 547.2 ms, Fastmash 495.1 ms 1.1× Large sorted summary, AMD desktop, native Linux: GNU 204.4 ms, Fastmash 240.2 ms 0.9× Times faster than GNU datamash 1.9, by median elapsed time. The dashed line is the same speed; shorter bars are slower.

* Large sorted summaries can be slower on Linux. All results and the method.

And better in other ways

FeatureGNU datamash 1.9Fastmash
ResultsDepend on the CPU and the C math libraryRepeatable across supported machines with the same version, input and settings
LocalesNeed locale data installed. Silently fall back to C without itBuilt in: numbers in 318 locales, sorting in 194
perc:100Can read past the end of the valuesReturns the largest value
rmdup on several key fieldsCrashes on an assertionA clear refusal
Malformed parameters such as strbin:2.0Misleading errors or undefined behaviorAn accurate error message
crosstab labels over 511 bytesTruncatedKept in full

Fastmash supports over 70 operations and modes with familiar field selectors, grouping and headers. Most scripts can switch by changing one word. Check the documented differences when migrating a workflow. Migrating from datamash · All differences

Get started

Fastmash is open source under the MIT or Apache-2.0 license. Source on GitHub.