Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Fastmash

Fast command-line statistics for delimited text.

Fastmash summarizes delimited text and quoted CSV, with grouping and statistics for each category. It also reshapes ordinary delimited text. It accepts the command language of GNU datamash, so existing datamash workflows can move over with few or no changes.

It also reads quoted CSV directly, gives your result columns stable names, calculates weighted means and selects the highest or lowest complete records. Inspect a table’s health or compare summaries from two exports with the same field selectors and portable arithmetic.

$ cat readings.tsv
site	reading
west	2
east	4
west	6
west	7
$ fastmash -H -s -g site count reading median reading < readings.tsv
Grouped summary
────────────────────────────────
  site  count(reading)  median(reading)
  east               1                4
  west               3                6
✔ Completed

Install it with one command, or see other ways:

curl -fsSL https://fastmash.io/install.sh | sh

Fastmash is faster than GNU datamash*

About five times faster on RefGene quartiles, with faster grouped summaries and decimal totals too.

How many times faster Fastmash is than GNU datamash on the core jobs, per host Intel laptop, native Linux AMD desktop, native Linux 0× 1× 2× 3× 4× 5× 6× Exon-count quantiles (RefGene) Exon-count quantiles (RefGene), Intel laptop, native Linux: GNU 111.0 ms, Fastmash 21.5 ms 5.2× Exon-count quantiles (RefGene), AMD desktop, native Linux: GNU 60.0 ms, Fastmash 12.6 ms 4.8× Transcripts per gene (RefGene) Transcripts per gene (RefGene), Intel laptop, native Linux: GNU 110.8 ms, Fastmash 73.0 ms 1.5× Transcripts per gene (RefGene), AMD desktop, native Linux: GNU 57.0 ms, Fastmash 37.9 ms 1.5× Exon statistics per gene (RefGene) Exon statistics per gene (RefGene), Intel laptop, native Linux: GNU 157.8 ms, Fastmash 86.8 ms 1.8× Exon statistics per gene (RefGene), AMD desktop, native Linux: GNU 74.8 ms, Fastmash 45.0 ms 1.7× Many small groups (genes) Many small groups (genes), Intel laptop, native Linux: GNU 159.8 ms, Fastmash 127.1 ms 1.3× Many small groups (genes), AMD desktop, native Linux: GNU 66.1 ms, Fastmash 53.1 ms 1.2× Gene example (datamash manual) Gene example (datamash manual), Intel laptop, native Linux: GNU 7.0 ms, Fastmash 4.2 ms 1.7× Gene example (datamash manual), AMD desktop, native Linux: GNU 3.3 ms, Fastmash 2.1 ms 1.6× Sum and mean, 1M decimals Sum and mean, 1M decimals, Intel laptop, native Linux: GNU 325.4 ms, Fastmash 87.5 ms 3.7× Sum and mean, 1M decimals, AMD desktop, native Linux: GNU 114.8 ms, Fastmash 37.2 ms 3.1× Sum and mean, 100k decimals Sum and mean, 100k decimals, Intel laptop, native Linux: GNU 32.8 ms, Fastmash 9.5 ms 3.5× Sum and mean, 100k decimals, AMD desktop, native Linux: GNU 11.6 ms, Fastmash 3.9 ms 3.0× Grouped decimals, 100k rows Grouped decimals, 100k rows, Intel laptop, native Linux: GNU 68.1 ms, Fastmash 38.7 ms 1.8× Grouped decimals, 100k rows, AMD desktop, native Linux: GNU 27.5 ms, Fastmash 18.0 ms 1.5× One dominant group One dominant group, Intel laptop, native Linux: GNU 21.6 ms, Fastmash 9.1 ms 2.4× One dominant group, AMD desktop, native Linux: GNU 9.3 ms, Fastmash 3.9 ms 2.4× Wine quality by grade Wine quality by grade, Intel laptop, native Linux: GNU 6.0 ms, Fastmash 2.6 ms 2.3× Wine quality by grade, AMD desktop, native Linux: GNU 3.0 ms, Fastmash 1.2 ms 2.5× Large sorted summary Large sorted summary, Intel laptop, native Linux: GNU 547.2 ms, Fastmash 495.1 ms 1.1× Large sorted summary, AMD desktop, native Linux: GNU 204.4 ms, Fastmash 240.2 ms 0.9× Times faster than GNU datamash 1.9, by median elapsed time. The dashed line is the same speed; shorter bars are slower.

* Large sorted summaries can be slower on Linux. All results and the method.

And better in other ways

FeatureGNU datamash 1.9Fastmash
ResultsDepend on the CPU and the C math libraryRepeatable across supported machines with the same version, input and settings
LocalesNeed locale data installed. Silently fall back to C without itBuilt in: numbers in 318 locales, sorting in 194
perc:100Can read past the end of the valuesReturns the largest value
rmdup on several key fieldsCrashes on an assertionA clear refusal
Malformed parameters such as strbin:2.0Misleading errors or undefined behaviorAn accurate error message
crosstab labels over 511 bytesTruncatedKept in full

Fastmash supports over 70 operations and modes with familiar field selectors, grouping and headers. Most scripts can switch by changing one word. Check the documented differences when migrating a workflow. Migrating from datamash · All differences

Get started

Fastmash is open source under the MIT or Apache-2.0 license. Source on GitHub.

Install

Fastmash runs on Linux, macOS 15 and later, and Windows 11. For Windows, use PowerShell 7.4 or later.

Choose a method

Use your system’s package manager when you can. It installs prebuilt programs and brings you later versions with your other updates.

Your systemRecommendedUpdates
Debian, Ubuntuapt repositorysudo apt update, then sudo apt install --only-upgrade fastmash
Fedora, RHEL and similardnf repositorysudo dnf upgrade fastmash
macOS, or Linux with HomebrewHomebrewbrew upgrade fastmash
Linux, macOS, or WSL2Quick installRun it again
WindowsNative archiveReplace the extracted folder

All of these install prebuilt programs. If you prefer to compile Fastmash yourself, see Cargo.

Debian and Ubuntu

The signed apt repository provides updates through apt on Linux. Install the tools needed to download and inspect its key:

sudo apt install ca-certificates curl gnupg
curl -fsSLo fastmash.gpg https://packages.fastmash.io/fastmash.gpg
gpg --show-keys --fingerprint fastmash.gpg

Check that the key fingerprint is:

76C1 7E23 B7D3 F8FD ED36 D31A 7BAC 7AF6 644B D290

Then add the repository and install Fastmash:

sudo install -m 644 fastmash.gpg /usr/share/keyrings/fastmash.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/fastmash.gpg] https://packages.fastmash.io/apt stable main" |
  sudo tee /etc/apt/sources.list.d/fastmash.list
sudo apt update
sudo apt install fastmash

Later versions arrive through sudo apt update and sudo apt upgrade. The key is trusted only for this repository.

For Debian 10, Ubuntu 20.04 and newer. Remove it with sudo apt remove fastmash.

Fedora, RHEL and similar

The signed dnf repository provides updates on Linux, with package and repository signature checks enabled:

sudo curl -fsSLo /etc/yum.repos.d/fastmash.repo https://packages.fastmash.io/fastmash.repo
sudo dnf install fastmash

dnf asks to trust the signing key on first use. Before confirming, check that its fingerprint is 76C1 7E23 B7D3 F8FD ED36 D31A 7BAC 7AF6 644B D290, the same key used by the apt repository above. Update with sudo dnf upgrade fastmash.

For Fedora, RHEL 8 and newer. Remove it with sudo dnf remove fastmash.

Homebrew

With Homebrew on macOS 15 or later, or on Linux:

brew install pederbe/tap/fastmash

Homebrew downloads the same tested programs as the Fastmash release archives. Installation needs no Rust compiler and does not rebuild Fastmash.

Update with brew upgrade fastmash and remove it with brew uninstall pederbe/tap/fastmash.

Quick install

On Linux, macOS or inside WSL2:

curl -fsSL https://fastmash.io/install.sh | sh

Downloads the latest release archive for your system, checks its checksum and installs it into ~/.local/bin. The installer prints a command to try when it finishes. If your shell cannot find fastmash, see Command not found. Run the same command again to update this installation.

Choose another writable directory with FASTMASH_INSTALL_DIR:

curl -fsSL https://fastmash.io/install.sh | env FASTMASH_INSTALL_DIR="$HOME/bin" sh

Use the same directory when updating. To read the installer’s help without installing anything:

curl -fsSL https://fastmash.io/install.sh | sh -s -- --help

If a download or checksum check fails, your existing installation stays in place. Check your connection and retry. For a permission error, choose a directory you can write to with FASTMASH_INSTALL_DIR.

Single packages and archives

These downloads install the latest release without adding a repository. You update them yourself.

Arch Linux package

Use the Arch package recipe to install the prebuilt executable through makepkg and pacman. No Rust compiler is needed. To update, rebuild the package from the current recipe.

Debian or RPM package

Download the .deb package for Debian or Ubuntu, or the .rpm package for Fedora, RHEL and similar systems, from the release downloads. Save it in an empty folder, then open a terminal there.

On Debian or Ubuntu:

sudo apt install ./fastmash_*.deb

On Fedora, RHEL and similar:

sudo dnf install ./fastmash-*.rpm

To update, repeat these steps with a fresh download.

Linux and macOS archives

  1. Open the release downloads and choose the archive for your system:

    SystemArchive ending
    Linux or WSL2x86_64-unknown-linux-musl.tar.gz
    macOSaarch64-apple-darwin.tar.gz
  2. Extract the archive, then open a terminal in the extracted folder containing fastmash.

  3. Move the executable into your local command directory and try it:

    mkdir -p ~/.local/bin
    mv -f fastmash ~/.local/bin/fastmash
    printf '1\n2\n' | ~/.local/bin/fastmash sum 1
    

The final command prints 3. If your shell cannot find fastmash by name, see Command not found. To update, repeat these steps with a fresh download.

With Cargo

If you already use Rust, Cargo offers two routes.

cargo-binstall downloads the same prebuilt executables as the release archives, without compiling. On Linux or WSL2, run:

cargo binstall fastmash --strategies crate-meta-data

On macOS or Windows, select its single executable:

cargo binstall fastmash --strategies crate-meta-data --bin fastmash

These commands install the published binaries and stop with an error if the download is unavailable.

To compile Fastmash from source with cargo install, use the latest stable Rust toolchain. On Linux or WSL2:

rustup target add x86_64-unknown-linux-musl
cargo install --locked --target x86_64-unknown-linux-musl fastmash

On macOS 15 or later, select its single executable:

cargo install --locked --bin fastmash fastmash

Check that it works

In Bash, or use the PowerShell example:

printf '1\t2\n3\t4\n' | fastmash sum 1 mean 2

The summary shows 4 for the sum and 3 for the mean. Next: the quick start.

Command not found

For a quick installation or an archive installed in the default directory, try the executable directly:

~/.local/bin/fastmash --version

PATH lists the directories your shell searches for commands, in order. If the direct command prints the version, add ~/.local/bin to PATH. In Fish:

fish_add_path "$HOME/.local/bin"

In Bash, add this line to ~/.bashrc. In Zsh, add it to ~/.zshrc, then open a new terminal:

export PATH="$HOME/.local/bin:$PATH"

If Bash starts a login shell, add the line to ~/.bash_profile instead. Replace ~/.local/bin with your chosen directory if you set FASTMASH_INSTALL_DIR.

To see which installation your shell uses:

command -v fastmash

Update that installation with its original method, or put your preferred installation directory first in PATH. In Fish, fish_add_path --move moves an existing directory to the front.

On Windows

Use PowerShell 7.4 or later on Windows 11.

  1. Open the release downloads and choose the archive ending in x86_64-pc-windows-msvc.tar.gz.

  2. Extract it and keep the extracted folder somewhere permanent.

  3. Search Windows for Edit environment variables for your account. Edit Path and add the folder containing fastmash.exe.

  4. Open a new PowerShell terminal and run:

    fastmash --version
    

It prints the installed version. Continue with the PowerShell quick start for files and pipelines. To update, replace the extracted folder and keep its location on Path. To uninstall, remove that folder and its Path entry. No Rust compiler or separate Visual C++ runtime installation is needed.

For Linux workflows, you can also use WSL2 and follow the Linux instructions. Keep WSL2 data in the Linux file system rather than under /mnt/c.

Uninstall

Package managers remove Fastmash with their own commands, shown above. For the quick installer or a downloaded archive:

rm ~/.local/bin/fastmash

If you chose another directory with FASTMASH_INSTALL_DIR, remove its fastmash file instead. Removing the executable does not remove your data.

If you installed with Cargo or cargo-binstall:

cargo uninstall fastmash

Fastmash keeps no configuration files, caches or background services.

See System requirements for building from source and verifying release archives.

Quick start

This tour takes about five minutes. It assumes Fastmash is installed. Every example reads from standard input, so you can paste it straight into Bash. For Windows, start with PowerShell 7.4+. The Fastmash operations and options are the same.

Locale. The examples use decimal points. If your shell uses a locale with decimal commas or an unsupported locale (see Locales), run export LC_ALL=C.UTF-8 first.

Your first summary

Fastmash reads records (lines) of fields separated by tabs, and applies one or more operations to selected fields:

$ printf '1\t2\n3\t4\n' | fastmash sum 1 mean 2
Summary
────────────────────────────────
  sum(field-1)  mean(field-2)
             4              3
✔ Completed

sum 1 adds up field 1 and mean 2 averages field 2. The summary shows both results in the order you requested.

PowerShell 7.4+

For a CSV file, pass its bytes directly through cmd input redirection. Try this sample, or use your own sales.csv with an amount field:

'item,amount', 'a,2', 'b,4' | Set-Content -Encoding utf8 sales.csv
cmd /d /v:off /c 'fastmash --csv -H sum amount < "sales.csv"'
sum(amount)
6

--csv accepts Windows line endings. PowerShell does not support < input redirection itself. Get-Content decodes a file into text, so use the cmd recipe when you want to preserve its original bytes.

You can also send text directly:

'item,amount', 'a,2', 'b,4' | fastmash --csv -H sum amount

This produces the same result. For a native program pipeline, extract the amounts and pass them to another Fastmash command:

cmd /d /v:off /c 'fastmash --csv-in --header-in cut amount < "sales.csv"' |
    fastmash sum 1

The summary shows 6. PowerShell 7.4+ preserves bytes between native programs and when redirecting their output with >. PowerShell string pipelines create text, including Windows line endings. See Fields and input for how ordinary text and CSV handle those endings.

After a command, $LASTEXITCODE gives its status. In scripts, follow PowerShell exit-status handling.

Several fields at once

A field list or range applies an operation to each field:

$ printf '1\t2\t3\n4\t5\t6\n' | fastmash sum 1-3
Summary
────────────────────────────────
  sum(field-1)  sum(field-2)  sum(field-3)
             5             7             9
✔ Completed

Grouping

-g groups records by a key field and summarizes each group:

$ printf 'A\t1\nA\t2\nB\t4\n' | fastmash -g 1 sum 2 count 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)  count(field-2)
        A             3               2
        B             4               1
✔ Completed

Groups are made from adjacent records with the same key. If equal keys can be scattered through the input, add -s to sort first:

$ printf 'b\t2\na\t1\na\t3\n' | fastmash -s -g 1 sum 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)
        a             4
        b             2
✔ Completed

Headers and named fields

With a header line, use -H to name fields and label the output:

$ printf 'site\treading\nwest\t2\neast\t4\nwest\t6\n' | fastmash -H -s -g site sum reading mean reading
Grouped summary
────────────────────────────────
  site  sum(reading)  mean(reading)
  east             4              4
  west             8              4
✔ Completed

--header-in reads field names. -H also uses them in result labels.

Other separators

-t sets a single-byte field separator, and -W splits on runs of spaces and tabs:

$ printf '1,2\n3,4\n' | fastmash -t, sum 1-2
Summary
────────────────────────────────
  sum(field-1)  sum(field-2)
             4             6
✔ Completed
$ printf '1  2\n3   4\n' | fastmash -W sum 2
Summary
────────────────────────────────
  sum(field-2)
             6
✔ Completed

-t, splits on every comma, preserving datamash’s literal separator behavior. For quoted CSV, use --csv to read and write CSV or choose input/output independently with --csv-in and --csv-out:

$ printf 'region,revenue\n"west,coast",2\neast,4\n"west,coast",6\n' | fastmash --csv -H -s -g region --result-name=1:total sum revenue
GroupBy(region),total
east,4
"west,coast",8

Format switches do not infer headers. Quoted CSV and result names describes the supported modes and quoting rules.

Statistics

$ printf '2\n4\n4\n4\n5\n5\n7\n9\n' | fastmash median 1 q1 1 q3 1 pstdev 1 perc:90 1
Summary
────────────────────────────────
  median(field-1)  q1(field-1)  q3(field-1)  pstdev(field-1)  perc:90(field-1)
              4.5            4          5.5                2               7.6
✔ Completed

See Operations for the full list: means, variances, quantiles, modes, skewness and kurtosis, normality tests, correlation and more.

Weighted means

Give each value a contribution weight with wmean VALUE:WEIGHT:

$ printf '10\t1\n20\t3\n' | fastmash wmean 1:2
Summary
────────────────────────────────
  wmean(field-1,field-2)
                    17.5
✔ Completed

Weights must be finite and nonnegative. A nonempty calculation needs positive retained weight. See Weighted mean.

Keep the highest records

Top-N selection retains complete records and keeps earlier input records on ties:

$ printf 'A\t8\nB\t10\nC\t10\nD\t9\n' | fastmash top:2 2
Top-N selection
────────────────────────────────
  field-1  field-2
        B       10
        C       10
✔ Completed

Use bottom:N for the lowest records, and -s -g for a cutoff in each group. See Highest and lowest records.

Per-row operations

Some operations transform each record instead of summarizing:

$ printf '3.14159\thello\n' | fastmash round 1 md5 2
Per-row results
────────────────────────────────
  round(field-1)  md5(field-2)
               3  5d41402abc4b2a76b9719d911017c592
✔ Completed

Missing values

--narm skips NA, N/A and NaN values:

$ printf '1\nNA\n3\n' | fastmash --narm mean 1
Summary
────────────────────────────────
  mean(field-1)
              2
✔ Completed

Next steps

Migrating from GNU datamash

Fastmash accepts GNU datamash 1.9’s command language. In most scripts, switching is a matter of replacing datamash with fastmash.

A safe way to switch

  1. Install Fastmash alongside datamash. Don’t alias or replace datamash; keep both available while you compare.

  2. Run your existing command with both programs on the same input and with the same locale settings, and compare the output and the exit status:

    export LC_ALL=C.UTF-8
    datamash -s -g 1 sum 2 < data.tsv > gnu.out;  echo "datamash: $?"
    fastmash -s -g 1 sum 2 < data.tsv > fast.out; echo "fastmash: $?"
    cmp gnu.out fast.out && echo identical
    
  3. If the outputs differ, check Differences from GNU datamash. If the difference isn’t listed there, please report it.

  4. Change the script, and keep checking exit statuses as before.

What works unchanged

  • Every GNU datamash 1.9 operation, mode and alias, except the explicit line mode (write fastmash base64 1, not fastmash line base64 1)
  • Field numbers, lists, ranges, pairs and header names
  • All options except --sort-cmd, including short-option clusters (-sg1), abbreviated long options (--whitesp), options after operands, and --
  • POSIXLY_CORRECT handling
  • Data output to files and pipelines, header names and the --full output
  • Exit statuses 0 and 1 for success and ordinary errors

What to check

Locale settings. Fastmash has the glibc 2.43 locale rules built in: numbers work in C, POSIX and every glibc locale with a one-byte decimal separator, @ modifier locales included, and sorting in C, POSIX, C.UTF-8 and 194 language locales checked against glibc. A name glibc has no locale for behaves as C, as in GNU datamash. Commands that read or print numbers exit with status 77 in ps_AF, and in a character set other than UTF-8 whose thousands separator is not ASCII (such as fr_FR in Latin-1); sorted commands do outside the sorting locales. GNU datamash falls back to C where a locale is not installed, while Fastmash does not. Set LC_ALL=C.UTF-8 (or C) in scripts. In language locales, Fastmash orders sorted keys as GNU sort does under glibc, letters and digits by Unicode collation and punctuation, symbols and spaces by glibc’s own table; keys that differ only in punctuation next to digits (12-A and 1-2A), vulgar fractions such as ¼ and some letters from outside the language’s alphabet can still sort differently. Locales.

Last digits of results. Fastmash’s portable arithmetic can differ from a local GNU build in the last printed digit of some results, or in the sign of a nan. If a script compares numbers textually, compare with a tolerance or fix the precision with --round. Numbers.

Exit status 77. Fastmash refuses some inputs that GNU datamash handles unsafely, such as perc:2.5 or rmdup with several keys. A script that treats any nonzero status as failure needs no change.

--sort-cmd. Not supported. To use a different sorter, sort the input first and leave out -s:

LC_ALL=C sort -s -t "$(printf '\t')" -k1,1 data.tsv | fastmash -g 1 sum 2

Test suites. GNU datamash’s hidden testing options (---print-inf, ---print-nan, ---print-progname, ---rmdup-test) are not provided.

Remember

-t, splits on commas and does not parse quoted CSV, in both programs. Fastmash adds explicit --csv-in, --csv-out and --csv switches for quoted CSV; they leave existing separator commands unchanged. Result names, weighted mean, Top-N selection, dataset comparison and table health are also optional extensions. spearson is the sample Pearson correlation, in both programs.

Compared with other tools

Fastmash runs GNU datamash commands with portable results, and extends that workflow with quoted CSV, weighted means, record selection, table health and dataset comparison. The tools below serve different data-processing needs. Facts about other tools come from their documentation, checked September 2026.

At a glance

FastmashGNU datamashMillerqsvcsvtk
Command languagedatamash’sdatamash’sIts own verbs and DSLIts own subcommandsIts own subcommands
Written inRustCGoRustGo
LicenceMIT or Apache-2.0GPL-3.0-or-laterBSD-2-ClauseMITMIT
InputDelimited text and explicit quoted CSVTab or single-byte delimited textCSV (RFC 4180), TSV, JSON and moreCSV, TSV, Excel, Parquet and moreCSV and TSV
Quoted CSV fieldsYes, in supported calculation, selection and comparison modesNoYesYesYes
Grouped statisticsYesYesYes (stats1 -g)Through pivotp or sqlpYes (summary -g)
Joins and expression filtersNoNoYesYesYes
Table reshapingTranspose and crosstabTranspose and crosstabYesYesYes
PlatformsLinux, macOS 15+, Windows 11Linux, macOS, Windows and othersLinux, macOS, Windows, BSDLinux, macOS, WindowsLinux, macOS, Windows, BSD

GNU datamash

The original, and the reference Fastmash is measured against. It runs almost everywhere and is packaged by every Linux distribution, conda-forge and Homebrew.

Choose Fastmash when you run datamash commands on Linux and want them faster (about five times on RefGene quartiles in the Linux benchmarks), want the same digits on every machine, or want clear errors where datamash 1.9 aborts or reads past its data. Most commands transfer directly. The differences are listed on one page.

Stay with GNU datamash when you need Intel macOS or a platform outside Fastmash’s supported targets, or when you need a GNU-maintained tool. Large sorted summaries can still favor GNU on Linux. The benchmarks show the costs and measurement conditions.

Miller

A complete toolkit for record-oriented data: it reads and writes CSV (with quoting), TSV, JSON and several other formats, and has a full expression language for filtering, computing new fields, joining, sorting, sampling and reshaping. Its stats1 verb computes grouped statistics (count, sum, mean, median and any percentile, mode, variance, skewness and more) without sorting the input first.

Choose Miller when your data is JSON, or the job needs more than statistics: filtering, joining or transforming records. Choose Fastmash when the job is a datamash-style summary of delimited text and you want familiar commands with portable numerical results.

qsv

A very fast, very broad CSV toolkit in Rust: indexing, joins (through Polars), SQL queries, validation, sampling, format conversion to Parquet, Excel and databases, and whole-column statistics. Grouped aggregation goes through pivotp (count, sum, mean, median, quantiles, first, last) or sqlp.

Choose qsv when you work with large CSV files and want one tool for exploring, querying and converting them. Choose Fastmash when you want datamash’s operations and syntax, for example sample and population standard deviations, skewness and kurtosis, mode, collapse or crosstab per group, with a documented compatibility contract.

csvtk

A cross-platform CSV/TSV toolkit popular in bioinformatics, with subcommands for selecting, filtering, joining, sorting, reshaping, converting and plotting. csvtk summary computes grouped statistics (count, sum, mean, median, quartiles, standard deviation, first, last, unique values, collapse), printing two decimals by default.

Choose csvtk when you want a single, familiar toolkit for everyday table work, including quoted CSV and compressed files. Choose Fastmash when you need datamash-style statistics and commands, portable numerical results, or its measured speed on large inputs.

Using them together

These tools combine well. Fastmash can read quoted CSV directly:

fastmash --csv -H -s -g region mean price < data.csv

Use another tool to filter or join records, then Fastmash to summarize them. For ordinary text workflows, a quote-aware conversion remains useful:

mlr --icsv --otsv cat data.csv | fastmash -H -s -g region mean price

User guide

Use Fastmash to summarize fields, combine groups, transform records and inspect tables. Commands take these forms:

fastmash [OPTIONS] OPERATION FIELDS [OPERATION FIELDS ...]
fastmash [OPTIONS] -g KEYS OPERATION FIELDS ...
fastmash [OPTIONS] MODE ...
fastmash [OPTIONS] top:N FIELD
fastmash [OPTIONS] compare BEFORE AFTER [rank RESULT [absolute|percent] [limit N]] OPERATION FIELDS ...
fastmash [INPUT OPTIONS] health [CONTROLS]

Fastmash reads records from standard input, splits each into fields, and applies the requested operations. It writes results to standard output and diagnostics to standard error, and reports success or failure through its exit status.

PageCovers
Fields and inputField selectors, separators, headers, record terminators, comments, missing values
Quoted CSV and result namesExplicit CSV formats, quoting rules and stable output labels
OperationsEvery summary statistic, with parameters and edge cases
Weighted meanContribution weights, complete-pair omission and numerical limits
Grouping and sorting-g, -s, case-insensitive grouping, crosstab
Highest and lowest recordsComplete-record selection, stable ties, groups and capacity limits
Dataset comparisonBefore/after summaries, global key alignment and change ranking
Table health reportsInspection, supplied validation rules and versioned TSV output
Per-row operations and modesRounding, binning, checksums, paths, cut, transpose, check, rmdup
Numbers, output and localesNumber formats, --format, --round, locales, portable results
Errors, exit statuses and resourcesWhat can fail, how it is reported, memory and disk use
Differences from GNU datamashEvery intentional difference

fastmash --help prints a complete summary of operations and options.

Fields and input

Records and fields

Ordinary delimited-text input is a sequence of records, each ended by a newline (or by a NUL byte with -z). The last record may omit its terminator. Each record is split into fields, by default at every tab.

Carriage returns are kept as data in ordinary text, so files with Windows line endings may need converting first (for example with tr -d '\r'). Explicit quoted CSV input accepts LF and CRLF record endings and permits line breaks inside quoted fields.

Selecting fields

Operations take a selector naming the fields to use:

SelectorMeaning
3Field 3 (fields are numbered from 1)
1,4Fields 1 and 4
2-5Fields 2 through 5
1,3-4Field 1, then fields 3 and 4
priceThe field named price in the input header
2:3The pair of fields 2 and 3, for paired statistics
1:2,3:4Two pairs

A list or range applies the operation to each field in turn, producing one result per field. Repeating a field repeats its result. Ranges must be ascending, and a range cannot be combined with pairs.

Field separators

OptionEffect
(default)Fields are separated by a tab
-t X, --field-separator=XFields are separated by the single byte X
-W, --whitespaceFields are separated by runs of spaces and tabs. Leading whitespace is skipped
--output-delimiter=XResults are separated by X

The output separator follows the input separator unless --output-delimiter is given. With -W, the output separator is a tab.

$ printf 'a;1\nb;2\n' | fastmash -t ';' sum 2
Summary
────────────────────────────────
  sum(field-2)
             3
✔ Completed

-t, does not understand CSV quoting: a comma inside a quoted field still splits it. Use --csv-in to read quoted CSV, or --csv for both input and output. Format switches do not imply headers. CSV input conflicts with -t, -W and -C. Either CSV direction conflicts with -z and --vnlog. See Quoted CSV and result names.

Headers

OptionEffect
--header-inThe first record names the fields. It is not treated as data
--header-outPrint a header line naming each result
-H, --headersBoth of the above

With an input header, selectors can use field names. Names are exact and case-sensitive. If two fields share a name, the first one is used.

$ printf 'price\tunits\n3\t2\n5\t4\n' | fastmash -H sum units mean price
Summary
────────────────────────────────
  sum(units)  mean(price)
           6            4
✔ Completed

NUL-terminated records

-z ends records with NUL bytes instead of newlines, for input from tools such as find -print0. Output records are NUL-terminated too.

Comments

-C (--skip-comments) skips records whose first non-blank character is # or ;. Comment markers later in a record are data. Skipped lines do not count towards line numbers in error messages.

--vnlog reads vnlog files: whitespace-separated columns with a # name name … header line and # comments. In vnlog data, everything after a # is ignored, even inside a field, and output uses spaces with a # header.

Missing values

--narm skips fields whose exact text is NA, N/A or NaN, in any letter case, for every operation:

$ printf '1\nNA\n3\n' | fastmash --narm sum 1 count 1
Summary
────────────────────────────────
  sum(field-1)  count(field-1)
             4               2
✔ Completed

Only these exact words count as missing. An empty field is not missing: a numerical operation reports it as an invalid value. A record that lacks the selected field is an error.

For paired statistics such as covariance, each side skips missing values independently, so remaining values may pair across different records. Weighted mean instead omits the whole value/weight pair after validating supplied partners.

Other options

OptionEffect
-f, --fullPrint a complete input record before each group’s results. The record is the group’s first, unless first, last, rand or an extremum selects another. GNU datamash’s deprecation warning for this option is kept
-S N, --seed NMake rand reproducible. Without a seed, rand is not reproducible. Its selection, like GNU datamash’s, is slightly biased and not cryptographic
--no-strict, --filler XAccept ragged rows and set the missing-cell text for transpose and crosstab. They don’t fill in missing fields for summary operations
--End option processing

Output headers name each result after its operation and field, such as sum(price), and grouping keys as GroupBy(site).

Options can appear after operations, and short options can be combined (-sg1) as in GNU datamash. If the POSIXLY_CORRECT environment variable is set, option processing stops at the first operation.

Quoted CSV and custom result names

Use --csv to read and write quoted CSV. Add -H when the first record names the fields. For example, summarize revenue by region:

$ printf 'region,revenue\nwest,2\neast,4\nwest,6\n' | fastmash --csv -H -s -g region sum revenue
GroupBy(region),sum(revenue)
east,4
west,8

--csv-in selects only CSV input, and --csv-out selects only CSV output. Ordinary delimited text remains the default. Use custom result names to give calculated columns stable labels, such as total instead of sum(revenue).

Supported workflows

WorkflowCSV input/outputResult names
Aggregate and per-row reportsYes, including supported grouping and full copied fieldsCalculated output columns with an explicit output header
Weighted meanYes, including adjacent and sorted groupsOne name per value/weight result
Top-N selectionYes, including adjacent and sorted groupsNo: selection copies the source fields
Dataset comparisonYes. Both sources use the same input formatOne name throughout each summary’s before/after/change columns
Crosstab, field modes and legacy table modesNoNo
Table healthNo. Ordinary text input and readable or TSV outputNo

Calculations on quoted CSV

Use --csv-in for strict quoted CSV input with ordinary text output, or --csv for CSV input and output together. --csv is equivalent to --csv-in --csv-out. Repeating these exact-only switches is idempotent. None enables headers. Top-N selection supports decoded CSV input for whole-dataset, adjacent-group and sorted-group selection with either output format. Sorted selection retains complete decoded records and original-input ties through bounded spill. Its copied output header uses the original source header or first accepted source record, independently of sorted encounter order. Its ranking field uses the decoded field-local numerical contract, and copied data preserves complete bytes. Aggregate and per-row calculations support their existing positional lists, numeric ranges, escaped named selectors, paired selectors and parameters. Aggregate grouping and --full work with decoded CSV fields. Add -s to bring equal decoded grouping keys together, using bounded spill when the input exceeds the maintained sort-memory target. Per-row operations retain their ordinary conflict with grouping keys. With no grouping keys, -s keeps the streaming workflow and source order, including per-row and --full reports. Output-only CSV retains sorted text-input workflows.

For example, a quoted export can be summed directly, without a quote-aware preprocessing step, while preserving a comma in its decoded column name:

printf '"amount,raw",note\r\n"2","two\nlines"\n4,"a""b"' | LC_ALL=C fastmash --csv -H --result-name='1:total,"daily"' sum 'amount\,raw'
"total,""daily"""
6

Grouping decoded fields

Grouping keys compare decoded values, so west and "west" belong to the same adjacent group. Repeated keys in a later run stay separate without sorting:

printf '"region,name",value\n"west,coast",2\n"west,coast",4\neast,3\n"west,coast",5\n' | LC_ALL=C fastmash --csv -H -g 'region\,name' --result-name=1:total sum value
"GroupBy(region,name)",total
"west,coast",6
east,3
"west,coast",5

Multiple and named keys retain their normal request order, exact name lookup and first-duplicate-name rules. Adjacent equality is byte-based, with ASCII folding under -i where the selected locale supports it. Equal-length keys compare up to NUL, while displayed keys retain their complete byte spans. Locale-aware numeric input and output keep their existing rules. A language locale does not silently merge differently spelled adjacent keys.

Sorting the same export merges the separate west,coast runs:

printf '"region,name",value\n"west,coast",2\n"west,coast",4\neast,3\n"west,coast",5\n' | LC_ALL=C fastmash --csv -H -s -g 'region\,name' --result-name=1:total sum value
"GroupBy(region,name)",total
east,3
"west,coast",11

Sorting compares complete decoded keys with the existing locale and case policy, then preserves source order for tied keys. Group equality remains distinct from locale ordering: differently spelled keys which the locale ties can still form separate adjacent groups. Headers bind before sorting. Without an input header, a generated output header takes its field width from the first sorted record. Copied records keep their actual widths and original logical record/physical-line locations through reordering and representative selection.

CSV sorting uses the same sort-memory policy as ordinary text, including FASTMASH_SORT_MEMORY_BYTES, and spills sorted runs to temporary files when needed. Complete decoded fields and their original input locations survive sorting. The memory target covers sort chunks, not the whole process. An oversized record, merge heads and retained statistical or text samples have their own storage needs. Samples are not spilled. Temporary I/O errors fail with status 1, and checked allocation failures refuse with status 77. Later failures may leave earlier output.

Copying complete records

Full reports copy each decoded source field before the calculation results. For example, a per-row report can retain a comma in a source field while selecting its numeric field with a custom result label:

printf 'place,value\n"west,coast",2\neast,4\n' | LC_ALL=C fastmash --csv -H --full --result-name=1:reading cut value
place,value,reading
"west,coast",2,2
east,4,4

For an aggregate, the representative starts as the group’s first record. first, last, rand and improving extrema may select another record under their existing rules. Extrema ties retain the earlier selection. With multiple operations the representative need not supply every result. Aggregate --full retains its existing GNU compatibility warning. Per-row --full does not warn. Copied-field labels replace GroupBy labels, and result-name indices still count only operation-produced results. Without sorting, header width comes from the first logical record. Each copied data record keeps its actual width, without padding or filler. Required fields must still exist, including with --no-strict or --filler. Copied fields preserve complete bytes, including NUL, independently of an operation’s display rules.

CSV syntax and bytes

A double quote opens quoting only at the start of a field. Inside quotes, commas, CR and LF are data, and two double quotes decode to one. After a closing quote, only comma, LF, CRLF or end of input is allowed. Quotes within unquoted fields, stray bytes after closing quotes, unfinished quoted fields and bare CR outside quotes are errors, including in unused fields. Spaces are data, with no implicit trimming. LF and CRLF endings may be mixed. A complete final record may omit its ending.

An empty file has no records. Each blank LF or CRLF record has one empty field. Quoted and unquoted empty fields decode identically. A trailing comma supplies a final empty field, while a terminal record ending supplies no extra record. Different widths are permitted when all requested fields exist. An absent field still errors. An existing empty field is valid text but invalid numeric input. --narm retains the existing exact, case-insensitive NA/N/A/NaN matching and independent paired-side compaction. It does not skip empty fields or trim spaces.

Decoding preserves bytes, including tabs, NUL, non-UTF-8 and internal CRLF. Operations keep their existing display and transformation rules: cut stops display at NUL, while base64 and checksums consume the complete decoded field. The active numerical locale and presentation settings apply to each isolated decoded numeric field. An unused numeric neighbor cannot affect conversion. Ordinary text numerical conversion keeps its existing behavior.

A leading UTF-8 BOM is data. Before an unquoted input header it becomes part of the name and generated label, following the normal byte-escaped selector rules. Before a quoted first field it is data followed by an illegal quote, so that export fails. There is no BOM stripping or transcoding. This dialect has stated byte and blank-record extensions. It is not a claim of full RFC 4180 conformance.

With --csv-in, text results use the normal tab default and accept an explicit --output-delimiter. Decoded delimiters or line breaks in result fields may make that text report ambiguous. Use --csv when field boundaries must survive output. CSV input conflicts with explicit -t, -W and -C, including their GNU abbreviations and redundant comma delimiters. Either CSV direction conflicts with -z and --vnlog, in either option order, with error status 1. Legacy modes refuse CSV before input. Health rejects all CSV switches and remains text-only.

Errors and locations

CSV validation errors identify the original logical record number, counting an input header, and its starting physical line. Syntax errors also identify the detected physical line and byte position, counting raw bytes including quotes. Unfinished quotes identify unexpected end of input. LF inside quotes advances the physical line. CRLF counts as one line break. Header lookup errors include the header’s location when it exists. I/O failures keep their I/O context without inventing a record or turning a read failure into a quoted-field EOF error. Malformed CSV uses status 1, checked capacity refusals use 77, and output finalization retains normal precedence. Later failures can leave partial output.

CSV output from delimited text

Use --csv-out to produce quoted CSV from ordinary delimited-text input. It works with aggregate and per-row operations, including adjacent or sorted groups, explicit headers and copied --full fields. Repeating the switch is idempotent. It does not enable either input or output headers. Top-N selection also supports CSV output from complete selected text records, including sorted groups. It adds no calculation columns.

For example, tab-separated observations can become a CSV report with a stable result label, including keys and names containing commas or quotation marks:

printf 'place\treading\nlab,west\t2\nlab,west\t4\n' | LC_ALL=C fastmash --csv-out -H -g place --result-name='1:total,"daily"' sum reading
GroupBy(place),"total,""daily"""
"lab,west",6

Every logical output field is encoded separately: complete generated labels, custom names, grouping keys, formatted results and each copied --full field. Commas separate fields and LF ends each emitted record. A field containing a comma, double quote, CR or LF is quoted, with internal double quotes doubled. An empty field prints as "". Embedded line breaks retain their bytes. Output is canonical. It does not retain an input field’s quoting style or line ending.

Decimal-comma formatted numbers and collapse or unique lists stay within one quoted result field. The collapse delimiter is still the list’s internal separator. CSV adds no escaping scheme inside those lists. Encoding preserves the result bytes, including non-UTF-8 and NUL bytes, under the existing operation and header display rules. For example, raw keys and copied fields retain NUL, while selected text results and generated field-name displays stop at NUL as they do in ordinary output. A leading UTF-8 BOM remains field data.

Output-only CSV allows text-input -t, -W and -C. An explicit --output-delimiter, -z or --vnlog conflicts with CSV output and fails with status 1, regardless of option order or a redundant delimiter value. Crosstab, legacy Table modes and field modes refuse CSV with status 77 before input. Health reports remain text-only and reject CSV switches using their status-1 calculation-control diagnostic. Only the exact CSV switch spellings are accepted. Existing GNU option abbreviations retain their meanings.

CSV output retains empty-input and header-only timing and the normal failure policy. A later input, calculation or transport failure can leave partial output.

Custom result names

Give calculated output columns stable names in your reports. Select output headers with --header-out or -H, then repeat --result-name=INDEX:NAME for the labels you want to replace. The separate form --result-name INDEX:NAME works too.

For example, a report can give its sum a stable schema name while retaining the generated label for its mean:

printf 'reading\n2\n4\n' | LC_ALL=C fastmash -H --result-name=1:total sum reading mean reading
Summary
────────────────────────────────
  total  mean(reading)
      6              3
✔ Completed

Indices follow operation-produced output order after selector expansion, starting at 1. A list or range contributes each expanded result, repeated operations contribute their results again, and a paired operation contributes one result. Each field selected by cut also contributes a result. Grouping keys and the copied fields from --full do not consume indices or receive replacement labels.

In this example, sum 2,1-2 produces three results in the order 2, 1, 2. Names can be supplied out of order, and unnamed results keep their generated labels:

printf '1\t2\n3\t4\n' | LC_ALL=C fastmash --header-out --result-name=3:again --result-name=1:total sum 2,1-2
Summary
────────────────────────────────
  total  sum(field-1)  again
      6             4      6
✔ Completed

Names replace labels only. They do not change selector lookup, required-field checks, calculation values, output order, or when headers appear. Empty input still produces no invented header. With -H, header-only input retains its normal output header without a result row. Later input or output failures can leave partial output, as with ordinary generated headers.

INDEX must be an unsigned, nonzero decimal integer within the expanded result count. Repeating a target, omitting a name, or supplying an empty name is an error. Duplicate name values are allowed. Split at the first colon, so a name can itself contain colons. Name bytes are preserved as supplied by the operating system. For text output, names cannot contain the effective output delimiter or record terminator. CSV output allows comma, quote, colon, CR and LF bytes in names and encodes them as one field. Invalid names fail before any output.

Result names work for ordinary aggregate and per-row operations, including adjacent and sorted groups. Crosstab, table modes (transpose, check, rmdup) and field modes (reverse, noop) refuse naming before reading input. Only the exact --result-name spelling is accepted. Existing GNU option abbreviations retain their normal meanings. --vnlog alone does not satisfy the requirement for explicit output headers. Add --header-out or -H.

GNU datamash 1.9 generates labels such as sum(reading). Stable custom report labels otherwise need a wrapper or a downstream renaming step. Naming inside the command avoids that extra step and follows the existing header timing.

Operations

Operations summarize a field over all records, or over each group. Operation names are case-insensitive. Parameters follow a colon: perc:90, trimmean:0.1.

For operations that transform each record instead, see Per-row operations and modes.

Counting and text

OperationResult
countNumber of values, including text and empty fields
countuniqueNumber of distinct values (case-sensitive unless -i)
unique, uniqSorted, comma-separated list of distinct values
collapseComma-separated list of all values, in input order
firstFirst value
lastLast value
randOne value chosen at random (reproducible with --seed)

-c X (--collapse-delimiter) changes the separator used by unique and collapse.

Basic statistics

OperationResult
sumSum
meanArithmetic mean
min, maxMinimum, maximum
absmin, absmaxValue with the smallest or largest magnitude, keeping its sign
rangemax minus min

Other means

OperationResult
geomeanGeometric mean
harmmeanHarmonic mean
msMean of the squared values
rmsRoot mean square
trimmean[:P]Mean after removing the fraction P of values from each end (default 0, with 0.5 giving the median)
wmean VALUE:WEIGHTWeighted mean, with finite nonnegative contribution weights

wmean takes one value/weight pair per result, for example wmean price:units with named fields or wmean 2:3 with positions. It supports text or quoted CSV, headers, custom result names and adjacent or sorted groups. A nonempty calculation needs positive retained weight. --narm omits whole pairs after checking supplied partners. See Weighted mean for examples, missing values and numerical range limits.

Quantiles and order statistics

OperationResult
medianMiddle value, or the mean of the two middle values
q1, q3First and third quartiles
iqrInterquartile range, q3 minus q1
perc[:N]Nth percentile, for integer N from 1 to 100 (default 95)
modeMost frequent value (the smallest, on ties)
antimodeLeast frequent value (the smallest, on ties)

Quartiles and percentiles use linear interpolation between order statistics (Hyndman and Fan’s type 7, the default in R and NumPy).

Dispersion

OperationResult
pvar, svarPopulation and sample variance
pstdev, sstdevPopulation and sample standard deviation
madMedian absolute deviation, scaled by 1.4826
madrawMedian absolute deviation, unscaled

Shape and normality

OperationResult
pskew, sskewPopulation and sample skewness
pkurt, skurtPopulation and sample excess kurtosis
jarqueJarque–Bera normality test p-value
dpoD’Agostino–Pearson omnibus normality test p-value

Paired statistics

These take a pair of fields, LEFT:RIGHT:

OperationResult
pcov, scovPopulation and sample covariance
ppearson, spearsonPopulation and sample Pearson correlation coefficient
dotprodDot product
$ printf '1\t2\n2\t4\n3\t7\n' | fastmash scov 1:2 spearson 1:2
Summary
────────────────────────────────
  scov(field-1,field-2)  spearson(field-1,field-2)
                    2.5           0.99339926779878
✔ Completed

spearson is the sample Pearson coefficient, not Spearman’s rank correlation.

Small samples and missing data

  • jarque and dpo keep GNU datamash’s formulas, including its tail cancellation: very small p-values can print as 0.
  • Sample statistics need enough values: svar and sstdev give nan for a single value. sskew needs at least three and skurt at least four.
  • When every value in a group is missing (with --narm), count and sum give 0. mean, median and most statistics give nan. min and max give -inf and inf. first and last give N/A.
  • An empty input produces no output line.

Memory use

Most operations keep a running total. Quantiles, modes, dispersion, shape, normality and paired statistics must keep every value of a group, and text operations like unique and collapse keep their text. Memory then grows with the size of the largest group, and this storage is not spilled to disk. Related operations on the same field share storage: quantiles, mode and antimode share one sorted copy, the variance family another, and the moment and normality statistics another.

Weighted mean

Use wmean VALUE:WEIGHT to average values according to their contribution weights. The result is the sum of weighted values divided by the total weight. Weights must be finite and nonnegative. They may be fractions and need not sum to one. Zero contributes nothing.

For example, values 10 and 20 with weights 1 and 3 produce 17.5:

printf '10\t1\n20\t3\n' | env LC_ALL=C fastmash wmean 1:2
Summary
────────────────────────────────
  wmean(field-1,field-2)
                    17.5
✔ Completed

Named reports and adjacent groups

Use explicit input and output headers with -H, or select them independently with --header-in and --header-out. Named selectors use the existing case-sensitive first match for duplicate names. A backslash escapes one byte in a name. Positional and named fields may be mixed. Pair lists, repeated pairs, shared weight fields and same-field pairs each produce a result in request order.

printf 'site\tvalue\tweight\na\t10\t1\na\t20\t3\nb\t5\t2\n' | env LC_ALL=C fastmash -H -g site --result-name=1:average wmean value:weight count value
Grouped summary
────────────────────────────────
  site  average  count(value)
     a     17.5             2
     b        5             1
✔ Completed

Without a result name, the generated label identifies both roles, for example wmean(value,weight) or wmean(field-1,field-2). Each expanded pair consumes one result-name index. Grouping keys and copied --full fields consume none. Use -H for named result labels. No count or total-weight column is added automatically. Empty input produces no report. Header-only input retains its ordinary output header and emits no data result. This includes sorted reports with named fields when input ends cleanly before the requested header. A present header still has to contain the requested names, and a failed read is an input error.

Grouping uses the existing group definition: consecutive records with equal grouping keys. Multiple keys and existing case and locale controls apply. Separated runs of the same key remain separate. The calculation resets between groups. Every requested grouping key must exist, including with --full and on omitted pairs. An empty key is still present. A preceding group completes before the next group’s pair is collected, preserving its result or completion failure. wmean leaves the ordinary --full representative unchanged, initially the first record. Another requested operation may replace it. It does not choose the largest-weight record. Aggregate --full retains its existing deprecation warning.

For ordinary-text input, existing separators, whitespace splitting, comment filtering, NUL record termination, numeric locales, --format and --round apply. Unused ragged or byte-oriented content remains accepted when every required numeric and key field exists. This calculation does not inspect whole-table health.

Sorted text reports

Use -s to bring interleaved grouping keys together. Multiple keys, named fields, headers, result names and other aggregate operations retain their usual behavior:

printf 'site\tvalue\tweight\nb\t5\t2\na\t10\t1\nb\t7\t2\na\t20\t3\n' | env LC_ALL=C fastmash -H -s -g site --result-name=1:average wmean value:weight
Grouped summary
────────────────────────────────
  site  average
     a     17.5
     b        6
✔ Completed

The same calculation accepts a regular file, for example env LC_ALL=C fastmash -H -s -g site wmean value:weight < measurements.tsv. Sorting, including hash grouping and replay, preserves the encounter order of each group’s contributions. It does not rearrange values or weights to improve the arithmetic. Existing case and supported-locale controls determine key ordering and equality. With -W, blanks before a key can affect sorting before ordinary group equality is applied. With no grouping keys, -s keeps the whole-dataset encounter order.

Both pair fields are retained, including reversed pairs and shared fields. --full keeps complete representative records using the existing selection rule. Sort storage has its own memory policy and can spill. The weighted accumulator still keeps two totals per request. Ordinary sorted numeric errors use the established sorted record position, rather than claiming an original-input location. Temporary-storage, replay, read, output and close failures prevent successful status, and numerical-range refusals remain unchanged.

Select --csv-out independently to encode results, keys, copied fields and generated or supplied labels through the existing output encoder:

printf 'site\tvalue\tweight\nb\t5\t2\na\t10\t1\na\t20\t3\n' | env LC_ALL=C fastmash --csv-out -H -s -g site wmean value:weight
GroupBy(site),"wmean(value,weight)"
a,17.5
b,5

CSV output follows the shared encoding rules, including quoting generated labels such as wmean(value,weight) as one field.

Quoted CSV reports

Use --csv-in for CSV input with ordinary text output, --csv-out for CSV output from ordinary text, or --csv for both. The switches do not imply headers. Numeric conversion and named selectors see decoded field bytes, including quoted names with commas, quotes or line breaks:

printf 'site,"value,raw",weight\r\nb,5,2\na,"10",1\nb,7,2\n"a",20,3\n' | env LC_ALL=C fastmash --csv -H -s -g site --result-name=1:average wmean 'value\,raw:weight'
GroupBy(site),average
a,17.5
b,6

Omit -s to report each adjacent run separately, or omit -g site to report one whole-dataset mean. Regular-file input uses the same calculation. Sorting compares decoded keys, so a and "a" name the same key. Multiple keys and existing case and supported-locale controls apply. Stable sorting and bounded spill retain complete needed fields and each group’s contribution order.

CSV output preserves complete decoded keys and copied --full fields using canonical encoding. CSV-to-text output retains ordinary text framing, so a decoded tab or newline can appear as a delimiter or record break. Choose CSV output when those bytes need unambiguous field boundaries.

Strict syntax validation covers every field and record, including unused content, omitted pairs and zero weights, even after an infinite or NaN result is known. The shared CSV grammar and option conflicts apply. Fastmash does not infer the input format.

Numeric, invalid-weight, missing-field and header errors retain the original logical record and starting physical line through sorting and spill. Syntax errors also identify the detected physical line and byte position. Completion errors identify the weighted pair and group without assigning a source record. Read, temporary-storage, output and close failures prevent successful status. Earlier output can remain under the ordinary report rules.

Missing pairs and failures

Values 10 and 20 with weights 0.25 and 0.75 also produce 17.5. Negative or nonfinite weights are input errors with status 1. Both signed zeros are zero weights. A zero-weight record still needs valid supplied fields, but its product is not evaluated, even for an admitted infinite or NaN value. Positive weights permit admitted infinite and NaN values under the ordinary arithmetic rules. Later records are still validated after such a result is known.

With --narm, an exact whole-field NA, N/A or NaN token, ignoring case, omits the entire value/weight pair for that request. Supplied nonmissing fields are validated in value-then-weight order before deciding omission. Thus value NA with weight 2 is omitted, while value NA with weight oops, -1 or infinity fails. Empty fields, absent fields, malformed numbers and conversion range failures remain errors. Surrounding spaces, signed NaNs and NaN payloads do not become missing values. Without --narm, the ordinary numeric conversion rules apply.

Omission is local to each request. Another weighted pair or an unweighted operation can still use the same record. Grouping keys and group changes are observed before omission. A nonempty dataset or group with only zero or omitted weights fails with status 1, identifying the weighted pair and, when grouped, its keys. It does not silently disappear or produce a placeholder mean. Completion errors do not invent a source record location. Earlier completed groups, a header or part of the failing output record may already have been written. Input, output and close failures also prevent successful status.

Numerical trace and limits

The calculation uses 80-bit software arithmetic, without narrowing operands to 64-bit floating point. It starts both totals at positive zero. For each retained positive-weight pair in encounter order it rounds the weight-times-value product, adds and rounds that product into the weighted total, then adds and rounds the weight into the weight total. Completion divides the totals with one rounding. Each step uses nearest-even rounding. There is no fused product/addition or rescaling. Results can depend on record order. Arbitrary weight scaling is not promised to preserve every rounded bit.

A numerical capacity refusal with status 77 identifies a finite product that overflows or rounds a nonzero value to zero, an overflowing finite weighted total, an overflowing weight total, or an overflowing finite final division. The check happens when the intermediate is formed, even if later records might cancel it. Deliberate infinite value inputs are distinguished from finite overflow. Finite subnormal intermediates, ordinary rounding and cancellation, and final-result underflow remain permitted.

These limits can refuse finite inputs whose mathematical mean is representable. Value 2 with weight 1e4932 refuses at its product. Value 1e-4000 with weight 1e-4000 refuses when its nonzero product rounds to zero. Scaling weights in advance may change the evaluation trace and rounded result.

Unsorted wmean retains two numeric totals per request, independent of input record count. Full row context and other requested operations keep their own storage policies. This is not a total-process-memory or performance comparison.

Grouping and sorting

Groups

-g KEYS (also written groupby KEYS, grouping KEYS or gb KEYS) makes Fastmash summarize each group of records that share the same key values. Output rows start with the key values, followed by the results:

$ printf 'A\tx\t1\nA\tx\t2\nA\ty\t5\nB\ty\t4\n' | fastmash -g 1,2 sum 3
Grouped summary
────────────────────────────────
  field-1  field-2  sum(field-3)
        A        x             3
        A        y             5
        B        y             4
✔ Completed

A group is a run of adjacent records with equal keys. When the key changes, the group ends and its results are printed. This makes grouping a single pass over the input, with memory proportional to one group.

If equal keys are not adjacent, the same key appears in more than one group:

$ printf 'b\t2\na\t1\nb\t3\n' | fastmash -g 1 sum 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)
        b             2
        a             1
        b             3
✔ Completed

Sorting first: -s

-s sorts the records by their keys before grouping, so every key forms one group:

$ printf 'b\t2\na\t1\nb\t3\n' | fastmash -s -g 1 sum 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)
        a             1
        b             5
✔ Completed

The sort is stable: records with equal keys keep their input order, so first, last and collapse give predictable results.

Key order depends on the locale. In C and C.UTF-8, keys are ordered byte by byte. In the supported language locales, such as en_US.UTF-8, fr_FR.UTF-8 or pl_PL.UTF-8, keys are ordered alphabetically, as GNU sort orders them, using built-in Unicode collation rules and glibc’s table of the punctuation and symbols it ignores.

If your input is already sorted, leave out -s: the result is the same and the job is faster.

Large inputs

Fastmash sorts inside its own executable. It holds a sort buffer in memory and writes sorted runs to temporary files when needed. Windows uses its system temporary directory. On Linux and macOS, TMPDIR selects the directory, with /tmp as the default. The operating system removes the files when Fastmash exits, including after interruption.

For eligible input with repeated keys, hash grouping collects operation state per key and sorts only the groups. If this cannot preserve sorted behavior, it rereads the input and sorts it before writing results. Language-locale pipes can retain input for that replay. C-locale pipes normally sort directly. File and pipe input can therefore have different time and memory costs. A file that changes between readings contributes its second-reading contents.

Sorting can spill, but the values retained by statistics such as quantiles stay in memory. A sort-memory target is not a total-process memory limit. See memory and disk requirements.

Resource controls

FASTMASH_SORT_MEMORY_BYTES sets the sort-buffer memory target, and FASTMASH_SORT_TRACE (set to anything) reports each decision on standard error. Neither is a limit on Fastmash’s total memory use.

FASTMASH_GROUPING=sort always sorts. For piped input, FASTMASH_PIPE_GROUPING=sort selects sorting and =hash permits hash grouping in every supported locale. --vnlog and rand always sort. These controls change the route, not the result.

When the selected calculations support parallel sorting, records are prepared on up to 8 threads. On Linux, numerical calculations can merge spilled records and calculate in parallel on two threads. Both use the processors available to Fastmash, or OMP_NUM_THREADS if it is set, capped by OMP_THREAD_LIMIT. Set OMP_NUM_THREADS=1 to sort on one thread.

Sorted rmdup keeps complete records and stable key order, with the same temporary-storage policy. Case folding affects ordering, while duplicate keys are compared exactly. The architecture describes buffer sizing, parallel preparation and replay eligibility.

Ignoring case: -i

-i compares keys ignoring ASCII letter case, for grouping and sorting. The output shows each key as it first appeared. -i also affects countunique and unique. Under the Turkic character types (tr_TR, az_AZ and a few others), where glibc does not fold i and I, -i is refused. Set LC_CTYPE=C.UTF-8 to fold ASCII letters.

Pivot tables: crosstab

crosstab KEY1,KEY2 (or ct) builds a two-way table with one row per value of the first key and one column per value of the second. By default each cell is a count. You can name one operation instead:

$ printf 'a\tx\t1\na\ty\t2\nb\tx\t3\n' | fastmash -s crosstab 1,2 sum 3
	x	y
a	1	2
b	3	N/A

Missing combinations show the --filler text (default N/A). Rows and columns are ordered byte by byte. Use -s unless each combination’s records are already adjacent.

Highest and lowest records

Use top:N FIELD to keep the N records with the highest values in one numeric field. bottom:N FIELD keeps the lowest. Each selected record retains all of its fields.

Select the highest records

$ printf 'A\t8\tfirst observation\nB\t10\tsecond observation\nC\t10\tthird observation\nD\t9\tfourth observation\n' | fastmash top:2 2
Top-N selection
────────────────────────────────
  field-1  field-2  field-3
        B       10  second observation
        C       10  third observation
✔ Completed

B and C tie at 10. Earlier input records win ties, including at the cutoff. The command emits at most N records in rank order. Duplicates remain separate observations, and fewer eligible records produce fewer results.

Use a field number or, with -H or --header-in, a field name. Lists, ranges and field pairs are not accepted for ranking. Selection cannot be combined with calculations or other modes.

Ranking and missing values

Ranking uses the active numerical profile and supported locale, including its binary80 precision. Numbers that convert to the same value tie, regardless of their spelling. Both zero signs tie. Infinities sort beyond finite values in their numeric direction. Any unskipped NaN is an error because it is unordered. Only the ranking field must be numeric. Accompanying fields are copied without numeric checks.

With --narm, records whose ranking field is exactly NA, N/A or NaN, in any letter case, are omitted. Matching does not trim whitespace. Empty or absent fields, malformed numbers, range failures, signed NaNs and NaN payloads remain errors. Empty and all-missing datasets emit no data records. Every accepted record is checked, including late records that cannot enter the selection.

Copied fields and CSV

Selection copies complete logical fields in their original order, retaining source numeric spelling, trailing empty fields, ragged widths and arbitrary accompanying bytes. It adds no score, rank or result column. Ordinary separators, whitespace input, output delimiters, comment filtering and NUL-terminated records use the existing controls. Whitespace separator runs become the effective output delimiter, and a final unterminated record receives an output terminator. Ordinary output does not escape embedded output delimiters or record terminators in copied fields. Use --csv-out to encode selected text records as CSV, including sorted text groups. Use --csv-in to select decoded CSV records with ordinary output, or --csv for CSV input and output together. These exact-only switches are idempotent and do not enable headers.

CSV follows the shared input and output rules. Every field is checked, including fields on omitted records or candidates that cannot win. Output preserves complete decoded fields and numeric spelling, with canonical CSV quoting.

--full is accepted with the same output and no deprecation warning. -s without grouping keys leaves this whole-dataset selection unchanged.

Selection within groups

With -g FIELD[,FIELD...], each consecutive group has its own cutoff. Groups appear in encounter order, and rank order applies within each group. For example, fastmash -H -g category top:2 reading selects the highest two records in each adjacent category, binding category and reading from the input header. A later repeated category remains a separate group. Selected records retain every field once, without an extra grouping-key prefix. Multiple keys and -i follow the existing group equality and supported locale controls. No-key -i leaves numeric ranking unchanged. Empty key text is a key value, while an absent required key is an error. Every required key and group boundary is checked before --narm omission, so an all-missing intervening group still separates equal-key runs on either side. CSV keys compare decoded field values, so different quote spellings of the same text belong to the same adjacent group.

Add -s to prepare interleaved categories under the existing sorted group workflow, for example fastmash -s -H -g category top:2 reading. Groups then follow the existing key order, with ranking applied inside each group. Sorting keeps complete records and original input identity through memory and spill, so earlier source records still win equal-rank ties. Sorting order and group equality retain their existing distinct meanings: whitespace key separators, case controls, language-locale spellings and NUL-terminated input can affect the prepared runs without introducing global key normalization. Selection uses the existing native sort for text and decoded CSV sort for quoted CSV. Pipe and regular-file input use the same selection route for each format.

For example, fastmash --csv -s -H -g category top:2 reading selects the highest two records per category from an interleaved CSV export. Sorting uses decoded key bytes, so quoted and unquoted spellings of the same key sort together. Each selected record keeps its complete decoded fields, including multiline notes, through sorting and spill. CSV output encodes them again. Locale ordering still does not merge differently spelled keys that existing group equality treats separately. No global key normalization is introduced.

Headers

--header-in consumes the first accepted record as the input header and excludes it from ranking. Names resolve to the first matching label when a header contains duplicates. --header-out emits one copied-field output header for the whole command, without operation wrappers or appended group labels. -H selects both controls. With an input header, copied labels follow the existing display convention, stopping at the first NUL byte. Without one, labels are field-1, field-2 and so on, using the first accepted source data record’s width even when that record is omitted or does not win. Later ragged records do not revise the header, and sorting does not change its source schema. Positional ranking fields may extend beyond the input header’s labels when the data supplies them. Empty input invents no header. Header-only input with both controls and all-missing nonempty input can emit a requested header without data. Name lookup fails before header output.

Completion and failures

The entire input must be inspected before ungrouped data is emitted. Completed adjacent groups can emit as the next group begins. A read failure or malformed record does not emit the incomplete current selection, but earlier groups and the output header may remain. Output writes and finalization must succeed. A later output failure can leave partial bytes. Sorted grouping finishes input intake before emitting selected data. A read or temporary-sort failure emits no incomplete sorted group. Later ranking or key errors can leave already completed groups. Diagnostics retain original accepted record numbers, including the input header and excluding filtered comments. CSV input errors identify the original logical record and its starting physical line. Syntax errors also retain their detected location.

Memory and limits

N is a mandatory positive decimal integer, at most 18446744073709551615. Leading zeros are accepted. At most N candidate records are retained for the current group or whole dataset, with storage growing as records arrive and candidate state released between groups. A large requested N does not allocate N records in advance. N limits candidate count, not bytes or total process memory: large records or a large N can still exhaust memory. Checked resource failures refuse with status 77. Fastmash does not truncate fields, reduce N or approximate the winners. Sorted grouping has a separate sort-memory target, controlled by FASTMASH_SORT_MEMORY_BYTES, and spills input when needed under the maintained sort policy. Candidate retention remains bounded by N for the active group and does not spill. Neither limit is a guarantee of total process memory.

Option compatibility

CSV input conflicts with explicit text-input separators and comment filtering. Any CSV format conflicts with NUL termination and --vnlog. CSV output also conflicts with an explicit output delimiter. Format conflicts are errors with status 1, independently of option order. CSV input with ordinary output can use an explicit output delimiter, subject to ordinary framing limits. A named field without -H or --header-in is a command error. Explicit result names, numeric presentation controls, collapse delimiter, filler, random seed, --no-strict, --vnlog and --sort-cmd are unsupported for selection and refuse with status 77. Malformed options and conflicting CSV controls retain error status 1. Ordinary commands keep their existing behavior.

Dataset comparison

Compare statistical summaries from two exports, including categories that were added or removed. For example, create two small sales exports:

printf 'category,revenue,weight\nwest,100,1\neast,50,2\n' > before.csv
printf 'category,revenue,weight\neast,75,2\nwest,80,1\nnorth,20,1\n' > after.csv

Compare revenue by category:

fastmash --csv -H -g category compare before.csv after.csv sum revenue
key(category),presence,rank,rank_state,before(sum(revenue)),after(sum(revenue)),difference(sum(revenue)),difference_state(sum(revenue)),percentage(sum(revenue)),percentage_state(sum(revenue))
east,matched,"",not_requested,50,75,25,available,50,available
north,added,"",not_requested,"",20,"",not_matched,"",not_matched
west,matched,"",not_requested,100,80,-20,available,-20,available

The east summary rises from 50 to 75, a change of 25 or 50%. west falls by 20%, and north appears as an added category with no before value. Without -g, the command compares one summary for each whole dataset.

Source files and supported operations

The examples below also use tab-separated copies of the same exports:

printf 'category\trevenue\tweight\nwest\t100\t1\neast\t50\t2\n' > before.tsv
printf 'category\trevenue\tweight\neast\t75\t2\nwest\t80\t1\nnorth\t20\t1\n' > after.tsv

BEFORE and AFTER are literal paths. A standalone - supplies stdin on one side:

printf 'category\trevenue\tweight\nwest\t80\t1\n' | fastmash -H -g category compare before.tsv - sum revenue

Files are opened independently, even when both paths are identical. The before source is read first, followed by after. Files are read as supplied, without a snapshot or locking. Use the ordinary -- option terminator for paths beginning with -. The same separator, whitespace and comment controls, record terminator, numeric locale, missing-value policy and operations apply to both sides.

Comparison supports whole datasets and positional or named comparison keys with single, list, range and paired operation selectors. All numerical aggregates, existing aliases and parameters are supported, including wmean VALUE:WEIGHT. Text-result aggregates such as first, unique and collapse are refused.

Input and output formats

Input and output formats are explicit and independent. The same input format applies to both sources. Comparison does not infer a format or support different formats on the two sides.

fastmash --csv-in -H -g 1 compare before.csv after.csv sum 2
fastmash --csv-out -H -g 1 compare before.tsv after.tsv sum 2
fastmash --csv -H -g category compare before.csv after.csv rank 1 percent limit 10 sum revenue

--csv-in reads strict comma-separated byte fields and retains ordinary text output. --csv-out encodes each complete output field as CSV, with commas and LF record endings. --csv enables both. These switches are exact-only, repeatable and header-neutral. Text input still supports its ordinary separator, whitespace, comment filtering and NUL record termination where allowed.

CSV uses the shared syntax and encoding rules. Names, numeric values and comparison keys use decoded fields, so different quote spellings of the same key consolidate. Every field is checked, including unused content, omitted pairs and records below a ranking cutoff. Output encodes each complete report field independently, quoting empty cells and locale decimal commas.

CSV input conflicts with -t, -W and -C. Either CSV direction conflicts with -z and --vnlog. CSV output also conflicts with an explicit --output-delimiter. These are command errors regardless of option order. CSV-to-text output permits an explicit delimiter, but embedded delimiters or line breaks can make text ambiguous. Use CSV output to preserve field boundaries.

Input and output headers

Use --header-in to consume the first accepted input-header record on each source. -H also requests an output header. A header-only source is valid and has no summary. An entirely empty source, including one containing only filtered comments, is an error when input headers are requested. Named fields must resolve on both headers even when a source has no data. Headers are never inferred.

Named selectors bind independently, so the same names can occupy different positions in the two exports:

fastmash -H -g category compare before.tsv after.tsv sum revenue
fastmash -H -g category compare before.tsv after.tsv wmean revenue:weight

Names use ordinary exact, case-sensitive lookup. The first duplicate label wins and backslash escapes one byte, for example metric\,one. Positional and mixed selectors retain the requested positions on both sides. Unselected header fields may differ. With only --header-in, positional data fields may extend beyond the header labels. When producing an output header, selected labels in its preferred source must exist under the ordinary header renderer’s rules. Unused after labels do not constrain positional data.

--header-out or -H requests exactly one fixed output header, including when both sources have no data. There is no implicit comparison header. Labels use the before input header when available, otherwise ordinary field-N labels. Keys are labeled key(FIELD_LABEL), followed by presence, rank and rank_state. For each ordinary generated summary label L, the six labels are before(L), after(L), difference(L), difference_state(L), percentage(L) and percentage_state(L). Label display retains ordinary conventions, including stopping at NUL. This never truncates comparison-key identity or spelling.

Result names replace L throughout one summary’s complete block:

fastmash -H --result-name=1:total compare before.tsv after.tsv sum revenue mean revenue
fastmash -H --result-name=2:total compare before.tsv after.tsv sum 2,3

Indexes count expanded summary results in request order, once per result. They do not count the six derived columns, rename keys or control columns, or identify a ranking result by name. Names must be nonempty. Each index can be supplied once, while different results may share a name. Text names must not contain the output delimiter or record terminator. CSV output permits and encodes those bytes. Numerical formatting applies to summary and change values independently of their labels.

Both source headers, required named binding, every input and all calculations must succeed before this output header is written.

Report columns

Without keys, each nonempty dataset has one summary. Output has presence, rank and rank_state, followed by six columns per expanded result in request order. Without ranking, every rank is empty and every rank state is not_requested:

ColumnMeaning
beforeCompleted numerical summary from BEFORE
afterCompleted numerical summary from AFTER
differenceSigned after minus before
difference_stateavailable, unordered or not_matched
percentageSigned ordinary percentage change, without a percent suffix
percentage_stateavailable, not_matched, zero_baseline, nonfinite_baseline or nonfinite_after

Presence is matched, added or removed. An empty source has no summary. An absent side and both changes have empty cells with not_matched change states. Two empty sources produce no data. Accepted records still establish a summary when --narm omits every value. Those operations keep their existing empty-sample results or completion errors, including weighted mean’s requirement for positive retained weight. Paired operations retain their established independent compaction. Weighted mean validates partners and omits whole pairs.

Comparison keys

Use the existing -g selectors to identify corresponding categories:

fastmash -H -g 1 compare before.tsv after.tsv sum 2 mean 2
fastmash -H -g 1,3 compare before.tsv after.tsv wmean 2:3

Every distinct key has one summary per source, even when its records are interleaved. Each repeated raw record contributes normally. All operations keep the original encounter order within each key, including rounding-sensitive weighted mean calculations. -s is accepted and gives exactly the same report. This consolidation differs from ordinary adjacent groups. It does not change their equality or the locale ordering of ordinary sorted groups.

A key is an ordered tuple of complete field byte strings. Field boundaries, lengths and bytes after NUL all matter: ('a', 'bc') differs from ('ab', 'c'), and 01 differs from 1. Whitespace is retained. -i folds only ASCII A through Z across complete fields. Non-ASCII bytes remain unchanged. Empty key text is valid, while a missing required key field is an error. Keys are extracted before any operation omits missing values, so all-omitted categories remain in the report and keep their ordinary completion results or errors.

Reports align the complete union of keys after both sources finish. Keys precede the report columns above. A matched or removed key displays its first before spelling. An added key displays its first after spelling, including under -i. Unranked reports follow lexicographic order over normalized key tuples, using unsigned bytes and shorter prefixes first. This order is independent of locale collation and input adjacency.

For a finite nonzero baseline, percentage is (after - before) / before * 100. The denominator retains its sign: -100 changing to -50 has difference 50 and percentage -50. A change from -100 to -150 has difference -50 and percentage 50. Both zero signs make percentage unavailable. Reasons are checked in this order: not_matched, zero_baseline, nonfinite_baseline, nonfinite_after. NaNs and infinities remain visible in the before and after summary cells. NaN differences, including equal-sign infinities, are unordered. Other infinite differences remain available.

Ranking and limits

Place rank RESULT [absolute|percent] after the source paths and before all operations. The default measure is absolute. RESULT selects one 1-based expanded summary result, counting list and range expansion in request order. Result names and the six derived columns do not affect its index. All requested result blocks remain in the report:

fastmash -H -g 1 compare before.tsv after.tsv rank 1 sum 2
fastmash -H -g 1 compare before.tsv after.tsv rank 2 percent limit 10 sum 2-3
fastmash -H -g category --result-name=1:total compare before.tsv after.tsv rank 1 percent sum revenue

Ranking orders matched keys by descending magnitude of the selected available difference or percentage. Signed cells keep their signs, including negative baseline percentages. For example, A changing from 100 to 130 ranks ahead of B changing from 10 to 20 by absolute change (30 versus 10). Percentage ranking puts B first (100 versus 30). Matched zero changes are eligible. Available infinite differences rank above finite magnitudes in absolute ranking. NaNs and unavailable changes never participate.

Equal magnitudes, including opposite signs, both zero signs and infinities, tie by the normalized complete comparison-key order. Input order and displayed rounding cannot change winners. Ranks are consecutive plain unsigned decimal numbers, even with hexadecimal or other numerical formats.

An optional limit N follows ranking and keeps at most N eligible keys, without padding. Eligible keys below the cutoff are omitted. Keys that cannot be ranked still appear after the ranked portion in canonical key order, with an empty rank and rank state one_sided for added/removed keys or unavailable otherwise. The selected result’s change-state columns provide the exact reason. Thus N bounds ranked keys, and total report records can exceed N. With no limit, every rankable key appears. An entirely unrankable dataset still reports every key.

RESULT and N accept leading zeros, reject signs and overflow, and must be positive. N can be as large as 18446744073709551615, without reserving N entries in advance. Ranking and limit occur once in that order. Limit requires ranking, and neither modifier can follow or interrupt operations. Invalid modifiers or result indexes fail before sources are opened. Every source, key, requested summary and change must finish before cutoff selection and output, so a limit cannot hide a later error or a failure in an unselected result.

Precision and completion

Changes use completed binary80 numerical values. Subtraction, division by the signed baseline and multiplication by exact 100 each round nearest-even, independently. Displayed values are never used as calculation inputs. Current numeric formatting and rounding controls apply to every numerical cell. Finite overflow in any new arithmetic stage refuses with status 77 and names the result and stage. Subnormal differences and exact cancellation are allowed. Distinct completed binary80 summaries have a nonzero relative difference of at least 2^-64, so percentage division and multiplication cannot underflow. Equal summaries produce exact zero. The ordered trace can refuse extreme pairs even when their final mathematical percentage would fit, and it does not promise one correctly rounded evaluation of the full formula. Existing summary operations retain their own numerical limits.

Every requested summary and change, both source scans and input completion must succeed before comparison writes stdout, including headers. Source errors identify before or after and the supplied path. CSV input errors retain the original logical record and its starting physical line. Syntax failures also retain the detected physical line and byte when available. Completion/change failures name the side where applicable, result and key without inventing an input record. Output write or finalization errors prevent success and can leave partial output. Key text, per-key operation state and required statistical samples remain in memory under existing checked resource limits. Comparison streams intake without retaining complete raw records for consolidation. It does not spill samples, approximate summaries or promise a fixed total-memory bound.

Comparison refuses full rows, other modes, --vnlog, --no-strict, explicit filler, collapse delimiter, random seed and --sort-cmd. Malformed commands and conflicting format controls are errors with status 1.

Table health reports

Use health to inspect a table for inconsistent widths, missing values and mixed value types. Inspection leaves the input unchanged.

Inspect a table

Here price contains both a number and text:

printf 'item\tprice\na\t10\nb\toops\n' | fastmash --header-in health examples 1

The report shows the field observations and the finding:

Table health report
⚠ Inspection complete · advisory findings
Data records: 2 · header records: 1 · accepted records: 3
Baseline width: 2 fields (input header)
Maximum data-record width: 2 fields

Field observations (data-record cells):
  field 1, name "item"
    Absent 0 · empty 0 · missing 0
    Integer 0 · other number 0 · text 2
    Expected type: text (inferred)
  field 2, name "price"
    Absent 0 · empty 0 · missing 0
    Integer 1 · other number 0 · text 1
    Expected type: undetermined (observation)

Findings (counts may overlap):
Advisory findings do not fail validation. Violations of supplied rules do.
⚠ Mixed value types [observation, advisory]
  1 field; field 2, name "price"
  examples retained 1, omitted 1
    Check whether this field should contain numbers or text, then supply a type rule if needed.
    accepted record 2, field 2: observed integer; sample "10"; truncated no


Input and reporting
Field separator: tab (\t)
Records end with: newline (\n)
Numeric decimal separator: "."
Input header: yes
Comment filtering: none
Annotated input (--vnlog): no
Record numbers start at 1, include the input header and exclude filtered records.
Example limit: 1 per finding
Sample byte limit: 128 raw bytes

The finding is advisory, so inspection succeeds. If you know that price must be numeric, supply that rule and request validation:

printf 'item\tprice\na\t10\nb\toops\n' | fastmash --header-in health type price number validate

Validation returns status 1 for the declared violation. Inferred findings alone do not make validation fail. required FIELD checks that a value is present, and nonmissing FIELD also rejects NA-like missing indicators.

Use examples N to choose the number of examples per finding, or examples 0 to omit them. For script-friendly output, add tsv and use the versioned report schema below. Health accepts ordinary text input, not quoted CSV.

Report contents

The table health report states the data-record, header-record and accepted-record counts, effective input settings, numeric decimal separator, baseline width, maximum data-record width and example limits. Readable reports explain each finding and suggest what to check. Empty and header-only reports state that there are no data-record values to check. width N supplies the baseline and checks both data records and an existing input header. N can be zero. Width does not declare individual fields. Without a supplied width, an input header supplies labels and the baseline rather than data values. Without a header, the baseline is the most frequent data-record width across the complete input, choosing the smaller width on a tie. With neither a supplied width, data nor a header, the baseline is unavailable.

Record width mismatch counts data records that differ from the selected baseline. For a supplied width it is a declared violation. Otherwise it is advisory. Header width mismatch separately counts an input header disagreeing with a supplied width, including when there are no data records. Each empty header position produces a Blank header names finding. Each position sharing a nonempty exact-byte label produces a Repeated header names finding. Both findings are advisory. A complete report succeeds even when findings exist. Record and header-field counts and cell counts can overlap. Adding them does not give a count of distinct bad records.

observation describes what the input contains, inferred marks a pattern found in the input, and declared marks a rule you supplied.

Missing values and lexical types

Every field appearing in the input header, a data record or a positional declaration has absent_values, empty_values, missing_values, integer_values, other_number_values and text_values counters, including zero counts. The six categories partition the data-record count for each tracked field. Readable reports use ordinary labels such as absent and Expected type. Readable finding titles also use ordinary words. TSV output keeps the identifiers documented in its schema. An absent field is a position beyond a record’s width. An empty field is present with zero-length bytes. missing_values counts exact whole-field NA, N/A and NaN tokens in any letter case, without trimming. Whitespace-only values and tokens with extra characters are separate values.

An integer encoding allows the numeric grammar’s leading ASCII whitespace, an optional sign and decimal digits consuming the whole field. Fractional, exponent and hexadecimal encodings are other numbers even when their mathematical value would be integral. Numbers must consume the whole field under the effective locale’s numeric grammar, including recognized infinity and NaN encodings such as -inf and nan(payload). A numeric prefix followed by extra bytes, trailing whitespace or an embedded NUL is text. Classification uses syntax, without conversion, rounding or arithmetic magnitude limits.

Absent, empty and candidate-missing values do not participate in inference. If numeric values outnumber text, the inferred health expectation is integer when all numbers have integer encodings, otherwise number. If text outnumbers numbers, it is text. A tie or no eligible values leaves it undetermined. The report labels these expectations as inferred and shows their counts. Undetermined is an observation. These guesses cannot establish the intended meaning of an identifier or distinguish a legitimate code from a missing token.

Positive counters produce Absent fields, Empty field values and Missing value indicators findings in that order, each counting data-record cells at its field position. These are observations: an NA-like token can be a legitimate code. Inspection does not change values or remove records. The input header is excluded from cell counts, and header-only input has zero observations. Fields appearing late include earlier absent positions. Positions outside the header have an empty label.

Mixed value types follows the missingness findings and counts one field whenever numeric and text encodings coexist. Type mismatch follows it and counts text cells incompatible with an inferred integer or number expectation. Inferred text does not reject numbers, and mixed integer/fractional numbers can share a number expectation. Both findings remain advisory, including an inconclusive mixed-type observation when numeric and text counts tie.

Supplied rules and validation

type FIELD integer, type FIELD number and type FIELD text supply a known health expectation, replacing inference for that field. Integer and number use the lexical rules above. Text permits arbitrary field bytes. Type checks exempt absent, empty and candidate-missing values. A supplied integer rule rejects other numeric encodings as well as text, even when their mathematical value is integral. Mismatches are declared Type mismatch violations. The field’s inferred mismatch is suppressed, while its counters and Mixed value types observation remain visible.

required FIELD rejects absent and empty cells. nonmissing FIELD additionally rejects candidate-missing indicators and works without required. When both are supplied, nonmissing is the one effective presence rule. One declared Presence rule not met finding per field counts failed cells without duplicating the rules. A literal NA code can pass required/text checks and fail nonmissing.

FIELD follows the existing single-selector spelling, without lists, ranges or pairs. In ordinary input, bare positive decimal integers are 1-based positions. Identifiers select exact input-header names. Existing backslash escaping permits special bytes and numeric names, such as \01 for a header named 01. Under vnlog, numeric selectors remain names and never fall back to positions. Named rules require an input header and a unique full-byte match. Missing and duplicated names are command errors before data processing or report output. Ordinary positions remain usable with duplicated headers. Names and positions resolving to the same field share their declarations and conflict checks. Never-observed positional declarations are tracked without inventing intervening fields or allocating through the largest declared position.

All controls may occur in any order. Identical declarations are idempotent. Conflicting types, widths or example limits are command errors. Repeated validate switches are harmless. validate requires at least one supplied type, presence or width rule. It emits the same complete report as ordinary inspection and returns status 1 for declared violations, otherwise 0. Observations and inferred suspicions alone do not fail validation. Empty input has zero data records. Header-only input has no per-cell or data-width violations, while a supplied width still checks its header. There is no minimum-record rule.

Examples and locations

examples N retains the earliest N examples per finding, with three by default. Mixed types reserve the earliest numeric and earliest text witnesses when N is at least two, then fill remaining slots with the earliest unused eligible cells. With N equal to one, they retain the earliest eligible cell. Examples are printed in input order. Mixed-type omissions count eligible numeric/text cells minus retained examples, separately from the finding’s count of one field. N is a nonnegative decimal integer. Zero suppresses examples. Repeating the same limit is harmless, while conflicting limits are command errors. Each sample retains at most 128 raw bytes. The report discloses retained and omitted examples and marks truncated samples. Header labels remain complete metadata.

Locations are 1-based accepted-record ordinals, including the input header and excluding filtered comments and annotations. They are not physical line numbers. Header findings include their field position. Width samples contain the raw record without its terminator. Header-width samples preserve the raw accepted header, including vnlog’s prefix and trailing blanks, within the same byte limit. Field observations use their accepted-record ordinal and field position. Absent and empty samples contain no bytes and state which condition was observed. Backslash, TAB, LF, CR and NUL are escaped as \\, \t, \n, \r and \0. Other bytes outside printable ASCII use \xHH with uppercase hexadecimal digits. This keeps binary bytes visible. For ordinary text input, select a field shown with byte escapes by its field number or original header text. With --vnlog, use the original header name.

Supported input and controls

Input follows the existing record intake and field separation: TAB by default, literal -t separators, -W whitespace, -z NUL terminators, -C comment filtering, input headers and vnlog. -H/--headers contributes its input-header meaning. The report supplies its own headings. Empty records can have zero fields, trailing empty fields are retained, and a final unterminated record is accepted. Pipes and regular files produce the same report for the same input.

Sorting, grouping, full rows, numerical formatting, missing-value removal, standalone output headers, result names and result delimiters are calculation controls and fail in health. Quoted CSV input and output remain unavailable. Unknown or incomplete controls are command errors.

TSV for scripts

tsv selects the same collected report in schema version 1. Readable output is the default. The switch can occur anywhere among health controls, and repeating it is harmless. Validation emits identical report bytes with or without validate. Declared violations determine its exit status. TSV output uses the same schema for terminals, files and pipes.

The mandatory header has 13 columns in this order:

row_kind	category	basis	field_index	field_name	count	count_unit	record_index	expected	observed	sample	sample_truncated	examples_omitted

Report records always use TAB separators and LF endings, including with -z input. Inapplicable cells are empty. Zero counters are 0, integers use unsigned decimal spelling and field positions are 1-based. String cells use the byte escapes described above. Decoding them recovers labels and retained samples without confusing literal escape-looking text with byte escapes. Field names retain the complete header bytes. Positions beyond the header have an empty name.

The report begins with exactly these summary rows, in this order. Cells not listed are empty.

CategoryPopulated cells
schema_versionobserved: 1
data_recordsbasis: observation, count, count_unit: data_records
header_recordsbasis: observation, count, count_unit: header_records
accepted_recordsbasis: observation, count, count_unit: records
baseline_widthbasis: declared or inferred, count, count_unit: fields. All three empty when unavailable
baseline_width_sourceobserved: declared, header, modal or unavailable
maximum_observed_widthbasis: observation, count, count_unit: fields. count empty with no data records
separator_kindobserved: literal or whitespace
separatorobserved: literal separator bytes, empty for whitespace separation
record_terminatorobserved: lf or nul
numeric_decimal_separatorobserved: effective locale’s decimal-separator byte
input_headerobserved: yes or no for the selected setting
comment_filterobserved: none, comments or vnlog
vnlogobserved: yes or no
record_numberingobserved: accepted_records_1based
example_limitbasis: observation, count, count_unit: examples
sample_byte_limitbasis: observation, count: 128, count_unit: bytes

Next, each tracked field in ascending position has six field rows in order: absent_values, empty_values, missing_values, integer_values, other_number_values and text_values. Each repeats field_index and field_name, with basis observation, count and count_unit cells. These counters partition the data records exactly, including zeros. A type_expectation row follows with basis declared or inferred and expected integer, number or text. Undetermined instead has basis observation and expected undetermined. A supplied presence rule adds one presence_requirement row with basis declared and expected required or nonmissing.

Positive finding rows follow in this category order, then basis order observation/inferred/declared, then ascending field position:

CategoryBasiscount_unitexpected
header_width_mismatchdeclaredheader_recordsdeclared width
width_mismatchinferred or declareddata_recordsbaseline width
blank_headerobservationheader_fieldsempty
duplicate_headerobservationheader_fieldsempty
absent_fieldobservationcellsempty
empty_fieldobservationcellsempty
missing_indicatorobservationcellsempty
mixed_typesobservationfieldsempty
type_mismatchinferred or declaredcellsinteger or number
presence_mismatchdeclaredcellsrequired or nonmissing

Finding rows populate basis, applicable field identifiers, count, count_unit, expected and examples_omitted. Their record_index, observed, sample and sample_truncated are empty. Width findings have no field identifiers. Do not add overlapping findings into a count of distinct bad records.

Each finding is immediately followed by its retained example rows in input order. They repeat category, basis, applicable field identifiers and expected, then populate record_index, observed, sample and sample_truncated (yes or no). Their count, count_unit and examples_omitted are empty. Width observed values are field counts, with a raw-record sample. Header observations use blank or duplicate, with a header-field sample. Field and presence observations use absent, empty or missing_indicator. Absent/empty samples are empty. Mixed/type examples use integer, other_number or text. Samples retain at most 128 raw bytes, independently of the full scan and labels. Mixed-type omissions count eligible cells rather than the finding’s one field. No other row categories occur in schema version 1. Future schema changes must identify their version.

Field counters, width summaries, header bytes and bounded examples remain in memory. Storage can grow with field count, distinct widths and the chosen example limit. There is no fixed memory guarantee or spill backend. Checked allocation or counter capacity failures refuse the command before report emission. A read failure prevents a report, and a failed report write returns failure. No numerical arithmetic is performed during inspection.

Per-row operations and modes

Per-row operations

Per-row operations produce one output record for each input record. They can be combined with each other, but not with summary operations in the same command.

Fields

OperationResult
cut FIELDS, echo FIELDSPrint the selected fields of each record
$ printf 'a\tb\tc\n' | fastmash cut 3,1
Per-row results
────────────────────────────────
  cut(field-3)  cut(field-1)
             c             a
✔ Completed

Numbers

OperationResult
roundNearest integer, halves away from zero
floor, ceilRound down, round up
truncRound toward zero
fracFractional part, keeping the sign
bin[:WIDTH]Lower bound of the bucket of width WIDTH (default 100) containing the value
strbin[:COUNT]Hash the field’s bytes into a bucket number from 0 to COUNT−1 (default 10)
getnum[:TYPE]Extract the first number found in a text field

getnum types: n digits, i signed integer, p digits and dots (the default), d signed digits and dots, h hexadecimal, o octal. Text with no number gives 0.

$ printf 'item-42.5kg\n' | fastmash getnum 1
Per-row results
────────────────────────────────
  getnum(field-1)
             42.5
✔ Completed
$ printf '17\n183\n' | fastmash bin:50 1
Per-row results
────────────────────────────────
  bin(field-1)
             0
           150
✔ Completed

Text

OperationResult
base64, debase64Encode or decode the field as Base64
md5, sha1, sha224, sha256, sha384, sha512Lowercase hexadecimal checksum of the field’s bytes
dirname, basenameDirectory part, or last component, of a path
extname, barenameFile extension, or file name without it

MD5 and SHA-1 are provided for compatibility and checksums, not for security.

Path operations work on the text alone. They don’t touch the filesystem.

Field modes

ModeEffect
reverseReverse the order of fields in each record
noop, nopRead the input and print nothing (useful for validation). With --full, print each record
$ printf 'a\tb\tc\n' | fastmash reverse
c	b	a

Table modes

ModeEffect
transposeSwap rows and columns
check [N lines] [N fields]Verify every record has the same number of fields, and optionally the expected counts
rmdup FIELD, dedup FIELDKeep the first record for each distinct value of one key field
health [CONTROLS]Inspect structural issues, missing values and lexical types, optionally validating supplied rules
$ printf 'a\tb\n1\t2\n' | fastmash transpose
a	1
b	2
$ printf 'x\t1\ny\t2\nx\t3\n' | fastmash rmdup 1
x	1
y	2

transpose and reverse require every record to have the same number of fields. With --no-strict they accept ragged input, and transpose fills missing cells with the --filler text (default N/A).

check prints a short confirmation and exits with status 0 when the table is consistent, or reports the first inconsistent record and exits with status 1.

Table health reports inspect the complete accepted input, with readable or schema-version-1 TSV output and bounded examples. Inferred findings are advisory. validate fails only for violations of supplied type, presence or width rules. Health accepts ordinary text input, not quoted CSV.

Selecting complete records

top:N FIELD and bottom:N FIELD keep the highest or lowest N records by one numeric field. They copy every field in rank order, with earlier input records winning ties. Add -g for a separate cutoff per group and -s for interleaved keys. Text and quoted CSV, headers and sorted spill are supported. See Highest and lowest records.

Comparing two datasets

compare BEFORE AFTER OPERATION FIELDS summarizes two raw datasets and reports before, after, signed difference and percentage for each result. -g aligns keys globally even when they are interleaved, and named fields bind independently to each input header. Optional ranking highlights the largest changes while keeping added, removed and unavailable keys visible. See Dataset comparison for report columns and limits.

Numbers, output and locales

Use quoted CSV to encode complete output fields, and --result-name=INDEX:NAME with an output header to replace generated labels.

Reading numbers

Numerical operations accept decimal numbers (42, -3.5, 1e-9), hexadecimal floating-point (0x1.8p3), inf and nan. Leading blanks are accepted. Anything else, including trailing blanks or an empty field, is an error that names the line and field.

Printing numbers

By default, results are printed with up to 14 significant digits, like printf "%.14g":

$ printf '1\n2\n' | fastmash mean 1 geomean 1
Summary
────────────────────────────────
  mean(field-1)  geomean(field-1)
            1.5   1.4142135623731
✔ Completed

Two options change how results are printed. Neither changes how they are computed.

OptionEffect
-R N, --round=NPrint N digits after the decimal point (1 to 50)
--format=FMTUse a printf-style format with one %e, %f, %g or %a conversion

Use --format=%.0f for no decimal places. The round operation instead rounds each selected input value to an integer. It does not set a result’s display format.

$ printf '1\n2\n' | fastmash -R 3 mean 1
Summary
────────────────────────────────
  mean(field-1)
          1.500
✔ Completed
$ printf '1234567\n' | fastmash --format "%'.2f" sum 1
Summary
────────────────────────────────
  sum(field-1)
    1234567.00
✔ Completed

The ' flag groups digits with the locale’s thousands separator and grouping: 1,234,567.00 in en_US.UTF-8, 1.234.567,00 in de_DE.UTF-8, 12,34,567.00 in as_IN.UTF-8. In C it does nothing.

Limits

--format strings longer than 99 bytes are an error (status 1), as in GNU datamash. So are column and operation names in a command longer than 511 bytes and, when the field delimiter could continue a number (-t ., -t e or a digit, for example), numeric fields longer than 511 bytes. Otherwise numbers and results have no fixed length limit.

Precision

Fastmash computes with 80-bit extended precision (a 64-bit significand, about 19 significant decimal digits), the same precision GNU datamash uses on x86-64 Linux. Sums are accumulated in input order.

Portable results

Fastmash implements its arithmetic in software, including its logarithms and exponentials. As a result, one Fastmash version gives identical results on every supported machine, independent of the CPU model or the system’s math library.

This also means Fastmash’s output can occasionally differ from a GNU datamash build in the last printed digit, or in the sign of a nan. See Differences from GNU datamash.

The rules for reading, computing and printing numbers are versioned together as a numerical profile. A Fastmash release that changes any numerical output says so in its changelog.

The current profile is portable-binary80-v4. It retains v3 arithmetic, conversion and random-number generation, and uses ICU4X 2.4’s Unicode 18 and CLDR 49 collation. Changed group ordering can assign the shared seeded random draws to different groups, so rand may select another number even with the same seed. For example, with four values 1 through 4 in each of the groups U+10EFB and U+10EC9, LC_ALL=en_US.UTF-8 fastmash -S0 -sg1 rand 2 selected 3 for U+10EFB and 4 for U+10EC9 under v3. V4 puts U+10EC9 first and selects 3 for it and 4 for U+10EFB.

Locales

The locale decides how numbers are written and the order of sorted keys. Fastmash has the rules of the glibc locales built in, taken from glibc 2.43, so it does not need locale data installed on the system.

Numbers. In C, POSIX and C.UTF-8, and in the UTF-8 locales glibc provides (such as en_US.UTF-8, fr_FR.UTF-8, nb_NO.UTF-8 or hi_IN.UTF-8), numbers are read and printed with that locale’s decimal separator, and the ' format flag uses its thousands separator and grouping. An @ modifier glibc has a locale for, such as sr_RS@latin or de_DE@euro, uses that locale’s rules, which can differ from its base’s. Glibc drops any other modifier, and so does Fastmash. A character set other than UTF-8 (a .ISO-8859-1 codeset, or a name such as de_DE whose default is Latin-1) keeps the locale’s rules where its separators are ASCII, and refuses numbers where they are not (fr_FR’s thousands separator is U+202F). A name glibc has no locale for, such as xx_YY.UTF-8, behaves as C, as in GNU datamash. The one glibc locale whose decimal separator is not a single byte, ps_AF, is not supported. Thousands separators are never accepted in input, as in GNU datamash.

Sorted keys.

LocaleSorted key order
C, POSIX, C.UTF-8Byte order
194 glibc locales in 123 languages, such as en_US, de_DE, fr_FR, es_ES, it_IT, pt_BR, nl_NL, pl_PL, ro_RO, hr_HR, ru_RU and he_ILAlphabetical, as GNU sort under glibc (Unicode collation for letters and digits, glibc’s table for punctuation, symbols and spaces)
Other locales, such as cs_CZ, da_DK, el_GR, fi_FI, hu_HU, ja_JP, nb_NO, sv_SE, tr_TR, uk_UA and the Chinese localesSorting is refused

A language locale is supported for sorting when Fastmash’s order of its alphabet (its standard and auxiliary letters in the Unicode CLDR data, alone, in pairs and in each case, and digits) matches GNU sort under glibc exactly. fastmash --help lists the languages. Punctuation, symbols and spaces were checked the same way in every one of them and order as in GNU sort, except next to digits in some keys (12-A and 1-2A) and for vulgar fractions such as ¼. See Differences.

Fastmash reads LC_NUMERIC (for numbers), LC_COLLATE (for sorting) and LC_CTYPE (for quotation marks and -i) from the first non-empty value of LC_ALL, that variable, and LANG, falling back to C.

With an unsupported locale, commands that neither read nor print numbers (transpose, cut, first, unique, checksums and the like, and count, countunique and strbin unless --format or --round is given) work normally. Commands that do are refused with exit status 77 rather than risk reading numbers with the wrong decimal separator, and the message tells you which setting to change:

fastmash: unsupported numeric locale ‘ps_AF.UTF-8’ from LANG
hint: set LC_NUMERIC=C.UTF-8, or a UTF-8 locale whose decimal separator is one byte

Two situations make GNU datamash differ, because it uses the installed locale data instead:

  • If the locale is not installed, GNU datamash (and every other C program) falls back to C for everything, while Fastmash still uses the locale’s rules. Check with locale -a.
  • A different glibc version can have slightly different data: for example, glibc changed fr_FR’s thousands separator to a narrow no-break space. This only affects the ' format flag and sorting.

For scripts, setting LC_ALL=C.UTF-8 gives the same results everywhere.

Diagnostics are always in English.

Further reference

Terminal presentation covers the terminal display and its controls.

Terminal presentation

Fastmash makes terminal results easier to scan with purple headings, blue result values, green completion marks, amber advisories and red failures. Secondary metadata uses gray. All styles use your terminal’s default background.

Ordinary calculations and Top-N selection show labeled, aligned fields and a ✔ Completed footer after successful processing. Headers appear even without --header-out. Automatic headers use positional labels such as sum(field-1). Use -H or --header-out to request labels from an input header. Results still stream as they become available. Alignment uses at most 32 columns per field, while labels and values are always printed in full. A failure can leave earlier results visible, followed by a diagnostic. It does not receive a completion mark. Numerical values and their formatting are unchanged.

Readable table-health reports put the outcome and findings ahead of input metadata. ⚠ identifies advisory findings, and ✗ identifies violations of supplied expectations. Successful validation uses ✔ and explicitly refers to the supplied expectations. Inferred findings remain advisory even when validation passes. Field counts, examples, omissions and interpretation details remain in the report.

Files, pipes and explicit formats

Output sent to a file or another program keeps its existing plain format, including readable health reports. It gains no headings, padding, symbols or color codes, even with --color=always. Examples that write a report file show its data format.

Explicit CSV output, NUL-terminated output, an explicit output delimiter, vnlog, crosstab, dataset comparison, field and table transformations, health TSV and version output retain their data formats on terminals too. This lets you inspect those formats directly without a presentation layer changing their structure.

Controls

fastmash --color=auto sum 1
fastmash --no-color sum 1
fastmash --color=always --help

auto is the default. Each destination is checked separately. Redirecting stdout does not disable colored diagnostics on a terminal. TERM=dumb, a nonempty NO_COLOR, or failed terminal detection disables automatic styling. An empty NO_COLOR permits it. never and --no-color disable the terminal presentation, including bold, added headers and completion marks.

always overrides environment suppression on a terminal. It can also force color in redirected help and diagnostics, but never decorates redirected results or health reports. Diagnostics style only their program prefix and failure mark, leaving the explanation, hints and input bytes in the default foreground.

Repeated controls follow the last one reached by ordinary option scanning. -- ends scanning, and POSIXLY_CORRECT stops it at the first operand. Help and version stop scanning immediately, so put color controls before them.

Errors, exit statuses and resources

Exit statuses

StatusMeaning
0Success
1Error: a problem with the command or its input, such as an unknown option, a non-numeric value, a missing field or a write failure
77Refusal: Fastmash deliberately declined to produce a result, because a feature or locale is unsupported or a checked resource limit was reached
70Internal failure: Fastmash detected a violation of its numerical invariants. Please report it

When one failure follows another, such as a failure to write the output after an error, the status is the more serious one: 70 over 77 over 1. If a diagnostic itself cannot be written, the status is 1. Internal numerical or table-identity failures indicate a bug: please report them. The legacy “internal error: input field too long” diagnostic instead identifies the numeric field-length limit. SIGABRT can indicate a bug or an allocation failure that the process could not report as a refusal.

Diagnostics go to standard error, in the form:

fastmash: invalid numeric value in line 3 field 2: 'n/a'

Some are followed by a hint: line suggesting a fix.

Find and fix the cause

The diagnostic identifies the first problem encountered. Check the named field, record, source file or option before rerunning the command.

Message or situationWhat to check
invalid numeric valueConfirm that the selected field contains numbers and that the field separator is correct. If the first record names the fields, add --header-in
A requested field is missingField numbers start at 1. Tab is the default separator. Use -W for runs of spaces and tabs, -t, for literal commas, or --csv for quoted CSV
A header name is not foundNames are exact and case-sensitive. Check the spelling, separator and whether the first record really is a header
invalid CSVUse double quotes around a complete field. Write a quote inside it as two double quotes. The reported record and physical line locate the problem
A CSV option conflicts with another optionRemove text-input controls such as -t or -W when reading CSV. CSV output chooses its own comma separator, so omit --output-delimiter
A weighted mean has no positive retained weightAt least one retained value/weight pair must have a positive weight. Check zero weights and pairs omitted by --narm
sort temporary I/O errorCheck that the temporary directory exists, is writable and has free space. Choose another writable directory if needed
Memory allocation failedCheck available memory and process limits. Sorting can spill to disk, but retained samples and complete tables stay in memory
A read or write failedCheck the input/output paths, access permissions and free disk space. A closed or incorrectly opened standard stream must be fixed by the shell or program launching Fastmash

--narm skips only whole-field NA, N/A and NaN tokens, ignoring case. It does not skip arbitrary text, empty fields or absent fields. --round and --format change display only and cannot repair a numerical-range refusal.

The warning about -f/--full with “non-linewise operations” concerns aggregate operations such as sum. It does not stop the command or change its exit status. Omit --full if you only need the calculated results. Per-row operations such as round do not produce this warning.

Always check the exit status

Fastmash writes results progressively through a small output buffer, so a command that fails part-way may already have printed earlier groups. In scripts, check the exit status rather than whether output appeared. In Bash pipelines, enable set -o pipefail so an earlier command’s failure isn’t hidden:

bash -c '
set -o pipefail
if ! fastmash -s -g 1 sum 2 < data.tsv > totals.tsv; then
  echo "summary failed" >&2
  exit 1
fi
'

Dataset comparison completes both input scans and all calculations before writing its report, including the header. Output write failures can still leave partial bytes. Table health likewise completes inspection before report emission; health ... validate can emit a complete report and return 1 for declared violations. Advisory findings alone do not make validation fail.

PowerShell 7.4+

Read $LASTEXITCODE immediately after the native command. It uses the statuses above, including 1 for an error and 77 for a refusal. To return that status from a .ps1 script, using the quick start’s CSV file:

cmd /d /v:off /c 'fastmash --csv -H sum amount < "sales.csv"' > totals.csv
exit $LASTEXITCODE

The script returns 0 on success, or the failure status if the command fails. Treat totals.csv as complete only after success. In a native pipeline, $LASTEXITCODE reports the last native program’s status. Run stages separately and check each status when you need to detect a failure in an earlier stage.

Why refusals exist

Fastmash prefers a clear refusal to a doubtful answer. Examples:

  • A non-integer percentile such as perc:2.5. GNU datamash 1.9 reads an unset parameter for it and reports a misleading error.
  • Multiple key fields for rmdup, which GNU datamash 1.9 aborts on.
  • An unsupported locale, where numbers might be read with the wrong decimal separator.
  • A calculation that would need more memory than is available.

A refusal is not a claim that GNU datamash would reject the same input.

Memory

Fastmash has no fixed limits on record length, field count, number of operations or number of values. (The few fixed limits that remain are GNU datamash’s own, on --format strings, names in a command and, in one narrow case, numeric fields; see Numbers.) Data storage grows with checked allocations. If memory runs out, Fastmash refuses with status 77 where it can. Process initialization, including arguments and configuration, uses standard Rust allocation behavior and can terminate before reporting a refusal. The operating system may also stop a process that exhausts memory. In those cases, the shell or job runner reports the process’s termination status.

What grows with the input:

  • Operations that need every value (quantiles, dispersion, paired statistics, unique, collapse) keep the values of the current group.
  • rmdup, transpose and crosstab keep their tables in memory.
  • Top-N selection keeps at most N candidates for the active group or dataset; N limits record count, not bytes.
  • Dataset comparison keeps keys, per-key calculation state and required samples in memory; these do not spill.
  • Table health keeps field counters, widths, header labels and bounded examples in memory, with no fixed total-memory guarantee.
  • -s keeps a sort buffer, spilling to disk beyond the chunk target. Sorted numerical jobs keep the original records until they are processed.

Address-space limits

Some systems limit a process’s address space rather than its memory, for example ulimit -v or a batch scheduler’s virtual-memory limit such as h_vmem. The static Linux download uses musl and needs no glibc allocator settings. If a sorted job refuses under the limit, try OMP_NUM_THREADS=1 to reduce worker memory (see Large inputs).

For glibc source builds, the C library’s memory allocator gives each thread that allocates an arena of its own, which reserves 64 MiB of address space, more as it grows, and sorts in language locales, like the heavier calculations in the C locales, run on up to eight threads. Under such a limit, Fastmash therefore keeps the allocator to one arena for each 512 MiB of the limit, from one (below 1 GiB) to eight. Threads that share an arena wait for each other, so a sort that would also fit without the cap can take somewhat longer. A number of arenas that GLIBC_TUNABLES (glibc.malloc.arena_max) or MALLOC_ARENA_MAX sets is used instead, higher or lower. If a sorted job still refuses under the limit:

  • GLIBC_TUNABLES=glibc.malloc.arena_max=1 keeps the allocator to one arena at a limit of 1 GiB or more too;
  • OMP_NUM_THREADS=1 sorts on one thread (see Large inputs).

Disk

On Windows, sorting uses the system temporary directory selected by Windows. On Linux and macOS, sorting large inputs with -s writes temporary data to TMPDIR (default /tmp), using anonymous files that the operating system removes automatically. Where the filesystem has no anonymous files (O_TMPFILE), such as NFS or a container’s /tmp on Linux before 6.10, Fastmash creates private files with random names and removes their names at once, so they also disappear when Fastmash exits, including when it is interrupted. A sort that has to spill to disk without a usable TMPDIR stops with “sort temporary I/O error” (status 1).

Interruption

An interrupted command can leave partial output. Treat output as complete only when the exit status is 0.

Differences from GNU datamash

Fastmash aims to behave exactly like GNU datamash 1.9. This page lists the known intentional differences and explicit extensions. Other differences in GNU-compatible commands are bugs: please report it.

Comparisons were made against GNU datamash 1.9 on x86-64 Linux. Other GNU versions and builds can differ from each other as well.

Fastmash extensions

These features extend the command language through explicit options, operations or modes. Ordinary GNU-compatible commands keep the compatibility rules on this page. The linked guides describe each extension’s supported combinations and limits. No comparative workflow speed or memory claim is implied.

  • Quoted CSV and custom result names add explicit quoted input/output formats and supplied labels for calculation results.
  • Table health inspects structural and missing-value observations, inferred types and supplied expectations, with readable and TSV reports and optional validation.
  • Top-N selection returns the highest or lowest complete records by one numeric field, retaining original input order for ties.
  • Weighted mean adds wmean VALUE:WEIGHT with finite, nonnegative contribution weights and checked numerical range limits.
  • Dataset comparison summarizes two raw datasets and reports their numerical changes, with complete comparison keys and optional ranking.
  • Terminal presentation reference describes interactive output behavior.

Numbers

AreaGNU datamash 1.9Fastmash
ArithmeticThe CPU’s x87 unit and the C math librarySoftware arithmetic, including log and exp. Identical on every supported machine
Last digitsDepend on the build and the math libraryCan differ from GNU in the last printed digit for some transcendental results
NaN from invalid arithmeticPrints -nan on x86-64Always positive nan
NaN between two NaN inputsChosen by the x87 unitThe one with the larger payload, positive on a tie
round, floor, ceil, trunc, frac, bin of -nanGives 0Keeps -nan
log and exp at extreme argumentsComputed by the C libraryA few boundary cases that cannot be settled within fixed limits are refused (status 77)

Safety

AreaGNU datamash 1.9Fastmash
perc:100Can read past the end of the valuesReturns the largest value
Non-integer percentile, such as perc:2.5Reads an unset parameter. On x86-64 builds reports the misleading “invalid percentile value 0”Refused, status 77, with a clear message
rmdup with several key fieldsAborts on an assertionRefused, status 77
Empty operation name, a pair whose second field is a range (1:2-3), strbin:2.0, numeric getnum typeAssertion failures or undefined behaviorerror, status 1, with a message
The long --seed optionMishandles its argumentRequires a value, like -S
crosstab labels longer than 511 bytesTruncatedKept in full
Unseeded rand when the kernel’s random source failsPrints Error N and continuesRefused, status 77
Running out of memory“memory exhausted”, status 1Refused, status 77, naming what could not be stored. For an input record, “record memory allocation failed”

Features

AreaFastmash
--sort-cmdUnsupported. Sort the input separately
Explicit line modeUnsupported. Write the per-row operation directly
Hidden test options (---print-inf and others)Unsupported
LocalesBundled rules, with the availability and ordering differences below
Quotation marks in messagesCurly quotes in UTF-8 locales. Messages are always in English
-iASCII case folding, excluding column names and duplicate-removal keys. Refused under the Turkic character types az_AZ, crh_UA, ku_TR, tr_CY, tr_TR and tt_RU@iqtelif, where glibc does not fold i and I
Unwritable TMPDIRFails only when sorting needs to spill, with status 1. Spill thresholds differ from GNU sort
Size limitsGrowable records, operation lists and samples, with the documented compatibility limits

Temporary storage

Fastmash accounts for available memory and cgroup limits when sizing its sort buffer. GNU sort sizes from input files and host memory, ignoring cgroup limits, and spills sooner on piped input. Hash grouping can avoid sorting records. See grouping and sorting for resource controls and the architecture for buffer sizing.

Locale and sorting differences

Locale availability

Fastmash bundles glibc 2.43 numeric rules. It supports C, POSIX, C.UTF-8 and every glibc locale with a one-byte decimal separator, including known @ modifiers. Non-UTF-8 character sets work where their separators are ASCII. Other numeric separators are refused when a command needs numbers.

Sorting supports the C locales and 194 verified language locales. Unverified sorting locales are refused. Unknown locale names behave as C in both programs. For a known locale missing on the host, GNU datamash falls back to C, while Fastmash uses its bundled rules.

Key order

In supported language locales, letters and digits follow Unicode collation. Punctuation, symbols and spaces use glibc’s ordering table, including characters ignored until otherwise equal keys need a tie-breaker. For example, a-1 sorts before a_1 before a1, and aa before a+b.

In Spanish locales, gl_ES and pl_PL, spaces sort before letters and digits, as in GNU sort. This puts San José before Sanabria.

Punctuation beside digits

glibc skips a character at the accent level in some digit-and-punctuation runs ending before a letter. These keys can therefore appear in different orders:

ProgramOrder
GNU sort12 A, 12-A, 12A, 1-2A
Fastmash1-2A, 12 A, 12-A, 12A
GNU sortfile1.txt, File1.txt, file(1).txt
Fastmashfile(1).txt, file1.txt, File1.txt

A few symbols that glibc weighs rather than ignores also differ, including ¼ versus 1/4, and the numero sign in tk_TM.

Letters and combining marks

Letters outside the language’s alphabet can differ in order. These include Hangul syllables, Georgian capitals, some Arabic letters, CJK Extension A/B, compatibility ideographs and characters newer than glibc’s table.

A precomposed letter sorts after its base plus combining marks, as in GNU: e with U+0301 precedes é, and Е with U+0308 precedes Ё. glibc makes exceptions for some letters with two marks, including Vietnamese ổ and ǖ, putting them first at the end of a key or before a letter. Fastmash keeps them after the decomposed spelling. They also differ where glibc uses a second mark before an earlier difference, as in éǖ.

Combining marks written outside Unicode order, such as a with U+0301 then U+0323, follow Unicode collation in Fastmash rather than glibc’s order.

Tied spellings and grouping

Where glibc treats two spellings as the same letter, the next key decides. Fully tied keys stay in input order, with one group per run of a spelling. Examples include и plus a combining breve versus й, L plus a middle dot versus Ŀ, and some Arabic, Indic, Thai, Lao and Tibetan vowel forms.

In cy_GB, sq_AL and sq_MK, ll followed by a middle dot sorts as l and ŀ in Fastmash. glibc instead reads the locale’s ll and then the dot. Such a key ties with lŀ in Fastmash, producing a group per input run, while GNU orders them apart.

When group order differs, seeded rand can select a different number because groups share one random stream.

Invalid sorted keys

Language-locale keys must be valid UTF-8 without NUL bytes. Other keys are refused. With rmdup -s --vnlog, comment records sort with the data, as in GNU datamash, and a comment can become the second header. A comment whose key is invalid UTF-8 is therefore refused too.

Diagnostics

Error messages use the same wording as GNU datamash where it matters to scripts, but some messages differ, and they begin with the program name, fastmash:. Errors for Fastmash extensions include their own field-selection guidance, supported modes and examples of valid option syntax.

Common command errors also include a separate hint: line with valid syntax or a recovery step. The original error wording and exit status are preserved. Top-N selection errors use their own wording and include a ranking example. For byte-order recovery, LC_ALL must be unset before LC_COLLATE=C takes effect. Copy its value to LC_NUMERIC first to keep the same number format.

A few failure paths also differ in detail: for example, a read error in input sorted with -s can give just “read error: Input/output error”, where GNU datamash prints the system sort’s message and “read error (on close)”. When standard input cannot be read at all (opened for writing only), GNU datamash’s -s prints an error from the shell that runs sort but ends with status 0, where Fastmash reports the read error with status 1. And --help or --version into a closed pipe exits with status 77 rather than by SIGPIPE. Scripts should rely on exit statuses, not the exact text of error messages.

Ordinary commands use normal Unix broken-pipe handling. Blocked SIGPIPE and failed signal-mask queries cause a refusal.

With native sorting on Linux and macOS, a command with a named grouping key or sorted rmdup key and no input header reports fastmash: missing input header for named grouping key and exits with status 1, including when comment-only input leaves no header. These failure paths run inside Fastmash on both platforms. A failed input read still reports the read error and exits with status 1. The missing-header diagnostic does not hide it. A command containing wmean still succeeds without output when no input header arrives at clean EOF. OS-owned diagnostic wording is checked separately from portable results.

System requirements

The Linux download is statically linked and runs on both glibc and musl distributions, including Alpine. Follow the install steps.

Platform

RequirementWhy
Linux (x86_64-unknown-linux-musl)Static prebuilt binaries, with no glibc version requirement
Linux (x86_64-unknown-linux-gnu)Also supported for source builds using the system glibc
macOS 15 or later (aarch64-apple-darwin)Supported platform for the native archive, checked on macOS 15 and 26
Windows 11 (x86_64-pc-windows-msvc), PowerShell 7.4+Native archive, with the C runtime included
/proc mounted on LinuxSupplies memory and terminal information. Without it, Fastmash uses conservative sort memory and writes output in 8 KiB blocks, even to a terminal
A writable temporary directoryTemporary data for large sorted jobs

See native Windows installation or macOS installation. Intel macOS, older macOS versions, Windows 10 and ARM64 Windows are outside the supported scope.

Sorting runs inside Fastmash and needs no other program. Its temporary directory must be writable when a job spills to disk.

For commands under an address-space limit (ulimit -v, as some batch schedulers set), see Address-space limits.

See Grouping and sorting for memory use and temporary storage.

Locales

Fastmash has the glibc 2.43 locale rules built in, so you don’t need to install locale data. See Numbers, output and locales.

Verifying a release archive

The installer checks the archive’s SHA-256 checksum. To verify which GitHub workflow built the latest release, use the GitHub CLI to download and verify its archive:

Linux:

gh release download --repo pederbe/fastmash --pattern 'fastmash-*-x86_64-unknown-linux-*.tar.gz' --output fastmash.tar.gz --clobber
gh attestation verify fastmash.tar.gz --repo pederbe/fastmash

macOS:

gh release download --repo pederbe/fastmash --pattern 'fastmash-*-aarch64-apple-darwin.tar.gz' --output fastmash.tar.gz --clobber
gh attestation verify fastmash.tar.gz --repo pederbe/fastmash --signer-workflow pederbe/fastmash/.github/workflows/release.yml

The macOS archive is not Apple-notarized.

Building from source

Use the latest stable Rust and a C compiler (cc, or set CC).

git clone https://github.com/pederbe/fastmash
cd fastmash
rustup target add x86_64-unknown-linux-musl
cargo build --release --locked --target x86_64-unknown-linux-musl
# the program is target/x86_64-unknown-linux-musl/release/fastmash

To build on macOS 15 or later, run this command in Bash:

MACOSX_DEPLOYMENT_TARGET=15.0 cargo build --release --locked --bin fastmash --target aarch64-apple-darwin
# the program is target/aarch64-apple-darwin/release/fastmash

The ordinary Linux cargo build --release --locked command still builds against the system glibc. The macOS build needs Apple’s Command Line Tools. Installing the archive needs no Rust compiler or Command Line Tools.

cargo install and source builds use your own compiler and settings. They omit the prebuilt binaries’ branch-alignment tuning for some Intel processors, so their speed can differ slightly. Keep --locked: it builds with the tested dependency versions.

Benchmarks

Fastmash is faster than GNU datamash on quantiles, grouped summaries and number-heavy tables. These Linux benchmarks compare both programs on an Intel laptop and an AMD desktop, using gene annotations, wine measurements and synthetic tables.

How many times faster Fastmash is than GNU datamash on the core jobs, per host Intel laptop, native Linux AMD desktop, native Linux 0× 1× 2× 3× 4× 5× 6× Exon-count quantiles (RefGene) Exon-count quantiles (RefGene), Intel laptop, native Linux: GNU 111.0 ms, Fastmash 21.5 ms 5.2× Exon-count quantiles (RefGene), AMD desktop, native Linux: GNU 60.0 ms, Fastmash 12.6 ms 4.8× Transcripts per gene (RefGene) Transcripts per gene (RefGene), Intel laptop, native Linux: GNU 110.8 ms, Fastmash 73.0 ms 1.5× Transcripts per gene (RefGene), AMD desktop, native Linux: GNU 57.0 ms, Fastmash 37.9 ms 1.5× Exon statistics per gene (RefGene) Exon statistics per gene (RefGene), Intel laptop, native Linux: GNU 157.8 ms, Fastmash 86.8 ms 1.8× Exon statistics per gene (RefGene), AMD desktop, native Linux: GNU 74.8 ms, Fastmash 45.0 ms 1.7× Many small groups (genes) Many small groups (genes), Intel laptop, native Linux: GNU 159.8 ms, Fastmash 127.1 ms 1.3× Many small groups (genes), AMD desktop, native Linux: GNU 66.1 ms, Fastmash 53.1 ms 1.2× Gene example (datamash manual) Gene example (datamash manual), Intel laptop, native Linux: GNU 7.0 ms, Fastmash 4.2 ms 1.7× Gene example (datamash manual), AMD desktop, native Linux: GNU 3.3 ms, Fastmash 2.1 ms 1.6× Sum and mean, 1M decimals Sum and mean, 1M decimals, Intel laptop, native Linux: GNU 325.4 ms, Fastmash 87.5 ms 3.7× Sum and mean, 1M decimals, AMD desktop, native Linux: GNU 114.8 ms, Fastmash 37.2 ms 3.1× Sum and mean, 100k decimals Sum and mean, 100k decimals, Intel laptop, native Linux: GNU 32.8 ms, Fastmash 9.5 ms 3.5× Sum and mean, 100k decimals, AMD desktop, native Linux: GNU 11.6 ms, Fastmash 3.9 ms 3.0× Grouped decimals, 100k rows Grouped decimals, 100k rows, Intel laptop, native Linux: GNU 68.1 ms, Fastmash 38.7 ms 1.8× Grouped decimals, 100k rows, AMD desktop, native Linux: GNU 27.5 ms, Fastmash 18.0 ms 1.5× One dominant group One dominant group, Intel laptop, native Linux: GNU 21.6 ms, Fastmash 9.1 ms 2.4× One dominant group, AMD desktop, native Linux: GNU 9.3 ms, Fastmash 3.9 ms 2.4× Wine quality by grade Wine quality by grade, Intel laptop, native Linux: GNU 6.0 ms, Fastmash 2.6 ms 2.3× Wine quality by grade, AMD desktop, native Linux: GNU 3.0 ms, Fastmash 1.2 ms 2.5× Large sorted summary Large sorted summary, Intel laptop, native Linux: GNU 547.2 ms, Fastmash 495.1 ms 1.1× Large sorted summary, AMD desktop, native Linux: GNU 204.4 ms, Fastmash 240.2 ms 0.9× Times faster than GNU datamash 1.9, by median elapsed time. The dashed line is the same speed; shorter bars are slower.

Large sorted summaries can be slower than GNU datamash.

Longer bars mean faster runs. The dashed line marks equal speed. See the method and timings for the commands, elapsed times and machines.

The animated demo replays the laptop’s RefGene quartile job. Playback is slowed 20 times so that both runs are visible. Its clocks show the measured times.

Run them yourself

The benchmark kit runs eight of these jobs on your machine with both programs, checks that they agree, and prints a table you can share:

curl -fsSLO https://raw.githubusercontent.com/pederbe/fastmash/main/bench/fastmash-bench.py
python3 fastmash-bench.py

The best benchmark is your own job. Compare both programs on your data:

export LC_ALL=C.UTF-8
time datamash -s -g 1 median 2 < your-data.tsv > /dev/null
time fastmash -s -g 1 median 2 < your-data.tsv > /dev/null

For stable numbers, repeat each command several times, alternate the two programs, and use a quiet machine. hyperfine does this for you:

hyperfine --warmup 2 \
  'datamash -s -g 1 median 2 < your-data.tsv' \
  'fastmash -s -g 1 median 2 < your-data.tsv'

If Fastmash is slower on a job you care about, please tell us: slow real-world jobs are what we optimize next.

Benchmark method

We compare Fastmash with GNU datamash on the same files and machine. The benchmark kit checks that their output matches before timing either program.

Each job runs in two rounds, with one warm-up and seven timed runs per program in each round. Program order alternates, and job order is shuffled between rounds. The tables show the mean of the two median elapsed times in milliseconds. Lower times are better.

Both programs read regular files with a warm file cache, use the C locale and write timed output to /dev/null. Temporary files use disk-backed storage. GNU datamash’s -s commands include the time taken by the system sort.

Machines

HostCPUSystem
LaptopIntel Core i7-8550UCachyOS, native Linux
DesktopAMD Ryzen 7 9800X3DCachyOS, native Linux

Data and commands

The benchmark kit downloads RefGene genome annotations and UCI Wine Quality measurements, and generates synthetic decimal tables from a fixed seed. The remaining jobs exercise small groups, a dominant group and a large sorted summary.

Field numbers in each command refer to its input table. The commands below are identical for both programs.

Job and argumentsIntel laptop, GNU → Fastmash (ms)AMD desktop, GNU → Fastmash (ms)
Exon-count quantiles (RefGene)
q1 9 median 9 q3 9 iqr 9
111.0 → 21.5 (5.16×)60.0 → 12.6 (4.76×)
Transcripts per gene (RefGene)
-s -g 13 count 2 collapse 2
110.8 → 73.0 (1.52×)57.0 → 37.9 (1.50×)
Exon statistics per gene (RefGene)
-s -g 13 count 9 min 9 max 9 mean 9 median 9
157.8 → 86.8 (1.82×)74.8 → 45.0 (1.66×)
Many small groups (genes)
-s -g 1 count 2 collapse 2
159.8 → 127.1 (1.26×)66.1 → 53.1 (1.24×)
Gene example (datamash manual)
-H -s -g 13 count 2
7.0 → 4.2 (1.67×)3.3 → 2.1 (1.57×)
Sum and mean, 1M decimals
-H sum 3 mean 3
325.4 → 87.5 (3.72×)114.8 → 37.2 (3.09×)
Sum and mean, 100k decimals
-H sum 3 mean 3
32.8 → 9.5 (3.45×)11.6 → 3.9 (2.97×)
Grouped decimals, 100k rows
-H -s -g 2 count 3 mean 3 median 3
68.1 → 38.7 (1.76×)27.5 → 18.0 (1.53×)
One dominant group
-s -g 1 count 2 collapse 2
21.6 → 9.1 (2.37×)9.3 → 3.9 (2.38×)
Wine quality by grade
-t ';' -H -s -g 12 count 1 mean 11 median 11
6.0 → 2.6 (2.31×)3.0 → 1.2 (2.50×)
Large sorted summary
-s -g 1 count 2 collapse 2
547.2 → 495.1 (1.11×)204.4 → 240.2 (0.85×)
White wine summary
-t ';' -H mean 9 median 9 min 9 max 9
4.2 → 1.9 (2.21×)1.7 → 0.9 (1.89×)

FAQ

Is Fastmash a drop-in replacement for GNU datamash?

For most commands, yes: it accepts the same operations, options and selectors and aims to produce the same output. Check your scripts against Differences from GNU datamash, especially the locale settings and numerical last digits, and keep datamash installed while you compare.

Is it always faster?

Not yet. The published Linux benchmarks show Fastmash faster on many common jobs and slower on some others. Try it on your own data, and tell us about jobs where it is slower.

Why does Fastmash refuse my locale?

Some locales use sorting rules or numeric separators Fastmash does not support. Set LC_ALL=C.UTF-8 to use decimal points and byte ordering:

LC_ALL=C.UTF-8 fastmash -s -g 1 sum 2 < data.tsv

If you need language-specific ordering or decimal commas, see Locales for supported settings and limitations.

Can it read CSV files?

Yes. --csv-in reads strict quoted CSV, --csv-out encodes output fields, and --csv selects both. Quoted fields can contain commas, quotes and line breaks. Format switches do not infer headers. Aggregates, per-row operations, Top-N selection and dataset comparison support CSV, including their supported grouped workflows. Table health and legacy table/field modes remain ordinary-text only. -t, still means literal comma splitting, as in GNU datamash. See Quoted CSV and result names.

Can I check a table before calculating?

fastmash --header-in health < data.tsv reports inconsistent widths, missing values, mixed lexical types and examples. Add known rules such as type price number nonmissing id width 3 validate to fail on declared violations. Inferred findings stay advisory. A versioned TSV report is available for scripts. See Table health reports.

Can I compare two exports with different column orders?

Yes. With -H, named fields bind independently to each input header. fastmash -H -g category compare before.tsv after.tsv sum revenue reports each key’s before, after, signed difference and percentage, including added and removed keys. Keys consolidate across each whole dataset. They need not be adjacent. See Dataset comparison.

Can I use Fastmash in pipelines?

Yes. Fastmash reads standard input and writes results for the next program. Results written to a file or another program keep their plain data format.

Does sorting need another program?

No. Fastmash handles all sorting internally.

Why exit status 77?

Status 77 means Fastmash deliberately declined to produce a result: an unsupported feature, locale or condition, or a resource limit. It is distinct from status 1, which means the command or the input is wrong. See Errors, exit statuses and resources.

Why do results sometimes differ from GNU datamash in the last digit?

Fastmash computes with the same 80-bit precision, but in software, including its logarithms and exponentials. The same Fastmash version, input and settings give the same numerical results on every supported machine. GNU datamash’s results depend on the hardware and C library. See Portable results.

Does it work on macOS or Windows?

Yes. See the installation guide for native Windows and macOS requirements, downloads and setup. WSL2 remains available for Linux workflows.

How does Fastmash relate to GNU datamash?

Fastmash is an independent reimplementation of GNU datamash’s command language, written in Rust. It contains no GNU code and is not affiliated with the GNU Project. GNU datamash, by Assaf Gordon and contributors, defined the interface that Fastmash follows.

Contributing

Help improve Fastmash by reporting bugs, sharing real workloads, improving documentation or contributing code. The contribution guide covers development setup, checks and pull requests.

Design decisions explain the constraints behind the implementation.

Report security problems through the security policy.

Architecture

The architecture guide maps Fastmash’s crates, command pipeline and design constraints for contributors.

For sorting internals, see the technical sorting explanation. For using grouping and controlling sort resources, read Grouping and sorting.

Testing

The testing guide provides local commands, prerequisites and specialist checks for contributors.

Quick start

Choose checks for your change and use CI for platform coverage unavailable locally.

Website checks

Use the website checks for documentation and site changes.

Native macOS development gate

Follow the native macOS checks for macOS prerequisites and coverage limits.

Locale tools

The locale-tool instructions cover reference generation and qualification.

Glossary

The project glossary defines the vocabulary used in Fastmash’s code, tests, documentation and issues.

About

Fastmash was created by Peder Bergan, who designs, builds and maintains it.

The project started from a practical question: can a tool people use every day in data pipelines be made substantially faster, without asking them to change their commands, and without giving up correctness? Fastmash combines familiar GNU datamash commands with Rust software arithmetic and published performance measurements. Numerical results repeat across supported machines for the same input, settings and Fastmash version.

Acknowledgements

Fastmash follows the interface of GNU datamash, created by Assaf Gordon and maintained by its contributors. Fastmash is an independent project and is not affiliated with or endorsed by the GNU Project.

Contact