Fastmash
Fast command-line statistics for delimited text.
Fastmash summarizes delimited text and quoted CSV, with grouping and statistics for each category. It also reshapes ordinary delimited text. It accepts the command language of GNU datamash, so existing datamash workflows can move over with few or no changes.
It also reads quoted CSV directly, gives your result columns stable names, calculates weighted means and selects the highest or lowest complete records. Inspect a table’s health or compare summaries from two exports with the same field selectors and portable arithmetic.
$ cat readings.tsv
site reading
west 2
east 4
west 6
west 7
$ fastmash -H -s -g site count reading median reading < readings.tsv
Grouped summary
────────────────────────────────
site count(reading) median(reading)
east 1 4
west 3 6
✔ Completed
Install it with one command, or see other ways:
curl -fsSL https://fastmash.io/install.sh | sh
Fastmash is faster than GNU datamash*
About five times faster on RefGene quartiles, with faster grouped summaries and decimal totals too.
* Large sorted summaries can be slower on Linux. All results and the method.
And better in other ways
| Feature | GNU datamash 1.9 | Fastmash |
|---|---|---|
| Results | Depend on the CPU and the C math library | Repeatable across supported machines with the same version, input and settings |
| Locales | Need locale data installed. Silently fall back to C without it | Built in: numbers in 318 locales, sorting in 194 |
perc:100 | Can read past the end of the values | Returns the largest value |
rmdup on several key fields | Crashes on an assertion | A clear refusal |
Malformed parameters such as strbin:2.0 | Misleading errors or undefined behavior | An accurate error message |
crosstab labels over 511 bytes | Truncated | Kept in full |
Fastmash supports over 70 operations and modes with familiar field selectors, grouping and headers. Most scripts can switch by changing one word. Check the documented differences when migrating a workflow. Migrating from datamash · All differences
Get started
- Install Fastmash on Linux, macOS or Windows
- Quick start: a five-minute tour
- Operations: every statistic and transformation
- Quoted CSV and result names: structured input and useful report labels
- Highest and lowest records: keep the records behind the numbers
- Table health and dataset comparison: inspect data and understand changes
Fastmash is open source under the MIT or Apache-2.0 license. Source on GitHub.
Install
Fastmash runs on Linux, macOS 15 and later, and Windows 11. For Windows, use PowerShell 7.4 or later.
Choose a method
Use your system’s package manager when you can. It installs prebuilt programs and brings you later versions with your other updates.
| Your system | Recommended | Updates |
|---|---|---|
| Debian, Ubuntu | apt repository | sudo apt update, then sudo apt install --only-upgrade fastmash |
| Fedora, RHEL and similar | dnf repository | sudo dnf upgrade fastmash |
| macOS, or Linux with Homebrew | Homebrew | brew upgrade fastmash |
| Linux, macOS, or WSL2 | Quick install | Run it again |
| Windows | Native archive | Replace the extracted folder |
All of these install prebuilt programs. If you prefer to compile Fastmash yourself, see Cargo.
Debian and Ubuntu
The signed apt repository provides updates through apt on Linux. Install the tools needed to download and inspect its key:
sudo apt install ca-certificates curl gnupg
curl -fsSLo fastmash.gpg https://packages.fastmash.io/fastmash.gpg
gpg --show-keys --fingerprint fastmash.gpg
Check that the key fingerprint is:
76C1 7E23 B7D3 F8FD ED36 D31A 7BAC 7AF6 644B D290
Then add the repository and install Fastmash:
sudo install -m 644 fastmash.gpg /usr/share/keyrings/fastmash.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/fastmash.gpg] https://packages.fastmash.io/apt stable main" |
sudo tee /etc/apt/sources.list.d/fastmash.list
sudo apt update
sudo apt install fastmash
Later versions arrive through sudo apt update and sudo apt upgrade.
The key is trusted only for this repository.
For Debian 10, Ubuntu 20.04 and newer. Remove it with sudo apt remove fastmash.
Fedora, RHEL and similar
The signed dnf repository provides updates on Linux, with package and repository signature checks enabled:
sudo curl -fsSLo /etc/yum.repos.d/fastmash.repo https://packages.fastmash.io/fastmash.repo
sudo dnf install fastmash
dnf asks to trust the signing key on first use. Before confirming, check that
its fingerprint is 76C1 7E23 B7D3 F8FD ED36 D31A 7BAC 7AF6 644B D290,
the same key used by the apt repository above. Update with
sudo dnf upgrade fastmash.
For Fedora, RHEL 8 and newer. Remove it with sudo dnf remove fastmash.
Homebrew
With Homebrew on macOS 15 or later, or on Linux:
brew install pederbe/tap/fastmash
Homebrew downloads the same tested programs as the Fastmash release archives. Installation needs no Rust compiler and does not rebuild Fastmash.
Update with brew upgrade fastmash and remove it with
brew uninstall pederbe/tap/fastmash.
Quick install
On Linux, macOS or inside WSL2:
curl -fsSL https://fastmash.io/install.sh | sh
Downloads the latest release archive for your system, checks its checksum and
installs it into ~/.local/bin. The installer prints a command to try when it
finishes. If your shell cannot find fastmash, see Command not found.
Run the same command again to update this installation.
Choose another writable directory with FASTMASH_INSTALL_DIR:
curl -fsSL https://fastmash.io/install.sh | env FASTMASH_INSTALL_DIR="$HOME/bin" sh
Use the same directory when updating. To read the installer’s help without installing anything:
curl -fsSL https://fastmash.io/install.sh | sh -s -- --help
If a download or checksum check fails, your existing installation stays in
place. Check your connection and retry. For a permission error, choose a
directory you can write to with FASTMASH_INSTALL_DIR.
Single packages and archives
These downloads install the latest release without adding a repository. You update them yourself.
Arch Linux package
Use the Arch package recipe to install the prebuilt executable through makepkg and pacman. No Rust compiler is needed. To update, rebuild the package from the current recipe.
Debian or RPM package
Download the .deb package for Debian or Ubuntu, or the .rpm package for
Fedora, RHEL and similar systems, from the
release downloads.
Save it in an empty folder, then open a terminal there.
On Debian or Ubuntu:
sudo apt install ./fastmash_*.deb
On Fedora, RHEL and similar:
sudo dnf install ./fastmash-*.rpm
To update, repeat these steps with a fresh download.
Linux and macOS archives
-
Open the release downloads and choose the archive for your system:
System Archive ending Linux or WSL2 x86_64-unknown-linux-musl.tar.gzmacOS aarch64-apple-darwin.tar.gz -
Extract the archive, then open a terminal in the extracted folder containing
fastmash. -
Move the executable into your local command directory and try it:
mkdir -p ~/.local/bin mv -f fastmash ~/.local/bin/fastmash printf '1\n2\n' | ~/.local/bin/fastmash sum 1
The final command prints 3. If your shell cannot find fastmash by name,
see Command not found. To update, repeat these steps with
a fresh download.
With Cargo
If you already use Rust, Cargo offers two routes.
cargo-binstall downloads the same prebuilt executables as the release archives, without compiling. On Linux or WSL2, run:
cargo binstall fastmash --strategies crate-meta-data
On macOS or Windows, select its single executable:
cargo binstall fastmash --strategies crate-meta-data --bin fastmash
These commands install the published binaries and stop with an error if the download is unavailable.
To compile Fastmash from source with cargo install, use the latest stable
Rust toolchain. On Linux or WSL2:
rustup target add x86_64-unknown-linux-musl
cargo install --locked --target x86_64-unknown-linux-musl fastmash
On macOS 15 or later, select its single executable:
cargo install --locked --bin fastmash fastmash
Check that it works
In Bash, or use the PowerShell example:
printf '1\t2\n3\t4\n' | fastmash sum 1 mean 2
The summary shows 4 for the sum and 3 for the mean. Next: the
quick start.
Command not found
For a quick installation or an archive installed in the default directory, try the executable directly:
~/.local/bin/fastmash --version
PATH lists the directories your shell searches for commands, in order.
If the direct command prints the version, add ~/.local/bin to PATH. In Fish:
fish_add_path "$HOME/.local/bin"
In Bash, add this line to ~/.bashrc. In Zsh, add it to ~/.zshrc, then open
a new terminal:
export PATH="$HOME/.local/bin:$PATH"
If Bash starts a login shell, add the line to ~/.bash_profile instead.
Replace ~/.local/bin with your chosen directory if you set
FASTMASH_INSTALL_DIR.
To see which installation your shell uses:
command -v fastmash
Update that installation with its original method, or put your preferred
installation directory first in PATH. In Fish, fish_add_path --move
moves an existing directory to the front.
On Windows
Use PowerShell 7.4 or later on Windows 11.
-
Open the release downloads and choose the archive ending in
x86_64-pc-windows-msvc.tar.gz. -
Extract it and keep the extracted folder somewhere permanent.
-
Search Windows for Edit environment variables for your account. Edit Path and add the folder containing
fastmash.exe. -
Open a new PowerShell terminal and run:
fastmash --version
It prints the installed version. Continue with the
PowerShell quick start for files and pipelines.
To update, replace the extracted folder and keep its location on Path.
To uninstall, remove that folder and its Path entry. No Rust compiler or
separate Visual C++ runtime installation is needed.
For Linux workflows, you can also use WSL2
and follow the Linux instructions. Keep WSL2 data in the Linux file system rather
than under /mnt/c.
Uninstall
Package managers remove Fastmash with their own commands, shown above. For the quick installer or a downloaded archive:
rm ~/.local/bin/fastmash
If you chose another directory with FASTMASH_INSTALL_DIR, remove its
fastmash file instead. Removing the executable does not remove your data.
If you installed with Cargo or cargo-binstall:
cargo uninstall fastmash
Fastmash keeps no configuration files, caches or background services.
See System requirements for building from source and verifying release archives.
Quick start
This tour takes about five minutes. It assumes Fastmash is installed. Every example reads from standard input, so you can paste it straight into Bash. For Windows, start with PowerShell 7.4+. The Fastmash operations and options are the same.
Locale. The examples use decimal points. If your shell uses a locale with decimal commas or an unsupported locale (see Locales), run
export LC_ALL=C.UTF-8first.
Your first summary
Fastmash reads records (lines) of fields separated by tabs, and applies one or more operations to selected fields:
$ printf '1\t2\n3\t4\n' | fastmash sum 1 mean 2
Summary
────────────────────────────────
sum(field-1) mean(field-2)
4 3
✔ Completed
sum 1 adds up field 1 and mean 2 averages field 2. The summary shows both
results in the order you requested.
PowerShell 7.4+
For a CSV file, pass its bytes directly through cmd input redirection.
Try this sample, or use your own sales.csv with an amount field:
'item,amount', 'a,2', 'b,4' | Set-Content -Encoding utf8 sales.csv
cmd /d /v:off /c 'fastmash --csv -H sum amount < "sales.csv"'
sum(amount)
6
--csv accepts Windows line endings. PowerShell does not support < input
redirection itself. Get-Content decodes a file into text, so use the cmd
recipe when you want to preserve its original bytes.
You can also send text directly:
'item,amount', 'a,2', 'b,4' | fastmash --csv -H sum amount
This produces the same result. For a native program pipeline, extract the amounts and pass them to another Fastmash command:
cmd /d /v:off /c 'fastmash --csv-in --header-in cut amount < "sales.csv"' |
fastmash sum 1
The summary shows 6. PowerShell 7.4+ preserves bytes between native programs
and when redirecting their output with >. PowerShell string pipelines create
text, including Windows line endings. See Fields and input
for how ordinary text and CSV handle those endings.
After a command, $LASTEXITCODE gives its status. In scripts, follow
PowerShell exit-status handling.
Several fields at once
A field list or range applies an operation to each field:
$ printf '1\t2\t3\n4\t5\t6\n' | fastmash sum 1-3
Summary
────────────────────────────────
sum(field-1) sum(field-2) sum(field-3)
5 7 9
✔ Completed
Grouping
-g groups records by a key field and summarizes each group:
$ printf 'A\t1\nA\t2\nB\t4\n' | fastmash -g 1 sum 2 count 2
Grouped summary
────────────────────────────────
field-1 sum(field-2) count(field-2)
A 3 2
B 4 1
✔ Completed
Groups are made from adjacent records with the same key. If equal keys
can be scattered through the input, add -s to sort first:
$ printf 'b\t2\na\t1\na\t3\n' | fastmash -s -g 1 sum 2
Grouped summary
────────────────────────────────
field-1 sum(field-2)
a 4
b 2
✔ Completed
Headers and named fields
With a header line, use -H to name fields and label the output:
$ printf 'site\treading\nwest\t2\neast\t4\nwest\t6\n' | fastmash -H -s -g site sum reading mean reading
Grouped summary
────────────────────────────────
site sum(reading) mean(reading)
east 4 4
west 8 4
✔ Completed
--header-in reads field names. -H also uses them in result labels.
Other separators
-t sets a single-byte field separator, and -W splits on runs of spaces
and tabs:
$ printf '1,2\n3,4\n' | fastmash -t, sum 1-2
Summary
────────────────────────────────
sum(field-1) sum(field-2)
4 6
✔ Completed
$ printf '1 2\n3 4\n' | fastmash -W sum 2
Summary
────────────────────────────────
sum(field-2)
6
✔ Completed
-t, splits on every comma, preserving datamash’s literal separator behavior.
For quoted CSV, use --csv to read and write CSV or choose input/output
independently with --csv-in and --csv-out:
$ printf 'region,revenue\n"west,coast",2\neast,4\n"west,coast",6\n' | fastmash --csv -H -s -g region --result-name=1:total sum revenue
GroupBy(region),total
east,4
"west,coast",8
Format switches do not infer headers. Quoted CSV and result names describes the supported modes and quoting rules.
Statistics
$ printf '2\n4\n4\n4\n5\n5\n7\n9\n' | fastmash median 1 q1 1 q3 1 pstdev 1 perc:90 1
Summary
────────────────────────────────
median(field-1) q1(field-1) q3(field-1) pstdev(field-1) perc:90(field-1)
4.5 4 5.5 2 7.6
✔ Completed
See Operations for the full list: means, variances, quantiles, modes, skewness and kurtosis, normality tests, correlation and more.
Weighted means
Give each value a contribution weight with wmean VALUE:WEIGHT:
$ printf '10\t1\n20\t3\n' | fastmash wmean 1:2
Summary
────────────────────────────────
wmean(field-1,field-2)
17.5
✔ Completed
Weights must be finite and nonnegative. A nonempty calculation needs positive retained weight. See Weighted mean.
Keep the highest records
Top-N selection retains complete records and keeps earlier input records on ties:
$ printf 'A\t8\nB\t10\nC\t10\nD\t9\n' | fastmash top:2 2
Top-N selection
────────────────────────────────
field-1 field-2
B 10
C 10
✔ Completed
Use bottom:N for the lowest records, and -s -g for a cutoff in each group.
See Highest and lowest records.
Per-row operations
Some operations transform each record instead of summarizing:
$ printf '3.14159\thello\n' | fastmash round 1 md5 2
Per-row results
────────────────────────────────
round(field-1) md5(field-2)
3 5d41402abc4b2a76b9719d911017c592
✔ Completed
Missing values
--narm skips NA, N/A and NaN values:
$ printf '1\nNA\n3\n' | fastmash --narm mean 1
Summary
────────────────────────────────
mean(field-1)
2
✔ Completed
Next steps
- Fields and input: selectors, separators, headers, comments
- Grouping and sorting:
-g,-s,crosstab - Table health: inspect a table or validate supplied rules
- Dataset comparison: before/after summaries and ranked changes
- Migrating from GNU datamash
fastmash --helplists every operation and option
Migrating from GNU datamash
Fastmash accepts GNU datamash 1.9’s command language. In most scripts,
switching is a matter of replacing datamash with fastmash.
A safe way to switch
-
Install Fastmash alongside
datamash. Don’t alias or replacedatamash; keep both available while you compare. -
Run your existing command with both programs on the same input and with the same locale settings, and compare the output and the exit status:
export LC_ALL=C.UTF-8 datamash -s -g 1 sum 2 < data.tsv > gnu.out; echo "datamash: $?" fastmash -s -g 1 sum 2 < data.tsv > fast.out; echo "fastmash: $?" cmp gnu.out fast.out && echo identical -
If the outputs differ, check Differences from GNU datamash. If the difference isn’t listed there, please report it.
-
Change the script, and keep checking exit statuses as before.
What works unchanged
- Every GNU datamash 1.9 operation, mode and alias, except the explicit
linemode (writefastmash base64 1, notfastmash line base64 1) - Field numbers, lists, ranges, pairs and header names
- All options except
--sort-cmd, including short-option clusters (-sg1), abbreviated long options (--whitesp), options after operands, and-- POSIXLY_CORRECThandling- Data output to files and pipelines, header names and the
--fulloutput - Exit statuses 0 and 1 for success and ordinary errors
What to check
Locale settings. Fastmash has the glibc 2.43 locale rules built in:
numbers work in C, POSIX and every glibc locale with a one-byte decimal
separator, @ modifier locales included, and sorting in C, POSIX,
C.UTF-8 and 194 language locales checked against glibc. A name glibc has no
locale for behaves as C, as in GNU datamash. Commands that read or print
numbers exit with status 77 in ps_AF, and in a character set other than
UTF-8 whose thousands separator is not ASCII (such as fr_FR in Latin-1);
sorted commands do outside the sorting locales. GNU datamash falls back to C
where a locale is not installed, while Fastmash does not. Set
LC_ALL=C.UTF-8 (or C) in scripts. In language locales, Fastmash orders
sorted keys as GNU sort does under glibc, letters and digits by Unicode
collation and punctuation, symbols and spaces by glibc’s own table; keys that
differ only in punctuation next to digits (12-A and 1-2A), vulgar fractions
such as ¼ and some letters from outside the language’s alphabet can still
sort differently. Locales.
Last digits of results. Fastmash’s portable arithmetic can differ from a
local GNU build in the last printed digit of some results, or in the sign of
a nan. If a script compares numbers textually, compare with a tolerance or
fix the precision with --round. Numbers.
Exit status 77. Fastmash refuses some inputs that GNU datamash handles
unsafely, such as perc:2.5 or rmdup with several keys. A script that
treats any nonzero status as failure needs no change.
--sort-cmd. Not supported. To use a different sorter, sort the input
first and leave out -s:
LC_ALL=C sort -s -t "$(printf '\t')" -k1,1 data.tsv | fastmash -g 1 sum 2
Test suites. GNU datamash’s hidden testing options (---print-inf,
---print-nan, ---print-progname, ---rmdup-test) are not provided.
Remember
-t, splits on commas and does not parse quoted CSV, in both programs.
Fastmash adds explicit --csv-in, --csv-out and --csv switches for
quoted CSV; they leave existing separator commands
unchanged. Result names, weighted mean, Top-N selection, dataset comparison and
table health are also optional extensions.
spearson is the sample Pearson correlation, in both programs.
Compared with other tools
Fastmash runs GNU datamash commands with portable results, and extends that workflow with quoted CSV, weighted means, record selection, table health and dataset comparison. The tools below serve different data-processing needs. Facts about other tools come from their documentation, checked September 2026.
At a glance
| Fastmash | GNU datamash | Miller | qsv | csvtk | |
|---|---|---|---|---|---|
| Command language | datamash’s | datamash’s | Its own verbs and DSL | Its own subcommands | Its own subcommands |
| Written in | Rust | C | Go | Rust | Go |
| Licence | MIT or Apache-2.0 | GPL-3.0-or-later | BSD-2-Clause | MIT | MIT |
| Input | Delimited text and explicit quoted CSV | Tab or single-byte delimited text | CSV (RFC 4180), TSV, JSON and more | CSV, TSV, Excel, Parquet and more | CSV and TSV |
| Quoted CSV fields | Yes, in supported calculation, selection and comparison modes | No | Yes | Yes | Yes |
| Grouped statistics | Yes | Yes | Yes (stats1 -g) | Through pivotp or sqlp | Yes (summary -g) |
| Joins and expression filters | No | No | Yes | Yes | Yes |
| Table reshaping | Transpose and crosstab | Transpose and crosstab | Yes | Yes | Yes |
| Platforms | Linux, macOS 15+, Windows 11 | Linux, macOS, Windows and others | Linux, macOS, Windows, BSD | Linux, macOS, Windows | Linux, macOS, Windows, BSD |
GNU datamash
The original, and the reference Fastmash is measured against. It runs almost everywhere and is packaged by every Linux distribution, conda-forge and Homebrew.
Choose Fastmash when you run datamash commands on Linux and want them faster (about five times on RefGene quartiles in the Linux benchmarks), want the same digits on every machine, or want clear errors where datamash 1.9 aborts or reads past its data. Most commands transfer directly. The differences are listed on one page.
Stay with GNU datamash when you need Intel macOS or a platform outside Fastmash’s supported targets, or when you need a GNU-maintained tool. Large sorted summaries can still favor GNU on Linux. The benchmarks show the costs and measurement conditions.
Miller
A complete toolkit for record-oriented data: it reads and writes CSV (with
quoting), TSV, JSON and several other formats, and has a full expression
language for filtering, computing new fields, joining, sorting, sampling and
reshaping. Its stats1 verb computes grouped statistics (count, sum, mean,
median and any percentile, mode, variance, skewness and more) without
sorting the input first.
Choose Miller when your data is JSON, or the job needs more than statistics: filtering, joining or transforming records. Choose Fastmash when the job is a datamash-style summary of delimited text and you want familiar commands with portable numerical results.
qsv
A very fast, very broad CSV toolkit in Rust: indexing, joins (through Polars),
SQL queries, validation, sampling, format conversion to Parquet, Excel and
databases, and whole-column statistics. Grouped aggregation goes through
pivotp (count, sum, mean, median, quantiles, first, last) or sqlp.
Choose qsv when you work with large CSV files and want one tool for exploring, querying and converting them. Choose Fastmash when you want datamash’s operations and syntax, for example sample and population standard deviations, skewness and kurtosis, mode, collapse or crosstab per group, with a documented compatibility contract.
csvtk
A cross-platform CSV/TSV toolkit popular in bioinformatics, with subcommands
for selecting, filtering, joining, sorting, reshaping, converting
and plotting. csvtk summary computes grouped statistics (count, sum, mean,
median, quartiles, standard deviation, first, last, unique values, collapse),
printing two decimals by default.
Choose csvtk when you want a single, familiar toolkit for everyday table work, including quoted CSV and compressed files. Choose Fastmash when you need datamash-style statistics and commands, portable numerical results, or its measured speed on large inputs.
Using them together
These tools combine well. Fastmash can read quoted CSV directly:
fastmash --csv -H -s -g region mean price < data.csv
Use another tool to filter or join records, then Fastmash to summarize them. For ordinary text workflows, a quote-aware conversion remains useful:
mlr --icsv --otsv cat data.csv | fastmash -H -s -g region mean price
User guide
Use Fastmash to summarize fields, combine groups, transform records and inspect tables. Commands take these forms:
fastmash [OPTIONS] OPERATION FIELDS [OPERATION FIELDS ...]
fastmash [OPTIONS] -g KEYS OPERATION FIELDS ...
fastmash [OPTIONS] MODE ...
fastmash [OPTIONS] top:N FIELD
fastmash [OPTIONS] compare BEFORE AFTER [rank RESULT [absolute|percent] [limit N]] OPERATION FIELDS ...
fastmash [INPUT OPTIONS] health [CONTROLS]
Fastmash reads records from standard input, splits each into fields, and applies the requested operations. It writes results to standard output and diagnostics to standard error, and reports success or failure through its exit status.
| Page | Covers |
|---|---|
| Fields and input | Field selectors, separators, headers, record terminators, comments, missing values |
| Quoted CSV and result names | Explicit CSV formats, quoting rules and stable output labels |
| Operations | Every summary statistic, with parameters and edge cases |
| Weighted mean | Contribution weights, complete-pair omission and numerical limits |
| Grouping and sorting | -g, -s, case-insensitive grouping, crosstab |
| Highest and lowest records | Complete-record selection, stable ties, groups and capacity limits |
| Dataset comparison | Before/after summaries, global key alignment and change ranking |
| Table health reports | Inspection, supplied validation rules and versioned TSV output |
| Per-row operations and modes | Rounding, binning, checksums, paths, cut, transpose, check, rmdup |
| Numbers, output and locales | Number formats, --format, --round, locales, portable results |
| Errors, exit statuses and resources | What can fail, how it is reported, memory and disk use |
| Differences from GNU datamash | Every intentional difference |
fastmash --help prints a complete summary of operations and options.
Fields and input
Records and fields
Ordinary delimited-text input is a sequence of records, each ended by a
newline (or by a NUL byte with -z). The last record may omit its terminator.
Each record is split into fields, by default at every tab.
Carriage returns are kept as data in ordinary text, so files with Windows line
endings may need converting first (for example with tr -d '\r'). Explicit
quoted CSV input accepts LF and CRLF record endings
and permits line breaks inside quoted fields.
Selecting fields
Operations take a selector naming the fields to use:
| Selector | Meaning |
|---|---|
3 | Field 3 (fields are numbered from 1) |
1,4 | Fields 1 and 4 |
2-5 | Fields 2 through 5 |
1,3-4 | Field 1, then fields 3 and 4 |
price | The field named price in the input header |
2:3 | The pair of fields 2 and 3, for paired statistics |
1:2,3:4 | Two pairs |
A list or range applies the operation to each field in turn, producing one result per field. Repeating a field repeats its result. Ranges must be ascending, and a range cannot be combined with pairs.
Field separators
| Option | Effect |
|---|---|
| (default) | Fields are separated by a tab |
-t X, --field-separator=X | Fields are separated by the single byte X |
-W, --whitespace | Fields are separated by runs of spaces and tabs. Leading whitespace is skipped |
--output-delimiter=X | Results are separated by X |
The output separator follows the input separator unless --output-delimiter
is given. With -W, the output separator is a tab.
$ printf 'a;1\nb;2\n' | fastmash -t ';' sum 2
Summary
────────────────────────────────
sum(field-2)
3
✔ Completed
-t, does not understand CSV quoting: a comma inside a quoted field still
splits it. Use --csv-in to read quoted CSV, or --csv for both input and
output. Format switches do not imply headers. CSV input conflicts with -t,
-W and -C. Either CSV direction conflicts with -z and --vnlog.
See Quoted CSV and result names.
Headers
| Option | Effect |
|---|---|
--header-in | The first record names the fields. It is not treated as data |
--header-out | Print a header line naming each result |
-H, --headers | Both of the above |
With an input header, selectors can use field names. Names are exact and case-sensitive. If two fields share a name, the first one is used.
$ printf 'price\tunits\n3\t2\n5\t4\n' | fastmash -H sum units mean price
Summary
────────────────────────────────
sum(units) mean(price)
6 4
✔ Completed
NUL-terminated records
-z ends records with NUL bytes instead of newlines, for input from tools
such as find -print0. Output records are NUL-terminated too.
Comments
-C (--skip-comments) skips records whose first non-blank character is #
or ;. Comment markers later in a record are data. Skipped lines do not count
towards line numbers in error messages.
--vnlog reads vnlog files: whitespace-separated
columns with a # name name … header line and # comments. In vnlog data,
everything after a # is ignored, even inside a field, and output uses spaces
with a # header.
Missing values
--narm skips fields whose exact text is NA, N/A or NaN, in any letter
case, for every operation:
$ printf '1\nNA\n3\n' | fastmash --narm sum 1 count 1
Summary
────────────────────────────────
sum(field-1) count(field-1)
4 2
✔ Completed
Only these exact words count as missing. An empty field is not missing: a numerical operation reports it as an invalid value. A record that lacks the selected field is an error.
For paired statistics such as covariance, each side skips missing values independently, so remaining values may pair across different records. Weighted mean instead omits the whole value/weight pair after validating supplied partners.
Other options
| Option | Effect |
|---|---|
-f, --full | Print a complete input record before each group’s results. The record is the group’s first, unless first, last, rand or an extremum selects another. GNU datamash’s deprecation warning for this option is kept |
-S N, --seed N | Make rand reproducible. Without a seed, rand is not reproducible. Its selection, like GNU datamash’s, is slightly biased and not cryptographic |
--no-strict, --filler X | Accept ragged rows and set the missing-cell text for transpose and crosstab. They don’t fill in missing fields for summary operations |
-- | End option processing |
Output headers name each result after its operation and field, such as
sum(price), and grouping keys as GroupBy(site).
Options can appear after operations, and short options can be combined
(-sg1) as in GNU datamash. If the POSIXLY_CORRECT environment variable is
set, option processing stops at the first operation.
Quoted CSV and custom result names
Use --csv to read and write quoted CSV. Add -H when the first record names
the fields. For example, summarize revenue by region:
$ printf 'region,revenue\nwest,2\neast,4\nwest,6\n' | fastmash --csv -H -s -g region sum revenue
GroupBy(region),sum(revenue)
east,4
west,8
--csv-in selects only CSV input, and --csv-out selects only CSV output.
Ordinary delimited text remains the default. Use
custom result names to give calculated columns stable
labels, such as total instead of sum(revenue).
Supported workflows
| Workflow | CSV input/output | Result names |
|---|---|---|
| Aggregate and per-row reports | Yes, including supported grouping and full copied fields | Calculated output columns with an explicit output header |
| Weighted mean | Yes, including adjacent and sorted groups | One name per value/weight result |
| Top-N selection | Yes, including adjacent and sorted groups | No: selection copies the source fields |
| Dataset comparison | Yes. Both sources use the same input format | One name throughout each summary’s before/after/change columns |
| Crosstab, field modes and legacy table modes | No | No |
| Table health | No. Ordinary text input and readable or TSV output | No |
Calculations on quoted CSV
Use --csv-in for strict quoted CSV input with ordinary text output, or --csv
for CSV input and output together. --csv is equivalent to --csv-in --csv-out.
Repeating these exact-only switches is idempotent. None enables headers.
Top-N selection supports decoded CSV input for whole-dataset,
adjacent-group and sorted-group selection with either output format. Sorted
selection retains complete decoded records and original-input ties through
bounded spill. Its copied output header uses the original source header or
first accepted source record, independently of sorted encounter order.
Its ranking field uses the decoded field-local numerical
contract, and copied data preserves complete bytes.
Aggregate and per-row calculations support their existing positional
lists, numeric ranges, escaped named selectors, paired selectors and parameters.
Aggregate grouping and --full work with decoded CSV fields. Add -s to bring
equal decoded grouping keys together, using bounded spill when the input exceeds
the maintained sort-memory target. Per-row operations retain their ordinary
conflict with grouping keys. With no grouping keys, -s keeps the streaming
workflow and source order, including per-row and --full reports. Output-only CSV retains sorted
text-input workflows.
For example, a quoted export can be summed directly, without a quote-aware preprocessing step, while preserving a comma in its decoded column name:
printf '"amount,raw",note\r\n"2","two\nlines"\n4,"a""b"' | LC_ALL=C fastmash --csv -H --result-name='1:total,"daily"' sum 'amount\,raw'
"total,""daily"""
6
Grouping decoded fields
Grouping keys compare decoded values, so west and "west" belong to the same
adjacent group. Repeated keys in a later run stay separate without sorting:
printf '"region,name",value\n"west,coast",2\n"west,coast",4\neast,3\n"west,coast",5\n' | LC_ALL=C fastmash --csv -H -g 'region\,name' --result-name=1:total sum value
"GroupBy(region,name)",total
"west,coast",6
east,3
"west,coast",5
Multiple and named keys retain their normal request order, exact name lookup and
first-duplicate-name rules. Adjacent equality is byte-based, with ASCII folding
under -i where the selected locale supports it. Equal-length keys compare up
to NUL, while displayed keys retain their complete byte spans. Locale-aware
numeric input and output keep their existing rules. A language locale does not
silently merge differently spelled adjacent keys.
Sorting the same export merges the separate west,coast runs:
printf '"region,name",value\n"west,coast",2\n"west,coast",4\neast,3\n"west,coast",5\n' | LC_ALL=C fastmash --csv -H -s -g 'region\,name' --result-name=1:total sum value
"GroupBy(region,name)",total
east,3
"west,coast",11
Sorting compares complete decoded keys with the existing locale and case policy, then preserves source order for tied keys. Group equality remains distinct from locale ordering: differently spelled keys which the locale ties can still form separate adjacent groups. Headers bind before sorting. Without an input header, a generated output header takes its field width from the first sorted record. Copied records keep their actual widths and original logical record/physical-line locations through reordering and representative selection.
CSV sorting uses the same sort-memory policy as ordinary text, including
FASTMASH_SORT_MEMORY_BYTES, and spills sorted runs to temporary files when
needed. Complete decoded fields and their original input locations survive sorting.
The memory target covers sort chunks, not the whole process. An oversized record,
merge heads and retained statistical or text samples have their own storage needs.
Samples are not spilled. Temporary I/O errors fail with status 1, and checked
allocation failures refuse with status 77. Later failures may leave earlier output.
Copying complete records
Full reports copy each decoded source field before the calculation results. For example, a per-row report can retain a comma in a source field while selecting its numeric field with a custom result label:
printf 'place,value\n"west,coast",2\neast,4\n' | LC_ALL=C fastmash --csv -H --full --result-name=1:reading cut value
place,value,reading
"west,coast",2,2
east,4,4
For an aggregate, the representative starts as the group’s first record.
first, last, rand and improving extrema may select another record under
their existing rules. Extrema ties retain the earlier selection. With multiple
operations the representative need not supply every result. Aggregate --full
retains its existing GNU compatibility warning. Per-row --full does not warn.
Copied-field labels replace GroupBy labels, and result-name indices still count
only operation-produced results. Without sorting, header width comes from the
first logical record. Each copied data record keeps its actual width, without padding or
filler. Required fields must still exist, including with --no-strict or
--filler. Copied fields preserve complete bytes, including NUL, independently
of an operation’s display rules.
CSV syntax and bytes
A double quote opens quoting only at the start of a field. Inside quotes, commas, CR and LF are data, and two double quotes decode to one. After a closing quote, only comma, LF, CRLF or end of input is allowed. Quotes within unquoted fields, stray bytes after closing quotes, unfinished quoted fields and bare CR outside quotes are errors, including in unused fields. Spaces are data, with no implicit trimming. LF and CRLF endings may be mixed. A complete final record may omit its ending.
An empty file has no records. Each blank LF or CRLF record has one empty field. Quoted and unquoted empty fields decode identically. A trailing comma supplies
a final empty field, while a terminal record ending supplies no extra record.
Different widths are permitted when all requested fields exist. An absent field
still errors. An existing empty field is valid text but invalid numeric input.
--narm retains the existing exact, case-insensitive NA/N/A/NaN matching and
independent paired-side compaction. It does not skip empty fields or trim spaces.
Decoding preserves bytes, including tabs, NUL, non-UTF-8 and internal CRLF.
Operations keep their existing display and transformation rules: cut stops
display at NUL, while base64 and checksums consume the complete decoded field.
The active numerical locale and presentation settings apply to each isolated
decoded numeric field. An unused numeric neighbor cannot affect conversion.
Ordinary text numerical conversion keeps its existing behavior.
A leading UTF-8 BOM is data. Before an unquoted input header it becomes part of the name and generated label, following the normal byte-escaped selector rules. Before a quoted first field it is data followed by an illegal quote, so that export fails. There is no BOM stripping or transcoding. This dialect has stated byte and blank-record extensions. It is not a claim of full RFC 4180 conformance.
With --csv-in, text results use the normal tab default and accept an explicit
--output-delimiter. Decoded delimiters or line breaks in result fields may make
that text report ambiguous. Use --csv when field boundaries must survive output.
CSV input conflicts with explicit -t, -W and -C, including their GNU
abbreviations and redundant comma delimiters. Either CSV direction conflicts
with -z and --vnlog, in either option order, with error status 1. Legacy modes
refuse CSV before input. Health rejects all CSV switches and remains text-only.
Errors and locations
CSV validation errors identify the original logical record number, counting an input header, and its starting physical line. Syntax errors also identify the detected physical line and byte position, counting raw bytes including quotes. Unfinished quotes identify unexpected end of input. LF inside quotes advances the physical line. CRLF counts as one line break. Header lookup errors include the header’s location when it exists. I/O failures keep their I/O context without inventing a record or turning a read failure into a quoted-field EOF error. Malformed CSV uses status 1, checked capacity refusals use 77, and output finalization retains normal precedence. Later failures can leave partial output.
CSV output from delimited text
Use --csv-out to produce quoted CSV from ordinary delimited-text input.
It works with aggregate and per-row operations, including adjacent or sorted
groups, explicit headers and copied --full fields. Repeating the switch is
idempotent. It does not enable either input or output headers.
Top-N selection also supports CSV output from complete
selected text records, including sorted groups. It adds no calculation columns.
For example, tab-separated observations can become a CSV report with a stable result label, including keys and names containing commas or quotation marks:
printf 'place\treading\nlab,west\t2\nlab,west\t4\n' | LC_ALL=C fastmash --csv-out -H -g place --result-name='1:total,"daily"' sum reading
GroupBy(place),"total,""daily"""
"lab,west",6
Every logical output field is encoded separately: complete generated labels,
custom names, grouping keys, formatted results and each copied --full field.
Commas separate fields and LF ends each emitted record. A field containing a
comma, double quote, CR or LF is quoted, with internal double quotes doubled.
An empty field prints as "". Embedded line breaks retain their bytes. Output
is canonical. It does not retain an input field’s quoting style or line ending.
Decimal-comma formatted numbers and collapse or unique lists stay within one
quoted result field. The collapse delimiter is still the list’s internal
separator. CSV adds no escaping scheme inside those lists. Encoding preserves
the result bytes, including non-UTF-8 and NUL bytes, under the existing operation
and header display rules. For example, raw keys and copied fields retain NUL,
while selected text results and generated field-name displays stop at NUL as
they do in ordinary output. A leading UTF-8 BOM remains field data.
Output-only CSV allows text-input -t, -W and -C. An explicit
--output-delimiter, -z or --vnlog conflicts with CSV output and fails with
status 1, regardless of option order or a redundant delimiter value. Crosstab,
legacy Table modes and field modes refuse CSV with status 77 before input.
Health reports remain text-only and reject CSV switches using their status-1
calculation-control diagnostic. Only the exact CSV switch spellings are accepted. Existing GNU option abbreviations retain their meanings.
CSV output retains empty-input and header-only timing and the normal failure policy. A later input, calculation or transport failure can leave partial output.
Custom result names
Give calculated output columns stable names in your reports. Select output headers with
--header-out or -H, then repeat --result-name=INDEX:NAME for the labels you
want to replace. The separate form --result-name INDEX:NAME works too.
For example, a report can give its sum a stable schema name while retaining the generated label for its mean:
printf 'reading\n2\n4\n' | LC_ALL=C fastmash -H --result-name=1:total sum reading mean reading
Summary
────────────────────────────────
total mean(reading)
6 3
✔ Completed
Indices follow operation-produced output order after selector expansion, starting
at 1. A list or range contributes each expanded result, repeated operations
contribute their results again, and a paired operation contributes one result.
Each field selected by cut also contributes a result. Grouping keys and the
copied fields from --full do not consume indices or receive replacement labels.
In this example, sum 2,1-2 produces three results in the order 2, 1, 2. Names
can be supplied out of order, and unnamed results keep their generated labels:
printf '1\t2\n3\t4\n' | LC_ALL=C fastmash --header-out --result-name=3:again --result-name=1:total sum 2,1-2
Summary
────────────────────────────────
total sum(field-1) again
6 4 6
✔ Completed
Names replace labels only. They do not change selector lookup, required-field
checks, calculation values, output order, or when headers appear. Empty input
still produces no invented header. With -H, header-only input retains its
normal output header without a result row. Later input or output failures can
leave partial output, as with ordinary generated headers.
INDEX must be an unsigned, nonzero decimal integer within the expanded result
count. Repeating a target, omitting a name, or supplying an empty name is an error.
Duplicate name values are allowed. Split at the first colon, so a name can itself
contain colons. Name bytes are preserved as supplied by the operating system.
For text output, names cannot contain the effective output delimiter or record
terminator. CSV output allows comma, quote, colon, CR and LF bytes in names and
encodes them as one field. Invalid names fail before any output.
Result names work for ordinary aggregate and per-row operations, including
adjacent and sorted groups. Crosstab, table modes (transpose, check, rmdup)
and field modes (reverse, noop) refuse naming before reading input. Only the
exact --result-name spelling is accepted. Existing GNU option abbreviations
retain their normal meanings. --vnlog alone does not satisfy the requirement
for explicit output headers. Add --header-out or -H.
GNU datamash 1.9 generates labels such as sum(reading). Stable custom report
labels otherwise need a wrapper or a downstream renaming step. Naming inside
the command avoids that extra step and follows the existing header timing.
Operations
Operations summarize a field over all records, or over each
group. Operation names are case-insensitive. Parameters follow a
colon: perc:90, trimmean:0.1.
For operations that transform each record instead, see Per-row operations and modes.
Counting and text
| Operation | Result |
|---|---|
count | Number of values, including text and empty fields |
countunique | Number of distinct values (case-sensitive unless -i) |
unique, uniq | Sorted, comma-separated list of distinct values |
collapse | Comma-separated list of all values, in input order |
first | First value |
last | Last value |
rand | One value chosen at random (reproducible with --seed) |
-c X (--collapse-delimiter) changes the separator used by unique and collapse.
Basic statistics
| Operation | Result |
|---|---|
sum | Sum |
mean | Arithmetic mean |
min, max | Minimum, maximum |
absmin, absmax | Value with the smallest or largest magnitude, keeping its sign |
range | max minus min |
Other means
| Operation | Result |
|---|---|
geomean | Geometric mean |
harmmean | Harmonic mean |
ms | Mean of the squared values |
rms | Root mean square |
trimmean[:P] | Mean after removing the fraction P of values from each end (default 0, with 0.5 giving the median) |
wmean VALUE:WEIGHT | Weighted mean, with finite nonnegative contribution weights |
wmean takes one value/weight pair per result, for example wmean price:units
with named fields or wmean 2:3 with positions. It supports text or quoted CSV,
headers, custom result names and adjacent or sorted groups. A nonempty
calculation needs positive retained weight. --narm omits whole pairs after
checking supplied partners. See Weighted mean for examples,
missing values and numerical range limits.
Quantiles and order statistics
| Operation | Result |
|---|---|
median | Middle value, or the mean of the two middle values |
q1, q3 | First and third quartiles |
iqr | Interquartile range, q3 minus q1 |
perc[:N] | Nth percentile, for integer N from 1 to 100 (default 95) |
mode | Most frequent value (the smallest, on ties) |
antimode | Least frequent value (the smallest, on ties) |
Quartiles and percentiles use linear interpolation between order statistics (Hyndman and Fan’s type 7, the default in R and NumPy).
Dispersion
| Operation | Result |
|---|---|
pvar, svar | Population and sample variance |
pstdev, sstdev | Population and sample standard deviation |
mad | Median absolute deviation, scaled by 1.4826 |
madraw | Median absolute deviation, unscaled |
Shape and normality
| Operation | Result |
|---|---|
pskew, sskew | Population and sample skewness |
pkurt, skurt | Population and sample excess kurtosis |
jarque | Jarque–Bera normality test p-value |
dpo | D’Agostino–Pearson omnibus normality test p-value |
Paired statistics
These take a pair of fields, LEFT:RIGHT:
| Operation | Result |
|---|---|
pcov, scov | Population and sample covariance |
ppearson, spearson | Population and sample Pearson correlation coefficient |
dotprod | Dot product |
$ printf '1\t2\n2\t4\n3\t7\n' | fastmash scov 1:2 spearson 1:2
Summary
────────────────────────────────
scov(field-1,field-2) spearson(field-1,field-2)
2.5 0.99339926779878
✔ Completed
spearson is the sample Pearson coefficient, not Spearman’s rank correlation.
Small samples and missing data
jarqueanddpokeep GNU datamash’s formulas, including its tail cancellation: very small p-values can print as0.- Sample statistics need enough values:
svarandsstdevgivenanfor a single value.sskewneeds at least three andskurtat least four. - When every value in a group is missing (with
--narm),countandsumgive0.mean,medianand most statistics givenan.minandmaxgive-infandinf.firstandlastgiveN/A. - An empty input produces no output line.
Memory use
Most operations keep a running total. Quantiles, modes, dispersion, shape,
normality and paired statistics must keep every value of a group, and text
operations like unique and collapse keep their text. Memory then grows
with the size of the largest group, and this storage is not spilled to disk.
Related operations on the same field share storage: quantiles, mode and
antimode share one sorted copy, the variance family another, and the
moment and normality statistics another.
Weighted mean
Use wmean VALUE:WEIGHT to average values according to their contribution
weights. The result is the sum of weighted values divided by the total weight.
Weights must be finite and nonnegative. They may be fractions and need not sum
to one. Zero contributes nothing.
For example, values 10 and 20 with weights 1 and 3 produce 17.5:
printf '10\t1\n20\t3\n' | env LC_ALL=C fastmash wmean 1:2
Summary
────────────────────────────────
wmean(field-1,field-2)
17.5
✔ Completed
Named reports and adjacent groups
Use explicit input and output headers with -H, or select them independently
with --header-in and --header-out. Named selectors use the existing
case-sensitive first match for duplicate names. A backslash escapes one byte
in a name. Positional and named fields may be mixed. Pair lists, repeated pairs,
shared weight fields and same-field pairs each produce a result in request order.
printf 'site\tvalue\tweight\na\t10\t1\na\t20\t3\nb\t5\t2\n' | env LC_ALL=C fastmash -H -g site --result-name=1:average wmean value:weight count value
Grouped summary
────────────────────────────────
site average count(value)
a 17.5 2
b 5 1
✔ Completed
Without a result name, the generated label identifies both roles, for example
wmean(value,weight) or wmean(field-1,field-2). Each expanded pair consumes
one result-name index. Grouping keys and copied --full fields consume none.
Use -H for named result labels. No count or total-weight column is added
automatically.
Empty input produces no report. Header-only input retains its ordinary output
header and emits no data result. This includes sorted reports with named fields
when input ends cleanly before the requested header. A present header still has
to contain the requested names, and a failed read is an input error.
Grouping uses the existing group definition: consecutive records with equal
grouping keys. Multiple keys and existing case and locale controls apply.
Separated runs of the same key remain separate. The calculation resets between
groups. Every requested grouping key must exist, including with --full and on
omitted pairs. An empty key is still present. A preceding group completes before
the next group’s pair is collected, preserving its result or completion failure.
wmean leaves the ordinary --full representative unchanged, initially
the first record. Another requested operation may replace it. It does not choose
the largest-weight record. Aggregate --full retains its existing deprecation
warning.
For ordinary-text input, existing separators,
whitespace splitting, comment filtering, NUL record termination, numeric locales,
--format and --round apply. Unused ragged or byte-oriented content remains
accepted when every required numeric and key field exists. This calculation
does not inspect whole-table health.
Sorted text reports
Use -s to bring interleaved grouping keys together. Multiple keys, named fields,
headers, result names and other aggregate operations retain their usual behavior:
printf 'site\tvalue\tweight\nb\t5\t2\na\t10\t1\nb\t7\t2\na\t20\t3\n' | env LC_ALL=C fastmash -H -s -g site --result-name=1:average wmean value:weight
Grouped summary
────────────────────────────────
site average
a 17.5
b 6
✔ Completed
The same calculation accepts a regular file, for example
env LC_ALL=C fastmash -H -s -g site wmean value:weight < measurements.tsv.
Sorting, including hash grouping and replay, preserves the encounter order of
each group’s contributions. It does not rearrange values or weights to improve
the arithmetic.
Existing case and supported-locale controls determine key ordering and equality.
With -W, blanks before a key can affect sorting before ordinary group equality
is applied. With no grouping keys, -s keeps the whole-dataset encounter order.
Both pair fields are retained, including reversed pairs and shared fields.
--full keeps complete representative records using the existing selection rule.
Sort storage has its own memory policy and can spill. The weighted accumulator
still keeps two totals per request. Ordinary sorted numeric errors use the
established sorted record position, rather than claiming an original-input
location. Temporary-storage, replay, read, output and close failures prevent
successful status, and numerical-range refusals remain unchanged.
Select --csv-out independently to encode results, keys, copied fields and
generated or supplied labels through the existing output encoder:
printf 'site\tvalue\tweight\nb\t5\t2\na\t10\t1\na\t20\t3\n' | env LC_ALL=C fastmash --csv-out -H -s -g site wmean value:weight
GroupBy(site),"wmean(value,weight)"
a,17.5
b,5
CSV output follows the shared encoding rules,
including quoting generated labels such as wmean(value,weight) as one field.
Quoted CSV reports
Use --csv-in for CSV input with ordinary text output, --csv-out for CSV
output from ordinary text, or --csv for both. The switches do not imply
headers. Numeric conversion and named selectors see decoded field bytes,
including quoted names with commas, quotes or line breaks:
printf 'site,"value,raw",weight\r\nb,5,2\na,"10",1\nb,7,2\n"a",20,3\n' | env LC_ALL=C fastmash --csv -H -s -g site --result-name=1:average wmean 'value\,raw:weight'
GroupBy(site),average
a,17.5
b,6
Omit -s to report each adjacent run separately, or omit -g site to report
one whole-dataset mean. Regular-file input uses the same calculation. Sorting
compares decoded keys, so a and "a" name the same key. Multiple keys and
existing case and supported-locale controls apply. Stable sorting and bounded
spill retain complete needed fields and each group’s contribution order.
CSV output preserves complete decoded keys and copied --full fields using
canonical encoding. CSV-to-text output retains ordinary text framing, so a
decoded tab or newline can appear as a delimiter or record break. Choose CSV
output when those bytes need unambiguous field boundaries.
Strict syntax validation covers every field and record, including unused content, omitted pairs and zero weights, even after an infinite or NaN result is known. The shared CSV grammar and option conflicts apply. Fastmash does not infer the input format.
Numeric, invalid-weight, missing-field and header errors retain the original logical record and starting physical line through sorting and spill. Syntax errors also identify the detected physical line and byte position. Completion errors identify the weighted pair and group without assigning a source record. Read, temporary-storage, output and close failures prevent successful status. Earlier output can remain under the ordinary report rules.
Missing pairs and failures
Values 10 and 20 with weights 0.25 and 0.75 also produce 17.5. Negative or nonfinite weights are input errors with status 1. Both signed zeros are zero weights. A zero-weight record still needs valid supplied fields, but its product is not evaluated, even for an admitted infinite or NaN value. Positive weights permit admitted infinite and NaN values under the ordinary arithmetic rules. Later records are still validated after such a result is known.
With --narm, an exact whole-field NA, N/A or NaN token, ignoring case,
omits the entire value/weight pair for that request. Supplied nonmissing fields
are validated in value-then-weight order before deciding omission.
Thus value NA with weight 2 is omitted, while
value NA with weight oops, -1 or infinity fails. Empty fields, absent fields,
malformed numbers and conversion range failures remain errors. Surrounding
spaces, signed NaNs and NaN payloads do not become missing values. Without
--narm, the ordinary numeric conversion rules apply.
Omission is local to each request. Another weighted pair or an unweighted operation can still use the same record. Grouping keys and group changes are observed before omission. A nonempty dataset or group with only zero or omitted weights fails with status 1, identifying the weighted pair and, when grouped, its keys. It does not silently disappear or produce a placeholder mean. Completion errors do not invent a source record location. Earlier completed groups, a header or part of the failing output record may already have been written. Input, output and close failures also prevent successful status.
Numerical trace and limits
The calculation uses 80-bit software arithmetic, without narrowing operands to 64-bit floating point. It starts both totals at positive zero. For each retained positive-weight pair in encounter order it rounds the weight-times-value product, adds and rounds that product into the weighted total, then adds and rounds the weight into the weight total. Completion divides the totals with one rounding. Each step uses nearest-even rounding. There is no fused product/addition or rescaling. Results can depend on record order. Arbitrary weight scaling is not promised to preserve every rounded bit.
A numerical capacity refusal with status 77 identifies a finite product that overflows or rounds a nonzero value to zero, an overflowing finite weighted total, an overflowing weight total, or an overflowing finite final division. The check happens when the intermediate is formed, even if later records might cancel it. Deliberate infinite value inputs are distinguished from finite overflow. Finite subnormal intermediates, ordinary rounding and cancellation, and final-result underflow remain permitted.
These limits can refuse finite inputs whose mathematical mean is representable.
Value 2 with weight 1e4932 refuses at its product. Value 1e-4000 with weight
1e-4000 refuses when its nonzero product rounds to zero. Scaling weights in
advance may change the evaluation trace and rounded result.
Unsorted wmean retains two numeric totals per request, independent of input
record count. Full row context and other requested operations keep their own
storage policies. This is not a total-process-memory or performance comparison.
Grouping and sorting
Groups
-g KEYS (also written groupby KEYS, grouping KEYS or gb KEYS) makes
Fastmash summarize each group of records that share the same key values.
Output rows start with the key values, followed by the results:
$ printf 'A\tx\t1\nA\tx\t2\nA\ty\t5\nB\ty\t4\n' | fastmash -g 1,2 sum 3
Grouped summary
────────────────────────────────
field-1 field-2 sum(field-3)
A x 3
A y 5
B y 4
✔ Completed
A group is a run of adjacent records with equal keys. When the key changes, the group ends and its results are printed. This makes grouping a single pass over the input, with memory proportional to one group.
If equal keys are not adjacent, the same key appears in more than one group:
$ printf 'b\t2\na\t1\nb\t3\n' | fastmash -g 1 sum 2
Grouped summary
────────────────────────────────
field-1 sum(field-2)
b 2
a 1
b 3
✔ Completed
Sorting first: -s
-s sorts the records by their keys before grouping, so every key forms one group:
$ printf 'b\t2\na\t1\nb\t3\n' | fastmash -s -g 1 sum 2
Grouped summary
────────────────────────────────
field-1 sum(field-2)
a 1
b 5
✔ Completed
The sort is stable: records with equal keys keep their input order, so
first, last and collapse give predictable results.
Key order depends on the locale. In C and C.UTF-8,
keys are ordered byte by byte. In the supported language locales, such as
en_US.UTF-8, fr_FR.UTF-8 or pl_PL.UTF-8, keys are ordered alphabetically,
as GNU sort orders them, using built-in Unicode collation rules and glibc’s
table of the punctuation and symbols it ignores.
If your input is already sorted, leave out -s: the result is the same and
the job is faster.
Large inputs
Fastmash sorts inside its own executable. It holds a sort buffer in memory and
writes sorted runs to temporary files when needed. Windows uses its system
temporary directory. On Linux and macOS, TMPDIR selects the directory,
with /tmp as the default. The operating system removes the files
when Fastmash exits, including after interruption.
For eligible input with repeated keys, hash grouping collects operation state per key and sorts only the groups. If this cannot preserve sorted behavior, it rereads the input and sorts it before writing results. Language-locale pipes can retain input for that replay. C-locale pipes normally sort directly. File and pipe input can therefore have different time and memory costs. A file that changes between readings contributes its second-reading contents.
Sorting can spill, but the values retained by statistics such as quantiles stay in memory. A sort-memory target is not a total-process memory limit. See memory and disk requirements.
Resource controls
FASTMASH_SORT_MEMORY_BYTES sets the sort-buffer memory target, and
FASTMASH_SORT_TRACE (set to anything) reports each decision on standard
error. Neither is a limit on Fastmash’s total memory use.
FASTMASH_GROUPING=sort always sorts. For piped input,
FASTMASH_PIPE_GROUPING=sort selects sorting and =hash permits hash grouping
in every supported locale. --vnlog and rand always sort. These controls
change the route, not the result.
When the selected calculations support parallel sorting, records are prepared
on up to 8 threads. On Linux, numerical calculations
can merge spilled records and calculate in parallel on two threads. Both use
the processors available to Fastmash, or OMP_NUM_THREADS if it is set,
capped by OMP_THREAD_LIMIT. Set OMP_NUM_THREADS=1 to sort on one thread.
Sorted rmdup keeps complete records and stable key order, with the same
temporary-storage policy. Case folding affects ordering, while duplicate keys
are compared exactly. The architecture
describes buffer sizing, parallel preparation and replay eligibility.
Ignoring case: -i
-i compares keys ignoring ASCII letter case, for grouping and sorting. The
output shows each key as it first appeared. -i also affects countunique
and unique. Under the Turkic character types (tr_TR, az_AZ and a few
others), where glibc does not fold i and I, -i is refused. Set
LC_CTYPE=C.UTF-8 to fold ASCII letters.
Pivot tables: crosstab
crosstab KEY1,KEY2 (or ct) builds a two-way table with one row per value of the first
key and one column per value of the second. By default each cell is a count. You can name one operation instead:
$ printf 'a\tx\t1\na\ty\t2\nb\tx\t3\n' | fastmash -s crosstab 1,2 sum 3
x y
a 1 2
b 3 N/A
Missing combinations show the --filler text (default N/A). Rows and columns
are ordered byte by byte. Use -s unless each combination’s records are already
adjacent.
Highest and lowest records
Use top:N FIELD to keep the N records with the highest values in one numeric
field. bottom:N FIELD keeps the lowest. Each selected record retains all of
its fields.
Select the highest records
$ printf 'A\t8\tfirst observation\nB\t10\tsecond observation\nC\t10\tthird observation\nD\t9\tfourth observation\n' | fastmash top:2 2
Top-N selection
────────────────────────────────
field-1 field-2 field-3
B 10 second observation
C 10 third observation
✔ Completed
B and C tie at 10. Earlier input records win ties, including at the cutoff. The command emits at most N records in rank order. Duplicates remain separate observations, and fewer eligible records produce fewer results.
Use a field number or, with -H or --header-in, a field name. Lists, ranges
and field pairs are not accepted for ranking. Selection cannot be combined
with calculations or other modes.
Ranking and missing values
Ranking uses the active numerical profile and supported locale, including its binary80 precision. Numbers that convert to the same value tie, regardless of their spelling. Both zero signs tie. Infinities sort beyond finite values in their numeric direction. Any unskipped NaN is an error because it is unordered. Only the ranking field must be numeric. Accompanying fields are copied without numeric checks.
With --narm, records whose ranking field is exactly NA, N/A or NaN, in
any letter case, are omitted. Matching does not trim whitespace. Empty or absent
fields, malformed numbers, range failures, signed NaNs and NaN payloads remain
errors. Empty and all-missing datasets emit no data records. Every accepted
record is checked, including late records that cannot enter the selection.
Copied fields and CSV
Selection copies complete logical fields in their original order, retaining
source numeric spelling, trailing empty fields, ragged widths and arbitrary
accompanying bytes. It adds no score, rank or result column. Ordinary separators,
whitespace input, output delimiters, comment filtering and NUL-terminated records
use the existing controls. Whitespace separator runs become the effective output
delimiter, and a final unterminated record receives an output terminator.
Ordinary output does not escape embedded output delimiters or record terminators
in copied fields.
Use --csv-out to encode selected text records as CSV, including sorted text
groups. Use --csv-in to select decoded CSV records with ordinary output, or
--csv for CSV input and output together. These exact-only switches are
idempotent and do not enable headers.
CSV follows the shared input and output rules. Every field is checked, including fields on omitted records or candidates that cannot win. Output preserves complete decoded fields and numeric spelling, with canonical CSV quoting.
--full is accepted with the same output and no deprecation warning. -s
without grouping keys leaves this whole-dataset selection unchanged.
Selection within groups
With -g FIELD[,FIELD...], each consecutive group has its own cutoff. Groups
appear in encounter order, and rank order applies within each group. For
example, fastmash -H -g category top:2 reading selects the highest two
records in each adjacent category, binding category and reading from the
input header. A later repeated category remains a separate group. Selected
records retain every field once, without an extra grouping-key prefix.
Multiple keys and -i follow the existing group equality and supported locale
controls. No-key -i leaves numeric ranking unchanged. Empty key text is a key
value, while an absent required key is an error. Every required key and group
boundary is checked before --narm omission, so an all-missing intervening
group still separates equal-key runs on either side.
CSV keys compare decoded field values, so different quote spellings of the
same text belong to the same adjacent group.
Add -s to prepare interleaved categories under the existing sorted group
workflow, for example fastmash -s -H -g category top:2 reading. Groups then
follow the existing key order, with ranking applied inside each group. Sorting
keeps complete records and original input identity through memory and spill,
so earlier source records still win equal-rank ties. Sorting order and group
equality retain their existing distinct meanings: whitespace key separators,
case controls, language-locale spellings and NUL-terminated input can affect
the prepared runs without introducing global key normalization. Selection uses
the existing native sort for text and decoded CSV sort for quoted CSV.
Pipe and regular-file input use the same selection route for each format.
For example, fastmash --csv -s -H -g category top:2 reading selects the
highest two records per category from an interleaved CSV export. Sorting uses
decoded key bytes, so quoted and unquoted spellings of the same key sort
together. Each selected record keeps its complete decoded fields, including
multiline notes, through sorting and spill. CSV output encodes them again.
Locale ordering still does not merge differently spelled keys that existing
group equality treats separately. No global key normalization is introduced.
Headers
--header-in consumes the first accepted record as the input header and
excludes it from ranking. Names resolve to the first matching label when a
header contains duplicates. --header-out emits one copied-field output header
for the whole command, without operation wrappers or appended group labels.
-H selects both controls. With an input header, copied labels follow the
existing display convention, stopping at the first NUL byte. Without one,
labels are field-1, field-2 and so on, using the first accepted source data
record’s width even when that record is omitted or does not win. Later ragged
records do not revise the header, and sorting does not change its source schema.
Positional ranking fields may extend beyond
the input header’s labels when the data supplies them. Empty input invents no
header. Header-only input with both controls and all-missing nonempty input can
emit a requested header without data. Name lookup fails before header output.
Completion and failures
The entire input must be inspected before ungrouped data is emitted. Completed adjacent groups can emit as the next group begins. A read failure or malformed record does not emit the incomplete current selection, but earlier groups and the output header may remain. Output writes and finalization must succeed. A later output failure can leave partial bytes. Sorted grouping finishes input intake before emitting selected data. A read or temporary-sort failure emits no incomplete sorted group. Later ranking or key errors can leave already completed groups. Diagnostics retain original accepted record numbers, including the input header and excluding filtered comments. CSV input errors identify the original logical record and its starting physical line. Syntax errors also retain their detected location.
Memory and limits
N is a mandatory positive decimal integer, at most 18446744073709551615.
Leading zeros are accepted. At most N candidate records are retained for the current group or whole dataset,
with storage growing as records arrive and candidate state released between
groups. A large requested N does not allocate N records in advance. N limits
candidate count, not bytes or total process memory: large records or a large N
can still exhaust memory. Checked resource failures refuse with status 77.
Fastmash does not truncate fields, reduce N or approximate the winners.
Sorted grouping has a separate sort-memory target, controlled by
FASTMASH_SORT_MEMORY_BYTES, and spills input when needed under the maintained
sort policy. Candidate retention remains bounded by N for the active group
and does not spill. Neither limit is a guarantee of total process memory.
Option compatibility
CSV input conflicts with explicit text-input separators and
comment filtering. Any CSV format conflicts with NUL termination and --vnlog.
CSV output also conflicts with an explicit output delimiter. Format conflicts
are errors with status 1, independently of option order. CSV input with ordinary
output can use an explicit output delimiter, subject to ordinary framing limits.
A named field without -H or --header-in is a command error.
Explicit result names, numeric presentation controls, collapse delimiter,
filler, random seed, --no-strict, --vnlog and --sort-cmd are unsupported
for selection and refuse with status 77. Malformed options and conflicting CSV
controls retain error status 1. Ordinary commands keep their existing behavior.
Dataset comparison
Compare statistical summaries from two exports, including categories that were added or removed. For example, create two small sales exports:
printf 'category,revenue,weight\nwest,100,1\neast,50,2\n' > before.csv
printf 'category,revenue,weight\neast,75,2\nwest,80,1\nnorth,20,1\n' > after.csv
Compare revenue by category:
fastmash --csv -H -g category compare before.csv after.csv sum revenue
key(category),presence,rank,rank_state,before(sum(revenue)),after(sum(revenue)),difference(sum(revenue)),difference_state(sum(revenue)),percentage(sum(revenue)),percentage_state(sum(revenue))
east,matched,"",not_requested,50,75,25,available,50,available
north,added,"",not_requested,"",20,"",not_matched,"",not_matched
west,matched,"",not_requested,100,80,-20,available,-20,available
The east summary rises from 50 to 75, a change of 25 or 50%. west falls by
20%, and north appears as an added category with no before value. Without -g, the command
compares one summary for each whole dataset.
Source files and supported operations
The examples below also use tab-separated copies of the same exports:
printf 'category\trevenue\tweight\nwest\t100\t1\neast\t50\t2\n' > before.tsv
printf 'category\trevenue\tweight\neast\t75\t2\nwest\t80\t1\nnorth\t20\t1\n' > after.tsv
BEFORE and AFTER are literal paths. A standalone - supplies stdin on one side:
printf 'category\trevenue\tweight\nwest\t80\t1\n' | fastmash -H -g category compare before.tsv - sum revenue
Files are opened independently, even when both paths are identical. The before
source is read first, followed by after. Files are read as supplied, without a
snapshot or locking. Use the ordinary -- option terminator for paths beginning
with -. The same separator, whitespace and comment controls, record terminator,
numeric locale, missing-value policy and operations apply to both sides.
Comparison supports whole datasets and positional or named comparison keys with
single, list, range and paired operation selectors. All numerical aggregates, existing
aliases and parameters are supported, including wmean VALUE:WEIGHT.
Text-result aggregates such as first, unique and collapse are refused.
Input and output formats
Input and output formats are explicit and independent. The same input format applies to both sources. Comparison does not infer a format or support different formats on the two sides.
fastmash --csv-in -H -g 1 compare before.csv after.csv sum 2
fastmash --csv-out -H -g 1 compare before.tsv after.tsv sum 2
fastmash --csv -H -g category compare before.csv after.csv rank 1 percent limit 10 sum revenue
--csv-in reads strict comma-separated byte fields and retains ordinary text
output. --csv-out encodes each complete output field as CSV, with commas and
LF record endings. --csv enables both. These switches are exact-only,
repeatable and header-neutral. Text input still supports its ordinary separator,
whitespace, comment filtering and NUL record termination where allowed.
CSV uses the shared syntax and encoding rules. Names, numeric values and comparison keys use decoded fields, so different quote spellings of the same key consolidate. Every field is checked, including unused content, omitted pairs and records below a ranking cutoff. Output encodes each complete report field independently, quoting empty cells and locale decimal commas.
CSV input conflicts with -t, -W and -C. Either CSV direction conflicts
with -z and --vnlog. CSV output also conflicts with an explicit
--output-delimiter. These are command errors regardless of option order.
CSV-to-text output permits an explicit delimiter, but embedded delimiters or
line breaks can make text ambiguous. Use CSV output to preserve field boundaries.
Input and output headers
Use --header-in to consume the first accepted input-header record on each
source. -H also requests an output header. A header-only source is valid and
has no summary. An entirely empty source, including one containing only filtered
comments, is an error when input headers are requested. Named fields must resolve
on both headers even when a source has no data. Headers are never inferred.
Named selectors bind independently, so the same names can occupy different positions in the two exports:
fastmash -H -g category compare before.tsv after.tsv sum revenue
fastmash -H -g category compare before.tsv after.tsv wmean revenue:weight
Names use ordinary exact, case-sensitive lookup. The first duplicate label wins
and backslash escapes one byte, for example metric\,one. Positional and mixed
selectors retain the requested positions on both sides. Unselected header fields
may differ. With only --header-in, positional data fields may extend beyond the
header labels. When producing an output header, selected labels in its preferred
source must exist under the ordinary header renderer’s rules. Unused after
labels do not constrain positional data.
--header-out or -H requests exactly one fixed output header, including when
both sources have no data. There is no implicit comparison header. Labels use
the before input header when available, otherwise ordinary field-N labels.
Keys are labeled key(FIELD_LABEL), followed by presence, rank and
rank_state. For each ordinary generated summary label L, the six labels are
before(L), after(L), difference(L), difference_state(L), percentage(L)
and percentage_state(L). Label display retains ordinary conventions, including
stopping at NUL. This never truncates comparison-key identity or spelling.
Result names replace L throughout one summary’s complete block:
fastmash -H --result-name=1:total compare before.tsv after.tsv sum revenue mean revenue
fastmash -H --result-name=2:total compare before.tsv after.tsv sum 2,3
Indexes count expanded summary results in request order, once per result. They do not count the six derived columns, rename keys or control columns, or identify a ranking result by name. Names must be nonempty. Each index can be supplied once, while different results may share a name. Text names must not contain the output delimiter or record terminator. CSV output permits and encodes those bytes. Numerical formatting applies to summary and change values independently of their labels.
Both source headers, required named binding, every input and all calculations must succeed before this output header is written.
Report columns
Without keys, each nonempty dataset has one summary. Output has presence,
rank and rank_state, followed by six columns per expanded result in request
order. Without ranking, every rank is empty and every rank state is
not_requested:
| Column | Meaning |
|---|---|
| before | Completed numerical summary from BEFORE |
| after | Completed numerical summary from AFTER |
| difference | Signed after minus before |
| difference_state | available, unordered or not_matched |
| percentage | Signed ordinary percentage change, without a percent suffix |
| percentage_state | available, not_matched, zero_baseline, nonfinite_baseline or nonfinite_after |
Presence is matched, added or removed. An empty source has no summary. An absent side and both changes have empty cells with not_matched change
states. Two empty sources produce no data. Accepted records still establish a
summary when --narm omits every value. Those operations keep their existing
empty-sample results or completion errors, including weighted mean’s requirement
for positive retained weight. Paired operations retain their established
independent compaction. Weighted mean validates partners and omits whole pairs.
Comparison keys
Use the existing -g selectors to identify corresponding categories:
fastmash -H -g 1 compare before.tsv after.tsv sum 2 mean 2
fastmash -H -g 1,3 compare before.tsv after.tsv wmean 2:3
Every distinct key has one summary per source, even when its records are
interleaved. Each repeated raw record contributes normally. All operations keep
the original encounter order within each key, including rounding-sensitive
weighted mean calculations. -s is accepted and gives exactly the same report.
This consolidation differs from ordinary adjacent groups. It does not change
their equality or the locale ordering of ordinary sorted groups.
A key is an ordered tuple of complete field byte strings. Field boundaries,
lengths and bytes after NUL all matter: ('a', 'bc') differs from ('ab', 'c'),
and 01 differs from 1. Whitespace is retained. -i folds only ASCII A
through Z across complete fields. Non-ASCII bytes remain unchanged. Empty key
text is valid, while a missing required key field is an error. Keys are extracted
before any operation omits missing values, so all-omitted categories remain in
the report and keep their ordinary completion results or errors.
Reports align the complete union of keys after both sources finish. Keys precede
the report columns above. A matched or removed key displays its first before
spelling. An added key displays its first after spelling, including under -i.
Unranked reports follow lexicographic order over normalized key tuples, using
unsigned bytes and shorter prefixes first. This order is independent of locale
collation and input adjacency.
For a finite nonzero baseline, percentage is (after - before) / before * 100.
The denominator retains its sign: -100 changing to -50 has difference 50 and
percentage -50. A change from -100 to -150 has difference -50 and percentage 50.
Both zero signs make percentage unavailable. Reasons are checked in this order:
not_matched, zero_baseline, nonfinite_baseline, nonfinite_after.
NaNs and infinities remain visible in the before and after summary cells.
NaN differences, including equal-sign infinities, are unordered. Other infinite
differences remain available.
Ranking and limits
Place rank RESULT [absolute|percent] after the source paths and before all
operations. The default measure is absolute. RESULT selects one 1-based
expanded summary result, counting list and range expansion in request order. Result names and the six derived columns do not affect its index. All requested
result blocks remain in the report:
fastmash -H -g 1 compare before.tsv after.tsv rank 1 sum 2
fastmash -H -g 1 compare before.tsv after.tsv rank 2 percent limit 10 sum 2-3
fastmash -H -g category --result-name=1:total compare before.tsv after.tsv rank 1 percent sum revenue
Ranking orders matched keys by descending magnitude of the selected available difference or percentage. Signed cells keep their signs, including negative baseline percentages. For example, A changing from 100 to 130 ranks ahead of B changing from 10 to 20 by absolute change (30 versus 10). Percentage ranking puts B first (100 versus 30). Matched zero changes are eligible. Available infinite differences rank above finite magnitudes in absolute ranking. NaNs and unavailable changes never participate.
Equal magnitudes, including opposite signs, both zero signs and infinities, tie by the normalized complete comparison-key order. Input order and displayed rounding cannot change winners. Ranks are consecutive plain unsigned decimal numbers, even with hexadecimal or other numerical formats.
An optional limit N follows ranking and keeps at most N eligible keys, without
padding. Eligible keys below the cutoff are omitted. Keys that cannot be ranked
still appear after the ranked portion in canonical key order, with an empty rank and
rank state one_sided for added/removed keys or unavailable otherwise. The
selected result’s change-state columns provide the exact reason. Thus N bounds
ranked keys, and total report records can exceed N. With no limit, every
rankable key appears. An entirely unrankable dataset still reports every key.
RESULT and N accept leading zeros, reject signs and overflow, and must be positive. N can be as large as 18446744073709551615, without reserving N entries in advance. Ranking and limit occur once in that order. Limit requires ranking, and neither modifier can follow or interrupt operations. Invalid modifiers or result indexes fail before sources are opened. Every source, key, requested summary and change must finish before cutoff selection and output, so a limit cannot hide a later error or a failure in an unselected result.
Precision and completion
Changes use completed binary80 numerical values. Subtraction, division by the signed baseline and multiplication by exact 100 each round nearest-even, independently. Displayed values are never used as calculation inputs. Current numeric formatting and rounding controls apply to every numerical cell. Finite overflow in any new arithmetic stage refuses with status 77 and names the result and stage. Subnormal differences and exact cancellation are allowed. Distinct completed binary80 summaries have a nonzero relative difference of at least 2^-64, so percentage division and multiplication cannot underflow. Equal summaries produce exact zero. The ordered trace can refuse extreme pairs even when their final mathematical percentage would fit, and it does not promise one correctly rounded evaluation of the full formula. Existing summary operations retain their own numerical limits.
Every requested summary and change, both source scans and input completion must succeed before comparison writes stdout, including headers. Source errors identify before or after and the supplied path. CSV input errors retain the original logical record and its starting physical line. Syntax failures also retain the detected physical line and byte when available. Completion/change failures name the side where applicable, result and key without inventing an input record. Output write or finalization errors prevent success and can leave partial output. Key text, per-key operation state and required statistical samples remain in memory under existing checked resource limits. Comparison streams intake without retaining complete raw records for consolidation. It does not spill samples, approximate summaries or promise a fixed total-memory bound.
Comparison refuses full rows, other modes, --vnlog, --no-strict, explicit
filler, collapse delimiter, random seed and --sort-cmd. Malformed commands and
conflicting format controls are errors with status 1.
Table health reports
Use health to inspect a table for inconsistent widths, missing values and
mixed value types. Inspection leaves the input unchanged.
Inspect a table
Here price contains both a number and text:
printf 'item\tprice\na\t10\nb\toops\n' | fastmash --header-in health examples 1
The report shows the field observations and the finding:
Table health report
⚠ Inspection complete · advisory findings
Data records: 2 · header records: 1 · accepted records: 3
Baseline width: 2 fields (input header)
Maximum data-record width: 2 fields
Field observations (data-record cells):
field 1, name "item"
Absent 0 · empty 0 · missing 0
Integer 0 · other number 0 · text 2
Expected type: text (inferred)
field 2, name "price"
Absent 0 · empty 0 · missing 0
Integer 1 · other number 0 · text 1
Expected type: undetermined (observation)
Findings (counts may overlap):
Advisory findings do not fail validation. Violations of supplied rules do.
⚠ Mixed value types [observation, advisory]
1 field; field 2, name "price"
examples retained 1, omitted 1
Check whether this field should contain numbers or text, then supply a type rule if needed.
accepted record 2, field 2: observed integer; sample "10"; truncated no
Input and reporting
Field separator: tab (\t)
Records end with: newline (\n)
Numeric decimal separator: "."
Input header: yes
Comment filtering: none
Annotated input (--vnlog): no
Record numbers start at 1, include the input header and exclude filtered records.
Example limit: 1 per finding
Sample byte limit: 128 raw bytes
The finding is advisory, so inspection succeeds. If you know that price must
be numeric, supply that rule and request validation:
printf 'item\tprice\na\t10\nb\toops\n' | fastmash --header-in health type price number validate
Validation returns status 1 for the declared violation. Inferred findings alone
do not make validation fail. required FIELD checks that a value is present,
and nonmissing FIELD also rejects NA-like missing indicators.
Use examples N to choose the number of examples per finding, or examples 0
to omit them. For script-friendly output, add tsv and use the
versioned report schema below. Health accepts ordinary
text input, not quoted CSV.
Report contents
The table health report states the data-record, header-record and accepted-record
counts, effective input settings, numeric decimal separator, baseline width,
maximum data-record width and example limits. Readable reports explain each
finding and suggest what to check. Empty and header-only reports state that
there are no data-record values to check. width N supplies the baseline and
checks both data records and an existing input header. N can be zero. Width does
not declare individual fields. Without a supplied width, an input header supplies
labels and the baseline rather than data values. Without a header, the baseline
is the most frequent data-record width across the complete input, choosing the
smaller width on a tie. With neither a supplied width, data nor a header, the
baseline is unavailable.
Record width mismatch counts data records that differ from the selected baseline. For a supplied width it is a declared violation. Otherwise it is advisory. Header width mismatch separately counts an input header disagreeing with a supplied width, including when there are no data records. Each empty header position produces a Blank header names finding. Each position sharing a nonempty exact-byte label produces a Repeated header names finding. Both findings are advisory. A complete report succeeds even when findings exist. Record and header-field counts and cell counts can overlap. Adding them does not give a count of distinct bad records.
observation describes what the input contains, inferred marks a pattern found
in the input, and declared marks a rule you supplied.
Missing values and lexical types
Every field appearing in the input header, a data record or a positional
declaration has absent_values, empty_values, missing_values, integer_values,
other_number_values and text_values counters, including zero counts. The six
categories partition the data-record count for each tracked field.
Readable reports use ordinary labels such as absent and Expected type.
Readable finding titles also use ordinary words. TSV output keeps the identifiers
documented in its schema.
An absent field is a position beyond a record’s width. An empty field is present with
zero-length bytes. missing_values counts exact whole-field NA, N/A and NaN
tokens in any letter case, without trimming. Whitespace-only values and tokens
with extra characters are separate values.
An integer encoding allows the numeric grammar’s leading ASCII whitespace, an
optional sign and decimal digits consuming the whole field. Fractional, exponent
and hexadecimal encodings are other numbers even when their mathematical value
would be integral. Numbers must consume the whole field under the effective
locale’s numeric grammar, including recognized infinity and NaN encodings such
as -inf and nan(payload). A numeric prefix followed by extra bytes, trailing
whitespace or an embedded NUL is text. Classification uses syntax, without
conversion, rounding or arithmetic magnitude limits.
Absent, empty and candidate-missing values do not participate in inference.
If numeric values outnumber text, the inferred health expectation is integer
when all numbers have integer encodings, otherwise number. If text outnumbers
numbers, it is text. A tie or no eligible values leaves it undetermined.
The report labels these expectations as inferred and shows their counts. Undetermined is an observation. These guesses cannot establish the intended
meaning of an identifier or distinguish a legitimate code from a missing token.
Positive counters produce Absent fields, Empty field values and Missing value indicators findings in that order, each counting data-record cells at its field position. These are observations: an NA-like token can be a legitimate code. Inspection does not change values or remove records. The input header is excluded from cell counts, and header-only input has zero observations. Fields appearing late include earlier absent positions. Positions outside the header have an empty label.
Mixed value types follows the missingness findings and counts one field whenever numeric and text encodings coexist. Type mismatch follows it and counts text cells incompatible with an inferred integer or number expectation. Inferred text does not reject numbers, and mixed integer/fractional numbers can share a number expectation. Both findings remain advisory, including an inconclusive mixed-type observation when numeric and text counts tie.
Supplied rules and validation
type FIELD integer, type FIELD number and type FIELD text supply a known
health expectation, replacing inference for that field. Integer and number use
the lexical rules above. Text permits arbitrary field bytes. Type checks exempt
absent, empty and candidate-missing values. A supplied integer rule rejects other
numeric encodings as well as text, even when their mathematical value is integral.
Mismatches are declared Type mismatch violations. The field’s inferred mismatch
is suppressed, while its counters and Mixed value types observation remain visible.
required FIELD rejects absent and empty cells. nonmissing FIELD additionally
rejects candidate-missing indicators and works without required. When both are
supplied, nonmissing is the one effective presence rule. One declared
Presence rule not met finding per field counts failed cells without duplicating
the rules. A literal NA code can pass required/text checks and fail nonmissing.
FIELD follows the existing single-selector spelling, without lists, ranges or
pairs. In ordinary input, bare positive decimal integers are 1-based positions. Identifiers select exact input-header names. Existing backslash escaping permits
special bytes and numeric names, such as \01 for a header named 01.
Under vnlog, numeric selectors remain names and never fall back to positions.
Named rules require an input header and a unique full-byte match. Missing and
duplicated names are command errors before data processing or report output.
Ordinary positions remain usable with duplicated headers. Names and positions
resolving to the same field share their declarations and conflict checks.
Never-observed positional declarations are tracked without inventing intervening
fields or allocating through the largest declared position.
All controls may occur in any order. Identical declarations are idempotent. Conflicting types, widths or example limits are command errors. Repeated
validate switches are harmless. validate requires at least one supplied
type, presence or width rule. It emits the same complete report as ordinary
inspection and returns status 1 for declared violations, otherwise 0.
Observations and inferred suspicions alone do not fail validation. Empty input
has zero data records. Header-only input has no per-cell or data-width violations,
while a supplied width still checks its header. There is no minimum-record rule.
Examples and locations
examples N retains the earliest N examples per finding, with three by default.
Mixed types reserve the earliest numeric and earliest text witnesses when N is at
least two, then fill remaining slots with the earliest unused eligible cells.
With N equal to one, they retain the earliest eligible cell. Examples are printed
in input order. Mixed-type omissions count eligible numeric/text cells minus
retained examples, separately from the finding’s count of one field.
N is a nonnegative decimal integer. Zero suppresses examples. Repeating the same
limit is harmless, while conflicting limits are command errors. Each sample
retains at most 128 raw bytes. The report discloses retained and omitted examples
and marks truncated samples. Header labels remain complete metadata.
Locations are 1-based accepted-record ordinals, including the input header and
excluding filtered comments and annotations. They are not physical line numbers.
Header findings include their field position. Width samples contain the raw
record without its terminator. Header-width samples preserve the raw accepted
header, including vnlog’s prefix and trailing blanks, within the same byte limit.
Field observations use their accepted-record ordinal and field position. Absent and empty samples contain no bytes and state
which condition was observed. Backslash, TAB, LF, CR and NUL are escaped as
\\, \t, \n, \r and \0. Other bytes outside printable ASCII use
\xHH with uppercase hexadecimal digits. This keeps binary bytes visible.
For ordinary text input, select a field shown with byte escapes by its field
number or original header text. With --vnlog, use the original header name.
Supported input and controls
Input follows the existing record intake and field separation: TAB by default,
literal -t separators, -W whitespace, -z NUL terminators, -C comment
filtering, input headers and vnlog. -H/--headers contributes its input-header
meaning. The report supplies its own headings. Empty records can have zero
fields, trailing empty fields are retained, and a final unterminated record is
accepted. Pipes and regular files produce the same report for the same input.
Sorting, grouping, full rows, numerical formatting, missing-value removal,
standalone output headers, result names and result delimiters are calculation
controls and fail in health. Quoted CSV input and output remain unavailable.
Unknown or incomplete controls are command errors.
TSV for scripts
tsv selects the same collected report in schema version 1. Readable output is
the default. The switch can occur anywhere among health controls, and repeating
it is harmless. Validation emits identical report bytes with or without
validate. Declared violations determine its exit status.
TSV output uses the same schema for terminals, files and pipes.
The mandatory header has 13 columns in this order:
row_kind category basis field_index field_name count count_unit record_index expected observed sample sample_truncated examples_omitted
Report records always use TAB separators and LF endings, including with -z
input. Inapplicable cells are empty. Zero counters are 0, integers use unsigned
decimal spelling and field positions are 1-based. String cells use the byte
escapes described above. Decoding them recovers labels and retained samples
without confusing literal escape-looking text with byte escapes. Field names
retain the complete header bytes. Positions beyond the header have an empty name.
The report begins with exactly these summary rows, in this order. Cells not listed are empty.
| Category | Populated cells |
|---|---|
| schema_version | observed: 1 |
| data_records | basis: observation, count, count_unit: data_records |
| header_records | basis: observation, count, count_unit: header_records |
| accepted_records | basis: observation, count, count_unit: records |
| baseline_width | basis: declared or inferred, count, count_unit: fields. All three empty when unavailable |
| baseline_width_source | observed: declared, header, modal or unavailable |
| maximum_observed_width | basis: observation, count, count_unit: fields. count empty with no data records |
| separator_kind | observed: literal or whitespace |
| separator | observed: literal separator bytes, empty for whitespace separation |
| record_terminator | observed: lf or nul |
| numeric_decimal_separator | observed: effective locale’s decimal-separator byte |
| input_header | observed: yes or no for the selected setting |
| comment_filter | observed: none, comments or vnlog |
| vnlog | observed: yes or no |
| record_numbering | observed: accepted_records_1based |
| example_limit | basis: observation, count, count_unit: examples |
| sample_byte_limit | basis: observation, count: 128, count_unit: bytes |
Next, each tracked field in ascending position has six field rows in order:
absent_values, empty_values, missing_values, integer_values,
other_number_values and text_values. Each repeats field_index and field_name,
with basis observation, count and count_unit cells. These counters partition
the data records exactly, including zeros. A type_expectation row follows with
basis declared or inferred and expected integer, number or text. Undetermined instead has basis observation and expected undetermined.
A supplied presence rule adds one presence_requirement row with basis
declared and expected required or nonmissing.
Positive finding rows follow in this category order, then basis order
observation/inferred/declared, then ascending field position:
| Category | Basis | count_unit | expected |
|---|---|---|---|
| header_width_mismatch | declared | header_records | declared width |
| width_mismatch | inferred or declared | data_records | baseline width |
| blank_header | observation | header_fields | empty |
| duplicate_header | observation | header_fields | empty |
| absent_field | observation | cells | empty |
| empty_field | observation | cells | empty |
| missing_indicator | observation | cells | empty |
| mixed_types | observation | fields | empty |
| type_mismatch | inferred or declared | cells | integer or number |
| presence_mismatch | declared | cells | required or nonmissing |
Finding rows populate basis, applicable field identifiers, count, count_unit, expected and examples_omitted. Their record_index, observed, sample and sample_truncated are empty. Width findings have no field identifiers. Do not add overlapping findings into a count of distinct bad records.
Each finding is immediately followed by its retained example rows in input
order. They repeat category, basis, applicable field identifiers and expected,
then populate record_index, observed, sample and sample_truncated (yes or no).
Their count, count_unit and examples_omitted are empty. Width observed values
are field counts, with a raw-record sample. Header observations use blank or
duplicate, with a header-field sample. Field and presence observations use
absent, empty or missing_indicator. Absent/empty samples are empty.
Mixed/type examples use integer, other_number or text.
Samples retain at most 128 raw bytes, independently of the full scan and labels.
Mixed-type omissions count eligible cells rather than the finding’s one field.
No other row categories occur in schema version 1. Future schema changes must
identify their version.
Field counters, width summaries, header bytes and bounded examples remain in memory. Storage can grow with field count, distinct widths and the chosen example limit. There is no fixed memory guarantee or spill backend. Checked allocation or counter capacity failures refuse the command before report emission. A read failure prevents a report, and a failed report write returns failure. No numerical arithmetic is performed during inspection.
Per-row operations and modes
Per-row operations
Per-row operations produce one output record for each input record. They can be combined with each other, but not with summary operations in the same command.
Fields
| Operation | Result |
|---|---|
cut FIELDS, echo FIELDS | Print the selected fields of each record |
$ printf 'a\tb\tc\n' | fastmash cut 3,1
Per-row results
────────────────────────────────
cut(field-3) cut(field-1)
c a
✔ Completed
Numbers
| Operation | Result |
|---|---|
round | Nearest integer, halves away from zero |
floor, ceil | Round down, round up |
trunc | Round toward zero |
frac | Fractional part, keeping the sign |
bin[:WIDTH] | Lower bound of the bucket of width WIDTH (default 100) containing the value |
strbin[:COUNT] | Hash the field’s bytes into a bucket number from 0 to COUNT−1 (default 10) |
getnum[:TYPE] | Extract the first number found in a text field |
getnum types: n digits, i signed integer, p digits and dots (the
default), d signed digits and dots, h hexadecimal, o octal. Text with no
number gives 0.
$ printf 'item-42.5kg\n' | fastmash getnum 1
Per-row results
────────────────────────────────
getnum(field-1)
42.5
✔ Completed
$ printf '17\n183\n' | fastmash bin:50 1
Per-row results
────────────────────────────────
bin(field-1)
0
150
✔ Completed
Text
| Operation | Result |
|---|---|
base64, debase64 | Encode or decode the field as Base64 |
md5, sha1, sha224, sha256, sha384, sha512 | Lowercase hexadecimal checksum of the field’s bytes |
dirname, basename | Directory part, or last component, of a path |
extname, barename | File extension, or file name without it |
MD5 and SHA-1 are provided for compatibility and checksums, not for security.
Path operations work on the text alone. They don’t touch the filesystem.
Field modes
| Mode | Effect |
|---|---|
reverse | Reverse the order of fields in each record |
noop, nop | Read the input and print nothing (useful for validation). With --full, print each record |
$ printf 'a\tb\tc\n' | fastmash reverse
c b a
Table modes
| Mode | Effect |
|---|---|
transpose | Swap rows and columns |
check [N lines] [N fields] | Verify every record has the same number of fields, and optionally the expected counts |
rmdup FIELD, dedup FIELD | Keep the first record for each distinct value of one key field |
health [CONTROLS] | Inspect structural issues, missing values and lexical types, optionally validating supplied rules |
$ printf 'a\tb\n1\t2\n' | fastmash transpose
a 1
b 2
$ printf 'x\t1\ny\t2\nx\t3\n' | fastmash rmdup 1
x 1
y 2
transpose and reverse require every record to have the same number of
fields. With --no-strict they accept ragged input, and transpose fills
missing cells with the --filler text (default N/A).
check prints a short confirmation and exits with status 0 when the table is
consistent, or reports the first inconsistent record and exits with status 1.
Table health reports inspect the complete accepted input,
with readable or schema-version-1 TSV output and bounded examples. Inferred
findings are advisory. validate fails only for violations of supplied type,
presence or width rules. Health accepts ordinary text input, not quoted CSV.
Selecting complete records
top:N FIELD and bottom:N FIELD keep the highest or lowest N records by one
numeric field. They copy every field in rank order, with earlier input records
winning ties. Add -g for a separate cutoff per group and -s for interleaved
keys. Text and quoted CSV, headers and sorted spill are supported. See
Highest and lowest records.
Comparing two datasets
compare BEFORE AFTER OPERATION FIELDS summarizes two raw datasets and reports
before, after, signed difference and percentage for each result. -g aligns
keys globally even when they are interleaved, and named fields bind independently
to each input header. Optional ranking highlights the largest changes while
keeping added, removed and unavailable keys visible. See
Dataset comparison for report columns and limits.
Numbers, output and locales
Use quoted CSV to encode complete output fields,
and --result-name=INDEX:NAME with an output header to replace generated labels.
Reading numbers
Numerical operations accept decimal numbers (42, -3.5, 1e-9),
hexadecimal floating-point (0x1.8p3), inf and nan. Leading blanks are
accepted. Anything else, including trailing blanks or an empty field, is an
error that names the line and field.
Printing numbers
By default, results are printed with up to 14 significant digits, like
printf "%.14g":
$ printf '1\n2\n' | fastmash mean 1 geomean 1
Summary
────────────────────────────────
mean(field-1) geomean(field-1)
1.5 1.4142135623731
✔ Completed
Two options change how results are printed. Neither changes how they are computed.
| Option | Effect |
|---|---|
-R N, --round=N | Print N digits after the decimal point (1 to 50) |
--format=FMT | Use a printf-style format with one %e, %f, %g or %a conversion |
Use --format=%.0f for no decimal places. The round operation instead rounds
each selected input value to an integer. It does not set a result’s display format.
$ printf '1\n2\n' | fastmash -R 3 mean 1
Summary
────────────────────────────────
mean(field-1)
1.500
✔ Completed
$ printf '1234567\n' | fastmash --format "%'.2f" sum 1
Summary
────────────────────────────────
sum(field-1)
1234567.00
✔ Completed
The ' flag groups digits with the locale’s thousands separator and grouping:
1,234,567.00 in en_US.UTF-8, 1.234.567,00 in de_DE.UTF-8,
12,34,567.00 in as_IN.UTF-8. In C it does nothing.
Limits
--format strings longer than 99 bytes are an error (status 1), as in GNU
datamash. So are column and operation names in a command longer than 511
bytes and, when the field delimiter could continue a number (-t ., -t e or
a digit, for example), numeric fields longer than 511 bytes. Otherwise numbers
and results have no fixed length limit.
Precision
Fastmash computes with 80-bit extended precision (a 64-bit significand, about 19 significant decimal digits), the same precision GNU datamash uses on x86-64 Linux. Sums are accumulated in input order.
Portable results
Fastmash implements its arithmetic in software, including its logarithms and exponentials. As a result, one Fastmash version gives identical results on every supported machine, independent of the CPU model or the system’s math library.
This also means Fastmash’s output can occasionally differ from a GNU datamash
build in the last printed digit, or in the sign of a nan. See
Differences from GNU datamash.
The rules for reading, computing and printing numbers are versioned together as a numerical profile. A Fastmash release that changes any numerical output says so in its changelog.
The current profile is portable-binary80-v4. It retains v3 arithmetic,
conversion and random-number generation, and uses ICU4X 2.4’s Unicode 18 and
CLDR 49 collation. Changed group ordering can assign the shared seeded random
draws to different groups, so rand may select another number even with the
same seed. For example, with four values 1 through 4 in each of the groups
U+10EFB and U+10EC9, LC_ALL=en_US.UTF-8 fastmash -S0 -sg1 rand 2 selected 3
for U+10EFB and 4 for U+10EC9 under v3. V4 puts U+10EC9 first and selects 3
for it and 4 for U+10EFB.
Locales
The locale decides how numbers are written and the order of sorted keys. Fastmash has the rules of the glibc locales built in, taken from glibc 2.43, so it does not need locale data installed on the system.
Numbers. In C, POSIX and C.UTF-8, and in the UTF-8 locales glibc
provides (such as en_US.UTF-8, fr_FR.UTF-8, nb_NO.UTF-8 or
hi_IN.UTF-8), numbers are read and printed with that locale’s decimal
separator, and the ' format flag uses its thousands separator and grouping.
An @ modifier glibc has a locale for, such as sr_RS@latin or de_DE@euro,
uses that locale’s rules, which can differ from its base’s. Glibc drops any
other modifier, and so does Fastmash. A character set other than UTF-8 (a
.ISO-8859-1 codeset, or a name such as de_DE whose default is Latin-1)
keeps the locale’s rules where its separators are ASCII, and refuses numbers
where they are not (fr_FR’s thousands separator is U+202F). A name glibc has
no locale for, such as xx_YY.UTF-8, behaves as C, as in GNU datamash. The
one glibc locale whose decimal separator is not a single byte, ps_AF, is not
supported. Thousands separators are never accepted in input, as in GNU
datamash.
Sorted keys.
| Locale | Sorted key order |
|---|---|
C, POSIX, C.UTF-8 | Byte order |
194 glibc locales in 123 languages, such as en_US, de_DE, fr_FR, es_ES, it_IT, pt_BR, nl_NL, pl_PL, ro_RO, hr_HR, ru_RU and he_IL | Alphabetical, as GNU sort under glibc (Unicode collation for letters and digits, glibc’s table for punctuation, symbols and spaces) |
Other locales, such as cs_CZ, da_DK, el_GR, fi_FI, hu_HU, ja_JP, nb_NO, sv_SE, tr_TR, uk_UA and the Chinese locales | Sorting is refused |
A language locale is supported for sorting when Fastmash’s order of its
alphabet (its standard and auxiliary letters in the Unicode CLDR data, alone,
in pairs and in each case, and digits) matches GNU sort under glibc exactly. fastmash --help lists the
languages. Punctuation, symbols and spaces were checked the same way in every
one of them and order as in GNU sort, except next to digits in some keys
(12-A and 1-2A) and for vulgar fractions such as ¼. See
Differences.
Fastmash reads LC_NUMERIC (for numbers), LC_COLLATE (for sorting) and
LC_CTYPE (for quotation marks and -i) from the first non-empty value of
LC_ALL, that variable, and LANG, falling back to C.
With an unsupported locale, commands that
neither read nor print numbers (transpose, cut, first, unique,
checksums and the like, and count, countunique and strbin unless
--format or --round is given)
work normally. Commands that do are
refused with exit status 77 rather than risk reading numbers with the wrong
decimal separator, and the message tells you which setting to change:
fastmash: unsupported numeric locale ‘ps_AF.UTF-8’ from LANG
hint: set LC_NUMERIC=C.UTF-8, or a UTF-8 locale whose decimal separator is one byte
Two situations make GNU datamash differ, because it uses the installed locale data instead:
- If the locale is not installed, GNU datamash (and every other C program)
falls back to
Cfor everything, while Fastmash still uses the locale’s rules. Check withlocale -a. - A different glibc version can have slightly different data: for example,
glibc changed
fr_FR’s thousands separator to a narrow no-break space. This only affects the'format flag and sorting.
For scripts, setting LC_ALL=C.UTF-8 gives the same results everywhere.
Diagnostics are always in English.
Further reference
Terminal presentation covers the terminal display and its controls.
Terminal presentation
Fastmash makes terminal results easier to scan with purple headings, blue result values, green completion marks, amber advisories and red failures. Secondary metadata uses gray. All styles use your terminal’s default background.
Ordinary calculations and Top-N selection show labeled, aligned fields and a
✔ Completed footer after successful processing. Headers appear even without
--header-out. Automatic headers use positional labels such as sum(field-1).
Use -H or --header-out to request labels from an input header. Results still
stream as they become available. Alignment uses at most 32 columns per field,
while labels and values are always printed in full.
A failure can leave earlier results visible, followed by a diagnostic. It does
not receive a completion mark. Numerical values and their formatting are unchanged.
Readable table-health reports put the outcome and findings ahead of input
metadata. ⚠ identifies advisory findings, and ✗ identifies violations of
supplied expectations. Successful validation uses ✔ and explicitly refers to
the supplied expectations. Inferred findings remain advisory even when validation
passes. Field counts, examples, omissions and interpretation details remain in
the report.
Files, pipes and explicit formats
Output sent to a file or another program keeps its existing plain format,
including readable health reports. It gains no headings, padding, symbols or
color codes, even with --color=always. Examples that write a report file show
its data format.
Explicit CSV output, NUL-terminated output, an explicit output delimiter, vnlog, crosstab, dataset comparison, field and table transformations, health TSV and version output retain their data formats on terminals too. This lets you inspect those formats directly without a presentation layer changing their structure.
Controls
fastmash --color=auto sum 1
fastmash --no-color sum 1
fastmash --color=always --help
auto is the default. Each destination is checked separately. Redirecting stdout
does not disable colored diagnostics on a terminal. TERM=dumb, a nonempty
NO_COLOR, or failed terminal detection disables automatic styling. An empty
NO_COLOR permits it. never and --no-color disable the terminal presentation,
including bold, added headers and completion marks.
always overrides environment suppression on a terminal. It can also force
color in redirected help and diagnostics, but never decorates redirected results
or health reports. Diagnostics style only their program prefix and failure mark,
leaving the explanation, hints and input bytes in the default foreground.
Repeated controls follow the last one reached by ordinary option scanning. --
ends scanning, and POSIXLY_CORRECT stops it at the first operand. Help and version
stop scanning immediately, so put color controls before them.
Errors, exit statuses and resources
Exit statuses
| Status | Meaning |
|---|---|
0 | Success |
1 | Error: a problem with the command or its input, such as an unknown option, a non-numeric value, a missing field or a write failure |
77 | Refusal: Fastmash deliberately declined to produce a result, because a feature or locale is unsupported or a checked resource limit was reached |
70 | Internal failure: Fastmash detected a violation of its numerical invariants. Please report it |
When one failure follows another, such as a failure to write the output after
an error, the status is the more serious one: 70 over 77 over 1. If a
diagnostic itself cannot be written, the status is 1. Internal numerical or
table-identity failures indicate a bug: please report them. The legacy
“internal error: input field too long” diagnostic instead identifies the
numeric field-length limit. SIGABRT can indicate a bug or
an allocation failure that the process could not report as a refusal.
Diagnostics go to standard error, in the form:
fastmash: invalid numeric value in line 3 field 2: 'n/a'
Some are followed by a hint: line suggesting a fix.
Find and fix the cause
The diagnostic identifies the first problem encountered. Check the named field, record, source file or option before rerunning the command.
| Message or situation | What to check |
|---|---|
invalid numeric value | Confirm that the selected field contains numbers and that the field separator is correct. If the first record names the fields, add --header-in |
| A requested field is missing | Field numbers start at 1. Tab is the default separator. Use -W for runs of spaces and tabs, -t, for literal commas, or --csv for quoted CSV |
| A header name is not found | Names are exact and case-sensitive. Check the spelling, separator and whether the first record really is a header |
invalid CSV | Use double quotes around a complete field. Write a quote inside it as two double quotes. The reported record and physical line locate the problem |
| A CSV option conflicts with another option | Remove text-input controls such as -t or -W when reading CSV. CSV output chooses its own comma separator, so omit --output-delimiter |
| A weighted mean has no positive retained weight | At least one retained value/weight pair must have a positive weight. Check zero weights and pairs omitted by --narm |
sort temporary I/O error | Check that the temporary directory exists, is writable and has free space. Choose another writable directory if needed |
| Memory allocation failed | Check available memory and process limits. Sorting can spill to disk, but retained samples and complete tables stay in memory |
| A read or write failed | Check the input/output paths, access permissions and free disk space. A closed or incorrectly opened standard stream must be fixed by the shell or program launching Fastmash |
--narm skips only whole-field NA, N/A and NaN tokens, ignoring case. It
does not skip arbitrary text, empty fields or absent fields. --round and
--format change display only and cannot repair a numerical-range refusal.
The warning about -f/--full with “non-linewise operations” concerns aggregate
operations such as sum. It does not stop the command or change its exit
status. Omit --full if you only need the calculated results. Per-row operations
such as round do not produce this warning.
Always check the exit status
Fastmash writes results progressively through a small output buffer, so a
command that fails part-way may already have printed earlier groups. In scripts, check the exit
status rather than whether output appeared. In Bash pipelines, enable
set -o pipefail so an earlier command’s failure isn’t hidden:
bash -c '
set -o pipefail
if ! fastmash -s -g 1 sum 2 < data.tsv > totals.tsv; then
echo "summary failed" >&2
exit 1
fi
'
Dataset comparison completes both input scans and all calculations before
writing its report, including the header. Output write failures can still leave
partial bytes. Table health likewise completes inspection before report emission;
health ... validate can emit a complete report and return 1 for declared
violations. Advisory findings alone do not make validation fail.
PowerShell 7.4+
Read $LASTEXITCODE immediately after the native command. It uses the
statuses above, including 1 for an error and 77 for a refusal.
To return that status from a .ps1 script, using the
quick start’s CSV file:
cmd /d /v:off /c 'fastmash --csv -H sum amount < "sales.csv"' > totals.csv
exit $LASTEXITCODE
The script returns 0 on success, or the failure status if the command fails.
Treat totals.csv as complete only after success. In a native pipeline,
$LASTEXITCODE reports the last native program’s status. Run stages separately
and check each status when you need to detect a failure in an earlier stage.
Why refusals exist
Fastmash prefers a clear refusal to a doubtful answer. Examples:
- A non-integer percentile such as
perc:2.5. GNU datamash 1.9 reads an unset parameter for it and reports a misleading error. - Multiple key fields for
rmdup, which GNU datamash 1.9 aborts on. - An unsupported locale, where numbers might be read with the wrong decimal separator.
- A calculation that would need more memory than is available.
A refusal is not a claim that GNU datamash would reject the same input.
Memory
Fastmash has no fixed limits on record length, field count, number of
operations or number of values. (The few fixed limits that remain are GNU
datamash’s own, on --format strings, names in a command and, in one narrow
case, numeric fields; see Numbers.) Data storage grows with
checked allocations. If memory runs out, Fastmash refuses with status 77 where
it can. Process initialization, including arguments and configuration, uses
standard Rust allocation behavior and can terminate before reporting a refusal.
The operating system may also stop a process that exhausts memory. In those
cases, the shell or job runner reports the process’s termination status.
What grows with the input:
- Operations that need every value (quantiles, dispersion, paired statistics,
unique,collapse) keep the values of the current group. rmdup,transposeandcrosstabkeep their tables in memory.- Top-N selection keeps at most N candidates for the active group or dataset; N limits record count, not bytes.
- Dataset comparison keeps keys, per-key calculation state and required samples in memory; these do not spill.
- Table health keeps field counters, widths, header labels and bounded examples in memory, with no fixed total-memory guarantee.
-skeeps a sort buffer, spilling to disk beyond the chunk target. Sorted numerical jobs keep the original records until they are processed.
Address-space limits
Some systems limit a process’s address space rather than its memory, for
example ulimit -v or a batch scheduler’s virtual-memory limit such as
h_vmem. The static Linux download uses musl and needs no glibc allocator
settings. If a sorted job refuses under the limit, try OMP_NUM_THREADS=1
to reduce worker memory (see Large inputs).
For glibc source builds, the C library’s memory allocator gives each thread
that allocates an arena of its own, which reserves 64 MiB of address space,
more as it grows, and sorts in language locales, like the heavier
calculations in the C locales, run on up to eight threads.
Under such a limit, Fastmash therefore keeps the allocator to one arena for
each 512 MiB of the limit, from one (below 1 GiB) to eight. Threads that share
an arena wait for each other, so a sort that would also fit without the cap
can take somewhat longer. A number of arenas that GLIBC_TUNABLES
(glibc.malloc.arena_max) or MALLOC_ARENA_MAX sets is used instead, higher
or lower. If a sorted job still refuses under the limit:
GLIBC_TUNABLES=glibc.malloc.arena_max=1keeps the allocator to one arena at a limit of 1 GiB or more too;OMP_NUM_THREADS=1sorts on one thread (see Large inputs).
Disk
On Windows, sorting uses the system temporary directory selected by Windows.
On Linux and macOS, sorting large inputs with -s writes temporary data to
TMPDIR (default /tmp), using anonymous files that the operating system removes automatically.
Where the filesystem has no anonymous files (O_TMPFILE), such as NFS or a
container’s /tmp on Linux before 6.10, Fastmash creates private files with
random names and removes their names at once, so they also disappear when
Fastmash exits, including when it is interrupted. A sort that has to spill to
disk without a usable TMPDIR stops with “sort temporary I/O error” (status 1).
Interruption
An interrupted command can leave partial output. Treat output as complete only when the exit status is 0.
Differences from GNU datamash
Fastmash aims to behave exactly like GNU datamash 1.9. This page lists the known intentional differences and explicit extensions. Other differences in GNU-compatible commands are bugs: please report it.
Comparisons were made against GNU datamash 1.9 on x86-64 Linux. Other GNU versions and builds can differ from each other as well.
Fastmash extensions
These features extend the command language through explicit options, operations or modes. Ordinary GNU-compatible commands keep the compatibility rules on this page. The linked guides describe each extension’s supported combinations and limits. No comparative workflow speed or memory claim is implied.
- Quoted CSV and custom result names add explicit quoted input/output formats and supplied labels for calculation results.
- Table health inspects structural and missing-value observations, inferred types and supplied expectations, with readable and TSV reports and optional validation.
- Top-N selection returns the highest or lowest complete records by one numeric field, retaining original input order for ties.
- Weighted mean adds
wmean VALUE:WEIGHTwith finite, nonnegative contribution weights and checked numerical range limits. - Dataset comparison summarizes two raw datasets and reports their numerical changes, with complete comparison keys and optional ranking.
- Terminal presentation reference describes interactive output behavior.
Numbers
| Area | GNU datamash 1.9 | Fastmash |
|---|---|---|
| Arithmetic | The CPU’s x87 unit and the C math library | Software arithmetic, including log and exp. Identical on every supported machine |
| Last digits | Depend on the build and the math library | Can differ from GNU in the last printed digit for some transcendental results |
| NaN from invalid arithmetic | Prints -nan on x86-64 | Always positive nan |
| NaN between two NaN inputs | Chosen by the x87 unit | The one with the larger payload, positive on a tie |
round, floor, ceil, trunc, frac, bin of -nan | Gives 0 | Keeps -nan |
log and exp at extreme arguments | Computed by the C library | A few boundary cases that cannot be settled within fixed limits are refused (status 77) |
Safety
| Area | GNU datamash 1.9 | Fastmash |
|---|---|---|
perc:100 | Can read past the end of the values | Returns the largest value |
Non-integer percentile, such as perc:2.5 | Reads an unset parameter. On x86-64 builds reports the misleading “invalid percentile value 0” | Refused, status 77, with a clear message |
rmdup with several key fields | Aborts on an assertion | Refused, status 77 |
Empty operation name, a pair whose second field is a range (1:2-3), strbin:2.0, numeric getnum type | Assertion failures or undefined behavior | error, status 1, with a message |
The long --seed option | Mishandles its argument | Requires a value, like -S |
crosstab labels longer than 511 bytes | Truncated | Kept in full |
Unseeded rand when the kernel’s random source fails | Prints Error N and continues | Refused, status 77 |
| Running out of memory | “memory exhausted”, status 1 | Refused, status 77, naming what could not be stored. For an input record, “record memory allocation failed” |
Features
| Area | Fastmash |
|---|---|
--sort-cmd | Unsupported. Sort the input separately |
Explicit line mode | Unsupported. Write the per-row operation directly |
Hidden test options (---print-inf and others) | Unsupported |
| Locales | Bundled rules, with the availability and ordering differences below |
| Quotation marks in messages | Curly quotes in UTF-8 locales. Messages are always in English |
-i | ASCII case folding, excluding column names and duplicate-removal keys. Refused under the Turkic character types az_AZ, crh_UA, ku_TR, tr_CY, tr_TR and tt_RU@iqtelif, where glibc does not fold i and I |
Unwritable TMPDIR | Fails only when sorting needs to spill, with status 1. Spill thresholds differ from GNU sort |
| Size limits | Growable records, operation lists and samples, with the documented compatibility limits |
Temporary storage
Fastmash accounts for available memory and cgroup limits when sizing its sort buffer. GNU sort sizes from input files and host memory, ignoring cgroup limits, and spills sooner on piped input. Hash grouping can avoid sorting records. See grouping and sorting for resource controls and the architecture for buffer sizing.
Locale and sorting differences
Locale availability
Fastmash bundles glibc 2.43 numeric rules. It supports C, POSIX,
C.UTF-8 and every glibc locale with a one-byte decimal separator, including
known @ modifiers. Non-UTF-8 character sets work where their separators are
ASCII. Other numeric separators are refused when a command needs numbers.
Sorting supports the C locales and 194 verified language locales. Unverified
sorting locales are refused. Unknown locale names behave as C in both
programs. For a known locale missing on the host, GNU datamash falls back to
C, while Fastmash uses its bundled rules.
Key order
In supported language locales, letters and digits follow Unicode collation.
Punctuation, symbols and spaces use glibc’s ordering table, including characters
ignored until otherwise equal keys need a tie-breaker. For example, a-1
sorts before a_1 before a1, and aa before a+b.
In Spanish locales, gl_ES and pl_PL, spaces sort before letters and
digits, as in GNU sort. This puts San José before Sanabria.
Punctuation beside digits
glibc skips a character at the accent level in some digit-and-punctuation runs ending before a letter. These keys can therefore appear in different orders:
| Program | Order |
|---|---|
| GNU sort | 12 A, 12-A, 12A, 1-2A |
| Fastmash | 1-2A, 12 A, 12-A, 12A |
| GNU sort | file1.txt, File1.txt, file(1).txt |
| Fastmash | file(1).txt, file1.txt, File1.txt |
A few symbols that glibc weighs rather than ignores also differ, including
¼ versus 1/4, and the numero sign in tk_TM.
Letters and combining marks
Letters outside the language’s alphabet can differ in order. These include Hangul syllables, Georgian capitals, some Arabic letters, CJK Extension A/B, compatibility ideographs and characters newer than glibc’s table.
A precomposed letter sorts after its base plus combining marks, as in GNU:
e with U+0301 precedes é, and Е with U+0308 precedes Ё.
glibc makes exceptions for some letters with two marks, including Vietnamese
ổ and ǖ, putting them first at the end of a key or before a letter.
Fastmash keeps them after the decomposed spelling. They also differ where glibc
uses a second mark before an earlier difference, as in éǖ.
Combining marks written outside Unicode order, such as a with U+0301 then
U+0323, follow Unicode collation in Fastmash rather than glibc’s order.
Tied spellings and grouping
Where glibc treats two spellings as the same letter, the next key decides.
Fully tied keys stay in input order, with one group per run of a spelling.
Examples include и plus a combining breve versus й, L plus a middle
dot versus Ŀ, and some Arabic, Indic, Thai, Lao and Tibetan vowel forms.
In cy_GB, sq_AL and sq_MK, ll followed by a middle dot sorts as
l and ŀ in Fastmash. glibc instead reads the locale’s ll and then
the dot. Such a key ties with lŀ in Fastmash, producing a group per input
run, while GNU orders them apart.
When group order differs, seeded rand can select a different number because
groups share one random stream.
Invalid sorted keys
Language-locale keys must be valid UTF-8 without NUL bytes. Other keys are
refused. With rmdup -s --vnlog, comment records sort with the data, as in
GNU datamash, and a comment can become the second header. A comment whose key is
invalid UTF-8 is therefore refused too.
Diagnostics
Error messages use the same wording as GNU datamash where it matters to
scripts, but some messages differ, and they begin with the program name, fastmash:.
Errors for Fastmash extensions include their own field-selection guidance,
supported modes and examples of valid option syntax.
Common command errors also include a separate hint: line with valid syntax
or a recovery step. The original error wording and exit status are preserved.
Top-N selection errors use their own wording and include a ranking example.
For byte-order recovery, LC_ALL must be unset before LC_COLLATE=C takes
effect. Copy its value to LC_NUMERIC first to keep the same number format.
A few failure paths also differ in detail: for example, a read error in input sorted
with -s can give just “read error: Input/output error”, where GNU datamash
prints the system sort’s message and “read error (on close)”. When standard
input cannot be read at all (opened for writing only), GNU datamash’s -s
prints an error from the shell that runs sort but ends with status 0, where
Fastmash reports the read error with status 1. And --help or --version into a closed pipe exits with status
77 rather than by SIGPIPE. Scripts should rely on exit statuses, not the
exact text of error messages.
Ordinary commands use normal Unix broken-pipe handling. Blocked SIGPIPE and failed signal-mask queries cause a refusal.
With native sorting on Linux and macOS, a command with a named grouping key
or sorted rmdup key and no input header
reports fastmash: missing input header for named grouping key and exits with
status 1, including when comment-only input leaves no header. These failure
paths run inside Fastmash on both platforms. A failed input read still reports
the read error and exits with status 1. The missing-header diagnostic does not
hide it. A command containing wmean still succeeds without output when no
input header arrives at clean EOF. OS-owned diagnostic wording is checked
separately from portable results.
System requirements
The Linux download is statically linked and runs on both glibc and musl distributions, including Alpine. Follow the install steps.
Platform
| Requirement | Why |
|---|---|
Linux (x86_64-unknown-linux-musl) | Static prebuilt binaries, with no glibc version requirement |
Linux (x86_64-unknown-linux-gnu) | Also supported for source builds using the system glibc |
macOS 15 or later (aarch64-apple-darwin) | Supported platform for the native archive, checked on macOS 15 and 26 |
Windows 11 (x86_64-pc-windows-msvc), PowerShell 7.4+ | Native archive, with the C runtime included |
/proc mounted on Linux | Supplies memory and terminal information. Without it, Fastmash uses conservative sort memory and writes output in 8 KiB blocks, even to a terminal |
| A writable temporary directory | Temporary data for large sorted jobs |
See native Windows installation or macOS installation. Intel macOS, older macOS versions, Windows 10 and ARM64 Windows are outside the supported scope.
Sorting runs inside Fastmash and needs no other program. Its temporary directory must be writable when a job spills to disk.
For commands under an address-space limit (ulimit -v, as some batch schedulers set), see
Address-space limits.
See Grouping and sorting for memory use and temporary storage.
Locales
Fastmash has the glibc 2.43 locale rules built in, so you don’t need to install locale data. See Numbers, output and locales.
Verifying a release archive
The installer checks the archive’s SHA-256 checksum. To verify which GitHub workflow built the latest release, use the GitHub CLI to download and verify its archive:
Linux:
gh release download --repo pederbe/fastmash --pattern 'fastmash-*-x86_64-unknown-linux-*.tar.gz' --output fastmash.tar.gz --clobber
gh attestation verify fastmash.tar.gz --repo pederbe/fastmash
macOS:
gh release download --repo pederbe/fastmash --pattern 'fastmash-*-aarch64-apple-darwin.tar.gz' --output fastmash.tar.gz --clobber
gh attestation verify fastmash.tar.gz --repo pederbe/fastmash --signer-workflow pederbe/fastmash/.github/workflows/release.yml
The macOS archive is not Apple-notarized.
Building from source
Use the latest stable Rust and a C compiler (cc, or set CC).
git clone https://github.com/pederbe/fastmash
cd fastmash
rustup target add x86_64-unknown-linux-musl
cargo build --release --locked --target x86_64-unknown-linux-musl
# the program is target/x86_64-unknown-linux-musl/release/fastmash
To build on macOS 15 or later, run this command in Bash:
MACOSX_DEPLOYMENT_TARGET=15.0 cargo build --release --locked --bin fastmash --target aarch64-apple-darwin
# the program is target/aarch64-apple-darwin/release/fastmash
The ordinary Linux cargo build --release --locked command still builds
against the system glibc. The macOS build needs Apple’s Command Line Tools.
Installing the archive needs no Rust compiler or Command Line Tools.
cargo install and source builds use your own compiler and settings. They
omit the prebuilt binaries’ branch-alignment tuning for some Intel processors,
so their speed can differ slightly. Keep --locked: it builds with the
tested dependency versions.
Benchmarks
Fastmash is faster than GNU datamash on quantiles, grouped summaries and number-heavy tables. These Linux benchmarks compare both programs on an Intel laptop and an AMD desktop, using gene annotations, wine measurements and synthetic tables.
Large sorted summaries can be slower than GNU datamash.
Longer bars mean faster runs. The dashed line marks equal speed. See the method and timings for the commands, elapsed times and machines.
The animated demo replays the laptop’s RefGene quartile job. Playback is slowed 20 times so that both runs are visible. Its clocks show the measured times.
Run them yourself
The benchmark kit runs eight of these jobs on your machine with both programs, checks that they agree, and prints a table you can share:
curl -fsSLO https://raw.githubusercontent.com/pederbe/fastmash/main/bench/fastmash-bench.py
python3 fastmash-bench.py
The best benchmark is your own job. Compare both programs on your data:
export LC_ALL=C.UTF-8
time datamash -s -g 1 median 2 < your-data.tsv > /dev/null
time fastmash -s -g 1 median 2 < your-data.tsv > /dev/null
For stable numbers, repeat each command several times, alternate the two programs, and use a quiet machine. hyperfine does this for you:
hyperfine --warmup 2 \
'datamash -s -g 1 median 2 < your-data.tsv' \
'fastmash -s -g 1 median 2 < your-data.tsv'
If Fastmash is slower on a job you care about, please tell us: slow real-world jobs are what we optimize next.
Benchmark method
We compare Fastmash with GNU datamash on the same files and machine. The benchmark kit checks that their output matches before timing either program.
Each job runs in two rounds, with one warm-up and seven timed runs per program in each round. Program order alternates, and job order is shuffled between rounds. The tables show the mean of the two median elapsed times in milliseconds. Lower times are better.
Both programs read regular files with a warm file cache, use the C locale
and write timed output to /dev/null. Temporary files use disk-backed storage.
GNU datamash’s -s commands include the time taken by the system sort.
Machines
| Host | CPU | System |
|---|---|---|
| Laptop | Intel Core i7-8550U | CachyOS, native Linux |
| Desktop | AMD Ryzen 7 9800X3D | CachyOS, native Linux |
Data and commands
The benchmark kit downloads RefGene genome annotations and UCI Wine Quality measurements, and generates synthetic decimal tables from a fixed seed. The remaining jobs exercise small groups, a dominant group and a large sorted summary.
Field numbers in each command refer to its input table. The commands below are identical for both programs.
| Job and arguments | Intel laptop, GNU → Fastmash (ms) | AMD desktop, GNU → Fastmash (ms) |
|---|---|---|
Exon-count quantiles (RefGene)q1 9 median 9 q3 9 iqr 9 | 111.0 → 21.5 (5.16×) | 60.0 → 12.6 (4.76×) |
Transcripts per gene (RefGene)-s -g 13 count 2 collapse 2 | 110.8 → 73.0 (1.52×) | 57.0 → 37.9 (1.50×) |
Exon statistics per gene (RefGene)-s -g 13 count 9 min 9 max 9 mean 9 median 9 | 157.8 → 86.8 (1.82×) | 74.8 → 45.0 (1.66×) |
Many small groups (genes)-s -g 1 count 2 collapse 2 | 159.8 → 127.1 (1.26×) | 66.1 → 53.1 (1.24×) |
Gene example (datamash manual)-H -s -g 13 count 2 | 7.0 → 4.2 (1.67×) | 3.3 → 2.1 (1.57×) |
Sum and mean, 1M decimals-H sum 3 mean 3 | 325.4 → 87.5 (3.72×) | 114.8 → 37.2 (3.09×) |
Sum and mean, 100k decimals-H sum 3 mean 3 | 32.8 → 9.5 (3.45×) | 11.6 → 3.9 (2.97×) |
Grouped decimals, 100k rows-H -s -g 2 count 3 mean 3 median 3 | 68.1 → 38.7 (1.76×) | 27.5 → 18.0 (1.53×) |
One dominant group-s -g 1 count 2 collapse 2 | 21.6 → 9.1 (2.37×) | 9.3 → 3.9 (2.38×) |
Wine quality by grade-t ';' -H -s -g 12 count 1 mean 11 median 11 | 6.0 → 2.6 (2.31×) | 3.0 → 1.2 (2.50×) |
Large sorted summary-s -g 1 count 2 collapse 2 | 547.2 → 495.1 (1.11×) | 204.4 → 240.2 (0.85×) |
White wine summary-t ';' -H mean 9 median 9 min 9 max 9 | 4.2 → 1.9 (2.21×) | 1.7 → 0.9 (1.89×) |
FAQ
Is Fastmash a drop-in replacement for GNU datamash?
For most commands, yes: it accepts the same operations, options and selectors
and aims to produce the same output. Check your scripts against
Differences from GNU datamash, especially the locale
settings and numerical last digits, and keep datamash installed while you compare.
Is it always faster?
Not yet. The published Linux benchmarks show Fastmash faster on many common jobs and slower on some others. Try it on your own data, and tell us about jobs where it is slower.
Why does Fastmash refuse my locale?
Some locales use sorting rules or numeric separators Fastmash does not support.
Set LC_ALL=C.UTF-8 to use decimal points and byte ordering:
LC_ALL=C.UTF-8 fastmash -s -g 1 sum 2 < data.tsv
If you need language-specific ordering or decimal commas, see Locales for supported settings and limitations.
Can it read CSV files?
Yes. --csv-in reads strict quoted CSV, --csv-out encodes output fields, and
--csv selects both. Quoted fields can contain commas, quotes and line breaks. Format switches do not infer headers. Aggregates, per-row operations, Top-N
selection and dataset comparison support CSV, including their supported grouped
workflows. Table health and legacy table/field modes remain ordinary-text only.
-t, still means literal comma splitting, as in GNU datamash. See
Quoted CSV and result names.
Can I check a table before calculating?
fastmash --header-in health < data.tsv reports inconsistent widths, missing
values, mixed lexical types and examples. Add known rules such as
type price number nonmissing id width 3 validate to fail on declared
violations. Inferred findings stay advisory. A versioned TSV report is available
for scripts. See Table health reports.
Can I compare two exports with different column orders?
Yes. With -H, named fields bind independently to each input header.
fastmash -H -g category compare before.tsv after.tsv sum revenue reports each
key’s before, after, signed difference and percentage, including added and
removed keys. Keys consolidate across each whole dataset. They need not be
adjacent. See Dataset comparison.
Can I use Fastmash in pipelines?
Yes. Fastmash reads standard input and writes results for the next program. Results written to a file or another program keep their plain data format.
Does sorting need another program?
No. Fastmash handles all sorting internally.
Why exit status 77?
Status 77 means Fastmash deliberately declined to produce a result: an unsupported feature, locale or condition, or a resource limit. It is distinct from status 1, which means the command or the input is wrong. See Errors, exit statuses and resources.
Why do results sometimes differ from GNU datamash in the last digit?
Fastmash computes with the same 80-bit precision, but in software, including its logarithms and exponentials. The same Fastmash version, input and settings give the same numerical results on every supported machine. GNU datamash’s results depend on the hardware and C library. See Portable results.
Does it work on macOS or Windows?
Yes. See the installation guide for native Windows and macOS requirements, downloads and setup. WSL2 remains available for Linux workflows.
How does Fastmash relate to GNU datamash?
Fastmash is an independent reimplementation of GNU datamash’s command language, written in Rust. It contains no GNU code and is not affiliated with the GNU Project. GNU datamash, by Assaf Gordon and contributors, defined the interface that Fastmash follows.
Contributing
Help improve Fastmash by reporting bugs, sharing real workloads, improving documentation or contributing code. The contribution guide covers development setup, checks and pull requests.
Design decisions explain the constraints behind the implementation.
Report security problems through the security policy.
Architecture
The architecture guide maps Fastmash’s crates, command pipeline and design constraints for contributors.
For sorting internals, see the technical sorting explanation. For using grouping and controlling sort resources, read Grouping and sorting.
Testing
The testing guide provides local commands, prerequisites and specialist checks for contributors.
Quick start
Choose checks for your change and use CI for platform coverage unavailable locally.
Website checks
Use the website checks for documentation and site changes.
Native macOS development gate
Follow the native macOS checks for macOS prerequisites and coverage limits.
Locale tools
The locale-tool instructions cover reference generation and qualification.
Glossary
The project glossary defines the vocabulary used in Fastmash’s code, tests, documentation and issues.
About
Fastmash was created by Peder Bergan, who designs, builds and maintains it.
The project started from a practical question: can a tool people use every day in data pipelines be made substantially faster, without asking them to change their commands, and without giving up correctness? Fastmash combines familiar GNU datamash commands with Rust software arithmetic and published performance measurements. Numerical results repeat across supported machines for the same input, settings and Fastmash version.
Acknowledgements
Fastmash follows the interface of GNU datamash, created by Assaf Gordon and maintained by its contributors. Fastmash is an independent project and is not affiliated with or endorsed by the GNU Project.
Contact
- Bugs and feature requests: GitHub issues
- Questions and ideas: GitHub discussions
- Security: see the security policy
- Anything else: pederbe.dev