Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Quick start

This tour takes about five minutes. It assumes Fastmash is installed. Every example reads from standard input, so you can paste it straight into Bash. For Windows, start with PowerShell 7.4+. The Fastmash operations and options are the same.

Locale. The examples use decimal points. If your shell uses a locale with decimal commas or an unsupported locale (see Locales), run export LC_ALL=C.UTF-8 first.

Your first summary

Fastmash reads records (lines) of fields separated by tabs, and applies one or more operations to selected fields:

$ printf '1\t2\n3\t4\n' | fastmash sum 1 mean 2
Summary
────────────────────────────────
  sum(field-1)  mean(field-2)
             4              3
✔ Completed

sum 1 adds up field 1 and mean 2 averages field 2. The summary shows both results in the order you requested.

PowerShell 7.4+

For a CSV file, pass its bytes directly through cmd input redirection. Try this sample, or use your own sales.csv with an amount field:

'item,amount', 'a,2', 'b,4' | Set-Content -Encoding utf8 sales.csv
cmd /d /v:off /c 'fastmash --csv -H sum amount < "sales.csv"'
sum(amount)
6

--csv accepts Windows line endings. PowerShell does not support < input redirection itself. Get-Content decodes a file into text, so use the cmd recipe when you want to preserve its original bytes.

You can also send text directly:

'item,amount', 'a,2', 'b,4' | fastmash --csv -H sum amount

This produces the same result. For a native program pipeline, extract the amounts and pass them to another Fastmash command:

cmd /d /v:off /c 'fastmash --csv-in --header-in cut amount < "sales.csv"' |
    fastmash sum 1

The summary shows 6. PowerShell 7.4+ preserves bytes between native programs and when redirecting their output with >. PowerShell string pipelines create text, including Windows line endings. See Fields and input for how ordinary text and CSV handle those endings.

After a command, $LASTEXITCODE gives its status. In scripts, follow PowerShell exit-status handling.

Several fields at once

A field list or range applies an operation to each field:

$ printf '1\t2\t3\n4\t5\t6\n' | fastmash sum 1-3
Summary
────────────────────────────────
  sum(field-1)  sum(field-2)  sum(field-3)
             5             7             9
✔ Completed

Grouping

-g groups records by a key field and summarizes each group:

$ printf 'A\t1\nA\t2\nB\t4\n' | fastmash -g 1 sum 2 count 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)  count(field-2)
        A             3               2
        B             4               1
✔ Completed

Groups are made from adjacent records with the same key. If equal keys can be scattered through the input, add -s to sort first:

$ printf 'b\t2\na\t1\na\t3\n' | fastmash -s -g 1 sum 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)
        a             4
        b             2
✔ Completed

Headers and named fields

With a header line, use -H to name fields and label the output:

$ printf 'site\treading\nwest\t2\neast\t4\nwest\t6\n' | fastmash -H -s -g site sum reading mean reading
Grouped summary
────────────────────────────────
  site  sum(reading)  mean(reading)
  east             4              4
  west             8              4
✔ Completed

--header-in reads field names. -H also uses them in result labels.

Other separators

-t sets a single-byte field separator, and -W splits on runs of spaces and tabs:

$ printf '1,2\n3,4\n' | fastmash -t, sum 1-2
Summary
────────────────────────────────
  sum(field-1)  sum(field-2)
             4             6
✔ Completed
$ printf '1  2\n3   4\n' | fastmash -W sum 2
Summary
────────────────────────────────
  sum(field-2)
             6
✔ Completed

-t, splits on every comma, preserving datamash’s literal separator behavior. For quoted CSV, use --csv to read and write CSV or choose input/output independently with --csv-in and --csv-out:

$ printf 'region,revenue\n"west,coast",2\neast,4\n"west,coast",6\n' | fastmash --csv -H -s -g region --result-name=1:total sum revenue
GroupBy(region),total
east,4
"west,coast",8

Format switches do not infer headers. Quoted CSV and result names describes the supported modes and quoting rules.

Statistics

$ printf '2\n4\n4\n4\n5\n5\n7\n9\n' | fastmash median 1 q1 1 q3 1 pstdev 1 perc:90 1
Summary
────────────────────────────────
  median(field-1)  q1(field-1)  q3(field-1)  pstdev(field-1)  perc:90(field-1)
              4.5            4          5.5                2               7.6
✔ Completed

See Operations for the full list: means, variances, quantiles, modes, skewness and kurtosis, normality tests, correlation and more.

Weighted means

Give each value a contribution weight with wmean VALUE:WEIGHT:

$ printf '10\t1\n20\t3\n' | fastmash wmean 1:2
Summary
────────────────────────────────
  wmean(field-1,field-2)
                    17.5
✔ Completed

Weights must be finite and nonnegative. A nonempty calculation needs positive retained weight. See Weighted mean.

Keep the highest records

Top-N selection retains complete records and keeps earlier input records on ties:

$ printf 'A\t8\nB\t10\nC\t10\nD\t9\n' | fastmash top:2 2
Top-N selection
────────────────────────────────
  field-1  field-2
        B       10
        C       10
✔ Completed

Use bottom:N for the lowest records, and -s -g for a cutoff in each group. See Highest and lowest records.

Per-row operations

Some operations transform each record instead of summarizing:

$ printf '3.14159\thello\n' | fastmash round 1 md5 2
Per-row results
────────────────────────────────
  round(field-1)  md5(field-2)
               3  5d41402abc4b2a76b9719d911017c592
✔ Completed

Missing values

--narm skips NA, N/A and NaN values:

$ printf '1\nNA\n3\n' | fastmash --narm mean 1
Summary
────────────────────────────────
  mean(field-1)
              2
✔ Completed

Next steps