Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Grouping and sorting

Groups

-g KEYS (also written groupby KEYS, grouping KEYS or gb KEYS) makes Fastmash summarize each group of records that share the same key values. Output rows start with the key values, followed by the results:

$ printf 'A\tx\t1\nA\tx\t2\nA\ty\t5\nB\ty\t4\n' | fastmash -g 1,2 sum 3
Grouped summary
────────────────────────────────
  field-1  field-2  sum(field-3)
        A        x             3
        A        y             5
        B        y             4
✔ Completed

A group is a run of adjacent records with equal keys. When the key changes, the group ends and its results are printed. This makes grouping a single pass over the input, with memory proportional to one group.

If equal keys are not adjacent, the same key appears in more than one group:

$ printf 'b\t2\na\t1\nb\t3\n' | fastmash -g 1 sum 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)
        b             2
        a             1
        b             3
✔ Completed

Sorting first: -s

-s sorts the records by their keys before grouping, so every key forms one group:

$ printf 'b\t2\na\t1\nb\t3\n' | fastmash -s -g 1 sum 2
Grouped summary
────────────────────────────────
  field-1  sum(field-2)
        a             1
        b             5
✔ Completed

The sort is stable: records with equal keys keep their input order, so first, last and collapse give predictable results.

Key order depends on the locale. In C and C.UTF-8, keys are ordered byte by byte. In the supported language locales, such as en_US.UTF-8, fr_FR.UTF-8 or pl_PL.UTF-8, keys are ordered alphabetically, as GNU sort orders them, using built-in Unicode collation rules and glibc’s table of the punctuation and symbols it ignores.

If your input is already sorted, leave out -s: the result is the same and the job is faster.

Large inputs

Fastmash sorts inside its own executable. It holds a sort buffer in memory and writes sorted runs to temporary files when needed. Windows uses its system temporary directory. On Linux and macOS, TMPDIR selects the directory, with /tmp as the default. The operating system removes the files when Fastmash exits, including after interruption.

For eligible input with repeated keys, hash grouping collects operation state per key and sorts only the groups. If this cannot preserve sorted behavior, it rereads the input and sorts it before writing results. Language-locale pipes can retain input for that replay. C-locale pipes normally sort directly. File and pipe input can therefore have different time and memory costs. A file that changes between readings contributes its second-reading contents.

Sorting can spill, but the values retained by statistics such as quantiles stay in memory. A sort-memory target is not a total-process memory limit. See memory and disk requirements.

Resource controls

FASTMASH_SORT_MEMORY_BYTES sets the sort-buffer memory target, and FASTMASH_SORT_TRACE (set to anything) reports each decision on standard error. Neither is a limit on Fastmash’s total memory use.

FASTMASH_GROUPING=sort always sorts. For piped input, FASTMASH_PIPE_GROUPING=sort selects sorting and =hash permits hash grouping in every supported locale. --vnlog and rand always sort. These controls change the route, not the result.

When the selected calculations support parallel sorting, records are prepared on up to 8 threads. On Linux, numerical calculations can merge spilled records and calculate in parallel on two threads. Both use the processors available to Fastmash, or OMP_NUM_THREADS if it is set, capped by OMP_THREAD_LIMIT. Set OMP_NUM_THREADS=1 to sort on one thread.

Sorted rmdup keeps complete records and stable key order, with the same temporary-storage policy. Case folding affects ordering, while duplicate keys are compared exactly. The architecture describes buffer sizing, parallel preparation and replay eligibility.

Ignoring case: -i

-i compares keys ignoring ASCII letter case, for grouping and sorting. The output shows each key as it first appeared. -i also affects countunique and unique. Under the Turkic character types (tr_TR, az_AZ and a few others), where glibc does not fold i and I, -i is refused. Set LC_CTYPE=C.UTF-8 to fold ASCII letters.

Pivot tables: crosstab

crosstab KEY1,KEY2 (or ct) builds a two-way table with one row per value of the first key and one column per value of the second. By default each cell is a count. You can name one operation instead:

$ printf 'a\tx\t1\na\ty\t2\nb\tx\t3\n' | fastmash -s crosstab 1,2 sum 3
	x	y
a	1	2
b	3	N/A

Missing combinations show the --filler text (default N/A). Rows and columns are ordered byte by byte. Use -s unless each combination’s records are already adjacent.