Text Processing Combos: grep, awk, sed, sort, uniq & jq
Chain the classic text tools into one-liners that answer real questions about logs and data.
The real power of the command line is not any single tool — it is gluing the small ones together. Once you internalise a handful of combos you will reach for them daily.
The workhorse pattern
sort | uniq -c | sort -rn is the "top offenders" idiom. Feed it any list and it returns each distinct line with a count, most frequent first — top IPs, top error messages, top URLs.
Pick the right tool for the shape
| Job | Tool |
|---|---|
| Keep/drop whole lines | grep |
| Work with columns / do maths | awk |
| Rewrite text in a stream | sed |
| Order and de-duplicate | sort / uniq |
| Query JSON | jq |
jq for JSON
APIs and modern tools speak JSON, and jq slices it the way awk slices columns. jq -r '.items[].name' pulls a field out of every element; -r gives raw strings without quotes so the output pipes cleanly into other tools.
awk does more than you think
awk can sum, average and group. awk '{s+=$1} END{print s}' totals a column; adding an associative array (counts[$2]++) turns it into a group-by in one expression.
Example
# Top 10 client IPs hitting an nginx access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# Count HTTP status codes (field 9 in the combined log format)
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# Total bytes served (field 10), human-friendly
awk '{sum+=$10} END {printf "%.1f MB\n", sum/1024/1024}' access.log
# Sum request bytes grouped by status code (a group-by in one line)
awk '{bytes[$9]+=$10} END {for (c in bytes) print c, bytes[c]}' access.log
# Extract names of failed items from a JSON API response
curl -s https://api.example.com/jobs \
| jq -r '.jobs[] | select(.status=="failed") | .name'When to use it
- A DevOps engineer pipes 'curl -s https://api.example.com/metrics | jq .requests | sort -rn | head -5' to rank the busiest API endpoints.
- A log analyst uses 'grep "POST /api" access.log | awk {print $1} | sort | uniq -c | sort -rn' to find the top IP addresses hitting a specific endpoint.
- A data engineer chains 'cut -d, -f3 users.csv | sort | uniq -c | sort -rn | head' to count users by country without writing a script.
More examples
Rank top IPs from access log
Five-stage pipeline: extract IPs, sort, count duplicates, sort by count descending, take top 10.
awk '{print $1}' /var/log/nginx/access.log \
| sort \
| uniq -c \
| sort -rn \
| head -10
# 1423 203.0.113.5
# 891 198.51.100.2Parse JSON API output with jq
Chains curl and jq to fetch a JSON API and extract specific fields, avoiding manual text parsing.
# Fetch and filter JSON in one pipeline
curl -s https://api.github.com/repos/torvalds/linux/releases \
| jq '.[] | {tag: .tag_name, date: .published_at}' \
| head -20Extract and aggregate log fields
Uses awk associative arrays to group and aggregate a numeric column by category in a single pass.
# Sum total bytes served per HTTP status code
awk '{status=$9; bytes=$10; sum[status]+=bytes}
END {for (s in sum) printf "%s: %.1f MB\n", s, sum[s]/1024/1024}' \
/var/log/nginx/access.log | sort
Discussion