Column processing with awk

awk splits each line into fields and lets you pick, calculate and format columns.

Syntaxawk -F sep 'pattern { action }' file

awk is a small programming language for text tables. It automatically splits every line into fields — $1 is the first column, $2 the second, and $0 the whole line.

Basic use

awk '{ print $1 }' prints the first column of every line. Whitespace is the default separator; use -F to change it (for example -F, for CSV).

Patterns and maths

You can filter with a pattern and do arithmetic:

  • awk '$3 > 100' — lines where column 3 exceeds 100.
  • awk '{ sum += $2 } END { print sum }' — add up column 2.

Example

Example · bash
# Print the first and third columns
awk '{ print $1, $3 }' data.txt

# Split a CSV on commas, print column 2
awk -F, '{ print $2 }' users.csv

# Sum the numbers in column 2
awk '{ sum += $2 } END { print sum }' sales.txt

# Show only rows where column 3 is over 100
awk '$3 > 100' metrics.txt

When to use it

  • A sysadmin uses 'awk "{print $1, $7}" access.log' to extract only the IP and URL columns from an nginx log.
  • A developer calculates total disk usage with 'df | awk "NR>1 {sum += $3} END {print sum}"' without a spreadsheet.
  • A data engineer uses awk to reformat a CSV by swapping columns before loading it into a database.

More examples

Print specific columns

Uses awk -F to set the field delimiter and $N to select specific columns from structured text.

Example · bash
# Print the 1st and 5th fields of /etc/passwd
awk -F: '{print $1, $5}' /etc/passwd
# root  root
# alice Alice Smith

# Default delimiter is whitespace
df -h | awk '{print $1, $5}'

Filter rows with a condition

Uses awk conditions (NR>1 skips header, $5+0>80 numeric compare) to filter rows matching a threshold.

Example · bash
# Print lines where disk use exceeds 80%
df -h | awk 'NR>1 && $5+0 > 80 {print $6, $5}'
# /var/log 92%

# Print only non-empty lines
awk 'NF > 0' file.txt

Sum a numeric column

Accumulates a column value with a running sum variable and prints the formatted total in the END block.

Example · bash
# Sum bytes transferred from nginx access log (field 10)
awk '{sum += $10} END {printf "Total: %.2f MB\n", sum/1024/1024}' access.log
# Total: 1432.76 MB

Related

linux sedlinux cut sort uniq

Discussion

  • Be the first to comment on this lesson.
Column processing with awk — Linux | SoundsCode