Performance Debugging: load, CPU, memory & disk

Read load averages, top/htop, vmstat and iostat to find whether the CPU, memory or disk is the bottleneck.

When a box is slow, resist the urge to reboot. A calm five-minute triage almost always points at the culprit: CPU, memory, or disk I/O.

Start with load and top

uptime gives three load averages (1, 5, 15 min). Compare them to your core count from nproc: a load of 8 on 8 cores is fully busy but fine; a load of 30 on 4 cores means a queue is building. In top/htop, press P to sort by CPU and M to sort by memory.

Is it CPU or is it I/O wait?

The key number is %wa (I/O wait) in top. High %us/%sy means real CPU work; high %wa means the CPU is idle, stuck waiting for a slow disk. That single distinction changes the whole investigation.

vmstat and iostat

  • vmstat 1 — one line per second. Watch the si/so columns: any sustained swap in/out means you are out of RAM and paying for it dearly.
  • iostat -xz 1 — per-device I/O. The %util column near 100% and a rising await (ms per request) mean the disk is the bottleneck.

Memory reality check

free -h scares beginners because "free" looks tiny — but Linux deliberately uses spare RAM as cache. Look at the available column, not free. Cache is given back to programs instantly when needed.

Example

Example · bash
# How many cores am I comparing load against?
nproc
# 4

# Load vs cores at a glance
uptime
# ... load average: 7.80, 6.10, 3.40   <- queueing on a 4-core box

# One sample per second: watch si/so (swap) and wa (io wait)
vmstat 1
# procs -----memory----- ---swap-- -----io---- --system-- -----cpu-----
#  r  b   swpd   free ... si   so    bi    bo   in   cs us sy id wa st
#  2  1      0 128000 ...  0    0   340  1200  900 1500 12  4 20 64  0

# Per-disk pressure: %util near 100 and rising await = disk-bound
iostat -xz 1

# Find the D-state (uninterruptible) processes stuck on I/O
ps -eo state,pid,comm | awk '$1=="D"'

When to use it

  • A sysadmin checks 'uptime' and sees a load average of 24 on a 4-core server, immediately knowing the system is severely overloaded.
  • A developer uses 'vmstat 1 5' to sample memory and swap activity every second during a suspected memory leak in production.
  • A storage engineer runs 'iostat -xz 1' to measure disk I/O wait and saturation during a slow database query investigation.

More examples

Read load averages and CPU

Reads load averages from uptime and compares against nproc to judge whether the CPU is overloaded.

Example · bash
uptime
# 16:00:00 up 10 days, load average: 1.20, 0.85, 0.72
# Load avg = processes waiting for CPU (1m, 5m, 15m)

# Number of CPU cores (load should stay below this)
nproc
# 4

Sample memory with vmstat

Uses vmstat to sample CPU, memory, swap, and I/O stats; non-zero 'si'/'so' columns indicate swap pressure.

Example · bash
# 5 samples, 1 second apart
vmstat 1 5
# procs -memory------  ---swap-- -----io---- -cpu---
#  r  b   swpd   free   buff  cache   si  so   bi  bo  us sy id wa
#  1  0  12288 204800  65536 512000    0   0    2   8  12  3 84  1

Measure disk I/O with iostat

Uses iostat -xz to show per-device throughput and await (latency); %util near 100% means the disk is saturated.

Example · bash
# -x extended stats  -z skip idle  1 second interval
iostat -xz 1 3
# Device  r/s  w/s  rkB/s  wkB/s  await  %util
# sda     0.5  45.2  12.4   892.1   2.3    38.4

Related

linux ps toplinux whoamilinux disk usage

Discussion

  • Be the first to comment on this lesson.