Tracing & Introspection: strace, lsof & ss
When logs run dry, watch what a process actually does — its syscalls, open files and network sockets.
Sometimes the logs simply do not say why something is stuck or failing. That is when you look at the process directly, from the outside.
lsof — what has this open?
lsof ("list open files", and on Linux everything is a file) answers priceless questions: which process is holding a deleted file that is still eating disk, or who has a file locked.
lsof -i :8080— which process owns port 8080.lsof -p 1234— every file that PID 1234 has open.lsof +D /var/log— everything open under a directory.
ss — sockets, fast
ss -tulpn lists listening TCP/UDP ports and the owning program; ss -tp state established shows live connections. It is the modern, faster replacement for netstat.
strace — the syscall X-ray
strace shows every system call a process makes. It is verbose but unbeatable for "why can't it find this config file?" — you will literally see the openat(...) = -1 ENOENT for the path it is looking in but you never created.
strace -f -e trace=openat command— follow child processes, only file opens.strace -p 1234 -f— attach to a running process.strace -c command— a summary table of which syscalls cost the most time.
Example
# Which process is squatting on the port I need?
sudo lsof -i :3000
# node 981 alice 23u IPv4 ... TCP *:3000 (LISTEN)
# A deleted-but-still-open file is silently eating disk
sudo lsof -nP | grep '(deleted)'
# Live established connections and who owns them
sudo ss -tp state established
# Why can't the app find its config? Show the failing open()s
strace -f -e trace=openat myapp 2>&1 | grep ENOENT
# openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT
# Where is a running process spending its syscall time?
sudo strace -c -p 981When to use it
- A developer uses 'strace -p $(pgrep myapp)' to attach to a hung process and discover it is stuck on a read() call waiting for a network socket.
- A sysadmin runs 'lsof -p 1234' to list all files, sockets, and pipes a process has open during a file descriptor leak investigation.
- A security analyst uses 'lsof -i :443' to find which process is listening on port 443 when no documentation is available.
More examples
Trace system calls of a command
Uses strace -e to filter to openat syscalls and -c to produce a summary count of all syscalls made.
# Trace which files a command opens
strace -e trace=openat ls /etc 2>&1 | grep openat
# Count syscalls made by a program
strace -c ls /etc 2>&1 | tail -15List open files for a process
Shows lsof to audit open files by PID, reverse-lookup which processes hold a file, and rank FD usage.
# List all open files for a PID
lsof -p $(pgrep nginx | head -1)
# List processes using a specific file
lsof /var/log/nginx/access.log
# Count open file descriptors per process
lsof | awk '{print $1}' | sort | uniq -c | sort -rn | headFind who owns a port
Uses lsof -i to identify the process listening on a port, which is faster than scanning all processes.
# Find the process listening on port 8080
lsof -i :8080
# COMMAND PID USER FD TYPE NODE NAME
# node 4321 app 22u IPv4 TCP *:8080 (LISTEN)
# Or with ss for same info
ss -tlnp 'sport = :8080'
Discussion