⏱️ Performance Triage

Performance Triage: "the server is slow", a first-ten-minutes playbook
uptime; free -h; vmstat 1 5; iostat -xz 1 3; ss -s

"The server is slow" can mean a hundred things. The trick is not knowing every tool, it is asking the same few questions in the same order, so you find the saturated resource instead of guessing. This guide gives you that order, how to read the numbers, and the scenarios that cause most real tickets.
Most commands here read the live machine, so they are shown without output. The "reading the numbers" sections use real sample output that was run through real one-liners, so you can see exactly how to spot the problem.
Rule one: look before you touch. Capture what you see (with timestamps) BEFORE you restart anything. A restart fixes the symptom and destroys the evidence.

For every resource (CPU, memory, disk, network) ask three questions:
  Utilization   how busy is it?                (CPU 95% busy, disk 100% busy)
  Saturation    is work WAITING for it?        (run queue longer than the CPU count, swapping, disk queue, dropped packets)
  Errors        is it reporting failures?      (OOM kills, disk errors, packet errors, retransmits)
Saturation is the one people forget, and it is what users feel: a CPU at 60% with a long run queue is slower than a CPU at 90% with none. Work through the resources in this order and write down what you find. When one resource is saturated, you have your suspect; then find which PROCESS is using it.

Ten commands, in order. Run them, read them, do not change anything yet.
$ uptime                 # load averages (1, 5, 15 minutes): compare with the CPU count (nproc). Rising? falling? steady?
$ dmesg -T | tail -20    # kernel errors: OOM kills, disk errors, NIC resets, "blocked for more than 120 seconds" (needs sudo on some systems)
$ vmstat 1 5             # CPU, memory, swap and block I/O at once: the best single command
$ mpstat -P ALL 1 3      # per-core CPU (sysstat): one core at 100% while the rest are idle means one busy thread
$ pidstat 1 3             # which PROCESSES use CPU (sysstat)
$ iostat -xz 1 3         # per-disk utilisation and wait times (sysstat)
$ free -h                # memory: the "available" column is the one that matters
$ sar -n DEV 1 3         # network throughput per interface (sysstat)
$ ss -s                  # socket summary: counts of established, time-wait, orphaned connections
$ top -b -n 1 | head -15 # the usual suspects, once (or htop, interactively)
sysstat (iostat, mpstat, pidstat, sar) is a package you install ahead of time: sudo apt install sysstat, or dnf install sysstat. Enable its collection service so sar has history when something already happened: on Debian and Ubuntu set ENABLED="true" in /etc/default/sysstat and restart sysstat.

Load average is the average number of tasks that are RUNNING or WAITING to run, plus (on Linux) tasks stuck in uninterruptible sleep, usually waiting for disk or NFS. Three numbers: the last 1, 5 and 15 minutes.
$ cat /proc/loadavg
3.52 2.10 1.05 4/312 18873      (1, 5, 15 minute averages; running/total tasks; last PID)
Rules of thumb (divide by the number of CPU cores, nproc):
  load per core around 1.0   the CPUs are fully used; any more means things are queuing
  load per core far above 1  saturated: either CPU-bound or stuck on I/O
  1-minute above 15-minute   it is getting worse right now; 1-minute below 15-minute: it is recovering
High load with LOW CPU use means processes are waiting for something else, usually disk or network storage. Check which:
$ ps -eo state,pid,ppid,wchan:20,cmd | awk '$1 ~ /^D/' | head      # processes in "D" state (uninterruptible sleep) and where they wait
A few D-state processes for a moment are normal. Many of them for a long time means a stuck disk, a dead NFS server or a failing device.
Compare load with the core count in one line (sample numbers shown):
$ echo "3.52 2.10 1.05" | awk -v cores=4 '{printf "load per core: %.2f  (%s)\n", $1/cores, ($1/cores > 1 ? "queuing" : "ok")}'
load per core: 0.88  (ok)
$ echo "8.40 7.90 6.50" | awk -v cores=4 '{printf "load per core: %.2f  (%s)\n", $1/cores, ($1/cores > 1 ? "queuing" : "ok")}'
load per core: 2.10  (queuing)

vmstat is the one-line-per-second summary. Columns that matter: r (runnable tasks: compare to cores), b (blocked tasks), si/so (swap in/out), us (user CPU), sy (system/kernel CPU), id (idle), wa (waiting for I/O), st (stolen by the hypervisor).
Here is a real-format sample from a struggling server, and a one-liner that explains it. (The sample was typed in; on your machine you run vmstat 1 5 instead.)
$ awk 'NR > 2 { printf "r=%-2s b=%-2s swapping=%-3s cpu_busy=%-3s iowait=%-3s steal=%s\n", $1, $2, ($7 + $8 > 0 ? "YES" : "no"), $13 + $14 "%", $16 "%", $17 "%" }' vmstat.sample
r=9  b=0  swapping=no  cpu_busy=99% iowait=0%  steal=0%
r=11 b=0  swapping=no  cpu_busy=99% iowait=0%  steal=0%
r=3  b=8  swapping=no  cpu_busy=12% iowait=78% steal=0%
r=2  b=9  swapping=YES cpu_busy=11% iowait=84% steal=0%
How to read the four rows: the first two are CPU-bound (92% user, tasks waiting to run, no I/O wait). The third is I/O-bound (78% wait, 8 blocked tasks, lots of block input). The last one is the worst case: swapping (si/so above zero) AND high I/O wait: the machine ran out of memory and is thrashing.
Find the CPU hog
$ top -b -n 1 -o %CPU | head -12                # snapshot; in interactive top: press P to sort by CPU, 1 for per-core, H for threads
$ ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head
$ pidstat -u -p ALL 1 3 | awk 'NR > 3 && $8 > 20'    # processes above 20% CPU over three one-second samples
$ top -H -p 1234                                  # threads inside ONE process: which thread is spinning?
$ sudo perf top                                   # which function is burning CPU (needs perf; see the perf sheet)
$ sudo strace -c -f -p 1234                       # summary of system calls (Ctrl+C after a few seconds): what is it doing?
$ sudo strace -f -tt -T -p 1234 2>&1 | head -50   # live calls with timestamps and how long each took
CPU steal: on virtual machines, "st" above a few percent means the host is overcommitted; nothing inside the VM will fix it, so move or resize the VM. Throttled containers: if CPU limits are set, a container can be slow while the host is idle: check cpu.stat (nr_throttled) in its cgroup:
$ docker stats --no-stream
$ cat /sys/fs/cgroup/system.slice/myapp.service/cpu.stat
$ systemd-cgtop -n 1 --order=cpu

Linux uses spare memory for the file cache, so "free" is usually tiny on a healthy machine. What matters is AVAILABLE (memory that could be given to programs) and whether the machine is swapping.
$ free -h
$ grep -E 'MemTotal|MemAvailable|SwapTotal|SwapFree|Dirty|Slab' /proc/meminfo
$ vmstat 1 5                  # si/so above zero repeatedly = swapping = slow
$ ps -eo pid,user,%mem,rss,vsz,cmd --sort=-rss | head       # biggest by resident memory (RSS)
$ sudo smem -tk -c "pid user command swap pss" | tail      # fairer accounting (PSS) than RSS; needs smem
$ sudo slabtop -o | head -15 # kernel memory: when cache and processes do not add up
Compute available memory as a percentage (sample /proc/meminfo lines):
$ printf 'MemTotal:       16000000 kB\nMemAvailable:    2400000 kB\nSwapTotal:       2000000 kB\nSwapFree:         300000 kB\n' | awk '/MemTotal/ {t=$2} /MemAvailable/ {a=$2} /SwapTotal/ {st=$2} /SwapFree/ {sf=$2} END {printf "available: %.0f%%   swap used: %.0f%%\n", 100*a/t, 100*(st-sf)/st}'
available: 15%   swap used: 85%
Below about 10% available and a lot of swap in use is trouble. A growing swap number that never shrinks is a leak or an undersized machine; a one-off bump is not.
Out-of-memory kills: the kernel kills a process to survive. Look for them:
$ sudo dmesg -T | grep -iE 'out of memory|oom-killer|killed process'
$ sudo journalctl -k --since "1 day ago" | grep -iE 'oom|killed process'
$ cat /proc/$(pgrep -o myapp)/oom_score          # which process the kernel would pick first
$ systemctl show myapp -p MemoryMax,MemoryCurrent    # a systemd memory ceiling: a service can be OOM-killed inside its cgroup while the host has plenty free
Memory leaks: measure the same process over time instead of guessing.
$ while true; do echo "$(date +%T) $(ps -o rss= -p 1234)"; sleep 60; done | tee rss.log      # steady growth = leak
Swappiness and caches: do not drop caches as a "fix" (echo 3 > /proc/sys/vm/drop_caches): it only makes the next minutes slower. If you test with it, say so in your notes.

$ iostat -xz 1 3
$ iotop -oPa                    # which processes do I/O (needs root; package iotop)
$ pidstat -d 1 3                # per-process disk reads and writes
$ df -hT -x tmpfs -x devtmpfs; df -i     # full file systems and full inode tables are the most common "slow app" cause
$ sudo lsof +L1                 # deleted files still held open: space not freed until the process restarts
$ sudo smartctl -H /dev/sda     # drive health (package smartmontools)
$ dmesg -T | grep -iE 'i/o error|ata|nvme|blk_update'    # failing disk, cable or controller
How to read iostat -x: r/s and w/s (operations per second), rkB/s and wkB/s (throughput), r_await and w_await (average milliseconds an operation waited: the number users feel), aqu-sz (average queue length), %util (how busy the device was). Typical thresholds: a spinning disk with await above 20-30 ms is struggling; an SSD above a few ms is; %util near 100% on a spinning disk means saturated, but on NVMe and arrays it can sit at 100% without being slow (look at await and queue instead).
Sample iostat -x lines and a one-liner that flags the slow device:
$ awk 'NR > 1 { slow = ($6 > 20 || $7 > 20); printf "%-8s await r=%.1fms w=%.1fms util=%.0f%%  %s\n", $1, $6, $7, $9, (slow ? "<-- SLOW" : "ok") }' iostat.sample
sda      await r=2.1ms w=41.5ms util=100%  <-- SLOW
nvme0n1  await r=0.2ms w=0.3ms util=35%  ok
Typical culprits: a backup or log rotation running at the wrong hour, a database doing a full scan, a full disk, an array rebuilding after a failure (cat /proc/mdstat), journald or application logs growing without limit, or a runaway process writing a debug log.
Who is writing, and what are the biggest recent files?
$ sudo find /var -xdev -type f -size +500M -printf '%s %p\n' 2>/dev/null | sort -rn | head
$ sudo du -xh /var --max-depth=2 2>/dev/null | sort -rh | head -15
$ sudo journalctl --disk-usage

$ ss -s                                     # totals: estab, timewait, orphan
$ ss -tan state established | wc -l         # how many established TCP connections
$ ss -tan | awk 'NR > 1 {print $1}' | sort | uniq -c | sort -rn      # connections by state (many SYN-RECV = SYN flood or full backlog; many TIME-WAIT is usually harmless)
$ ss -ti dst 10.0.0.5 | head -8             # per-connection details: rtt, retransmissions, congestion window
$ nstat -az | grep -iE 'retrans|drop|overflow|listendrops'      # kernel counters: retransmits, drops, full listen queues
$ ip -s link show eth0                      # errors, drops and overruns on the interface
$ sudo ethtool -S eth0 | grep -iE 'err|drop|miss'
$ sar -n DEV,EDEV 1 3                       # throughput and errors per interface over time
$ mtr -rwc 20 example.com                   # loss and latency per hop
$ dig +stats example.com | grep 'Query time'    # slow DNS looks like a slow application: time it
$ curl -sS -o /dev/null -w 'dns %{time_namelookup}  connect %{time_connect}  tls %{time_appconnect}  first-byte %{time_starttransfer}  total %{time_total}\n' https://example.com/
The last curl is the fastest way to say WHERE the time goes: if "dns" is large the resolver is slow; "connect" large means network or firewall; "tls" large means certificate or CPU; "first-byte" large means the server is thinking; a big gap between first-byte and total means a large or slow download.
Sample: turn those timings into a verdict
$ echo "0.002 0.021 0.080 1.950 2.010" | awk '{ split("DNS connect TLS first-byte total", n, " "); prev = 0; for (i = 1; i <= 4; i++) { printf "%-11s +%.3fs\n", n[i], $i - prev; prev = $i }; printf "%-11s %.3fs\n", "download", $5 - $4 }'
DNS         +0.002s
connect     +0.019s
TLS         +0.059s
first-byte  +1.870s
download    0.060s

Kernels from 4.20 report how much time tasks spend STALLED waiting for CPU, memory or I/O. It is the cleanest single saturation signal (it must be enabled; most current distributions do).
$ cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
some avg10=0.00 avg60=0.00 avg300=0.00 total=12345678          (cpu)
some avg10=2.15 avg60=1.02 avg300=0.40 total=234567890         (memory)
full avg10=1.80 avg60=0.90 avg300=0.35 total=198765432         (memory: ALL tasks stalled)
Numbers are percentages of the last 10, 60 and 300 seconds. "some" means at least one task was stalled, "full" means every non-idle task was. Sustained values above a few percent mean real impact; above 10-20% users notice. systemd shows the same per service: systemd-cgtop, and systemctl status prints pressure for units that have it.
Parse it (sample lines):
$ printf 'some avg10=2.15 avg60=1.02 avg300=0.40 total=234567890\nfull avg10=1.80 avg60=0.90 avg300=0.35 total=198765432\n' | awk '{ split($2, a, "="); printf "%s stall (last 10s): %s%%  %s\n", $1, a[2], (a[2] > 5 ? "<-- HURTING" : "fine") }'
some stall (last 10s): 2.15%  fine
full stall (last 10s): 1.80%  fine

Symptom                                      Look at                                              Usual cause
High load, CPU mostly idle                   vmstat wa/b columns, ps D-state, iostat               disk or NFS stall; a failing device
CPU 100%, one process                        top, pidstat, perf top, strace -c                    runaway loop, busy retry loop, regex backtracking, cron overlap
CPU 100% in the kernel (sy high)             vmstat sy, perf top, ss -s                           interrupt storm, huge connection churn, many small syscalls
Everything slow, swap in use                 free -h, vmstat si/so, dmesg for OOM                 not enough RAM, a leaking process, memory ceiling too low
"Disk full" but du shows little              lsof +L1, df -i                                      deleted file held open; inodes exhausted
Slow only at certain times                   sar -u -f yesterday's file, journalctl, cron, timers backup, log rotation, a batch job, a timer at the same minute on every host
Web app slow, server idle                    curl -w timings, logs with request time, DB slow log DNS, upstream timeout, database lock, connection pool exhausted
Slow SSH login (long wait before password)   ssh -vvv, /var/log/auth.log                         reverse DNS lookups (UseDNS), GSSAPI authentication timeouts
Slow after reboot / first minutes            systemd-analyze blame, iostat, free                  cold caches, services starting, fsck, RAID resync (cat /proc/mdstat)
One VM slow, host fine                       vmstat st column                                     CPU steal: noisy neighbour, overcommitted host
Container slow, host idle                    docker stats, cpu.stat nr_throttled                  CPU or memory limits
Sudden slow, nothing changed                 journalctl -p warning --since "1 hour ago", dmesg    hardware, a deploy, certificate or DNS change, a dependency outage
Look at history, not just now: sar keeps it (if enabled):
$ sar -u -f /var/log/sysstat/sa$(date +%d)            # CPU today, 10-minute averages
$ sar -r -f /var/log/sysstat/sa$(date +%d)            # memory today
$ sar -q -f /var/log/sysstat/sa$(date +%d)            # load average and run queue today
$ sar -d -p -f /var/log/sysstat/sa$(date +%d)         # disks today
$ sar -u -s 09:00:00 -e 11:00:00 -f /var/log/sysstat/sa$(date +%d)   # a time window

A small script that records the evidence in one go, with timestamps. It only reads: it does not change the system. Tools that are not installed are skipped.
$ cat > triage.sh <<'EOF'
> #!/usr/bin/env bash
> # triage.sh: write a performance snapshot to ./triage-<host>-<time>.txt
> out="triage-$(hostname -s)-$(date -u +%Y%m%dT%H%M%SZ).txt"
> run() { echo; echo "== $*"; if command -v "${1%% *}" >/dev/null; then eval "$@" 2>&1 | head -${LINES_MAX:-60}; else echo "(not installed)"; fi; }
> {
>   echo "snapshot taken $(date -u '+%F %T UTC') on $(hostname)"
>   run uptime
>   run "cat /proc/loadavg"
>   run "free -h"
>   run "vmstat 1 5"
>   run "iostat -xz 1 2"
>   run "mpstat -P ALL 1 2"
>   run "ps -eo pid,ppid,user,stat,%cpu,%mem,etime,cmd --sort=-%cpu"
>   run "ps -eo pid,user,stat,wchan:20,cmd | awk '\$3 ~ /^D/'"
>   run "df -hT -x tmpfs -x devtmpfs"
>   run "df -i -x tmpfs -x devtmpfs"
>   run "ss -s"
>   run "dmesg -T | tail -30"
> } > "$out"
> echo "$out"
> EOF
$ bash -n triage.sh && echo "syntax OK"
syntax OK
$ bash triage.sh | sed -E 's/^triage-.*-[0-9]{8}T[0-9]{6}Z\.txt$/triage-HOSTNAME-TIMESTAMP.txt/'
triage-HOSTNAME-TIMESTAMP.txt
$ grep '^== ' triage-*.txt | head -6
== uptime
== cat /proc/loadavg
== free -h
== vmstat 1 5
== iostat -xz 1 2
== mpstat -P ALL 1 2
Read the file in order: load and memory first, then disks, then processes. The same file attached to a ticket saves a round of "can you run this for me". Run it once when the system is healthy, too, so you have a baseline to compare against: "slow" is relative.

Do not restart services first: you lose the evidence and the fix is usually temporary. Do not run heavy commands on a machine that is swapping (find /, du / and grep -r make it worse; use ionice -c3 nice -n 19 if you must). Do not drop caches, kill -9 random processes, or tune kernel settings you cannot explain. Do not change two things at once. Write down the time, the symptom, what you looked at and what you changed.
$ ionice -c3 nice -n 19 du -xh /var --max-depth=1 | sort -rh | head      # a polite way to run an unavoidable heavy command

Use the same order every time: uptime, dmesg, vmstat, disks, memory, network, then processes. Habit beats brilliance at 3 a.m.
Saturation and errors matter more than utilization: queues, swapping, drops, retransmits and OOM kills.
Compare with a baseline: install sysstat now so that "it was slow at 02:00" has an answer.
When the machine is fine and the application is slow, measure where the time goes (curl -w, request time in the web server log, the database slow log) before blaming the server.
The fastest fix is often a full disk, an exhausted inode table, a stuck NFS mount or a runaway cron job. Check the boring things first.
"It was working yesterday" has a cause: ask what changed (a deploy, an update, a certificate, a DNS record, a new customer) before touching the system.