🔬 strace Quick Reference

strace [options] command [args]
strace -p PID

strace shows the system calls a program makes to the Linux kernel: files it opens, network
connections it makes, processes it starts, signals it gets. When a program fails with a
useless error, strace often shows exactly which file, permission or connection it choked on.
Install it with: sudo apt install strace | sudo dnf install strace
Attaching to another user's process needs sudo. Tracing slows the program down a lot,
so be careful on busy production systems.

strace ls /tmp                         # run a command under strace
strace -o out.txt ls /tmp              # write the trace to a file (keeps the screen clean)
strace -p 1234                         # attach to a running process
sudo strace -p 1234                    # ...owned by someone else
strace -f ./script.sh                  # also follow child processes (forks, threads)
strace -ff -o trace ./server           # one file per process: trace.PID
strace -c ls /tmp                      # summary table of calls, counts and time
strace -p 1234 -p 5678                 # several processes
Stop an attached trace with Ctrl+C. That detaches and the program keeps running.

-o FILE    Write output to FILE                   -f     Follow child processes and threads
-ff        Separate output file per process      -p PID Attach to a running process
-s N       Show up to N characters of strings (default 32; try -s 200)
-v         Do not abbreviate arrays and structures    -x   Print non-ASCII strings in hex
-e EXPR    Filter what is traced (see below)     -c     Count and time calls, print a summary
-C         Like -c but also prints the normal trace
-t / -tt / -ttt   Timestamps: time of day, with microseconds, or epoch seconds
-r         Time since the previous call          -T     Time spent inside each call
-y         Show file paths next to file descriptors   -yy   Also show socket addresses
-i         Show the instruction pointer           -k     Print a stack trace for each call
-q         Suppress attach/detach messages        -qq    Suppress exit messages too
-D         Run the tracer as a detached grandchild (so ps shows less noise)
-E VAR=val Run the command with an environment variable set
-u USER    Run the command as another user        -z     Show only successful calls (newer versions)
-Z         Show only failed calls (newer versions)   -n   Number each line (newer versions)
--follow-forks, --output-separately     Long forms of -f and -ff
Good default for investigating: strace -f -tt -T -s 200 -o trace.txt COMMAND

openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
call   arguments                                 return value   error name and text
- Success returns 0 or a positive number (a file descriptor, or a count of bytes read/written)
- Failure returns -1 followed by an errno name (ENOENT) and its message
- read(3, "hello\n", 4096) = 6     means it read 6 bytes from file descriptor 3
- +++ exited with 0 +++            is the exit status
- --- SIGSEGV {...} ---            is a signal received (SEGV is a crash)
- <unfinished ...> and <... resumed>  show a call interrupted by another thread

strace -e trace=openat,open ls          # only these calls
strace -e trace=file cmd                # calls that take a filename (open, stat, unlink ...)
strace -e trace=process cmd             # fork, exec, exit, wait
strace -e trace=network cmd             # socket, connect, send, recv, bind ...
strace -e trace=memory cmd              # mmap, brk, munmap ...
strace -e trace=signal cmd              # kill, signal handlers
strace -e trace=desc cmd                # calls on file descriptors
strace -e trace=ipc cmd                 # shared memory, semaphores, queues
strace -e trace=%file,%network cmd      # the % form works in newer versions
strace -e trace=!write cmd              # everything except write (quote the ! in shells)
strace -e signal=none cmd               # hide signals
strace -e read=3 cmd                    # dump the data read from fd 3 (-e write=N for writes)
strace -e verbose=none cmd
strace -e status=failed cmd             # only failed calls (newer versions)
strace -e inject=openat:error=ENOENT cmd          # make a call fail on purpose (fault injection)
strace -e inject=openat:error=EACCES:when=3 cmd   # fail only the 3rd call
strace -e inject=read:delay_enter=500ms cmd       # slow down a call to test timeouts

strace -c ls /tmp                       # counts, errors and time per call
strace -c -f -p 1234                    # summary for a running process and its children
strace -T cmd                           # time spent in each call <0.000123>
strace -r cmd                           # relative time between calls
strace -tt -T -f cmd                    # timestamps plus per-call durations: find the slow call
Sample -c output: % time, seconds, usecs/call, calls, errors, syscall. Sort by the top row.
Anything with a huge count or time usually points at the problem (busy loops, slow reads).

openat / open       Open a file                  read / write        Read or write a descriptor
close               Close a descriptor           stat / fstat / lstat File information
access              Check permissions            execve              Start a program
clone / fork        Create a process or thread   wait4               Wait for a child
socket / connect    Create and connect a socket  bind / listen / accept  Server sockets
sendto / recvfrom   Network data                 poll / epoll_wait / select  Wait for events
mmap / brk          Memory                       futex               Lock waiting between threads
ioctl               Device control               getdents64          Read a directory
nanosleep / clock_nanosleep   Sleeping           unlink / rename     Delete or move a file

ENOENT     No such file or directory     EACCES   Permission denied
EEXIST     File exists                   EISDIR   Is a directory
ENOTDIR    Not a directory               EPERM    Operation not permitted
EAGAIN     Try again (normal for non-blocking I/O)    EINTR   Interrupted by a signal
ECONNREFUSED   Nothing listening there   ETIMEDOUT   Connection timed out
EADDRINUSE     Port already in use       EMFILE   Too many open files
ENOSPC     No space left on device       EPIPE    Broken pipe
ENOENT on a config file or library is the most common cause of "program will not start".

# Why won't it start? Which files does it try to open?
strace -f -e trace=openat,open,stat,access -o trace.txt ./myapp
grep ENOENT trace.txt                    # missing files
grep EACCES trace.txt                    # permission problems

# Which config files does this program read?
strace -e trace=openat -f cmd 2>&1 | grep -v ENOENT | grep -E '\.(conf|cfg|ini|yaml|json)'

# What is this hung process waiting on?
sudo strace -p 1234
#   stuck on read(4, ...) or futex(...) or poll(...) shows what it waits for

# What network connections does it make?
strace -f -e trace=network -s 200 -o net.txt curl https://example.com
grep connect net.txt

# What commands does a script really run?
strace -f -e trace=execve -o exec.txt ./install.sh
grep execve exec.txt

# See what a program writes (for example to the terminal or a log)
strace -e trace=write -s 500 cmd

# Which system calls are slow?
strace -c -S time cmd                    # summary sorted by time
strace -f -tt -T -o trace.txt cmd && sort -t'<' -k2 -n -r trace.txt | head

# How many files does it open?
strace -c -e trace=openat cmd

# Which environment variables and libraries does it use?
strace -e trace=openat,execve -f cmd 2>&1 | grep -E '\.so|/etc/'

# Find the log file a daemon writes to
sudo strace -f -e trace=openat,write -p $(pgrep -o daemon) 2>&1 | grep -E '\.log'

# Trace a program started by systemd or a service (attach after it starts)
sudo strace -f -p $(systemctl show -p MainPID --value myservice) -o svc.txt

# Test how an app handles a failing disk or missing file
strace -e inject=openat:error=ENOSPC cmd

# Trace SSH connection problems
strace -f -e trace=network,file -o ssh.txt ssh host

ltrace           Traces library calls (libc functions) instead of system calls
lsof -p PID      Files and sockets a process has open right now
ls -l /proc/PID/fd    Open descriptors straight from /proc
perf trace       Lower-overhead system call tracing (Linux perf)
bpftrace / bcc / eBPF tools   Very low-overhead tracing for production
gdb -p PID       Attach a debugger to inspect a stuck process
dtruss / dtrace  The macOS equivalents (strace itself is Linux only)
tcpdump          Network packets rather than system calls

"ptrace: Operation not permitted"   Use sudo, or the kernel restricts ptrace (see /proc/sys/kernel/yama/ptrace_scope)
"attach: ptrace(PTRACE_SEIZE) ... Permission denied"   Same: sudo, or set ptrace_scope to 0 temporarily
Output flies by                    Use -o file, then grep and less it
Strings are cut off                Add -s 200 (or larger)
Program runs very slowly           That is strace overhead: filter with -e trace=... or use -c
Child processes missing            Add -f (follow forks and threads)
Too much noise from read/write     Filter: -e trace=file, or exclude with -e trace=!read,write
Setuid programs do not trace as root   ptrace drops privileges: run strace as root, not the program
Containers                         Needs SYS_PTRACE capability (docker run --cap-add SYS_PTRACE)

    Start with strace -f -o trace.txt COMMAND, then grep the file for ENOENT, EACCES and ECONNREFUSED.

    Always add -f for anything that starts other processes, including scripts, shells and servers.

    Use -s 200 so you can read the strings, not just the first 32 characters.

    A process stuck in read, poll, epoll_wait or futex is waiting. The descriptor or lock in the arguments shows for what.

    Use -c first for a bird's-eye summary, then trace only the interesting calls with -e trace=.

    ENOENT lines on the way to finding a file (search paths) are often harmless. Look at the last one before the failure.

    strace slows programs down badly. On production, attach briefly, filter tightly, and detach with Ctrl+C.

    Traces may contain passwords, tokens and private data. Treat trace files as sensitive and delete them when done.

    For long-term or low-overhead tracing on busy systems, use perf trace or eBPF tools instead.