journalctl: reading and managing the systemd journal
journalctl -u nginx --since today -p warning
journalctl reads the journal, the log database kept by systemd-journald. Services, the kernel, sudo, ssh and anything started by systemd all log into it, so one tool answers most "what happened on this server" questions. The skill is filtering: a busy machine writes thousands of lines an hour, and the useful one is found by unit, time, priority or text, never by scrolling.
Every example below ran for real on Ubuntu 24.04 (systemd 255) as an ordinary user, using test messages written with logger under the tag "labapp" so the output stays small and private. The host name is shown as server1. Commands that need root or would change the system are shown without output.
What the journal is and who can read it
systemd-journald collects messages from services (stdout and stderr), syslog calls, the kernel and audit, and stores them in a binary, indexed format. Because it is indexed, filters are fast, and every entry carries fields (unit, PID, user, priority, boot ID) you can filter on. The old text files in /var/log (syslog, auth.log) may still exist as well, written by rsyslog.
Reading everything needs membership of the adm or systemd-journal group (or root); a user in neither sees only their own entries. Check with:
$ id -nG | tr ' ' '\n' | grep -E 'adm|systemd-journal'
adm
$ sudo usermod -aG systemd-journal USERNAME # give a colleague read access (log out and in)
Reading the log: the basics
Plain journalctl starts at the oldest entry in a pager. Start from the other end instead:
journalctl -e jump to the end of the log in the pager
journalctl -n 50 the last 50 lines, no pager scrolling needed
journalctl -r newest first (reverse)
journalctl --no-pager print straight to the terminal, for pipes and scripts
Here is a tagged service logging four lines at different severities. The columns are: time, host, identifier[PID], message.
$ journalctl -t labapp --no-pager
Oct 02 06:45:57 server1 labapp[9167]: service started, listening on port 8080
Oct 02 06:45:57 server1 labapp[9168]: cache is 91% full
Oct 02 06:45:57 server1 labapp[9169]: database connection refused
Oct 02 06:45:57 server1 labapp[9170]: disk write failed on /var/lib/labapp
-t (--identifier) matches the tag a program logs under. For services you will mostly use -u instead.
Filter by unit, boot and time
journalctl -u nginx one service (add -u again for several: -u nginx -u php8.3-fpm)
journalctl -u nginx -b only since the current boot
journalctl -u nginx -b -1 the previous boot (what happened before the crash)
journalctl --since today since midnight
journalctl --since "1 hour ago"
journalctl --since "2026-10-01 18:00" --until "2026-10-01 19:30"
journalctl -u nginx --since yesterday --until today
Times accept "yesterday", "today", "now", "-2h", "10 minutes ago" and full dates. The time zone is local unless you add --utc. Add -o short-precise when ordering of events within one second matters:
$ journalctl -t labapp --since "5 minutes ago" -o short-precise --no-pager | head -2
Oct 02 06:45:57.323381 server1 labapp[9167]: service started, listening on port 8080
Oct 02 06:45:57.327221 server1 labapp[9168]: cache is 91% full
--user-unit NAME reads a service of your own systemd --user session. _PID=1234 and _UID=1000 (field matches, see below) filter by process or user.
Filter by priority
Every entry has a priority from 0 (emerg) to 7 (debug). -p accepts a name or a number, and it means "this level AND everything more severe":
0 emerg 1 alert 2 crit 3 err 4 warning 5 notice 6 info 7 debug
$ journalctl -t labapp -p warning --no-pager
Oct 02 06:45:57 server1 labapp[9168]: cache is 91% full
Oct 02 06:45:57 server1 labapp[9169]: database connection refused
Oct 02 06:45:57 server1 labapp[9170]: disk write failed on /var/lib/labapp
Note that -p warning also returned the err and crit lines, not only the warning. For a range use FROM..TO:
$ journalctl -t labapp -p err..crit --no-pager
Oct 02 06:45:57 server1 labapp[9169]: database connection refused
Oct 02 06:45:57 server1 labapp[9170]: disk write failed on /var/lib/labapp
A good first look at a sick machine is journalctl -b -p err: everything that went wrong since boot, without the noise.
Search by text and by field
-g (--grep) takes a regular expression and is case-insensitive when the pattern is all lower case:
$ journalctl -t labapp -g 'refused|failed' --no-pager
Oct 02 06:45:57 server1 labapp[9169]: database connection refused
Oct 02 06:45:57 server1 labapp[9170]: disk write failed on /var/lib/labapp
Every entry is a set of fields. See them all with -o json, then match any field as FIELD=value (no -g needed, and much faster than grep):
$ journalctl -t labapp -n 1 -o json --no-pager | python3 -m json.tool | grep -E 'MESSAGE"|PRIORITY|SYSLOG_IDENTIFIER|_UID|_HOSTNAME'
"PRIORITY": "2",
"MESSAGE": "disk write failed on /var/lib/labapp",
"_HOSTNAME": "server1",
"SYSLOG_IDENTIFIER": "labapp",
"_UID": "1000",
$ journalctl _SYSTEMD_UNIT=ssh.service _UID=0 --since today
$ journalctl /usr/sbin/sshd # a path matches every entry from that executable
$ journalctl -F SYSLOG_IDENTIFIER | head # list every value a field has taken (find the tag to filter on)
Fields starting with an underscore are added by journald and cannot be forged by the program; the others are supplied by the sender. Several matches on different fields are ANDed, several on the same field ORed.
Output formats
-o chooses how entries are printed:
short the default, like syslog
short-iso ISO 8601 time stamps with the zone, good for sorting and sharing
short-precise microsecond time stamps
cat the message only, nothing else (good for scripts)
json, json-pretty every field, for jq or Python
verbose every field, human readable
$ journalctl -t labapp -n 2 -o short-iso --no-pager
2026-10-02T06:45:57-04:00 server1 labapp[9169]: database connection refused
2026-10-02T06:45:57-04:00 server1 labapp[9170]: disk write failed on /var/lib/labapp
$ journalctl -t labapp -n 2 -o cat --no-pager
database connection refused
disk write failed on /var/lib/labapp
-x adds a short explanation to well-known messages (a failed unit, a mount problem). The pager truncates long lines: press the right arrow, or use --no-pager.
Follow live
journalctl -f is tail -f for the journal, and it takes every filter:
journalctl -f everything, live
journalctl -u nginx -f follow one service
journalctl -u nginx -u php8.3-fpm -f two services interleaved (the order of events is preserved, which separate log files cannot do)
journalctl -f -p err only errors, live, across the whole machine
Press Ctrl+C to stop. A common debugging pattern: in one terminal run journalctl -u myapp -f, in another restart the service and watch it start.
Kernel messages and boots
-k (--dmesg) shows only kernel messages, like dmesg but with the history of previous boots too. -b N picks a boot; list them with --list-boots:
$ journalctl --list-boots --no-pager | tail -3
-2 caf527acacea4f85a2343bacf55d0328 Sun 2026-09-13 18:45:40 EDT Fri 2026-09-25 23:34:43 EDT
-1 169658968985498780af8e7d71cc749e Fri 2026-09-25 23:35:37 EDT Thu 2026-10-01 22:54:36 EDT
0 a4846a873e484ec496311793b262e1be Fri 2026-10-02 05:52:23 EDT Fri 2026-10-02 06:45:57 EDT
$ journalctl -k -b -1 -p err # kernel errors from the previous boot: the first place to look after an unexplained reboot
Old boots only exist if the journal is persistent (next section). If --list-boots shows a single line, the machine forgets everything at each reboot.
Disk usage, persistence and cleanup
$ journalctl --disk-usage
Archived and active journals take up 685.6M in the file system.
Where it is stored decides whether it survives a reboot: /run/log/journal is in memory and is lost at reboot; /var/log/journal is on disk. With the default Storage=auto, journald writes to disk only if /var/log/journal exists (Ubuntu creates it, some minimal images do not). To make it persistent:
$ sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal
$ sudo systemctl restart systemd-journald
Limit the size in /etc/systemd/journald.conf (or a drop-in in /etc/systemd/journald.conf.d/):
[Journal]
SystemMaxUse=500M total size cap for the journal
SystemKeepFree=1G always leave this much free
MaxRetentionSec=1month drop entries older than this
Storage=persistent always on disk
$ sudo systemctl restart systemd-journald
Free space right now, without changing the configuration:
$ sudo journalctl --vacuum-size=500M # delete the oldest archived files until the total is 500M
$ sudo journalctl --vacuum-time=2weeks # delete entries older than two weeks
$ sudo journalctl --rotate # close the active file so it can be vacuumed
Vacuuming removes whole archived files, so the result is usually a little under the target. journalctl --verify checks the files for corruption; it reads everything, so on a large journal expect it to take several minutes.
Writing your own entries
Logging from a script means the message lands next to everything else, with a priority and a tag you can filter on:
$ logger -t labapp -p user.err "database connection refused"
$ echo "backup finished" | systemd-cat -t backup -p info
$ journalctl -t backup -n 1
In shell scripts, log each important step (started, finished, failed) with a tag; months later journalctl -t backup -p err is the whole history of failures. For long-running services, systemd already captures their stdout and stderr, so a plain echo becomes a journal entry with no extra code (see the systemd Units and Timers page).
Recipes
journalctl -b -p err what went wrong since boot
journalctl -u SERVICE -b -n 100 --no-pager the last lines of a service after it failed to start
journalctl -u SERVICE --since "10 minutes ago" -f watch it while you reproduce the problem
journalctl -b -1 -k | tail -50 what the kernel said just before the last reboot or crash
journalctl _COMM=sudo --since today every sudo command run today
journalctl -u ssh -g 'Failed password' --since today -o cat | wc -l count failed SSH logins
journalctl -u ssh -g 'Failed password' -o cat | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head top offending addresses
journalctl --since "2026-10-01 18:00" --until "2026-10-01 19:00" -o short-iso > incident.log save a window as evidence
journalctl -o json -u SERVICE | jq -r '.MESSAGE' machine-readable processing
Common problems
No journal files were found / Hint: You are currently not seeing messages from other users you are not in the adm or systemd-journal group: use sudo or add yourself to the group
-- No entries -- the filter matched nothing: check the unit name with systemctl list-units, the time window, and -b (the entries may be in a previous boot)
Only one boot in --list-boots the journal is not persistent: create /var/log/journal (see above) and restart journald
journalctl -u myapp shows nothing but the app runs it was not started by systemd, so it has no unit: use -t, _COMM= or _PID=, or the log file the app writes itself
Lines are cut off at the screen edge the pager does not wrap: use the arrow keys, --no-pager, or pipe through cut or fold
The journal grew to many gigabytes no size limit: set SystemMaxUse and run journalctl --vacuum-size
Time stamps look an hour off the display uses the local zone: add --utc, or compare with timedatectl
Journal files corrupted after a power cut journald moves them aside and starts a new file by itself; journalctl --verify shows which files are damaged
Messages from a container are missing the container logs to its own stdout, so use docker logs (or configure the Docker journald log driver)
Tips
Filter first, read second: unit, time window, priority, then text. Start narrow and widen only if you find nothing.
journalctl -b -p err and journalctl -xeu SERVICE are the two commands to type when something fails.
Use -o cat or -o json when a script will read the output, and --no-pager in cron jobs and pipes.
Make the journal persistent and cap its size on every server, so you can read the previous boot and the disk never fills with logs.
Tag your own scripts with logger -t or systemd-cat -t so their entries are easy to find.
Do not delete files in /var/log/journal by hand while journald runs: use journalctl --vacuum-size or --vacuum-time.