systemd Units and Timers: beyond the basics
systemd-analyze verify myapp.service
The systemctl sheet covers starting, stopping and enabling services. This guide is about WRITING and DEBUGGING units: how a service file is built, how dependencies and ordering really work, templates, timers that replace cron, socket activation, resource limits and sandboxing, and what the cryptic exit codes mean.
The unit-file checks below were run with systemd 255 (Ubuntu 24.04) using systemd-analyze, which needs no root. Commands that start real services or touch /etc are shown without output.
Where unit files live, and who wins
/usr/lib/systemd/system/ units installed by packages (do not edit: updates overwrite them)
/etc/systemd/system/ your units and your overrides: the place you edit; wins over /usr/lib
/run/systemd/system/ temporary units (gone at reboot)
~/.config/systemd/user/ per-user units (systemctl --user), no root needed
/etc/systemd/system/NAME.service.d/*.conf drop-ins: small override files for one unit (the safe way to change a packaged unit)
$ systemctl cat nginx.service # the unit as systemd sees it: the file PLUS every drop-in, with paths
$ systemctl show nginx -p FragmentPath,DropInPaths # which files make up the unit
$ systemd-delta --type=overridden,extended # every unit you have changed from the vendor default
$ sudo systemctl daemon-reload # REQUIRED after creating or editing any unit file
systemd warns "unit file changed on disk, run systemctl daemon-reload" when you forget. It does not reread files by itself.
Anatomy of a service unit
A unit file is an INI file: sections in [brackets], one Key=Value per line, # for comments.
$ cat units/myapp.service
[Unit]
Description=My demo web app
Documentation=https://example.com/docs/myapp
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
DynamicUser=yes
StateDirectory=myapp
Environment=PORT=8080
ExecStart=/usr/bin/python3 -m http.server ${PORT} --bind 127.0.0.1
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
$ systemd-analyze verify units/myapp.service && echo "verify: no problems found"
verify: no problems found
systemd-analyze verify is your linter: it checks the syntax, the keys and whether ExecStart exists. Silence means success. Run it BEFORE you daemon-reload.
What each part does
[Unit] description and relationships: After= / Wants= / Requires= (see below)
[Service] how to run it: the command, user, restart behavior, limits
[Install] what "systemctl enable" does: WantedBy=multi-user.target means "start it at normal boot"
Type= simple/exec: the main process IS the service (exec also reports a missing program as a failure to start); forking: the program daemonizes itself (old style);
oneshot: a command that runs and exits (add RemainAfterExit=yes to stay "active"); notify: the program tells systemd when it is ready (sd_notify)
Restart= no, on-failure (non-zero exit, signals, timeouts), on-abnormal, on-abort, always. RestartSec= is the pause between attempts.
User=/Group=/DynamicUser=yes run unprivileged; DynamicUser creates a throw-away account (state goes in StateDirectory=, logs in LogsDirectory=, cache in CacheDirectory=)
ExecStartPre=/ExecStartPost=/ExecReload=/ExecStop= steps around the main command (a leading - means "ignore failure", a leading + means "run with full privileges")
WorkingDirectory=, Environment=KEY=value, EnvironmentFile=/etc/default/myapp (one KEY=value per line)
TimeoutStartSec=, TimeoutStopSec=, KillMode=control-group (default: stop everything in the service's group)
Typos are silently ignored by the service manager, but systemd-analyze verify catches them:
$ systemd-analyze verify units/typo.service
/home/demo/units/typo.service:2: Unknown key name 'Execstart' in section 'Service', ignoring.
/home/demo/units/typo.service:3: Failed to parse service restart specifier, ignoring: on-faliure
typo.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Unit typo.service has a bad unit file setting.
# (exit status 1)
$ systemd-analyze verify units/missing.service
missing.service: Command /opt/myapp/bin/run is not executable: No such file or directory
# (exit status 1)
Write ExecStart with an ABSOLUTE path and no shell syntax: systemd does not run your command through a shell, so pipes, redirects, && and $VAR expansion of shell variables do not work. If you need them: ExecStart=/bin/sh -c 'cmd1 | cmd2 > /var/log/out'. In a unit file, $VAR and ${VAR} are replaced by systemd from Environment= (not by a shell), and a literal % must be written %%.
Install it
$ sudo cp units/myapp.service /etc/systemd/system/
$ sudo systemctl daemon-reload && sudo systemctl enable --now myapp.service
$ systemctl status myapp.service --no-pager
$ journalctl -u myapp.service -f
Dependencies versus ordering: the most misunderstood part
Two separate questions: does B NEED A (a dependency), and must A start BEFORE B (an ordering)?
Wants=A start A too, but if A fails, B still starts. (Use this by default.)
Requires=A start A too, and if A fails to start, B fails. If A is stopped, B is stopped.
Requisite=A A must ALREADY be running, or B fails immediately.
BindsTo=A like Requires, and B also stops when A goes away unexpectedly (good for devices and mounts).
PartOf=A stopping or restarting A also stops or restarts B (it never starts B).
Conflicts=A starting B stops A.
After=A / Before=A ordering ONLY: start B after A has finished starting. It does NOT start A.
The classic mistake: After=postgresql.service alone means "if both are being started, wait for postgres first", but nothing starts postgres. You need Wants=postgresql.service (or Requires=) AS WELL.
Wants=postgresql.service
After=postgresql.service
"The network is up" means different things:
network.target the network stack is configured (not that you have an address yet)
network-online.target an address is usable: use this plus Wants= for services that connect out at start (and make sure a wait-online service is enabled: systemd-networkd-wait-online or NetworkManager-wait-online)
multi-user.target normal text-mode system; graphical.target adds the desktop
$ systemctl list-dependencies myapp.service # what it pulls in
$ systemctl list-dependencies --reverse myapp.service # what pulls IT in
$ systemd-analyze critical-chain myapp.service # the chain of units that delayed it, with times
$ systemd-analyze verify myapp.service # also catches ordering cycles
Change a packaged unit the safe way: drop-ins
$ sudo systemctl edit nginx.service # opens an empty override.conf (a drop-in), then reloads systemd for you
$ sudo systemctl edit --full nginx.service # edits a full COPY in /etc (only when you must change a lot)
$ sudo systemctl revert nginx.service # throw every change away and go back to the vendor unit
A drop-in only needs the lines you change. To REPLACE a list-type setting such as ExecStart= you must first clear it with an empty assignment, otherwise systemd complains about two ExecStart lines:
$ cat <<'EOF'
> # /etc/systemd/system/nginx.service.d/override.conf
> [Service]
> ExecStart=
> ExecStart=/usr/sbin/nginx -g 'daemon off; master_process on;' -c /etc/nginx/special.conf
> Restart=always
> LimitNOFILE=65535
> EOF
# /etc/systemd/system/nginx.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/nginx -g 'daemon off; master_process on;' -c /etc/nginx/special.conf
Restart=always
LimitNOFILE=65535
Settings that ADD to a list (Environment=, After=, Wants=) simply append; settings that replace a single value just override it. Check the result with systemctl cat and systemctl show -p ExecStart nginx.
Template units: one file, many instances
A file named name@.service is a template. The text after the @ when you start it is the instance, available as %i (and %I unescaped).
(The demo files call /usr/bin/echo so that verify can find the program. In real use ExecStart points at your own script, and verify will tell you if the path is wrong.)
$ systemd-analyze verify units/backup@.service && echo "template OK"
template OK
$ sudo systemctl start backup@home.service backup@etc.service # two instances from one file
$ systemctl status 'backup@*.service'
$ systemctl enable --now getty@tty3.service # real example: login prompts
Other specifiers you will use: %n the full unit name, %N the unit name without .service, %p the part before the @, %H the hostname, %u user, %t runtime directory (/run), %S state directory (/var/lib), %h home. Instance names that contain / or spaces must be escaped:
$ systemd-escape 'my path/with space'
my\x20path-with\x20space
$ systemd-escape --template=backup@.service '/srv/data'
backup@-srv-data.service
$ systemd-escape --unescape 'backup@-srv-data.service'
backup@/srv/data.service
Timers: cron with logging, dependencies and no overlap
A timer unit (NAME.timer) activates a service (NAME.service) of the same name. The service has no [Install] section; you enable the TIMER.
$ cat units/report.timer
[Unit]
Description=Run the report on weekdays at 09:00
[Timer]
OnCalendar=Mon..Fri *-*-* 09:00:00
Persistent=true
RandomizedDelaySec=5min
AccuracySec=1min
[Install]
WantedBy=timers.target
$ systemd-analyze verify units/report.service units/report.timer && echo "timer pair OK"
timer pair OK
$ sudo systemctl enable --now report.timer # enable the TIMER, not the service
$ systemctl list-timers --all --no-pager # next/last run of every timer
$ sudo systemctl start report.service # run the job now, outside the schedule (test it!)
$ journalctl -u report.service --since today # the job's output and exit status
Persistent=true if the machine was OFF at the scheduled time, run the job once at the next boot (cron's anacron behaviour)
RandomizedDelaySec= spread the start over a window: keeps a hundred machines from hitting a server at the same second
AccuracySec= systemd may start the job up to this much late to save wakeups (default 1 minute; set 1s when seconds matter)
WakeSystem=true wake a suspended machine to run it
Calendar expressions: test them before you trust them. systemd-analyze calendar shows how an expression is normalised and when it fires next (the base time is fixed here so the output is reproducible):
$ systemd-analyze calendar --base-time='2026-03-08 12:00:00' 'Mon..Fri *-*-* 09:00' | grep -E 'Normalized|Next elapse'
Normalized form: Mon..Fri *-*-* 09:00:00
Next elapse: Mon 2026-03-09 09:00:00 UTC
$ systemd-analyze calendar --base-time='2026-03-08 12:00:00' --iterations=4 '*-*-01 03:00' | grep -E 'Next elapse|Iteration'
Next elapse: Wed 2026-04-01 03:00:00 UTC
Iteration #2: Fri 2026-05-01 03:00:00 UTC
Iteration #3: Mon 2026-06-01 03:00:00 UTC
Iteration #4: Wed 2026-07-01 03:00:00 UTC
$ systemd-analyze calendar --base-time='2026-03-08 12:00:00' --iterations=3 '*:0/15' | grep -E 'Normalized|Next elapse|Iteration'
Normalized form: *-*-* *:00/15:00
Next elapse: Sun 2026-03-08 12:15:00 UTC
Iteration #2: Sun 2026-03-08 12:30:00 UTC
Iteration #3: Sun 2026-03-08 12:45:00 UTC
$ systemd-analyze calendar --base-time='2026-03-08 12:00:00' 'quarterly' | grep -E 'Normalized|Next elapse'
Normalized form: *-01,04,07,10-01 00:00:00
Next elapse: Wed 2026-04-01 00:00:00 UTC
$ systemd-analyze calendar --base-time='2026-03-08 12:00:00' 'Sun *-*-* 02:30' | grep -E 'Next elapse'
Next elapse: Sun 2026-03-15 02:30:00 UTC
$ systemd-analyze calendar 'daily' 'weekly' 'monthly' | grep -E 'Original|Normalized'
Original form: daily
Normalized form: *-*-* 00:00:00
Original form: weekly
Normalized form: Mon *-*-* 00:00:00
Original form: monthly
Normalized form: *-*-01 00:00:00
$ systemd-analyze calendar 'Mon..Fri 9am'
Failed to parse calendar specification 'Mon..Fri 9am': Invalid argument
# (exit status 1)
The shape is DayOfWeek Year-Month-Day Hour:Minute:Second, with * for any, a/b for "start a, every b" (*:0/15 = every 15 minutes), commas for lists, .. for ranges. Shortcuts: minutely, hourly, daily, weekly, monthly, quarterly, yearly. Add UTC at the end for a UTC schedule: "*-*-* 03:00:00 UTC".
Relative timers (no calendar): measured from boot or from the last run
OnBootSec=10min 10 minutes after boot
OnUnitActiveSec=1h 1 hour after the service last STARTED (so it repeats every hour, drifting by the run time)
OnUnitInactiveSec=1h 1 hour after the service last FINISHED (never overlaps itself)
$ systemd-analyze timespan 90min
Original: 90min
μs: 5400000000
Human: 1h 30min
$ systemd-analyze timespan 1d12h
Original: 1d12h
μs: 129600000000
Human: 1d 12h
$ systemd-analyze timespan 2w
Original: 2w
μs: 1209600000000
Human: 2w
cron to timer, at a glance
*/5 * * * * OnCalendar=*:0/5
0 * * * * OnCalendar=hourly
30 2 * * * OnCalendar=*-*-* 02:30:00
0 9 * * 1-5 OnCalendar=Mon..Fri *-*-* 09:00:00
0 0 1 * * OnCalendar=monthly (or *-*-01 00:00:00)
@reboot OnBootSec=1min
Why timers are nicer than cron: output goes to the journal with the job's name, a job cannot overlap itself (a timer will not start its service while it is still running), you can set resource limits and sandboxing on the job, dependencies and ordering work, and systemctl list-timers shows when it last ran and when it runs next.
One-off jobs without writing files: systemd-run
$ sudo systemd-run --on-calendar='*-*-* 03:30:00' --unit=nightly-clean /usr/local/bin/clean.sh # a transient timer
$ sudo systemd-run --on-active=10min --unit=reminder echo "ten minutes passed"
$ sudo systemd-run --unit=bigjob -p MemoryMax=2G -p CPUQuota=50% /usr/local/bin/heavy.sh # a one-off service with limits
$ sudo systemd-run --pty --collect -p User=www-data -p ProtectSystem=strict bash # a shell inside the same restrictions a service would have: the best way to reproduce "it works for me but not as a service"
Socket activation and path units
A .socket unit listens on a port or a path on behalf of a service and starts the service on the FIRST connection. Benefits: nothing runs until needed, the port is open instantly at boot, and the service can be restarted without dropping connections.
$ systemd-analyze verify units/echo.socket units/echo.service && echo "socket and service pair OK"
socket and service pair OK
Names pair up automatically: echo.socket starts echo.service. The program receives the already-open socket (file descriptor 3, or via LISTEN_FDS) instead of opening its own. Many services support this: sshd, cups and docker ship socket units.
A path unit starts a service when a file or folder changes:
$ cat <<'EOF'
> # incoming.path
> [Path]
> PathChanged=/srv/incoming
> DirectoryNotEmpty=/srv/incoming
>
> [Install]
> WantedBy=multi-user.target
> EOF
# incoming.path
[Path]
PathChanged=/srv/incoming
DirectoryNotEmpty=/srv/incoming
[Install]
WantedBy=multi-user.target
Enable incoming.path and it starts incoming.service whenever something arrives (PathExists= fires when a file exists, PathChanged= when it is written and closed, DirectoryNotEmpty= while files are waiting).
Limits and sandboxing: make the service weak on purpose
Resource limits (cgroups): these are enforced by the kernel, per service
MemoryMax=500M hard cap: above it the service is OOM-killed inside its own cgroup; MemoryHigh=400M soft cap: it gets slowed down first
CPUQuota=50% at most half of one CPU (200% = two CPUs); CPUWeight= and IOWeight= are relative shares
TasksMax=200 at most 200 processes/threads (stops fork bombs)
LimitNOFILE=65535 open files per process (the ulimit equivalent)
$ sudo systemctl set-property myapp.service MemoryMax=500M CPUQuota=50% # change on a RUNNING service, and keep it (add --runtime to make it temporary)
$ systemctl show myapp -p MemoryCurrent,CPUUsageNSec,TasksCurrent # live numbers
$ systemd-cgtop # top, but per service
Sandboxing: take away what the service does not need. systemd-analyze security scores a unit from 0 (hardened) to 10 (wide open). Real scores for the demo service above, before and after adding a block of restrictions (offline mode, no root needed):
$ systemd-analyze security --offline=true units/plain.service | tail -1
→ Overall exposure level for plain.service: 8.2 EXPOSED :-(
$ systemd-analyze security --offline=true units/hardened.service | tail -1
→ Overall exposure level for hardened.service: 1.9 OK :-)
$ systemd-analyze security myapp.service # score a LIVE service: the table shows every remaining exposure and a score for each
ProtectSystem=strict the whole file system is read-only for the service, except what you allow with ReadWritePaths= (or StateDirectory=)
ProtectHome=true /home, /root and /run/user are invisible; PrivateTmp=true: its own /tmp
NoNewPrivileges=true it can never gain more privileges (blocks setuid tricks)
RestrictAddressFamilies only the socket types it needs; SystemCallFilter=@system-service blocks system calls normal services never use
CapabilityBoundingSet= drop every Linux capability (add back only what is needed, e.g. CAP_NET_BIND_SERVICE for ports below 1024: AmbientCapabilities=CAP_NET_BIND_SERVICE)
Add the restrictions one at a time and test: a too-strict sandbox shows up as status=226/NAMESPACE (a ReadWritePaths= folder that does not exist) or permission errors in the journal. Do not chase a score of 0: a low score on a service that can only talk to localhost is plenty.
User units and lingering
$ mkdir -p ~/.config/systemd/user && $EDITOR ~/.config/systemd/user/syncthing.service
$ systemctl --user daemon-reload && systemctl --user enable --now syncthing.service
$ journalctl --user -u syncthing.service
$ loginctl enable-linger alice # keep alice's user services running when she is logged OUT, and start them at boot
User units run as that user and need no root. Without lingering they stop when the last session ends.
When a service will not start: read the exit status
$ systemctl status myapp.service --no-pager -l # state, the last log lines, and the exit status
$ journalctl -xeu myapp.service --no-pager | tail -40 # full log with explanations
$ systemctl reset-failed myapp.service # clear a "failed" state or a start-limit hit
Status Meaning and the usual fix
status=203/EXEC systemd could not run ExecStart: wrong path, not executable (chmod +x), wrong interpreter in the #! line, or a noexec mount
status=200/CHDIR WorkingDirectory= does not exist
status=217/USER the User= does not exist (or DynamicUser conflicts with a fixed User)
status=226/NAMESPACE a sandbox setting failed: often a ReadWritePaths= or BindPaths= folder that does not exist
status=1/FAILURE, 2, ... the program itself exited with an error: read ITS log lines above the status
status=127 command not found inside a shell command line: use absolute paths (PATH is minimal in services)
status=137 or code=killed, signal=KILL killed: OOM (journalctl -k | grep -i oom) or a MemoryMax= limit, or the timeout stopped it
Result: start-limit-hit it restarted too fast too often: fix the cause, then systemctl reset-failed (StartLimitIntervalSec=/StartLimitBurst= control the limit)
Result: timeout it did not become ready in TimeoutStartSec: a slow start, or Type=notify without the program notifying
"Unit ... not found" wrong name, forgot daemon-reload, or the file is not in a unit directory
"Failed to start ... Unit is masked" it was disabled hard: sudo systemctl unmask NAME
Works in your shell but not as a service? A service has a tiny environment: no login PATH, no HOME unless set, no tty, different user, maybe a sandbox. Reproduce it with systemd-run --pty (shown earlier) and set what is missing with Environment= or EnvironmentFile=.
Boot and startup time
$ systemd-analyze # total boot time (firmware, loader, kernel, userspace)
$ systemd-analyze blame | head # slowest units to start
$ systemd-analyze critical-chain # what the boot actually WAITED for
$ systemd-analyze plot > boot.svg # a picture of the whole boot (open in a browser)
Tips
Always run systemd-analyze verify on a new or edited unit before daemon-reload.
Never edit files under /usr/lib/systemd/system: use systemctl edit (a drop-in) so package updates cannot overwrite your change.
After= is only ordering. If B needs A, say Wants= or Requires= too.
Enable the timer, not the service. Test the job by hand with systemctl start NAME.service before you trust the schedule.
Give every long-running service a restart policy (Restart=on-failure with RestartSec=) and a limit (MemoryMax=), and every one-off job Type=oneshot.
Harden step by step with systemd-analyze security, and test after each directive.
If it works from your shell but not as a service, think: environment, user, working directory, sandbox.
"systemd does too much" is a fine opinion. It is also the reason the same twelve lines now give you a daemon with restart, limits, logging and sandboxing.
