📊 Checkmk Guide

Checkmk Raw Edition: monitoring that finds the services by itself
docker run -d --name cmk -p 8080:5000 -p 8000:8000 -e CMK_PASSWORD=... checkmk/check-mk-raw:2.4.0p37
su - cmk -c "cmk -vI web1"

Checkmk watches servers, services and devices and alerts you when they go wrong, like Nagios does. The big difference is who writes the list of things to check. With Nagios you write every service by hand. With Checkmk you add a host, install a small agent on it, and the server asks the agent what is there (disks, interfaces, memory, services, processes) and creates the services for you. Thresholds are not set per check in files but by rules in the web interface ("for all hosts that match this, use these limits"). It keeps graphs and metrics too.

The free version is the Raw Edition (a version string ending in .cre). It runs a classic Nagios core under a Checkmk web interface, which the lab confirmed. Paid editions add a faster core, a distributed setup and more; this guide covers what the free edition does.

Every example below ran for real: Checkmk Raw 2.4.0p37 (the current stable release when this was written), as the official Docker image, on a private Docker network with a second container, "web1" (Ubuntu 24.04 with nginx), as the monitored server, and a third host, "web2", that does not exist, to see what a dead host looks like. The image is built on Ubuntu 22.04, which does not matter for using it. A container has no systemd, so two steps that systemd does for you on a real machine were done by hand and are marked. Most actions were done through the REST API and the command line, which do exactly what the web interface does; the web pages named in this guide were opened in the lab, but not every button was clicked. Things that were not run (installing from the .deb package, updating a site, SNMP, HTTPS, distributed monitoring) are marked "not run in the lab".

Five terms.

  site        one complete monitoring system with its own web interface, its own core, its own configuration, run by its own Linux user. The user and the site have the same name (here: cmk). One server can host several sites
  omd         the command (as root) that creates, starts, stops, backs up and updates sites: omd create, omd start, omd status, omd backup
  core        the part that schedules and runs the checks. In the Raw Edition this is Nagios: omd config cmk show says CORE: nagios, and omd status lists a nagios service
  agent       a small program on each monitored host that reports everything it sees, in one text blob made of sections (<<<uptime>>>, <<<lnx_if>>> ...)
  discovery   the server reads the sections and creates the services (CPU utilization, Memory, Interface 2, Uptime ...) for you

The flow, in the order the lab did it:

  1. add the host to the site (a name and an address)
  2. install the agent on the host and register it with the site (a TLS certificate, so only your site can pull data)
  3. run service discovery; the services appear
  4. activate the changes (the core reloads)
  5. the checks run every minute; rules set the thresholds; notifications go to contacts

Two Checkmk habits to know early: changes made in the web interface or the API are pending until someone clicks Activate changes, and what happens to a host is decided by rules whose conditions match it, not by the host's own file.

The quickest way to try Checkmk is the official image. This is the command the lab used (a throwaway password; put a real one in your environment, and keep the site data in a volume):

$ docker volume create cmk-sites
$ docker run -d -it --name cmk --hostname cmk \
    -p 8080:5000 -p 8000:8000 \
    --tmpfs /opt/omd/sites/cmk/tmp:uid=1000,gid=1000 \
    -v cmk-sites:/omd/sites \
    -e CMK_SITE_ID=cmk -e CMK_PASSWORD=labpass-cmk1 \
    checkmk/check-mk-raw:2.4.0p37

  -p 8080:5000   the web interface (port 5000 inside the container)
  -p 8000:8000   the agent receiver: the port where agents register
  -v cmk-sites   the site's data. Without it the monitoring data is lost with the container
  CMK_SITE_ID    the site name. The first start creates it
  CMK_PASSWORD   the password of the first user, cmkadmin

The first start takes a few minutes. The container reports "healthy" when it is ready:

$ docker logs cmk | tail -3
### CREATING SITE 'cmk'
Adding /opt/omd/sites/cmk/tmp to /etc/fstab.
Going to set TMPFS to off.
$ docker inspect -f '{{.State.Health.Status}}' cmk
healthy

Open http://SERVER:8080/cmk/ and log in as cmkadmin. In the lab the login page answered with HTTP 200 at http://127.0.0.1:8081/cmk/check_mk/login.py (the lab mapped port 8081).

Look at what is running:

$ docker exec cmk omd version
OMD - Open Monitoring Distribution Version 2.4.0p37.cre
$ docker exec cmk omd sites
cmk              2.4.0p37.cre     default version
$ docker exec cmk su - cmk -c "omd status"
agent-receiver:     running
mkeventd:           running
rrdcached:          running
redis:              running
npcd:               running
automation-helper:  running
ui-job-scheduler:   running
nagios:             running
apache:             running
crontab:            running
---------------------------
Overall state:      running

Installing on a normal server instead: download the package for your system from checkmk.com (the download page lists check-mk-raw-2.4.0p37_0.noble_amd64.deb for Ubuntu 24.04), then sudo apt install ./check-mk-raw-2.4.0p37_0.noble_amd64.deb, sudo omd create mysite and sudo omd start mysite. That path was not run in the lab. The site name is also the user, so choose it once.

Everything about the site is done as the site user. su - cmk (or docker exec -it cmk su - cmk) gives you a shell with the Checkmk commands (cmk, lq, unixcat) in the path. As root, omd is the tool.

Site settings are read with omd config (as root; the container ran it directly):

$ omd config cmk show
AGENT_RECEIVER: on
AGENT_RECEIVER_PORT: 8000
APACHE_MODE: own
APACHE_TCP_ADDR: 0.0.0.0
APACHE_TCP_PORT: 5000
AUTOMATION_HELPER: on
AUTOSTART: on
CORE: nagios
LIVESTATUS_TCP: off
MKEVENTD: on
MULTISITE_AUTHORISATION: on
MULTISITE_COOKIE_AUTH: on
PNP4NAGIOS: on
TMPFS: off
...

The ones you will touch: APACHE_TCP_PORT and APACHE_TCP_ADDR (where the web interface listens; with a normal install, the system Apache forwards to the site), AGENT_RECEIVER_PORT (8000 here), LIVESTATUS_TCP (off: the data interface is only reachable locally, which is the safe default), ADMIN_MAIL, and AUTOSTART (start the site at boot). Change one with omd config cmk set VAR VALUE, with the site stopped (not run in the lab).

The commands in daily use:

  omd status [SITE]              show the services of the site (see the list above)
  omd stop|start|restart|reload SITE [SERVICE]   one service can be named: omd restart cmk apache
  omd backup SITE ARCHIVE        write a backup tarball
  omd restore ...                restore it (into an existing site, or create one)
  omd update SITE                move the site to another installed version (not run in the lab)
  omd su SITE                    a shell as the site user
  omd cp / mv / rm SITE ...      copy, rename, remove a site

Backup was run in the lab:

$ omd backup cmk /tmp/cmk-backup.tar.gz
$ ls -l /tmp/cmk-backup.tar.gz
-rw-r--r-- 1 root root 1692916 Oct 14 15:43 /tmp/cmk-backup.tar.gz
$ tar tzf /tmp/cmk-backup.tar.gz | head -3
cmk/version
cmk/
cmk/.bashrc

A freshly set-up site with two hosts is 1.7 MB and 1706 entries. The archive holds the configuration, the history and the graphs, so it is the file to copy off the machine. Restore was not run in the lab.

The interface has a Monitor side (what is happening) and a Setup side (what is configured). These pages were opened in the lab, with the title the page showed:

  /cmk/check_mk/view.py?view_name=allhosts        All hosts
  /cmk/check_mk/view.py?view_name=allservices     All services
  /cmk/check_mk/view.py?view_name=svcproblems     Service problems
  /cmk/check_mk/wato.py?mode=folder               Main (the folder tree of hosts)
  /cmk/check_mk/wato.py?mode=edit_host&host=web1  Properties of host web1
  /cmk/check_mk/wato.py?mode=changelog            Activate pending changes
  /cmk/check_mk/wato.py?mode=notifications        Notifications
  /cmk/check_mk/wato.py?mode=users                Users
  /cmk/check_mk/wato.py?mode=contact_groups       Contact groups
  /cmk/check_mk/wato.py?mode=timeperiods          Time periods

Menus move a little between versions, so remember the idea and not the clicks: hosts live in folders (the Main folder is the root), a host has attributes (address, agent type, contact groups...), and "Activate changes" is always the last step. Setup is the old name for the configuration module (Checkmk used to call it WATO, and the URLs still say wato.py).

A tip for scripting: you can open the same pages with a session cookie. The lab logged in with a POST to /cmk/check_mk/login.py (fields _username, _password and _login=1) and then read pages with the cookie. For automation, use an automation user and the REST API instead (next sections). A second point from the lab: putting _username and _secret in the URL of a page did not log a browser in, so do not rely on that.

Through the web interface: Setup, the folder where the host should live, Add host, name and address, Save. Through the REST API (the same thing, scriptable). First a tiny helper that talks to the API as cmkadmin:

#!/bin/bash
# capi.sh METHOD PATH [JSON]
curl -s -X "$1" -H 'Authorization: Bearer cmkadmin labpass-cmk1' -H 'Accept: application/json' \
     ${3:+-H 'Content-Type: application/json' -d "$3"} "http://127.0.0.1:8081/cmk/check_mk/api/1.0$2"

The Authorization header is "Bearer USERNAME PASSWORD" (the lab used cmkadmin's password; an automation user with its secret works the same way). The API describes itself at /cmk/check_mk/api/1.0/ui/ on a running site (not opened in the lab). Check it works:

$ ./capi.sh GET /version | python3 -m json.tool | head -8
{
    "site": "cmk",
    "group": "",
    "rest_api": { "revision": "0" },
    "versions": {
        "apache": [2, 4, 52],
        "checkmk": "2.4.0p37.cre",

Add the host:

$ ./capi.sh POST /domain-types/host_config/collections/all '{"host_name":"web1","folder":"/","attributes":{"ipaddress":"172.21.0.3","alias":"Web server 1"}}'
(returns the host, title "Web server 1")

and see that nothing is live yet:

$ ./capi.sh GET /domain-types/activation_run/collections/pending_changes
1 pending: create-host | cmkadmin

Activation. A change is pending until it is activated, and the API asks you to prove you saw the pending list. Asking without the ETag is refused:

Precondition required     (HTTP 428)

The ETag is in the header of the pending-changes answer. Send it back in If-Match:

import urllib.request, json
B = "http://127.0.0.1:8081/cmk/check_mk/api/1.0"
H = {"Authorization": "Bearer cmkadmin labpass-cmk1", "Accept": "application/json"}
g = urllib.request.urlopen(urllib.request.Request(B + "/domain-types/activation_run/collections/pending_changes", headers=H))
r = urllib.request.Request(B + "/domain-types/activation_run/actions/activate-changes/invoke",
        data=json.dumps({"redirect": False, "sites": ["cmk"], "force_foreign_changes": True}).encode(),
        method="POST", headers={**H, "Content-Type": "application/json", "If-Match": g.headers["ETag"]})
print(json.load(urllib.request.urlopen(r))["title"])        # Activation status: In progress.

About 10 seconds later the pending list was empty and the host was monitored:

$ su - cmk -c 'cmk --list-hosts'
web1
$ su - cmk -c 'lq "GET hosts\nColumns: name address state plugin_output\n"'
web1;172.21.0.3;0;OK - 172.21.0.3 rta 0.048ms lost 0%

A host with no agent is checked with ping only. The state 0 above is a ping result. Services come when the agent is there.

The agent is a package that the site itself serves, so the version always matches. On the host to monitor (web1):

$ curl -s -o agent.deb -w "HTTP %{http_code} %{size_download} bytes\n" http://cmk:5000/cmk/check_mk/agents/check-mk-agent_2.4.0p37-1_all.deb
HTTP 200 5129174 bytes
$ sudo apt install ./agent.deb
$ cmk-agent-ctl --version
cmk-agent-ctl 2.4.0p37

On the site, the same files are in ~/share/check_mk/agents/: the .deb, the .rpm, and agents for Windows, AIX, Solaris, FreeBSD and more. Use the file name for your site's version.

What the package installs: /usr/bin/check_mk_agent (the agent itself, a shell script), /usr/bin/cmk-agent-ctl (the controller), and systemd units: cmk-agent-ctl-daemon.service, check-mk-agent.socket and check-mk-agent-async.service. Directories: /etc/check_mk, /usr/lib/check_mk_agent/plugins (extra plugins) and /usr/lib/check_mk_agent/local (your own checks).

Registration. The controller makes sure that only your site can read the agent's data. You register the host once; this needs the site's agent receiver (port 8000) and a user who may register:

$ sudo cmk-agent-ctl register --hostname web1 --server cmk:8000 --site cmk --user cmkadmin --password PASSWORD --trust-cert
Registration complete.
$ cmk-agent-ctl status
Version: 2.4.0p37
Agent socket: operational
IP allowlist: any

Connection: cmk/cmk
        UUID: 48538339-1983-40d4-8276-acbd8f5df39c
        Local:
                Connection mode: pull-agent
                Connecting to receiver port: 8000
                Certificate issuer: Site 'cmk' agent signing CA
                Certificate validity: Wed, 14 Oct 2026 15:37:15 +0000 - Tue, 14 Oct 2031 15:37:15 +0000
        Remote:
                Connection mode: pull-agent
                Hostname: web1

The certificate is valid for five years. --trust-cert accepts the site's own certificate on first contact, which is fine in a lab. On a real network, verify the certificate instead of trusting it blindly (not run in the lab).

Two things the lab hit that systemd normally does for you:

1. The first registration failed with: ERROR [cmk_agent_ctl] Something seems wrong with the agent socket (/run/check-mk-agent.socket), aborting. The agent socket is the systemd unit check-mk-agent.socket, and a container has no systemd. The lab stood in for it with socat, which runs the agent for every connection: socat UNIX-LISTEN:/run/check-mk-agent.socket,fork,mode=666 EXEC:/usr/bin/check_mk_agent. On a real machine, systemctl status check-mk-agent.socket is the thing to look at (not run in the lab).
2. The daemon that answers the site's pulls was started by hand: cmk-agent-ctl daemon. On a real machine it is cmk-agent-ctl-daemon.service. It listens on port 6556:

$ ss -ltn | grep 6556
LISTEN 0      4096               *:6556             *:*

Allow port 6556 from the Checkmk server through any firewall. (Registration also needs port 8000 to be reachable, once.)

Look at what the agent says, from the site's side. This is the first command to run when something looks wrong:

$ su - cmk -c 'cmk -d web1 | head -14'
<<<check_mk>>>
Version: 2.4.0p37
AgentOS: linux
Hostname: web1
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
OSType: linux
OSName: Ubuntu
OSVersion: 24.04
OSPlatform: ubuntu
...
$ su - cmk -c 'cmk -d web1 | grep -c "^<<<"'
20

20 sections in the lab, from <<<check_mk>>> to <<<mounts>>>, <<<ps_lnx>>>, <<<uptime>>>, <<<lnx_if>>> and <<<tcp_conn_stats>>>. The agent also reports the details of its own registration (<<<cmk_agent_ctl_status>>>) and labels such as {"cmk/device_type":"container"}.

Discovery turns the agent's sections into services. From the command line, as the site user:

$ cmk -vI web1
Discovering services and host labels on: web1
web1:
+ FETCHING DATA
Get piggybacked data
+ ANALYSE DISCOVERED HOST LABELS
SUCCESS - Found 6 host labels
+ ANALYSE DISCOVERED SERVICES
+ EXECUTING DISCOVERY PLUGINS (20)
  1 checkmk_agent
  1 cpu_utilization_os
  1 diskstat
  1 lnx_if
  1 mem_used
  1 systemd_units_services_summary
  1 systemd_units_sockets_summary
  1 tcp_conn_stats
  1 uptime
SUCCESS - Found 9 services

(cmk -I discovers; cmk -II does a fresh discovery, which is what the lab used after adding a local check.) The web interface has the same step on the host's Services page in Setup (not clicked in the lab).

The result is stored (the "autochecks"). To make the core use it, reload it. This is what Activate changes does in the interface:

$ cmk -O
Precompiling host checks...OK
Validating Nagios configuration...OK
Reloading monitoring core...OK

(cmk -R restarts the core instead; not run in the lab. Prefer -O.) Livestatus shows what is now monitored:

$ lq "GET services\nColumns: host_name description state plugin_output\nFilter: host_name = web1\n"
web1;Uptime;0;Up since 2026-10-14 15:20:04, Uptime: 18 minutes 14 seconds
web1;TCP Connections;0;Established: 1
web1;Systemd Socket Summary;0;Total: 0, Disabled: 0, Failed: 0
web1;Systemd Service Summary;0;Total: 0, Disabled: 0, Failed: 0
web1;Memory;0;RAM: 0.22% - 69.4 MiB of 31.0 GiB
web1;Interface 2;0;[eth0], (up), MAC: F2:2B:1F:54:DA:6C, Speed: 10 GBit/s
web1;Disk IO SUMMARY;0;Read: 0.00 B/s, Write: 137 B/s
web1;Check_MK Agent;0;Version: 2.4.0p37, OS: linux, Agent plug-ins: 0, Local checks: 0
web1;Check_MK;0;[agent] Success, [piggyback] Success (but no data found for this host), execution time 0.7 sec
web1;Check_MK Discovery;0;
web1;CPU utilization;0;

(Output is empty on the first run of a check; a minute later Check_MK Discovery read: Services: all up to date, Host labels: all up to date, execution time 0.0 sec.)

Three of these are not about the host itself: Check_MK checks that the data could be fetched, Check_MK Agent checks the agent (its version, plugins and local checks), and Check_MK Discovery is the one that notices change. When something new appears on the machine it reports "unmonitored services" as a warning, which tells you your monitoring has fallen behind the machine. The lab saw the wording with a new local check (next section).

Note that a container showed no "Filesystem" service: the agent's disk section saw nothing worth monitoring in the container's overlay file system. On a real server the mounted file systems become Filesystem / and so on.

The quickest way to monitor something specific is a local check: a script in /usr/lib/check_mk_agent/local that prints one line per service. The format is: state, "name", metrics, text. The state is 0 OK, 1 WARNING, 2 CRITICAL or 3 UNKNOWN.

$ cat /usr/lib/check_mk_agent/local/backup_age
#!/bin/bash
AGE=3
if [ "$AGE" -lt 24 ]; then S=0; elif [ "$AGE" -lt 48 ]; then S=1; else S=2; fi
echo "$S \"Backup age\" age=$AGE;24;48 Last backup finished $AGE hours ago"
$ chmod +x /usr/lib/check_mk_agent/local/backup_age

The line that comes out has three parts: the state (0), the service name in quotes (Backup age), the metrics name=value;warn;crit (age=3;24;48, which also draws a graph) and the free text. A dash instead of metrics means none. From the site, the data is there at once:

$ cmk -d web1 | sed -n '/<<<local/,/^<<</p'
<<<local:sep(0)>>>
0 "Backup age" age=3;24;48 Last backup finished 3 hours ago

and the next discovery finds the service:

$ cmk -vII web1 | grep -E "^ +[0-9]+ |SUCCESS"
...
  1 local
...
SUCCESS - Found 10 services

After cmk -O the service exists: web1;Backup age;0;Last backup finished 3 hours ago, and the Check_MK Agent service now says Local checks: 1.

Check what discovery would add, without changing anything. A second local check (nginx_conf_age) was added to the agent and not yet discovered:

$ cmk --check-discovery web1
Services unmonitored: 1 (local: 1)(!), Host labels: all up to date, execution time 0.7 sec | execution_time=0.650 ...
Service unmonitored: local: Nginx config age

The "Check_MK Discovery" service runs this check on its own schedule: right after the new script was added it still said "all up to date", because it had not run again. Larger scripts can print any number of lines, each one a service. A script that is slow should not be a local check: put it in plugins/ with a cache time in the file name, or run it by cron and read a file (the Checkmk manual has the details; not run in the lab).

Checkmk does not have a file per check. It has rulesets, and each ruleset has rules like "for these hosts or services, use this value". The first matching rule wins for a given host and service. You can see the effect on a host with cmk -D:

$ cmk -D web1
web1
Addresses:              172.21.0.3
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], ...
Labels:                 [cmk/device_type:container], [cmk/os_name:Ubuntu], [cmk/os_version:24.04], ...
Host groups:            check_mk
Contact groups:         check-mk-notify, all
Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:
  TCP: 172.21.0.3:6556
Services:
  checktype            item        params                                                  description
  local                Backup age  {}                                                      Backup age
  cpu_utilization_os   None        {'util': (0.01, 0.02)}                                  CPU utilization
  mem_used             None        {'levels': (150.0, 200.0)}                              Memory
  lnx_if               2           {'errors': {'both': ('perc', (0.01, 0.1))}, ...}        Interface 2
  ...

(Shortened: the real output is much wider. This listing was taken after the CPU rule below was in place; the Memory line shows a default.) The params column is the answer to "which limits apply to this service?". When a threshold is wrong, start here.

A rule through the API. This one lowers the CPU levels on web1 so low that an alert fires, to prove that the rule is used. The ruleset is checkgroup_parameters:cpu_utilization_os; the value is written as a Python literal, here the dictionary {'util': (warn, crit)}, in percent:

$ cat rule-cpu.json
{"ruleset": "checkgroup_parameters:cpu_utilization_os",
 "folder": "/",
 "properties": {"description": "Lab: very low CPU levels on web1 to force an alert", "disabled": false},
 "value_raw": "{'util': (0.01, 0.02)}",
 "conditions": {"host_name": {"match_on": ["web1"], "operator": "one_of"}}}
$ ./capi.sh POST /domain-types/rule/collections/all "$(cat rule-cpu.json)"

The conditions decide who the rule applies to (host name, folder, host tags, labels, service names). The first attempt in the lab used ('fixed', (0.01, 0.02)) and the API refused it:

Problem in (sub-)field 'util'  (HTTP 400): Levels on total CPU utilization: The data type of the value does not match any of the allowed alternatives.

The error names the field, and the plain tuple was accepted. The easy way to learn the format of a value is to set it once in the web interface and then read the rule back through the API. After activating the changes (see "Add a host"), cmk -D shows 'util': (0.01, 0.02) and the service goes CRITICAL on its next run (checks run every minute):

$ lq "GET services\nColumns: description state state_type current_attempt max_check_attempts plugin_output\nFilter: host_name = web1\nFilter: description = CPU utilization\n"
CPU utilization;2;1;1;1;Total CPU: 1.33% (warn/crit at 0.01%/0.02%)

The state is 2 (CRITICAL), the state type is 1 (HARD) and the attempt 1 of 1: the default is that a problem is confirmed at once and alerts at once. Rules for the number of attempts exist (the ruleset "Maximum number of check attempts for service") if you want a Nagios-style SOFT period.

Notifications are also rules. A new site has one default rule, which the log names 'HTML email to all contacts about service/host status changes'. The rule says "when a state changes, mail all contacts of the host or service". Contacts are the users who belong to the contact groups of that host or service.

The lab found the trap in this sentence. The CPU service went CRITICAL, the rule matched, and the notification log said:

2026-10-14 15:40:20,065 [20] [cmk.base.notify] Got raw notification (web1;CPU utilization) context with 54 variables
2026-10-14 15:40:20,065 [20] [cmk.base.notify] Global rule 'HTML email to all contacts about service/host status changes'...
2026-10-14 15:40:20,065 [20] [cmk.base.notify]  -> matches!
2026-10-14 15:40:20,065 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.

Nothing was sent. "No notification has been created" means: the rule matched, and nobody was in its audience. The host had these contacts:

$ lq "GET hosts\nColumns: name contacts contact_groups\nFilter: name = web1\n"
web1;check-mk-notify;all,check-mk-notify
$ lq "GET contactgroups\nColumns: name members\n"
check_mk;check_mk
check-mk-notify;check-mk-notify
all;

The host was in the contact group all, and the group all had no members. The administrator cmkadmin was in no contact group (a user has to be put in one), and had no email address either. Both fixes are on the user: an email address and a contact group. Through the API:

$ ./capi.sh PUT /objects/user_config/cmkadmin '{"contact_options": {"email": "cmk@localhost"}}'
$ ./capi.sh PUT /objects/user_config/cmkadmin '{"contactgroups": ["all"]}'

Both PUT calls need If-Match with the user's ETag. In the lab the first attempt got HTTP 412: ETag didn't match. Expected 0448371c... and the value in the message changed on every call, so the reliable way was a small script that fetches the ETag, sends the PUT, and on a 412 sends it again with the value from the message (it worked on the second try). In the web interface the same two settings are on the user's properties page, and the group membership is on the same page.

Activate the changes, and check that the host now has the contact:

$ lq "GET hosts\nColumns: name contacts\nFilter: name = web1\n"
web1;check-mk-notify,cmkadmin

Then test the whole path without waiting for a failure, with a custom notification (the Livestatus command that the web interface's "Send custom notification" uses):

$ echo "COMMAND [$(date +%s)] SEND_CUSTOM_SVC_NOTIFICATION;web1;CPU utilization;1;cmkadmin;Lab test of notifications" | unixcat ~/tmp/run/live
$ tail -n 8 ~/var/log/notify.log
... Global rule 'HTML email to all contacts about service/host status changes'...
...  -> matches!
...    - adding notification of cmkadmin via mail
... Executing 1 notifications:
...   * notifying cmkadmin via mail, parameters: graphs_per_notification, notifications_with_graphs, ...
... sending command LOG;SERVICE NOTIFICATION: cmkadmin;web1;CPU utilization;CRITICAL;mail;Total CPU: 1.33% (warn/crit at 0.01%/0.02%)
...      executing /omd/sites/cmk/share/check_mk/notifications/mail
...      Output: Spooled mail to local mail transmission agent

The notify.log is the place to look; ~/var/log/notify.log in the site. A mail did arrive in the lab (the container had a local-only Postfix installed with sudo apt install postfix mailutils):

Subject: Checkmk: web1/CPU utilization Custom Notification (CRIT)
To: cmk@localhost
From: cmk@cmk

It is an HTML mail with a graph, so it is large (the mailbox was 73 KB after a few). "Spooled mail to local mail transmission agent" means Checkmk handed it to sendmail; whether it leaves the server is the job of your mail setup, which the lab did not configure beyond local delivery. Test mail delivery by hand first (echo test | mail -s test you@example.com).

Hosts notify too, and at once: the nonexistent web2 went DOWN and the log said HOST NOTIFICATION: cmkadmin;web2;DOWN;mail;CRITICAL - 172.21.0.99: Host unreachable @ 172.21.0.4. rta nan, lost 100%. The notification rules (Setup, Notifications) can do much more: choose contacts, times, services, send to chat or ticket tools through plugins, bulk, escalate. Those parts were not run in the lab.

The same operator actions as in Nagios, through the REST API. Acknowledge the CPU problem with a comment:

$ ./capi.sh POST /domain-types/acknowledge/collections/service '{"acknowledge_type": "service", "service_description": "CPU utilization", "host_name": "web1", "comment": "known: lab thresholds are deliberately tiny", "sticky": true, "notify": false, "persistent": false}'

Schedule a downtime for the host that is not built yet (times in ISO format):

$ ./capi.sh POST /domain-types/downtime/collections/host '{"downtime_type": "host", "host_name": "web2", "start_time": "2026-10-14T15:00:00Z", "end_time": "2026-10-15T15:00:00Z", "comment": "server not built yet"}'

Both took effect within seconds:

$ lq "GET services\nColumns: description state acknowledged\nFilter: host_name = web1\nFilter: description = CPU utilization\n"
CPU utilization;2;1
$ lq "GET hosts\nColumns: name state scheduled_downtime_depth\n"
web2;1;1
web1;0;0
$ lq "GET comments\nColumns: host_name service_description author comment entry_type\n"
web1;CPU utilization;cmkadmin;known: lab thresholds are deliberately tiny;4
web2;;cmkadmin;This host has been scheduled for fixed downtime from 2026-10-14 15:00:00 to 2026-10-15 15:00:00.  Notifications for the host will not be sent out during that time period.;2

The service stays CRITICAL (state 2) but acknowledged (the 1 at the end); the host is DOWN (state 1) and in downtime (depth 1). The comment on the downtime says it directly: "Notifications for the host will not be sent out during that time period."

Livestatus is the query language for the monitoring data, and you have already been using it. lq reads a query from the command line and prints the result; it talks to the site's socket (~/tmp/run/live), which unixcat can also use for commands. The query is text: GET table, Columns, Filter lines, one per line:

$ lq "GET hosts\nColumns: name address state state_type plugin_output\n"
web2;172.21.0.99;1;1;CRITICAL - 172.21.0.99: Host unreachable @ 172.21.0.4. rta nan, lost 100%
web1;172.21.0.3;0;1;OK - 172.21.0.3 rta 0.057ms lost 0%

Tables used in the lab: hosts, services, contactgroups and comments (contacts, downtimes and log also exist). States are numbers (services: 0 OK, 1 WARN, 2 CRIT, 3 UNKNOWN; hosts: 0 UP, 1 DOWN, 2 UNREACHABLE) and state_type is 1 for HARD. The data is only reachable locally unless you turn on LIVESTATUS_TCP, which is off by default and should stay off unless a trusted network and a firewall are in place.

Not everything can run an agent: a printer, a switch, a server that is not built yet. The host still gets a ping check (the host's own state), and you can add checks that run from the Checkmk server (HTTP, TCP, DNS...) or over SNMP. For the lab, web2 was added with the agent switched off, which is the tag_agent attribute set to no-agent:

$ ./capi.sh POST /domain-types/host_config/collections/all '{"host_name":"web2","folder":"/","attributes":{"ipaddress":"172.21.0.99","alias":"Web server 2 (not built yet)","tag_agent":"no-agent"}}'

and after activation, 30 seconds later:

$ lq "GET hosts\nColumns: name address state state_type plugin_output\n"
web2;172.21.0.99;1;1;CRITICAL - 172.21.0.99: Host unreachable @ 172.21.0.4. rta nan, lost 100%
web1;172.21.0.3;0;1;OK - 172.21.0.3 rta 0.057ms lost 0%

DOWN and HARD at once, with a HOST NOTIFICATION. Active checks of web services from the server (the "Check HTTP web service" rule) and SNMP monitoring are set up with rules too; they were not run in the lab.

Install (Docker)    docker run -d --name cmk -p 8080:5000 -p 8000:8000 -v cmk-sites:/omd/sites -e CMK_SITE_ID=cmk -e CMK_PASSWORD=... checkmk/check-mk-raw:2.4.0p37
Site (as root)      omd status   omd start|stop|restart|reload SITE   omd config SITE show   omd backup SITE FILE   su - SITE
Site (as site user) cmk -d HOST (agent data)   cmk -vI HOST (discover)   cmk -vII HOST (rediscover)   cmk -D HOST (config and params)   cmk -O (reload core)
Check discovery     cmk --check-discovery HOST
Agent               download: http://SERVER:PORT/SITE/check_mk/agents/    install, then: cmk-agent-ctl register --hostname H --server S:8000 --site SITE --user U --password P
Agent check         cmk-agent-ctl status     ss -ltn | grep 6556     check_mk_agent | head
Local check         /usr/lib/check_mk_agent/local/NAME     prints:  0 "Service name" metric=1;warn;crit Text
REST API            curl -H 'Authorization: Bearer USER PASSWORD' http://HOST/SITE/check_mk/api/1.0/...
Activate            GET .../domain-types/activation_run/collections/pending_changes (note ETag), then POST .../activate-changes/invoke with If-Match
Livestatus          lq "GET services\nColumns: host_name description state plugin_output\nFilter: state != 0\n"
Commands            echo "COMMAND [$(date +%s)] CMD;args" | unixcat ~/tmp/run/live
Notification log    ~/var/log/notify.log     test: SEND_CUSTOM_SVC_NOTIFICATION;host;service;1;user;comment
Contacts            user needs an email address AND a contact group, and the host needs that contact group
States              services 0 OK 1 WARN 2 CRIT 3 UNKNOWN    hosts 0 UP 1 DOWN 2 UNREACHABLE    state_type 1 = HARD

Agent registration: "Something seems wrong with the agent socket"   the agent socket (check-mk-agent.socket under systemd) is not running: systemctl status check-mk-agent.socket
cmk -d HOST shows nothing or times out     the daemon is not running (cmk-agent-ctl-daemon.service), or port 6556 is blocked, or the host address in the site is wrong
Host is monitored but has no services       you added it but did not discover and activate: cmk -vI HOST then cmk -O (or the Services page and Activate changes)
Changes do not take effect                 they are still pending: GET pending_changes, then activate. Nothing is live until then
REST API: Precondition required (428)        activation needs the ETag of the pending changes in If-Match
REST API: Precondition failed (412)         the object changed: use the ETag the message names. For users the value in the message changed every time, so retry with it
REST API: Problem in (sub-)field 400          the value format is wrong: read the field name; learn the format from a rule made in the web interface
Alert but no mail: "no notification has been created"   nobody was in the audience: give the user an email address and a contact group, and put the host in that group
Mail "spooled" but never arrives           Checkmk handed it to sendmail: the mail setup of the server is the problem, test with mail by hand
Check_MK Discovery service is WARNING         there are unmonitored services (the machine changed): cmk --check-discovery HOST shows which, then discover and activate
"UNKNOWN" for a local check                the script printed something else than "state name metrics text", or lacks the executable bit
Web page says "Invalid user input" or "Error"   a page was opened with a missing or wrong parameter; use the interface to get to it, not hand-made URLs
A service never alerts                  its rule may have a looser threshold first: cmk -D HOST shows which parameters apply, and the first matching rule wins
Browser login with _username and _secret in the URL did not log in   use the login form, or the REST API with an automation user

Look at cmk -d HOST first. If the data is not there, nothing downstream can be right.
Discover, then activate, then wait one minute. Most "it does not work" moments are one of those three steps.
Treat Check_MK Discovery as a feature: it is your reminder that the machine changed and the monitoring did not.
Set thresholds with rules and conditions (folders, tags, labels), not host by host. A new server should be monitored correctly just by being in the right folder.
Use cmk -D HOST to see the parameters that really apply. It ends most arguments about "why did it not alert".
Give every host a contact group and every contact group a member. Test notifications on a new site before you need them (a custom notification takes one command).
Start with few rules and defaults. The defaults are sensible; replace them where you have a reason.
Use local checks for the things only you know (is the backup recent, is the certificate valid for 30 days, is the queue short).
Keep the agent version in step with the site. The site serves the matching package; use it.
Script with the REST API and lq, not by clicking, when you add the fifteenth host.
Back up with omd backup on a schedule, and copy the file off the server. Keep the same version when you restore.
Keep LIVESTATUS_TCP off, the web interface behind HTTPS and the agent port reachable only from the Checkmk server.