📈 Zabbix Guide

Zabbix: monitoring with an agent, a database and a web interface
sudo apt install zabbix-server-pgsql zabbix-frontend-php zabbix-nginx-conf zabbix-sql-scripts zabbix-agent2
zabbix_get -s web1 -k agent.ping

Zabbix collects numbers and states from your machines, stores them in a database, compares them with rules called triggers, and tells people when something is wrong. A small program called the agent runs on each machine and answers questions ("what is the CPU load?", "is port 80 open?"). The server asks the agents on a schedule, keeps the history for as long as you tell it to, and shows everything in a web interface. It can also receive values pushed to it, probe from outside, read SNMP devices and use proxies for remote sites; this guide covers the core you need first.

Every example below ran for real: Zabbix 7.0.31 (the 7.0 LTS series, the long-term release when this was written; 8.0 LTS was announced for early 2027), installed from Zabbix's own repository on Ubuntu 24.04, with PostgreSQL 16, nginx and PHP 8.3, in two Docker containers on a private network: "zbx" (server, database, web interface, an agent for itself, a local mail server) and "web1" (an agent and nginx, the machine being watched). A container has no systemd, so the daemons were started by hand; the commands for the services on a real machine (systemctl) are marked "not run in the lab". Most configuration was done through the Zabbix API, because it is exact and repeatable; everything it does can also be done in the web interface, which was not clicked through in the lab. Menu paths such as Data collection > Hosts are given from the Zabbix documentation and were not clicked in the lab. Passwords in the examples are throwaway.

The pieces and their ports:

  zabbix-server     the daemon that polls, receives, evaluates triggers and sends alerts. Listens on 10051 (for pushed data and agents that report in)
  database          PostgreSQL (or MySQL): configuration and history. In this guide: PostgreSQL 16
  web frontend      PHP pages served by nginx (or Apache): the interface and the API at /api_jsonrpc.php
  zabbix-agent2     the agent on every monitored machine (zabbix-agent is the older C version). Listens on 10050
  zabbix_get        a command that asks an agent one question by hand: the best debugging tool
  zabbix_sender     a command that pushes a value into Zabbix from a script

Terms you will meet:

  host         a machine or device you monitor
  item         one thing collected from a host: a key such as system.cpu.load[all,avg1], how often to ask, and how long to keep it
  trigger      a condition on items that decides "problem" or "OK": last(/web1/net.tcp.service[tcp,,80])=0
  template     a bundle of items, triggers, graphs and discovery rules that you link to hosts, so 50 servers are configured once
  action       what to do on a problem (send a message, run a script), with conditions
  media type   a way to deliver messages (email, chat, SMS, webhook)

Passive and active checks. In a passive check the server connects to the agent and asks (agent listens on 10050). In an active check the agent connects to the server (10051), downloads its list of checks, and sends the results. Passive is simplest and is what this guide uses; active is better for agents behind a firewall or NAT (the template "Linux by Zabbix agent active" exists; not run in the lab).

Zabbix publishes its own repository; use it, because Ubuntu's archive carries an old version. Add the repository (this installs only the repository definition), then install the server, frontend, database and a local agent:

$ wget https://repo.zabbix.com/zabbix/7.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_7.0+ubuntu24.04_all.deb
$ sudo dpkg -i zabbix-release_latest_7.0+ubuntu24.04_all.deb
$ sudo apt update
$ apt-cache policy zabbix-server-pgsql | grep Candidate
  Candidate: 1:7.0.31-1+ubuntu24.04
$ sudo apt install zabbix-server-pgsql zabbix-frontend-php php8.3-pgsql zabbix-nginx-conf zabbix-sql-scripts zabbix-agent2 postgresql

What that brought in (versions as installed):

  postgresql-16       16.15-0ubuntu0.24.04.1
  nginx               1.24.0-2ubuntu7.18
  php8.3-fpm          8.3.6-0ubuntu0.24.04.11
  zabbix-server-pgsql, zabbix-frontend-php, zabbix-nginx-conf, zabbix-sql-scripts, zabbix-agent2    1:7.0.31-1+ubuntu24.04
$ zabbix_server --version | head -1
zabbix_server (Zabbix) 7.0.31

(For Apache instead of nginx use zabbix-apache-conf. For MySQL use zabbix-server-mysql.) Check the Zabbix documentation for the repository file of the release you want; 7.0 is the LTS line used here. 7.2 and 7.4 are shorter-lived releases.

The database. Create a role and a database, then load the schema that the package ships. The schema import took 10 seconds and no errors:

$ sudo -u postgres psql -c "CREATE USER zabbix WITH PASSWORD 'labpass-zbx1';"
$ sudo -u postgres createdb -O zabbix -E Unicode -T template0 zabbix
$ time (zcat /usr/share/zabbix-sql-scripts/postgresql/server.sql.gz | sudo -u zabbix psql -q zabbix)
real    0m9.922s
$ sudo -u zabbix psql -tAc "select count(*) from information_schema.tables where table_schema='public'" zabbix
203
$ sudo -u zabbix psql -tAc "select userid, username, roleid from users" zabbix
1|Admin|3
2|guest|4

(The lab's container had to start the cluster by hand with pg_ctlcluster 16 main start; on a real machine PostgreSQL is already running.) The database already has the two accounts: Admin (a super admin) and guest.

Tell the server about the database in /etc/zabbix/zabbix_server.conf. The packaged file has the right defaults for the name and user and a commented DBPassword line:

DBName=zabbix
DBUser=zabbix
DBPassword=labpass-zbx1
LogFile=/var/log/zabbix/zabbix_server.log
PidFile=/run/zabbix/zabbix_server.pid

Start it. On a real machine: sudo systemctl enable --now zabbix-server zabbix-agent2 (not run in the lab; the lab started the daemons directly, with zabbix_server -c /etc/zabbix/zabbix_server.conf). Look at the log:

$ tail -n 5 /var/log/zabbix/zabbix_server.log
 17283:20261015:003634.683 server #48 started [configuration syncer worker #1]
 17284:20261015:003634.684 server #49 started [internal poller #1]
 17289:20261015:003634.690 server #50 started [proxy group manager #1]
 17272:20261015:003637.678 Zabbix agent item "vfs.fs.get" on host "Zabbix server" failed: first network error, wait for 15 seconds
 17206:20261015:003641.725 item "Zabbix server:zabbix[process,ipmi poller,avg,busy]" became not supported: No "ipmi poller" processes started.

$ pgrep -c -x zabbix_server
53
$ ss -ltn | grep 10051
0.0.0.0:10051   [::]:10051

The server runs 53 processes. Among them: 16 preprocessing workers, 5 pollers, 5 trappers, 5 history pollers, 5 discovery workers, 4 history syncers, 3 alerters and 2 LLD workers, and one each of timer, task manager, service manager and trigger housekeeper. Two log lines are worth understanding now. The first says the server's own agent check failed: the host "Zabbix server" exists from the start, and it needs an agent running on the same machine (start zabbix_agent2 there; on a real install it starts with the package). The second is not an error: IPMI polling is not started by default, so those internal items say "not supported" until you enable it.

The frontend is PHP served by nginx. The package ships the nginx config at /etc/zabbix/nginx.conf (linked into /etc/nginx/conf.d/zabbix.conf) and a PHP-FPM pool at /etc/zabbix/php-fpm.conf, which listens on the socket /var/run/php/zabbix.sock as www-data. The nginx file has the listen port and the server name commented out; uncomment and set them:

$ grep -n -E "^\s*(listen|server_name)" /etc/zabbix/nginx.conf
2:        listen          8080;
3:        server_name     zbx;

The lab removed nginx's default site (rm /etc/nginx/sites-enabled/default), because it competes for the same configuration, and started PHP-FPM and nginx (systemctl restart nginx php8.3-fpm on a real machine; not run in the lab). Then open http://SERVER:8080/. The first visit normally runs a setup wizard that checks PHP, asks for the database details and writes /etc/zabbix/web/zabbix.conf.php. The lab wrote that file by hand instead, which is what the wizard does:

<?php
$DB["TYPE"]     = "POSTGRESQL";
$DB["SERVER"]   = "localhost";
$DB["PORT"]     = "0";
$DB["DATABASE"] = "zabbix";
$DB["USER"]     = "zabbix";
$DB["PASSWORD"] = "labpass-zbx1";
$DB["SCHEMA"]   = "";
$DB["ENCRYPTION"] = false;
... (the optional TLS and vault settings stay empty)
$ZBX_SERVER      = "localhost";
$ZBX_SERVER_PORT = "10051";
$ZBX_SERVER_NAME = "Lab Zabbix";

(Owned by www-data, mode 640: it holds the database password. The wizard's other fields were left out because the default values were used.) With that in place:

$ curl -s -o /dev/null -w "frontend: HTTP %{http_code}\n" http://localhost:8080/
frontend: HTTP 200
$ curl -s http://localhost:8080/index.php | grep -o "<title>[^<]*"
<title>Lab Zabbix: Zabbix

ZBX_SERVER_NAME ("Lab Zabbix") appears in the browser title and the page header; use it to tell your test and production servers apart. The first login is Admin with the password zabbix. Change it immediately. The lab did it through the API, because that is the safest and the fastest way to see that the old one stops working:

$ ./zapi.sh user.login '{"username":"Admin","password":"zabbix"}'          (zapi.sh is the helper in the API section below)
$ ./zapi.sh user.update '{"userid":"1","passwd":"NEWPASSWORD","current_passwd":"zabbix"}' TOKEN
{"jsonrpc":"2.0","result":{"userids":["1"]},"id":1}
$ ./zapi.sh user.login '{"username":"Admin","password":"zabbix"}'
{"jsonrpc":"2.0","error":{"code":-32500,"message":"Application error.","data":"Incorrect user name or password or account is temporarily blocked."},"id":1}

The change ended the current session as well (the next call said "Session terminated, re-login, please."), so you log in again with the new password.

Zabbix has a JSON-RPC API at /api_jsonrpc.php. Everything in the web interface goes through it, and it is how you create hundreds of hosts without clicking. A tiny helper, as used in the lab:

#!/bin/bash
# zapi.sh METHOD 'PARAMS-JSON' [TOKEN]
M=$1; P=${2:-{\}}; T=$3
curl -s -H 'Content-Type: application/json-rpc' ${T:+-H "Authorization: Bearer $T"} \
     -d "{\"jsonrpc\":\"2.0\",\"method\":\"$M\",\"params\":$P,\"id\":1}" http://localhost:8080/api_jsonrpc.php

The version call needs no login. A login returns a session token that you pass as a Bearer token:

$ ./zapi.sh apiinfo.version '[]'
{"jsonrpc":"2.0","result":"7.0.31","id":1}
$ ./zapi.sh user.login '{"username":"Admin","password":"zabbix"}'
{"jsonrpc":"2.0","result":"<32 characters>","id":1}
$ ./zapi.sh host.get '{"output":["hostid","host","status","available"]}' TOKEN
{"jsonrpc":"2.0","result":[{"hostid":"10084","host":"Zabbix server","status":"0"}],"id":1}

For scripts, use an API token instead of a password in a file. A token is created for a user, generated once (the string is shown only then), and works as a Bearer token without any login:

$ ./zapi.sh token.create '{"name":"lab-automation","userid":"1"}' TOKEN
{"jsonrpc":"2.0","result":{"tokenids":["1"]},"id":1}
$ ./zapi.sh token.generate '["1"]' TOKEN                       (returns the 64-character token string)
$ curl -s -H 'Content-Type: application/json-rpc' -H "Authorization: Bearer $APITOKEN" \
     -d '{"jsonrpc":"2.0","method":"host.get","params":{"output":["host"]},"id":1}' http://localhost:8080/api_jsonrpc.php
{"jsonrpc":"2.0","result":[{"hostid":"10683","host":"web1"},{"hostid":"10084","host":"Zabbix server"}],"id":1}

A wrong token, or no token on a protected method, gets the same refusal:

{"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid params.","data":"Not authorized."},"id":1}

and token.get shows when each token was last used (lastaccess), which is how you find ones nobody uses and can delete. A token has the rights of its user, so create the user for automation with only the permissions it needs.

On each machine to monitor (here web1, Ubuntu 24.04), add the same repository and install the agent:

$ wget https://repo.zabbix.com/zabbix/7.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_7.0+ubuntu24.04_all.deb
$ sudo dpkg -i zabbix-release_latest_7.0+ubuntu24.04_all.deb && sudo apt update
$ sudo apt install zabbix-agent2
$ zabbix_agent2 --version | head -1
zabbix_agent2 (Zabbix) 7.0.31

The packaged defaults point at the same machine, so they must be changed. Three lines in /etc/zabbix/zabbix_agent2.conf:

$ grep -E "^(Server|ServerActive|Hostname)=" /etc/zabbix/zabbix_agent2.conf
Server=127.0.0.1
ServerActive=127.0.0.1
Hostname=Zabbix server

  Server=zbx          the IP address or name of the Zabbix server that may ask this agent (passive checks). Anyone else is refused
  ServerActive=zbx    where this agent reports to for active checks
  Hostname=web1       the name this machine has in Zabbix. It must match the host's name in the frontend exactly

After editing, they read Server=zbx, ServerActive=zbx and Hostname=web1. Start the agent (sudo systemctl enable --now zabbix-agent2 on a real machine; not run in the lab) and test it locally. The agent can run a check itself with -t:

$ zabbix_agent2 -t agent.ping
agent.ping                                    [s|1]
$ zabbix_agent2 -t "net.tcp.service[http]"
net.tcp.service[http]                         [s|1]

Now the most useful habit in Zabbix: test from the server what the server will ask, with zabbix_get (apt install zabbix-get). If this works, the host will work; if not, no amount of frontend clicking helps:

$ for k in agent.ping agent.version system.hostname "system.cpu.load[all,avg1]" "vfs.fs.size[/,pfree]" "net.tcp.service[tcp,,80]"; do printf "%-30s" "$k"; zabbix_get -s web1 -k "$k"; done
agent.ping                    1
agent.version                 7.0.31
system.hostname               web1
system.cpu.load[all,avg1]     1.314453
vfs.fs.size[/,pfree]          76.425354
net.tcp.service[tcp,,80]      1
$ zabbix_get -s web1 -k no.such.key
ZBX_NOTSUPPORTED: Unknown metric no.such.key

A check that fails in the agent returns ZBX_NOTSUPPORTED followed by the reason. If the Server= line did not allow your server, the connection is refused, which is the second most common problem after a firewall on port 10050. The agent's own log, /var/log/zabbix/zabbix_agent2.log, says what it sees. In the lab, before the host existed on the server, it wrote:

[101] no active checks on server [zbx:10051]: host [web1] not found
[101] active checks on server started to fail
...
[101] active checks on server are active again

"host [web1] not found" means the Hostname does not match any host on the server (here: the host had not been added yet). Once the host was added, the next refresh said the checks were active again.

A host belongs to at least one host group (groups are for permissions and for finding things) and usually links one or more templates. Create them through the API (everything here is also under Data collection in the web interface):

$ ./zapi.sh hostgroup.create '{"name":"Web servers"}' TOKEN
{"jsonrpc":"2.0","result":{"groupids":["22"]},"id":1}
$ ./zapi.sh host.create '{"host":"web1","name":"Web server 1",
    "interfaces":[{"type":1,"main":1,"useip":0,"ip":"","dns":"web1","port":"10050"}],
    "groups":[{"groupid":"2"},{"groupid":"22"}],
    "templates":[{"templateid":"10001"}]}' TOKEN
{"jsonrpc":"2.0","result":{"hostids":["10683"]},"id":1}

The interface type 1 is an agent. useip 0 with a dns name tells Zabbix to connect by name (it resolves it each time, which survives an IP change). Group 2 is "Linux servers" (it exists from the start), 22 is the one just made, and the template 10001 is "Linux by Zabbix agent", found with:

$ ./zapi.sh template.get '{"output":["host"],"search":{"host":"Linux by Zabbix agent"}}' TOKEN
[('10001', 'Linux by Zabbix agent'), ('10343', 'Linux by Zabbix agent active')]

Within a minute the host's interface became available (the green indicator in the interface):

$ ./zapi.sh host.get '{"output":["host","status"],"selectInterfaces":["available","error"],"hostids":["10683"]}' TOKEN
web1 status 0 interfaces [{'available': '1', 'error': ''}]

The template created 43 items on the host at once, and the lab read some of them back (name, latest value, unit, how often):

agent.ping                                  1       every 1m
system.cpu.load[all,avg1]            1.254883       every 1m
system.cpu.util                     12.647599 %
system.cpu.util[,idle]              87.352401 %     every 1m
system.cpu.util[,user]               9.61367  %     every 1m
system.uptime                            2200 uptime every 30s
vm.memory.size[pavailable]           79.69874 %     every 1m
net.if.in["eth0"]                           0 bps   every 3m

and a set of triggers, all in the "not a problem" state (value 0), for example:

average  Linux: High memory utilization
average  Linux: Zabbix agent is not available
average  Linux: Lack of available memory
average  Linux: Load average is too high
average  Linux: FS [/etc/hostname]: Space is critically low
...

and three discovery rules, which create more items and triggers automatically as things appear on the machine:

net.if.discovery                Network interface discovery        1h
vfs.dev.discovery               Block devices discovery            1h
vfs.fs.dependent.discovery      Mounted filesystem discovery

That is low-level discovery. For every network interface, disk and mounted file system the agent reports, Zabbix creates the items and triggers from a prototype. Note what it found in the lab: the file systems named FS [/etc/hostname], FS [/etc/hosts] and FS [/etc/resolv.conf], because in a container those are the only mounts the agent sees. On a real server you get / and your data disks.

Open the template (Data collection > Templates in the interface) to see what you are linking: it is a good way to learn the item keys and the trigger expressions Zabbix itself recommends.

The template covers the machine. For a service you add an item and a trigger. This one checks whether anything answers on port 80. The item type 0 is "Zabbix agent" (the server asks the agent), value type 3 is a whole number, and it is checked every 30 seconds. It needs the host's interface id:

$ ./zapi.sh hostinterface.get '{"output":["interfaceid"],"hostids":["10683"]}' TOKEN       (the lab's was 31)
$ ./zapi.sh item.create '{"hostid":"10683","name":"HTTP port 80 is open","key_":"net.tcp.service[tcp,,80]",
    "type":0,"value_type":3,"delay":"30s","interfaceid":"31"}' TOKEN
{"jsonrpc":"2.0","result":{"itemids":["50871"]},"id":1}

The trigger says when it counts as a problem. Zabbix 7 trigger expressions name the host and key in slashes:

$ ./zapi.sh trigger.create '{"description":"nginx on {HOST.NAME} is not answering on port 80",
    "expression":"last(/web1/net.tcp.service[tcp,,80])=0","priority":4,
    "comments":"Check: systemctl status nginx"}' TOKEN
{"jsonrpc":"2.0","result":{"triggerids":["25290"]},"id":1}

  last(/web1/key)        the latest value
  avg(/web1/key,5m)      the average over 5 minutes (also min, max, sum, count, nodata, change)
  and, or, not            combine conditions
  {HOST.NAME}            macros that are filled in when the problem is shown or sent

The template's own load trigger shows a longer one (macros expanded):

min(/web1/system.cpu.load[all,avg1],5m)/last(/web1/system.cpu.num)>1.5
and last(/web1/system.cpu.load[all,avg5])>0
and last(/web1/system.cpu.load[all,avg15])>0

Read it: the lowest 1-minute load over the last 5 minutes, divided by the number of CPUs, above 1.5 (and the 5 and 15 minute loads exist). Using min() over a window means a single spike does not fire it. The priority is a number: 0 not classified, 1 information, 2 warning, 3 average, 4 high, 5 disaster. Choose it for the notifications you want (actions can filter on it). The trigger's recovery is automatic: when the expression is no longer true, the problem is resolved.

Test a trigger without waiting for a real failure: break the thing it watches, which is what the next sections do.

A fresh Zabbix sends nothing. The lab looked at what ships:

$ ./zapi.sh action.get '{"output":["actionid","name","status","eventsource"]}' TOKEN
{'actionid': '2', 'name': 'Auto discovery. Linux servers.', 'status': '1', 'eventsource': '1'}
{'actionid': '3', 'name': 'Report problems to Zabbix administrators', 'status': '1', 'eventsource': '0'}
{'actionid': '4', 'name': 'Report not supported items', 'status': '1', 'eventsource': '3'}
...
$ ./zapi.sh mediatype.get '{"output":["mediatypeid","name","type","status"]}' TOKEN
1 Email type 0 disabled
3 SMS type 2 disabled
4 Email (HTML) type 0 disabled
34 Gmail ... disabled     35 Gmail relay ... disabled     36 Office365 ... disabled

Every action and every media type is disabled (status 1 means disabled), and the Admin user has no media (no address to send to). Three things need to be done, in this order.

1. Make the server able to send mail. The lab installed a local-only Postfix (sudo apt install postfix mailutils, choosing "Local only") and tested it by hand first: echo test | mail -s test someone@zbx.example.com. For real mail use your own relay or a smarthost; the media type only needs to know where to hand the mail.
2. Enable and set up the Email media type:

$ ./zapi.sh mediatype.update '{"mediatypeid":"1","status":0,"smtp_server":"localhost","smtp_port":25,
    "smtp_helo":"zbx.example.com","smtp_email":"zabbix@zbx.example.com","smtp_security":0,"smtp_authentication":0}' TOKEN
{"jsonrpc":"2.0","result":{"mediatypeids":["1"]},"id":1}

3. Give the user an address, and enable the action. The first attempt used zabbixadmin@localhost and Zabbix refused it:

Invalid parameter "/1/medias/1/sendto/1": an email address is expected.

It validates the format, and "user@localhost" does not pass (no dot in the domain). With a proper name:

$ ./zapi.sh user.update '{"userid":"1","medias":[{"mediatypeid":"1","sendto":["zabbixadmin@zbx.example.com"],
    "active":0,"severity":63,"period":"1-7,00:00-24:00"}]}' TOKEN
$ ./zapi.sh action.update '{"actionid":"3","status":0}' TOKEN

The media entry has a severity mask (63 means all six severities), a time period ("1-7,00:00-24:00" is every day, all day) and the address. Action 3, "Report problems to Zabbix administrators", sends a message to the user group "Zabbix administrators" (id 7, which includes Admin) when a trigger fires, and a "resolved" message when it recovers. Create your own actions to route by host group, severity or tag (Alerts > Actions in the interface).

The lab stopped nginx on web1 and followed the problem through its whole life, polling the API. Times are UTC. First, the preconditions were checked (nginx up, the item at 1, no open problem), then:

00:51:40  nginx stopped on web1
00:51:51  problem opened (11 seconds later): "nginx on Web server 1 is not answering on port 80", severity High
00:51:51  email sent (the message id shows the same second)
00:53:02  acknowledged through the API, with a message
00:53:08  nginx started again
00:53:21  the next check returned 1: problem resolved, "Resolved in 1m 30s" email sent

The email, as it arrived (Zabbix sends the text base64-encoded; this is decoded):

Subject: Problem: nginx on Web server 1 is not answering on port 80
From: <zabbix@zbx.example.com>
To: <zabbixadmin@zbx.example.com>

Problem started at 00:51:51 on 2026.10.15
Problem name: nginx on Web server 1 is not answering on port 80
Host: Web server 1
Severity: High
Operational data: 0
Original problem ID: 59

and the recovery:

Subject: Resolved in 1m 30s: nginx on Web server 1 is not answering on port 80

Problem has been resolved at 00:53:21 on 2026.10.15
Problem name: nginx on Web server 1 is not answering on port 80
Problem duration: 1m 30s
Host: Web server 1
Severity: High
Original problem ID: 59

Only two emails for the whole incident: the acknowledgement sent nothing, because the shipped action has no step for updates. Add an "update operation" to the action if you want colleagues told when someone acknowledges.

The problem is listed with problem.get:

$ ./zapi.sh problem.get '{"output":["eventid","name","severity","acknowledged","clock","r_eventid"],"recent":true}' TOKEN
('59', 'nginx on Web server 1 is not answering on port 80', 'sev4', 'ack=0', 'OPEN')

(r_eventid is the id of the recovery event; "0" means still open.) Acknowledge it so that others see you are on it, adding a message. The action number is a sum of flags: 2 acknowledge, 4 add a message, so 6 does both (other flags exist for closing, changing severity and suppressing; see the API reference, not run in the lab):

$ ./zapi.sh event.acknowledge '{"eventids":"59","action":6,"message":"nginx stopped for a config change, back in 5 minutes"}' TOKEN
{"jsonrpc":"2.0","result":{"eventids":[59]},"id":1}
('59', ..., 'ack=1', 'OPEN')

The second item to notice: the check runs every 30 seconds, but the problem appeared 11 seconds after the stop, and it was resolved 13 seconds after the start. The speed comes from when the next check falls, so the time to detect is up to the check interval. For faster detection use a shorter delay or an active check, at the cost of more load.

Planned work should not page anyone. A maintenance period on a host (or a group) tells Zabbix to keep collecting but to hold the notifications. The lab created one for web1 and then broke nginx on purpose:

$ ./zapi.sh maintenance.create '{"name":"web1 nginx upgrade","active_since":NOW_MINUS_60,"active_till":NOW_PLUS_3600,
    "maintenance_type":0,"hosts":[{"hostid":"10683"}],
    "timeperiods":[{"timeperiod_type":0,"start_date":NOW_MINUS_60,"period":3600}]}' TOKEN
{'maintenanceids': ['1']}

(times are seconds since 1970; maintenance_type 0 means "with data collection": items are still read; type 1 stops collection.) Thirty-five seconds later, host.get showed maintenance_status 1: in maintenance. With nginx stopped for a minute:

problem 61  nginx on Web server 1 is not answering on port 80   suppressed=1   (suppression data: maintenance 1)
item still collected during maintenance: net.tcp.service = 0
emails: still 2 (the same two as before)

The problem existed, was recorded, and was flagged suppressed, but no email was sent: during maintenance, the monitoring still sees the truth and the people are not told. What happens when the window ends is the part to know. The lab deleted the maintenance (as if it had ended) while nginx was still down, and then started nginx a few seconds later: the mailbox got the "Problem:" email (late, at the end of the maintenance) and then "Resolved in 1m 0s". So a problem that is still open when maintenance ends is announced then.

Maintenance is under Data collection > Maintenance in the interface. Use groups instead of single hosts for a rolling upgrade, and prefer a real time period (an end time) over deleting it by hand.

Not everything can be asked. A nightly backup job knows how big its result was; it can tell Zabbix. For that you create an item of type trapper (type 2): Zabbix does not poll it, it waits for values, and accepts them only from the addresses you list:

$ ./zapi.sh item.create '{"hostid":"10683","name":"Backup size (pushed)","key_":"backup.size","type":2,
    "value_type":3,"units":"B","trapper_hosts":"zbx,127.0.0.1,172.0.0.0/8"}' TOKEN

Send a value from the machine (apt install zabbix-sender):

$ zabbix_sender -z zbx -s web1 -k backup.size -o 1453289472
Response from "zbx:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000302"
sent: 1; skipped: 0; total: 1
$ ./zapi.sh item.get '{"output":["name","lastvalue","units"],"hostids":["10683"],"filter":{"key_":"backup.size"}}' TOKEN
[{'name': 'Backup size (pushed)', 'lastvalue': '1453289472', 'units': 'B'}]

-z is the server, -s the host name exactly as it is in Zabbix, -k the key, -o the value. Several values go in a file, one per line (host key value), with -i:

$ printf "web1 backup.size 1500000000\nweb1 backup.size 1510000000\n" > vals.txt
$ zabbix_sender -z zbx -i vals.txt
Response from "zbx:10051": "processed: 2; failed: 0; total: 2; ..."

Two failures from the lab, both with the same message and one "processed: 0; failed: 1":

Response from "zbx:10051": "processed: 0; failed: 1; total: 1; seconds spent: 0.000058"

1. The value was sent a second after the item was created. The server learns about configuration changes every few seconds (the configuration cache, refreshed on a schedule), so the item did not exist yet for it. Wait about ten seconds, or force it with zabbix_server -R config_cache_reload (next section), and send again.
2. A host name that does not exist in Zabbix (the lab tried nosuchhost) fails the same way. The failure tells you something is wrong with the host name or the key, but not which.

A trigger on the pushed value works like any other. A trigger such as nodata(/web1/backup.size,24h)=1 fires when the job stops reporting, which catches a job that stopped running (the expression is from the Zabbix documentation; not run in the lab).

A macro is a name in braces with a value, and templates use them for every threshold, so you can change a limit for one host without touching the template. The Linux template's macros the lab looked at:

{$CPU.UTIL.CRIT}                  90
{$LOAD_AVG_PER_CPU.MAX.WARN}      1.5
{$MEMORY.AVAILABLE.MIN}           20M
{$MEMORY.UTIL.MAX}                90

The load trigger above uses {$LOAD_AVG_PER_CPU.MAX.WARN} and shows 1.5. For a build server where a high load is normal, override it on the host:

$ ./zapi.sh usermacro.create '{"hostid":"10683","macro":"{$LOAD_AVG_PER_CPU.MAX.WARN}","value":"4",
    "description":"web1 is a build box: higher load is normal"}' TOKEN
{"jsonrpc":"2.0","result":{"hostmacroids":["8085"]},"id":1}

The same trigger on web1 now reads ...>4 instead of ...>1.5 (trigger.get with expandExpression shows the expanded text), and every other host keeps 1.5. The order is: host macro, then the macros of the linked templates, then global macros. Put the explanation in the description, because six months later nobody remembers why.

By default the server and the agents talk in clear text, which is fine for a lab and not for a network you share. The simplest protection is a pre-shared key (PSK): a secret known only to the agent and the server, with an identity name. The lab set it up for web1.

On the agent, create the key, give the agent access to it and tell it to use it:

$ openssl rand -hex 32 | sudo tee /etc/zabbix/web1.psk
$ sudo chown zabbix:zabbix /etc/zabbix/web1.psk && sudo chmod 640 /etc/zabbix/web1.psk
$ cat >> /etc/zabbix/zabbix_agent2.conf <<EOF
TLSAccept=psk,unencrypted
TLSConnect=psk
TLSPSKIdentity=web1-psk
TLSPSKFile=/etc/zabbix/web1.psk
EOF

(psk,unencrypted at first, so that nothing breaks while you switch over.) Restart the agent. From the server (with the key file copied there), zabbix_get can now speak either way, and refuses a wrong identity:

$ zabbix_get -s web1 -k agent.ping
1
$ zabbix_get -s web1 -k agent.ping --tls-connect psk --tls-psk-identity web1-psk --tls-psk-file /etc/zabbix/web1.psk
1
$ zabbix_get -s web1 -k agent.ping --tls-connect psk --tls-psk-identity wrong --tls-psk-file /etc/zabbix/web1.psk
zabbix_get [18912]: Get value error: TCP successful, cannot establish TLS to [[web1]:10050]: SSL_connect() set result code to SSL_ERROR_SSL: ... TLS write fatal alert "decode error"

Then tell the server to use PSK for this host (Host > Encryption in the interface), and tighten the agent to PSK only:

$ ./zapi.sh host.update '{"hostid":"10683","tls_connect":2,"tls_accept":2,"tls_psk_identity":"web1-psk","tls_psk":"THE-64-HEX-KEY"}' TOKEN
$ sed -i 's/^TLSAccept=.*/TLSAccept=psk/' /etc/zabbix/zabbix_agent2.conf      (and restart the agent)
$ zabbix_get -s web1 -k agent.ping
zabbix_get [18942]: Get value error: cannot read from socket: [104] Connection reset by peer

Plain connections are now cut off, and the host stayed available (available: 1) because the server connects with the key. Keep the key file readable only by the Zabbix user. Certificates instead of a PSK are the other option (TLSCAFile, TLSCertFile and TLSKeyFile on the agent); not run in the lab.

Runtime control. The server can be told to do things without a restart, from the same machine:

$ zabbix_server -R config_cache_reload
Runtime control command was forwarded successfully
$ tail -n 4 /var/log/zabbix/zabbix_server.log
 17155:20261015:005202.282 forced reloading of the configuration cache
 17154:20261015:005202.283 forced reloading of the service manager cache
 17154:20261015:005202.285 finished forced reloading of the service manager cache
 17155:20261015:005202.302 finished forced reloading of the configuration cache
$ zabbix_server -R log_level_increase=housekeeper        (more detail for one process type; then log_level_decrease=housekeeper)

The log is the first place to look when something does not collect: it names the host, the item and the reason ("became not supported: ..."). The frontend shows the latest error for an unsupported item in its Info column.

Housekeeping. The server deletes old history on a schedule. The defaults, from the API:

$ ./zapi.sh housekeeping.get '{"output":["hk_history_mode","hk_history","hk_trends_mode","hk_trends","hk_events_mode"]}' TOKEN
{"hk_events_mode":"1","hk_history_mode":"1","hk_history":"31d","hk_trends":"365d","hk_trends_mode":"1"}

History (every value) is kept for 31 days, trends (hourly averages) for 365 days. Those two numbers, together with the number of items and how often they are checked, decide the size of the database; change them under Administration > Housekeeping. The packaged server config also has defaults you may need to tune as you grow, all commented out in the file: StartPollers=5, CacheSize=32M, HistoryCacheSize=16M, HousekeepingFrequency=1 (hours). The packaged file sets Timeout=4. The built-in template "Zabbix server health" (linked to the Zabbix server host from the start) has items and triggers for exactly this: busy processes and full caches.

The database. A new install is small; the lab's, after loading the schema and the default templates and collecting for an hour:

$ sudo -u zabbix psql -tAc "select pg_size_pretty(pg_database_size('zabbix'))" zabbix
72 MB
$ sudo -u zabbix psql -tAc "select relname, pg_size_pretty(pg_total_relation_size(oid)) from pg_class where relkind='r' and relnamespace='public'::regnamespace order by pg_total_relation_size(oid) desc limit 4" zabbix
items|14 MB
changelog|8704 kB
valuemap_mapping|4864 kB
item_preproc|3712 kB

In a real installation the history tables grow to dominate. Back up with pg_dump (custom format, which is compressed and restorable with pg_restore):

$ time (sudo -u zabbix pg_dump -Fc zabbix > /tmp/zabbix.dump)
real    0m0.820s
$ ls -l /tmp/zabbix.dump
-rw-r--r-- 1 root root 5135389 Oct 15 00:55 /tmp/zabbix.dump

That is 5 MB for 72 MB of database, mostly the default templates. On a big system, back up the configuration tables on their own and the history less often; Zabbix's documentation lists which tables are history. Upgrades (stop the server, install the new packages, start the new server, which upgrades the schema itself), proxies, SNMP monitoring, web scenarios and the services and SLA features were not run in the lab.

Repository          wget .../zabbix-release_latest_7.0+ubuntu24.04_all.deb ; dpkg -i ; apt update
Server packages     zabbix-server-pgsql zabbix-frontend-php php8.3-pgsql zabbix-nginx-conf zabbix-sql-scripts zabbix-agent2 postgresql
Database            createuser/createdb -O zabbix ; zcat server.sql.gz | sudo -u zabbix psql zabbix ; DBPassword= in zabbix_server.conf
Services            zabbix-server, zabbix-agent2, nginx, php8.3-fpm, postgresql     (systemctl status ...)
Ports               10051 server (from agents and senders)   10050 agent (from the server)   8080/80/443 frontend
Agent config        Server=  ServerActive=  Hostname=  in /etc/zabbix/zabbix_agent2.conf   (Hostname must equal the host name in the frontend)
Test an item        zabbix_get -s HOST -k KEY      on the agent: zabbix_agent2 -t KEY
Push a value        zabbix_sender -z SERVER -s HOST -k KEY -o VALUE        (trapper item; wait for the config cache)
Reload config       zabbix_server -R config_cache_reload
API                 POST /api_jsonrpc.php  Authorization: Bearer TOKEN     apiinfo.version, user.login, host.create, item.create, trigger.create, problem.get, event.acknowledge
Trigger syntax      last(/host/key)=0    avg(/host/key,5m)>X    nodata(/host/key,10m)=1      priority 0..5
Notifications       enable the media type, give the user an address, enable the action   (all off by default)
Maintenance         maintenance.create : problems are suppressed, nothing is sent, open problems are announced when it ends
Defaults to change  Admin password (zabbix); Server=127.0.0.1 and Hostname=Zabbix server in the agent file; history 31d / trends 365d

Host stays red (agent not available)         from the server run zabbix_get -s HOST -k agent.ping. Refused or timed out: firewall on 10050, or the agent is not running. "Connection reset": the server is not in Server= or the TLS settings do not match
agent log: "host [NAME] not found"           Hostname in the agent file does not match the host name in Zabbix (case counts), or the host was not created yet
ZBX_NOTSUPPORTED: Unknown metric              the key is misspelt or the plugin is not loaded; compare with the template's item
Item is "not supported" in the frontend       the Info icon shows the reason; also read the server log for 'became not supported'
No email, nothing in the log                  the media type is disabled, the user has no media, the action is disabled, or the action conditions do not match. All three are off by default
"an email address is expected"                user@localhost is refused; use a name with a dot in the domain
Email sent but nothing arrives                the mail server is the next place: Zabbix hands over the message. Test by hand (echo test | mail ...) first
zabbix_sender: "processed: 0; failed: 1"      wrong host name or key, or the item was created less than ~10 seconds ago (config cache). Retry, or zabbix_server -R config_cache_reload
Problem during planned work still paged       the host was not in the maintenance (check host groups), or the problem started before the window began
Trigger never fires                           check the expression with the item's latest data and the "last" value; macros expand per host (trigger.get with expandExpression)
Frontend shows "Zabbix server is not running"  the server process or its port 10051 is down, or $ZBX_SERVER in zabbix.conf.php is wrong. Read zabbix_server.log
Login with Admin / zabbix fails after changing it   as expected: the old password is rejected and the session is ended; use the new one
Server slow, queue growing                    pollers busy: raise StartPollers, check the process-busy internal items, and look at the database (disk, shared_buffers)
Database keeps growing                        history/trends keeping times too long for the number of items; check Administration > Housekeeping

Learn zabbix_get. It reproduces what the server asks, in one line, and settles most "is it the agent or the server" questions.
Set the passwords and the Server= lines before the first host: change Admin/zabbix, restrict Server= to the Zabbix server, and use PSK or certificates once the agents leave the lab.
Link templates and use macros for thresholds. A host that needs special treatment gets a macro, not a new template.
Put hosts in groups from day one (by role, by site, by owner). Permissions, maintenance and actions all use groups.
Make the notification chain work before anything depends on it: mail server, media type, user address, action. Break something on purpose and watch the email arrive.
Use maintenance windows for planned work, and remember that open problems are announced when the window ends.
Prefer triggers that look at a window (avg, min, count) over single values, so a blip does not page you.
Add nodata() triggers for anything pushed by scripts: a silent job is the worst kind of failure.
Use the API for anything repeated (hosts, groups, macros, maintenance), and API tokens instead of passwords in scripts.
Watch the server's own health items and the database size. A monitoring system that falls over silently is worse than none.
Keep the database backed up and test a restore on another machine: dump the configuration often and the history rarely.
Do not keep history longer than you will use it. Trends (365 days) are enough for capacity planning.