🔥 Prometheus Guide

Prometheus: a time-series database that collects numbers by asking for them
promtool check config /etc/prometheus/prometheus.yml
curl -s -u admin:PASSWORD "localhost:9090/api/v1/query?query=up"

Prometheus works by pulling. Every few seconds it fetches a web page of numbers (usually /metrics) from each target, stores them with a timestamp, and lets you ask questions about them in a query language called PromQL. Rules turn questions into alerts; a separate program, Alertmanager, decides who hears about them. Prometheus does not draw dashboards (Grafana does), keep logs, or store data for years. It is built to answer "what is happening now and over the last few weeks" very fast.

A metric is a name plus labels: node_cpu_seconds_total{cpu="0",mode="idle",instance="tgt:9100"}. Each different set of labels is a separate series. That is the power of it and also the way it goes wrong, so this guide keeps coming back to labels.

Every example below ran for real: Prometheus 3.15.0, node_exporter 1.12.1, Alertmanager 0.34.1 and blackbox_exporter 0.29.0 (the current releases when this was written), installed from the project's own tarballs on Ubuntu 24.04, in two Docker containers on a private network: "prom" (Prometheus, Alertmanager, blackbox, a local mail server and a tiny webhook receiver) and "tgt" (node_exporter and nginx, the thing being watched). A container has no systemd, so the daemons were started by hand; the systemd units shown are the standard ones and were not run in the lab. Ubuntu 24.04 also packages Prometheus, but an older series (2.45); the guide says where that matters.

Six programs, and you rarely need all of them at once.

  prometheus          the server: scrapes, stores, evaluates rules, answers queries on port 9090
  exporters           small programs that turn something into a /metrics page. node_exporter (port 9100) for Linux hosts; blackbox_exporter (9115) to probe things from the outside
  Alertmanager        receives alerts from Prometheus, groups them, silences them, and sends them out by email, chat or webhook (port 9093)
  promtool            the command-line checker and helper for Prometheus (config, rules, metrics, TSDB)
  amtool              the same for Alertmanager (config, routing tests, silences)
  Grafana             the usual dashboard tool (a separate project; not run in the lab)

The flow: exporters publish numbers, Prometheus scrapes them every scrape_interval, rules inside Prometheus evaluate every evaluation_interval, firing alerts are pushed to Alertmanager, and Alertmanager sends notifications.

Prometheus ships as a single static binary per program. Downloading the current release and checking it against the project's checksum file takes a minute. On the server:

$ V=3.15.0
$ curl -sLO https://github.com/prometheus/prometheus/releases/download/v$V/prometheus-$V.linux-amd64.tar.gz
$ curl -sLO https://github.com/prometheus/prometheus/releases/download/v$V/sha256sums.txt
$ grep prometheus-$V.linux-amd64.tar.gz sha256sums.txt
2a542df32eac02ee17b9d844fb2aa1de00dafa5476579ba8a3ba862e9d572ea0  prometheus-3.15.0.linux-amd64.tar.gz
$ sha256sum prometheus-$V.linux-amd64.tar.gz
2a542df32eac02ee17b9d844fb2aa1de00dafa5476579ba8a3ba862e9d572ea0  prometheus-3.15.0.linux-amd64.tar.gz

The two lines must be identical. The lab did the same for the other three (the checksum files are named sha256sums.txt in each release; replace linux-amd64 with linux-arm64 on ARM):

b51d8a76aa2a9156a55d501aca6276fae09e262259a5e4e831d2c2222f084e63  node_exporter-1.12.1.linux-amd64.tar.gz
265b9d1e55ef0d5306a436018af6d2b686c2ce051f03d968f7464ecb1372a7e8  alertmanager-0.34.1.linux-amd64.tar.gz
5512929259bb6164f68ebe20ce433ec43b8609e3a79c95324f5020e931488fb4  blackbox_exporter-0.29.0.linux-amd64.tar.gz

What is in the Prometheus tarball: LICENSE, NOTICE, prometheus, prometheus.yml (an example) and promtool. (Alertmanager: alertmanager, alertmanager.yml, amtool. node_exporter: just the program. blackbox_exporter: the program and blackbox.yml.) Install them with their own user and the usual folders:

$ tar xzf prometheus-$V.linux-amd64.tar.gz
$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
$ sudo install -m755 prometheus-$V.linux-amd64/{prometheus,promtool} /usr/local/bin/
$ sudo mkdir -p /etc/prometheus /var/lib/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
$ prometheus --version | head -1
prometheus, version 3.15.0 (branch: HEAD, revision: 5241a27fe3c6983549fccc32f6e65917408c63cd)
$ promtool --version | head -1
promtool, version 3.15.0 (branch: HEAD, revision: 5241a27fe3c6983549fccc32f6e65917408c63cd)

Configuration goes in /etc/prometheus, data in /var/lib/prometheus. On a real server you run it under systemd. This is the standard unit (not run in the lab, where the program was started by hand):

[Unit]
Description=Prometheus
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save it as /etc/systemd/system/prometheus.service, then sudo systemctl daemon-reload && sudo systemctl enable --now prometheus. The other three programs take the same shape (a user of their own, an ExecStart with their flags).

The Ubuntu packages instead. sudo apt install prometheus prometheus-node-exporter prometheus-alertmanager gives a working setup in one command, with systemd units, a prometheus user and log rotation, but older versions. The lab installed them in a clean container and read what they bring:

prometheus                  2.45.3+ds-2ubuntu0.3
prometheus-alertmanager     0.26.0+ds-1ubuntu0.3
prometheus-node-exporter    1.7.0-1ubuntu0.3
(prometheus-blackbox-exporter 0.24.0 is also in the archive)

The package keeps its configuration in /etc/prometheus/prometheus.yml (it ships with a 15-second interval and an external label "monitor: example"), passes extra flags through ARGS="" in /etc/default/prometheus, and its unit runs /usr/bin/prometheus $ARGS as the prometheus user with ExecReload=/bin/kill -HUP $MAINPID. Choose the tarballs for current features (Prometheus 3 changed defaults and some behaviour), or the packages for a system that updates through apt. Commands in this guide work on both except where the text says otherwise; the paths and the version are the differences.

A small configuration is enough to start. Prometheus can scrape itself:

$ cat /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

Always check the file before you use it. promtool reads it the same way Prometheus does:

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Start it (the lab ran it as the prometheus user; /var/log/prometheus.log had to be created and owned by that user first, which a systemd unit does not need):

$ prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus \
    --web.listen-address=0.0.0.0:9090 --web.enable-lifecycle --storage.tsdb.retention.time=15d

Two endpoints tell you it is alive and ready, the ones for load balancers and monitoring:

$ curl -s localhost:9090/-/healthy
Prometheus Server is Healthy.
$ curl -s localhost:9090/-/ready
Prometheus Server is Ready.

and the log says what it did, in order:

level=INFO source=main.go:933 msg="Starting Prometheus Server" mode=server version="(version=3.15.0, ...
level=INFO source=tls_config.go:381 msg="Listening on" component=web address=[::]:9090
level=INFO source=main.go:1523 msg="TSDB started"
level=INFO source=main.go:1721 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=INFO source=main.go:1484 msg="Server is ready to receive web requests."

The web interface is at http://SERVER:9090/ and has a query box and pages for alerts, rules and targets. The lab did not open it: it used the HTTP API, which returns the same information as JSON, and so does this guide.

node_exporter turns a Linux machine into numbers: CPU, memory, disks, network, load, file systems, and much more. On the machine to watch (tgt):

$ tar xzf node_exporter-1.12.1.linux-amd64.tar.gz
$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
$ sudo install -m755 node_exporter-1.12.1.linux-amd64/node_exporter /usr/local/bin/
$ sudo mkdir -p /var/lib/node_exporter/textfile && sudo chown -R node_exporter /var/lib/node_exporter
$ node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile

It listens on port 9100. Look at what it publishes: this is a plain text page you can read in a terminal.

$ curl -s localhost:9100/metrics | grep -c "^node_"
720
$ curl -s localhost:9100/metrics | grep -E "^node_(load1|memory_MemTotal_bytes)"
node_load1 0.6
node_memory_MemTotal_bytes 3.3322160128e+10

The format is simple: optional # HELP and # TYPE lines, then name{labels} value. The types are counter (only goes up, resets on restart: use rate()), gauge (goes up and down), and histogram (buckets, for latencies and sizes).

Your own numbers: the textfile collector. A script writes a file ending in .prom into the textfile directory, and node_exporter adds its contents to /metrics. Write to a temporary name and rename, so a scrape never reads a half-written file:

$ cat > /var/lib/node_exporter/textfile/backup.prom.tmp <<EOF
# HELP backup_last_success_timestamp_seconds Unix time of the last successful backup.
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
# HELP backup_size_bytes Size of the last backup archive.
# TYPE backup_size_bytes gauge
backup_size_bytes 1453289472
EOF
$ mv /var/lib/node_exporter/textfile/backup.prom.tmp /var/lib/node_exporter/textfile/backup.prom
$ curl -s localhost:9100/metrics | grep -E "^(backup_|node_textfile_scrape_error)"
backup_last_success_timestamp_seconds 1.792024354e+09
backup_size_bytes 1.453289472e+09
node_textfile_scrape_error 0

node_textfile_scrape_error 0 means all files parsed. Make the backup job write the file when it succeeds, and you can alert on "no backup for a day" without any other software.

promtool can lint a metrics page against the naming rules. Run on node_exporter itself it grumbles about a few names it has to keep, and on a bad file it explains the rules:

$ printf '# TYPE bad_name_total gauge\nbad_name_total 1\nmyMetricCamel 2\n' | promtool check metrics
bad_name_total no help text
bad_name_total non-counter metrics should not have "_total" suffix
myMetricCamel metric names should be written in 'snake_case' not 'camelCase'

The rules: snake_case names, base units (seconds, bytes), _total only on counters, and a HELP line.

A scrape config says which targets to fetch from. Add the node exporter as a second job (this is the first version; the file-based version comes next):

  - job_name: node
    static_configs:
      - targets: ["tgt:9100"]
        labels:
          env: lab
          role: web

Check, then reload without restarting. The reload endpoint only exists when Prometheus was started with --web.enable-lifecycle (the systemd reload by SIGHUP needs no flag, and does the same):

$ promtool check config /etc/prometheus/prometheus.yml | tail -2
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
$ curl -s -X POST -w "HTTP %{http_code}\n" localhost:9090/-/reload
HTTP 200

The target list shows health, address and timing:

$ curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | [.labels.job, .labels.instance, .health, .scrapeUrl, (.lastScrapeDuration*1000|round|tostring)+" ms"] | @tsv'
node        tgt:9100         up  http://tgt:9100/metrics  114 ms
prometheus  localhost:9090   up  http://localhost:9090/metrics  10 ms

Every scrape also writes a synthetic series, up, which is 1 when the scrape worked and 0 when it did not. It is the first thing to alert on:

$ curl -s --data-urlencode "query=up" localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value: .value[1]}'
{"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":"1"}
{"metric":{"__name__":"up","env":"lab","instance":"tgt:9100","job":"node","role":"web"},"value":"1"}

Every series gets the labels job and instance automatically, plus the ones you add.

File-based discovery. For more than a handful of targets, keep them in files that other tools (Ansible, a script, your CMDB) can write, and let Prometheus watch the files. No reload is needed when a file changes:

  - job_name: node
    file_sd_configs:
      - files: ["/etc/prometheus/targets/*.json"]
        refresh_interval: 1m

$ cat /etc/prometheus/targets/web.json
[
  {"targets": ["tgt:9100"], "labels": {"env": "lab", "role": "web", "team": "platform"}}
]

The lab changed "team" from platform to sre in that file and did not reload. Six seconds later the old value was still there; it was updated within the refresh interval, and the new label showed up on new samples. Which brings the point about labels:

$ curl -s -G localhost:9090/api/v1/series --data-urlencode 'match[]=up' | jq -c '.data[]'
{"__name__":"up","instance":"localhost:9090","job":"prometheus"}
{"__name__":"up","env":"lab","instance":"tgt:9100","job":"node","role":"web"}
{"__name__":"up","env":"lab","instance":"tgt:9100","job":"node","role":"web","team":"platform"}
{"__name__":"up","env":"lab","instance":"tgt:9100","job":"node","role":"web","team":"sre"}
{"__name__":"up","instance":"http://tgt/","job":"blackbox-http"}

The same host is now three series: the one without the label, the one with team=platform and the one with team=sre. Changing a label's value starts a new series, and old ones stay until they age out. Pick labels you will keep (environment, role, team), not ones that change (a version, a start time), and never put an unbounded value in a label (a user id, a URL with parameters): each distinct value is another series in memory.

Relabeling rewrites labels before a scrape (relabel_configs) or after it (metric_relabel_configs). The most common use is the one in the blackbox section below, and another is dropping metrics you do not need to save space. The scrape page's own targets table with before and after labels is the best way to debug it (the lab used /api/v1/targets, which shows both discoveredLabels and labels).

Open the query box (or use the API) and type an expression. These ran against the lab data; the helper q is a one-line wrapper around /api/v1/query that prints the labels and the value.

A metric name is the simplest query. It returns the latest value of every series with that name:

$ q node_load1
{env="lab",instance="tgt:9100",job="node",role="web",team="sre"}  1.24

Filter with labels in braces: = equals, != not, =~ regex, !~ not regex.

$ q 'node_cpu_seconds_total{mode!="idle"}'     (every non-idle CPU mode, per core)

Counters are for rate(). A counter such as node_cpu_seconds_total only ever grows, so you look at how fast it grows over a window in square brackets. The window must hold at least two scrapes (with a 15 second interval, [1m] is safe):

$ q 'rate(node_network_receive_bytes_total{device="eth0"}[1m])'
{device="eth0",...}  45.8                          (bytes per second)

Aggregate with sum, avg, min, max, count, topk, by or without. This is the CPU in use as a percentage, the classic one:

$ q '100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])))'
{instance="tgt:9100"}  12.288888888888716

Arithmetic between series pairs up the ones with the same labels:

$ q 'node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes'
{...}  0.7937339196019123                          (79 percent of memory available)

More that you will use all the time:

$ q 'count by (mode) (node_cpu_seconds_total)'              how many series per mode (8 here, one per core)
{mode="idle"}  8
{mode="user"}  8
...
$ q 'sum by (mode) (rate(node_cpu_seconds_total[1m]))'      cores worth of time per mode
{mode="idle"}  7.016888888888903
{mode="user"}  0.6393333333333329
$ q 'avg_over_time(node_load1[5m])'                          the average of a gauge over a window
{...}  1.276
$ q 'node_load1 offset 5m'                                   the value five minutes ago
{...}  1.29
$ q 'time() - backup_last_success_timestamp_seconds'         age of the last backup, in seconds
{...}  642.6730000972748
$ q 'up == 0'                                                 comparison filters: only the series that match
(empty result)                                                (nothing is down)
$ q 'absent(nonexistent_metric)'                             1 when a series does not exist (use it to catch a missing metric)
{}  1
$ q 'count(up) by (job)'
{job="prometheus"}  1
{job="node"}  1

An instant query gives one value per series at a moment; a range query gives a series of values over time, for graphs:

$ curl -s "localhost:9090/api/v1/query_range?query=node_load1&start=$S&end=$E&step=60" | jq -c '.status, .data.resultType, (.data.result[0].values | length), .data.result[0].values[0:2]'
"success"
"matrix"
6
[[1792025016,"1.66"],[1792025076,"1.63"]]

(Five minutes at a 60 second step: 6 points; each is [unix time, "value as text"].) A few notes from the lab. predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[10m], 86400) returned nothing, because in the lab's container node_exporter reported only three bind-mounted files (/etc/hostname, /etc/hosts and /etc/resolv.conf) and no "/" file system. Check which mount points exist (q node_filesystem_avail_bytes) before you write a disk alert. A scalar() query does not use the same output shape as a vector one, so your tooling has to handle both result types. And a query with an unknown metric does not fail: it returns an empty result, which is why absent() exists.

Other API calls: /api/v1/labels (all label names), /api/v1/label/job/values (the values of one label; the lab got ["blackbox-http","node","prometheus"]), /api/v1/series (series matching a selector), /api/v1/targets, /api/v1/rules, /api/v1/alerts and /api/v1/status/tsdb.

Rules live in files listed under rule_files and are evaluated every evaluation_interval. A recording rule saves the result of an expression as a new series (cheap to graph, and the name is yours). An alerting rule fires when its expression returns anything. The lab's file, /etc/prometheus/rules/node.yml:

groups:
  - name: node
    rules:
      - record: instance:node_cpu_utilisation:rate1m
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m]))

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"
          description: "{{ $labels.job }} target {{ $labels.instance }} has been unreachable for more than 1 minute."

      - alert: BackupTooOld
        expr: time() - backup_last_success_timestamp_seconds > 86400
        labels:
          severity: warning
        annotations:
          summary: "No successful backup on {{ $labels.instance }} for over a day"

for: 1m means the expression must be true for a full minute before the alert fires (it is "pending" in between), which absorbs short blips. Labels are what Alertmanager routes on; annotations are text for humans, with {{ $labels.name }} filled in. A naming convention for recording rules: level:metric:operations.

Check syntax:

$ promtool check rules /etc/prometheus/rules/node.yml
Checking /etc/prometheus/rules/node.yml
  SUCCESS: 3 rules found

and, better, write a test. promtool test rules feeds made-up samples to your rules and checks what fires. This one says "up is 1 for two minutes and then 0", so at the 2 minute mark nothing fires and by 4 minutes InstanceDown does:

$ cat /etc/prometheus/tests/node_test.yml
rule_files:
  - ../rules/node.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: up{job="node", instance="tgt:9100"}
        values: "1 1 0 0 0 0"
    alert_rule_test:
      - eval_time: 2m
        alertname: InstanceDown
        exp_alerts: []
      - eval_time: 4m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels:
              severity: critical
              job: node
              instance: tgt:9100
            exp_annotations:
              summary: "tgt:9100 is down"
              description: "node target tgt:9100 has been unreachable for more than 1 minute."
$ cd /etc/prometheus/tests && promtool test rules node_test.yml
  SUCCESS

Put the tests in their own folder. The lab first put node_test.yml next to the rules, and rule_files used a glob (/etc/prometheus/rules/*.yml), so Prometheus tried to load the test file as rules and the reload failed:

$ promtool check config /etc/prometheus/prometheus.yml
  FAILED:
/etc/prometheus/rules/node_test.yml: yaml: unmarshal errors:
  line 1: field rule_files not found in type rulefmt.RuleGroups
  line 3: field evaluation_interval not found in type rulefmt.RuleGroups
  line 4: field tests not found in type rulefmt.RuleGroups
$ curl -s -X POST -w "HTTP %{http_code}\n" localhost:9090/-/reload
failed to reload config: one or more errors occurred while applying the new configuration (--config.file="/etc/prometheus/prometheus.yml")
HTTP 500

A failed reload is safe: Prometheus keeps running on the old configuration and tells you. That is why promtool check config belongs before every reload.

Rules are connected to Prometheus in the main file, together with Alertmanager (next section):

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

After the reload the rules API shows each rule, its health, and the state of alerts (inactive, pending, firing):

$ curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[] | [.type, .name, .health, .state // "-"] | @tsv'
recording  instance:node_cpu_utilisation:rate1m  ok  -
alerting   InstanceDown                          ok  inactive
alerting   BackupTooOld                          ok  inactive

The recording rule is already queryable: q instance:node_cpu_utilisation:rate1m returned {instance="tgt:9100"} 0.05308333333333193.

Prometheus decides that something is wrong. Alertmanager decides what to do about it: group related alerts into one message, wait a moment in case more arrive, silence what you are working on, and route to the right place. Install and start it like the others:

$ tar xzf alertmanager-0.34.1.linux-amd64.tar.gz
$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager
$ sudo install -m755 alertmanager-0.34.1.linux-amd64/{alertmanager,amtool} /usr/local/bin/
$ sudo mkdir -p /etc/alertmanager /var/lib/alertmanager && sudo chown alertmanager /var/lib/alertmanager
$ alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager

The lab's configuration, with the two kinds of receiver almost everyone starts with (a webhook, which is how chat tools and ticket systems are fed, and email):

global:
  resolve_timeout: 5m
  smtp_smarthost: "localhost:25"
  smtp_from: "alertmanager@prom.example.com"
  smtp_require_tls: false

route:
  receiver: team-webhook
  group_by: ["alertname", "instance"]
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: oncall-email
      continue: true
    - receiver: team-webhook

receivers:
  - name: team-webhook
    webhook_configs:
      - url: "http://localhost:5001/alerts"
  - name: oncall-email
    email_configs:
      - to: "oncall@localhost"
        send_resolved: true

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: ["instance"]

How to read the timings: group_by decides which alerts are sent together (here: same alert name on the same host). group_wait (10s) is how long to wait after the first alert of a new group before sending, so that a burst arrives as one message. group_interval (1m) is how long to wait before telling you about new alerts in a group that has already been sent. repeat_interval (4h) is how often to repeat a notification for an alert that is still firing. The inhibit rule says: if a critical alert is firing for an instance, do not also send warnings for the same instance.

Check it, and test the routing before real alerts depend on it. amtool answers "where would an alert with these labels go?":

$ amtool check-config /etc/alertmanager/alertmanager.yml
Checking '/etc/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 2 receivers
 - 0 templates
$ amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=critical alertname=InstanceDown
oncall-email,team-webhook
$ amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=warning alertname=BackupTooOld
team-webhook
$ amtool config routes show --config.file=/etc/alertmanager/alertmanager.yml
Routing tree:
.
└── default-route  receiver: team-webhook
    ├── {severity="critical"}  continue: true  receiver: oncall-email
    └── default-route  receiver: team-webhook

The trap that this test caught: the first version of the config had only the critical route with continue: true and no second route, and the test said a critical alert went to oncall-email only:

$ amtool config routes test ... severity=critical alertname=InstanceDown
oncall-email

continue: true means "after this route matches, keep looking at the next SIBLING routes", not "also use the parent's receiver". So the parent's receiver (team-webhook) was never used for critical alerts. The second route (no matchers, so it matches everything) is the fix. Always run routes test with the alert labels you actually send.

Connect Prometheus (the alerting block above), and confirm that it knows about Alertmanager:

$ curl -s localhost:9090/api/v1/alertmanagers | jq -c '.data.activeAlertmanagers'
[{"url":"http://localhost:9093/api/v2/alerts"}]
$ curl -s localhost:9093/-/healthy ; curl -s localhost:9093/-/ready
OK
OK

Reload Alertmanager the same way as Prometheus: curl -s -X POST localhost:9093/-/reload, or SIGHUP.

The real test: stop the node exporter on tgt and watch. The lab polled both APIs every 15 seconds. This is the timeline (times are UTC):

00:34:22  node_exporter stopped
00:34:37  InstanceDown becomes pending      (up == 0 is true; "for: 1m" starts counting)
00:35:37  InstanceDown becomes firing       (one minute later)
00:35:38  Alertmanager has it as active
00:35:47  the webhook receives the notification   (group_wait: 10s after the alert arrived)
00:37:54  node_exporter started again
00:38:07  the alert ends
00:38:47  the webhook receives the RESOLVED notification

The webhook body is JSON, one object per notification, with the grouped alerts inside. This is the first one (trimmed):

{"receiver":"team-webhook","status":"firing","alerts":[{"status":"firing",
 "labels":{"alertname":"InstanceDown","env":"lab","instance":"tgt:9100","job":"node","role":"web","severity":"critical","team":"sre"},
 "annotations":{"description":"node target tgt:9100 has been unreachable for more than 1 minute.","summary":"tgt:9100 is down"},
 "startsAt":"2026-10-15T00:35:37.707Z","endsAt":"0001-01-01T00:00:00Z", ...}], "notification_reason":"first notification", ...}

and the resolved one has "status":"resolved", a real endsAt (00:38:07) and "notification_reason":"all alerts resolved". Whatever you point a webhook at (chat bridge, ticket tool, a script) reads exactly this.

Email lessons from the same test, in the order the lab learned them:

1. The first email did not arrive. The mail subject in the log, [FIRING:1] InstanceDown tgt:9100 (lab node web critical sre), proved Alertmanager had sent it, and the mail server bounced it: "cannot update mailbox /var/mail/oncall for user oncall". The cause was the mailbox file, which had been created as root. Alertmanager only hands mail to your mail server; delivery problems are in the mail server's logs, so test mail by hand first.
2. Email resolved messages are off by default. After send_resolved: true was added to email_configs, a resolved message arrived: [RESOLVED] DiskFull tgt:9100 (critical). (Webhooks send resolved messages by default.)
3. With the mailbox fixed, the next test alert arrived normally: [FIRING:1] DiskFull tgt:9100 (critical).

Test alerts without breaking anything. amtool can inject an alert directly into Alertmanager, which is a fast way to test routing, templates and receivers. Quote values that contain spaces (the lab got a warning from amtool for unquoted ones, and it parsed them in a fallback mode):

$ amtool --alertmanager.url=http://localhost:9093 alert add alertname=DiskFull severity=critical instance=tgt:9100 \
    --annotation=summary='"Disk almost full on tgt"' --start=$(date -u +%Y-%m-%dT%H:%M:%SZ)
$ amtool --alertmanager.url=http://localhost:9093 alert query
Alertname  Starts At                Summary                  State
DiskFull   2026-10-15 00:40:42 UTC  Disk almost full on tgt  active

(An alert with --end set to now ends it, which is how the lab triggered the resolved message.)

A silence mutes alerts that match labels, for a time, with a name and a reason. It is what you create when you start planned work:

$ amtool --alertmanager.url=http://localhost:9093 silence add alertname=DiskFull --comment="planned disk work" --duration=30m --author=admin
9d01515f-af15-43c1-881d-535b2e30ef63
$ amtool --alertmanager.url=http://localhost:9093 silence query
ID                                    Matchers              Ends At                  Created By  Comment
9d01515f-af15-43c1-881d-535b2e30ef63  alertname="DiskFull"  2026-10-15 01:11:07 UTC  admin       planned disk work

With the silence active, a new DiskFull alert was accepted but alert query listed nothing (silenced alerts are hidden unless you ask for them) and nothing was sent. Expire it with amtool silence expire ID.

A subtle result from the lab, which surprises people: after the silence was expired, the alert that was still open did not send a new "firing" message. Alertmanager remembered it had already notified for that group, and the next reminder is due after repeat_interval (4h here). So silences suppress, they do not queue: if the problem is still there when you remove the silence, you will not necessarily be told again before repeat_interval. Check the dashboard after you remove a silence.

Resolved messages are suppressed by a silence too, so silence for as short a time as the work needs.

Alertmanager also keeps its state (silences, which notifications went out) in --storage.path, so a restart does not forget them.

node_exporter shows what a machine says about itself. blackbox_exporter checks what a user would see: does the web page answer, does the port accept connections, does the certificate expire soon. Install it as before, with a small config that defines modules:

$ cat /etc/blackbox/blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      preferred_ip_protocol: ip4
  tcp_connect:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: ip4
$ blackbox_exporter --config.file=/etc/blackbox/blackbox.yml --web.listen-address=127.0.0.1:9115

Probing is a request to the exporter's /probe endpoint with the module and the target. Try it by hand first:

$ curl -s "localhost:9115/probe?module=http_2xx&target=http://tgt/" | grep -E "^(probe_success|probe_http_status_code|probe_duration_seconds)"
probe_duration_seconds 0.001835929
probe_http_status_code 200
probe_success 1
$ curl -s "localhost:9115/probe?module=tcp_connect&target=tgt:8080" | grep probe_success
probe_success 0

probe_success is the one to alert on. To make Prometheus do this on a schedule for a list of URLs, the targets are the URLs, and relabeling moves each URL into the target parameter and points the scrape at the exporter:

  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - http://tgt/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

The three lines mean: the address from the list becomes the target parameter; it also becomes the instance label (so graphs show the URL); and the real scrape address becomes the exporter. The targets API shows the result:

http://tgt/   http://localhost:9115/probe?module=http_2xx&target=http%3A%2F%2Ftgt%2F   up

Then the proof. With nginx running, probe_success was 1. After nginx -s stop on tgt, within a scrape or two probe_success was 0 and probe_http_status_code was 0, and after nginx was started again it went back to 1. A rule such as probe_success == 0 for 2m alerts when the site is down for users, whatever the web server's own metrics say. (Certificate expiry: probe_ssl_earliest_cert_expiry, available when the target is https; not run in the lab.)

Out of the box, Prometheus and every exporter answer anyone who can reach the port, and Prometheus can be told to reload (with --web.enable-lifecycle) by anyone. Basic authentication is built in. Create a bcrypt hash (htpasswd is in apache2-utils; -b takes the password on the command line, -B is bcrypt) and put it in a web config file:

$ HASH=$(htpasswd -nbBC 10 "" 'PASSWORD' | tr -d ':\n')
$ printf 'basic_auth_users:\n  admin: %s\n' "$HASH" > /etc/prometheus/web.yml
$ chown prometheus /etc/prometheus/web.yml; chmod 640 /etc/prometheus/web.yml
$ promtool check web-config /etc/prometheus/web.yml
/etc/prometheus/web.yml SUCCESS

(The lab's first attempt used htpasswd without -b; it prompted for a password instead of using the one given, produced an empty hash, and promtool check web-config said: FAILED: crypto/bcrypt: hashedSecret too short to be a bcrypted password. Prometheus would have refused to start with that file. Check before you restart.)

Restart with --web.config.file=/etc/prometheus/web.yml (a new flag needs a restart, not a reload). The results:

$ curl -s -o /dev/null -w "HTTP %{http_code}\n" "localhost:9090/api/v1/query?query=up"
HTTP 401
$ curl -s -o /dev/null -w "HTTP %{http_code}\n" -u admin:wrong "localhost:9090/api/v1/query?query=up"
HTTP 401
$ curl -s -o /dev/null -w "HTTP %{http_code}\n" -u admin:PASSWORD "localhost:9090/api/v1/query?query=up"
HTTP 200

Three things changed that are easy to miss, all seen in the lab:

1. /-/healthy and /-/ready are protected too (HTTP 401 without credentials). Load balancers and monitoring that probe them need the password.
2. Prometheus scraping itself broke: the target went down with "server returned HTTP status 401 Unauthorized", because its own scrape job had no password. Give the job one (a password file keeps it out of the config):

  - job_name: prometheus
    basic_auth:
      username: admin
      password_file: /etc/prometheus/prom_password
    static_configs:
      - targets: ["localhost:9090"]

3. The reload endpoint needs the login as well:

$ curl -s -o /dev/null -w "HTTP %{http_code}\n" -X POST localhost:9090/-/reload
HTTP 401
$ curl -s -o /dev/null -w "HTTP %{http_code}\n" -u admin:PASSWORD -X POST localhost:9090/-/reload
HTTP 200

and so does promtool when it talks to the server (--http.config.file with the same basic_auth block):

$ promtool query instant --http.config.file=/tmp/promtool-http.yml http://localhost:9090 'up{job="node"}'
up{env="lab", instance="tgt:9100", job="node", role="web", team="sre"} => 1 @[1792025195.118]

The same web config file can enable TLS (tls_server_config with a certificate and key) and use the same format for node_exporter, Alertmanager and the blackbox exporter, which is how you protect the exporters (not run in the lab). The admin API (snapshots, deleting series) is off unless you start with --web.enable-admin-api; the lab confirmed web.enable-admin-api=false in /api/v1/status/flags. Besides passwords: put exporters on a private network or behind a firewall that only allows the Prometheus server; node_exporter's port 9100 gives away a lot about a machine.

Prometheus keeps recent data in memory and on disk in a write-ahead log (wal and chunks_head folders), and writes a block to disk every two hours. The lab's data directory after about an hour of running:

$ ls /var/lib/prometheus
chunks_head  lock  queries.active  wal
$ du -sh /var/lib/prometheus
580K    /var/lib/prometheus
$ promtool tsdb list /var/lib/prometheus
BLOCK ULID   MIN TIME     MAX TIME     DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE

No finished block yet (the list is empty): everything was still in the head. How much room you need is series x samples per second x bytes per sample x retention, and the numbers come from your own server. The lab's, from the server's own metrics:

$ q prometheus_tsdb_head_series
{instance="localhost:9090",job="prometheus"}  3331
$ q 'rate(prometheus_tsdb_head_samples_appended_total[2m])'
{... type="float"}  22.27767111111111
$ curl -s localhost:9090/api/v1/status/tsdb | jq -c '.data.headStats'
{"numSeries":3354,"numLabelPairs":1099,"chunkCount":3354,"minTime":1792024308032,"maxTime":1792025314478}

That is three targets (the server, a node exporter, one probe), 3,331 series and 22 new samples per second. Watch prometheus_tsdb_head_series: the day it jumps is the day somebody put a user id in a label.

Retention is set by flags (changing them needs a restart). The lab's:

$ curl -s localhost:9090/api/v1/status/flags | jq -r '.data | to_entries[] | select(.key|test("retention|storage.tsdb.path|config.file|web.enable")) | "\(.key)=\(.value)"'
config.file=/etc/prometheus/prometheus.yml
storage.tsdb.path=/var/lib/prometheus
storage.tsdb.retention.size=0B
storage.tsdb.retention.time=15d
web.enable-admin-api=false
web.enable-lifecycle=true
...

The default retention is 15 days; --storage.tsdb.retention.size caps the disk used instead (0B means no size limit). Old blocks are deleted whole when either limit is reached. For long-term storage or several Prometheus servers in one place, Prometheus has remote_write to other systems and federation; both were not run in the lab.

Backups: with --web.enable-admin-api, a POST to /api/v1/admin/tsdb/snapshot makes a consistent copy of the data (not run in the lab). Without it, stop Prometheus and copy the directory. The configuration files are small and belong in version control.

Install             tarball: curl -sLO URL ; sha256sum -c ; tar xzf ; install -m755 prometheus promtool /usr/local/bin     or: apt install prometheus (2.45 on Ubuntu 24.04)
Check config        promtool check config prometheus.yml     promtool check rules rules.yml     promtool check web-config web.yml
Test rules          promtool test rules tests.yml            amtool check-config alertmanager.yml
Reload              curl -X POST localhost:9090/-/reload (needs --web.enable-lifecycle)     or kill -HUP PID     (add -u user:pass when auth is on)
Health              /-/healthy   /-/ready  (Prometheus 9090, Alertmanager 9093)
Targets / rules     /api/v1/targets   /api/v1/rules   /api/v1/alerts   /api/v1/alertmanagers
Query               curl --data-urlencode 'query=up' localhost:9090/api/v1/query        /api/v1/query_range with start, end, step
PromQL              rate(counter[1m])   sum by (label) (...)   avg_over_time(g[5m])   x offset 5m   absent(x)   topk(3, x)   up == 0
Alert pieces        expr + for + labels (routing) + annotations (text)         pending -> firing -> resolved
Alertmanager        amtool config routes test LABELS      amtool alert add/query      amtool silence add/query/expire
Exporters           node_exporter :9100 (textfile collector for your own numbers)     blackbox_exporter :9115 /probe?module=M&target=T
Security            --web.config.file=web.yml with basic_auth_users (bcrypt); probes, reloads and self-scrape then need the password
Storage             --storage.tsdb.retention.time=15d   --storage.tsdb.retention.size   prometheus_tsdb_head_series   promtool tsdb list DIR

Target shows "down" with a connection error      the exporter is not running, the port is blocked, or the address is wrong. Open the targets page (or /api/v1/targets) and read lastError
Target "down": 401 Unauthorized                  the target (or Prometheus itself) needs a password: add basic_auth to the scrape job
Reload returns HTTP 500                          the new config is invalid; Prometheus keeps the old one. Run promtool check config and fix. (A tests file inside the rules glob is a classic cause)
Reload returns HTTP 403 "Lifecycle API is not enabled."   start with --web.enable-lifecycle, or reload with kill -HUP
A query returns an empty result                  the metric name or a label is misspelt, or the series has no recent sample. Try the bare metric name, then add labels one by one
rate() returns nothing                           the window holds fewer than two samples: use at least 4x the scrape interval, for example [1m] at 15s
Two series where you expected one                a label changed (see the three "up" series above); they are different series until the old one ages out
Alert is "pending" and never fires               the for: time keeps restarting because the expression flickers; look at the expression over time
Alert fires in Prometheus but no notification    Prometheus cannot reach Alertmanager (see /api/v1/alertmanagers), or a route sent it somewhere you did not mean. amtool config routes test
Critical alerts only reach one receiver          continue: true does not return to the parent route's receiver: add a sibling route (see the Alertmanager section)
Email "sent" but nothing arrives                 Alertmanager handed it to the mail server; read the mail server's log and check the recipient's mailbox. Test mail by hand
No resolved email                                email_configs has send_resolved: false by default
Silence ended but no new notification            the alert was already notified within repeat_interval; check the alert view yourself
Prometheus does not start after the auth change  the web config file is invalid: promtool check web-config; a bcrypt hash must be 60 characters starting $2y$
Disk fills up                                    retention too long for the series count: check prometheus_tsdb_head_series and set retention.time or retention.size
node_exporter has no file system you expected    it hides some mounts (containers, tmpfs); check q node_filesystem_avail_bytes and the --collector.filesystem flags

Run promtool check config and promtool check rules before every reload, and put promtool test rules in your deployment pipeline. A failed reload is safe, but a typo in a rule is a silent hole.
Alert on up == 0 first. It is the one alert that tells you every other alert may be lying.
Alert on symptoms users feel (probe_success == 0, error rate, latency) before causes (CPU, memory). Add the causes as warnings.
Use for: on almost every alert. Short blips are normal; five minutes of the same thing is a problem.
Choose labels you will keep for years (env, team, role) and never use unbounded values. Watch the series count.
Write recording rules for expressions you graph or alert on often; name them level:metric:operation.
Test routing with amtool config routes test and send test alerts with amtool alert add before you trust a new route.
Keep group_by sensible (alertname and the instance or cluster): one message per problem, not one per series.
Silence for the shortest time the work needs, with a comment saying who and why.
Put every exporter and Prometheus itself behind a firewall or basic auth/TLS. Metrics are information an attacker would like.
Keep configuration in version control and use file_sd or an automation tool to generate target lists.
Do not use Prometheus as a place for logs or billing data: it is fast, approximate and forgetful by design.