🕹️ Ansible Semaphore Guide

Semaphore UI: a web interface and API in front of ansible-playbook
curl -s -X POST -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" localhost:3000/api/project/1/tasks -d '{"template_id":2}'
semaphore server --config /etc/semaphore/config.json

Ansible runs from a command line. That is fine for one person, until a team needs to run the same playbooks, see who ran what and when, keep SSH keys out of laptops, run jobs on a schedule, and start a run when somebody pushes to git. Semaphore (Semaphore UI) is a small server that does exactly that: it checks out your playbooks from git, runs ansible-playbook with an inventory and keys that it stores, keeps the output of every run, and shows it all in a web interface and an HTTP API.

It does not replace Ansible. Your playbooks stay plain playbooks in a git repository, and every example in the Ansible guides on this site runs unchanged. Semaphore is the part around them: who may run what, where the secrets live, when it runs, and the history.

Every example below ran for real: Semaphore 2.19.16 (the current release when this was written), installed from the project's .deb package on Ubuntu 24.04 with ansible-core 2.16.3 from Ubuntu, managing two Ubuntu 24.04 containers over SSH. The lab is a set of containers without systemd, so the server was started by hand with the same command the service uses; the systemd file is shown and marked as not run. The lab drove Semaphore through its API (the web interface calls the same API), so what the pages look like was not checked: where a result is something you would see on screen, this guide says so.

Semaphore has a small vocabulary. Learn these nouns and the rest follows. Everything lives inside a project.

  project         a workspace: its own keys, repositories, inventories, templates, people and history
  key store       secrets: SSH keys, login/password pairs, plain strings. Stored encrypted, never shown again after saving
  repository      a git repository (or a folder) that holds your playbooks
  inventory       the list of hosts (typed in, or a file in the repository), plus the key used to log in
  environment     extra variables for the playbook (and secrets), as JSON
  template        the recipe: this playbook, with this inventory, this repository and this environment
  task            one run of a template, with its status and its full output
  schedule        a cron expression that starts a template
  integration     a webhook URL that starts a template (from a git push, for example)
  runner          a separate machine that executes tasks instead of the server (optional)

The flow: you define a template once. A person, a schedule or a webhook starts a task. The server (or a runner) clones the repository, writes the inventory and key to a temporary folder, runs ansible-playbook, streams the output to the interface and stores it, and sends an alert if it failed.

Where it runs: one program (a Go binary with the interface inside), a database (SQLite for small setups, PostgreSQL or MySQL for larger ones), and ansible-playbook installed on the same machine (or on the runner). Semaphore does not bundle Ansible: it calls whatever ansible-playbook is on its PATH.

Semaphore is not in Ubuntu's repositories. The project publishes a .deb package (and a Docker image and tarballs). Download the package and the checksum file, and compare:

$ V=2.19.16
$ curl -sLO https://github.com/semaphoreui/semaphore/releases/download/v$V/semaphore_${V}_linux_amd64.deb
$ curl -sLO https://github.com/semaphoreui/semaphore/releases/download/v$V/semaphore_${V}_checksums.txt
$ sha256sum -c --ignore-missing semaphore_${V}_checksums.txt
semaphore_2.19.16_linux_amd64.deb: OK
$ sudo apt-get install -y ./semaphore_${V}_linux_amd64.deb
$ semaphore version
2.19.16-f8cefe6-1791363617

The release also has a signature file for the checksums (semaphore_${V}_checksums.txt.sig); the lab compared checksums but did not check the signature.

The package contains one file and nothing else. The lab listed it:

$ dpkg -L semaphore
/usr
/usr/bin
/usr/bin/semaphore

There is no semaphore user, no configuration, no data directory and no systemd unit: you make them (next section). Then install Ansible. Ubuntu 24.04 has ansible-core 2.16.3 (the small package with only the core modules) and ansible (the large package with many collections):

$ sudo apt-get install -y ansible-core git
$ ansible --version | head -1
ansible [core 2.16.3]

The version matters later: newer collections want a newer ansible-core (see the collections section). If you need a newer ansible-core, install it with pip into a virtual environment and put that environment first on the PATH of the semaphore service (not run in the lab).

The subcommands you will use:

$ semaphore --help
  server      run the server (alias: service)
  setup       interactive setup: database, paths, first user
  migrate     create or upgrade the database tables
  users       add, list, get, change and delete users (and API tokens, TOTP)
  runner      run as a remote runner (register, start, unregister)
  projects    manage projects
  vaults      manage access keys and other secrets

Make a user for the service, a configuration folder and a data folder:

$ sudo useradd --system --create-home --home-dir /var/lib/semaphore --shell /usr/sbin/nologin semaphore
$ sudo mkdir -p /etc/semaphore

The setup command asks questions and writes the configuration, creates the tables and makes the first admin user. It first asks which database. In the lab's version, BoltDB is listed as "NOT SUPPORTED", so the choices that work are MySQL, PostgreSQL and SQLite. SQLite needs nothing else and is fine for a team's playbooks. The lab chose 4:

$ sudo semaphore setup --config /etc/semaphore/config.json
What database to use:
   1 - MySQL
   2 - BoltDB (NOT SUPPORTED)
   3 - PostgreSQL
   4 - SQLite
 (default 1): 4
db Hostname (default /var/lib/semaphore/database.sqlite): /var/lib/semaphore/semaphore.db
Playbook path (default /tmp/semaphore): /var/lib/semaphore/tmp
Public URL (optional, example: https://example.com/semaphore):
Enable email alerts? (yes/no) (default no): no
... (telegram, slack, Rocket.Chat, Microsoft Teams, LDAP: no)
Executing migration v2.19.14 ...
Migrations Finished
 > Username: admin
 > Email: admin@example.com
 > Your name: Admin
 > Password:
 You are all setup Admin!

Two answers are not what their questions say. "db Hostname" for SQLite is the path of the database file. "Playbook path" becomes tmp_path: the folder where Semaphore clones your repositories for each run. Do not leave it at /tmp/semaphore on a real server.

Run setup as root (sudo), not as the semaphore user: the lab ran it as semaphore first and got a Go panic, "open /etc/semaphore/probe.json: permission denied", because /etc/semaphore belongs to root. Setup as root leaves root-owned files, so fix the ownership and, more importantly, the permissions:

$ sudo chown root:semaphore /etc/semaphore/config.json && sudo chmod 640 /etc/semaphore/config.json
$ sudo chown semaphore:semaphore /var/lib/semaphore/semaphore.db && sudo chmod 600 /var/lib/semaphore/semaphore.db
$ sudo install -d -o semaphore -g semaphore /var/lib/semaphore/tmp

The file setup writes is mode 644, readable by every user on the machine, and it contains secrets:

$ sudo cat /etc/semaphore/config.json
{
  "sqlite": { "host": "/var/lib/semaphore/semaphore.db" },
  "dialect": "sqlite",
  "tmp_path": "/var/lib/semaphore/tmp",
  "cookie_hash": "...",
  "cookie_encryption": "...",
  "access_key_encryption": "..."
}

The three random strings matter. cookie_hash and cookie_encryption protect login sessions. access_key_encryption encrypts every key in the key store (the SSH keys, passwords and tokens). If you lose it, the stored keys cannot be read any more (the backup section shows the error). Keep a copy of config.json with your backups, in a safe place.

Start the server (the lab ran this command by hand; it is the command the service runs), and check it answers:

$ cd /var/lib/semaphore && sudo -u semaphore semaphore server --config /etc/semaphore/config.json
SQLite @/var/lib/semaphore/semaphore.db semaphore
Tmp Path (projects home) /var/lib/semaphore/tmp
Semaphore 2.19.16-f8cefe6-1791363617
Interface
Port :3000
Server is running
$ curl -s localhost:3000/api/ping
pong

On a real server, run it with systemd. This is the usual unit file (not run in the lab, which has no systemd):

# /etc/systemd/system/semaphore.service
[Unit]
Description=Semaphore UI
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=semaphore
Group=semaphore
WorkingDirectory=/var/lib/semaphore
ExecStart=/usr/bin/semaphore server --config /etc/semaphore/config.json
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload && sudo systemctl enable --now semaphore

The web interface is at http://SERVER:3000, log in with the admin user from setup. For scripts, the API uses the same login. A login returns no body, only a session cookie (204). Create an API token with the cookie and use it as a Bearer token from then on:

$ curl -s -c /tmp/cj -X POST -H "Content-Type: application/json" -d '{"auth":"admin","password":"PASSWORD"}' -w "login: %{http_code}\n" localhost:3000/api/auth/login
login: 204
$ curl -s -b /tmp/cj localhost:3000/api/user | jq -c '{username,admin,can_create_project}'
{"username":"admin","admin":true,"can_create_project":true}
$ curl -s -b /tmp/cj -X POST localhost:3000/api/user/tokens | jq -c '{id,created,expired}'
{"id":"gqmfvikiltdxe2ui0t6czqcbp7xtru1c90q7zhu01hs=","created":"2026-10-17T03:31:13Z","expired":false}
$ curl -s -o /dev/null -w "no auth: %{http_code}\n" localhost:3000/api/projects
no auth: 401

The id in the answer is the token. It works as Authorization: Bearer TOKEN. Treat it like a password: it has the powers of the user who made it. A user's tokens are listed with GET /api/user/tokens (only the first characters matter for recognising them) and removed with DELETE /api/user/tokens/ID.

The rest of this guide uses the API, with a variable for the header, so the commands stay short. The lab also used a small helper that starts a task, waits for it and prints the output. Here it is, because it shows how little the API needs:

# semrun TEMPLATE_ID [extra task JSON]
T=$(cat /tmp/tok); H="Authorization: Bearer $T"; C="Content-Type: application/json"; G=localhost:3000/api
ID=$(curl -s -H "$H" -H "$C" -X POST $G/project/1/tasks -d "{\"template_id\":$1${2:+,$2}}" | jq -r .id)
echo "task $ID started"
for i in $(seq 1 60); do S=$(curl -s -H "$H" $G/project/1/tasks/$ID | jq -r .status); case $S in success|error|stopped) break;; esac; sleep 2; done
echo "status: $S"
curl -s -H "$H" $G/project/1/tasks/$ID/output | jq -r '.[].output'

The output has terminal colour codes in it; pipe it through sed -r "s/\x1b\[[0-9;]*m//g" to read it as plain text.

Build the objects in the order a template needs them. First the playbooks, in a git repository. The lab used a repository on the same machine (a bare repository, owned by the semaphore user, cloned from a file:// URL). In real use this is your Git server over https or ssh:

$ sudo install -d -o semaphore -g semaphore /srv/git
$ sudo -u semaphore git init -q --bare -b main /srv/git/ops.git
$ sudo -u semaphore git clone -q /srv/git/ops.git /var/lib/semaphore/ops-work

The lab's first playbooks (playbooks/ping.yml, playbooks/baseline.yml, playbooks/maintenance.yml and templates/motd.j2) are ordinary Ansible:

# playbooks/baseline.yml
---
- name: Baseline packages and a login banner
  hosts: all
  become: true
  vars:
    motd_message: "Managed by Ansible"
  tasks:
    - name: Install the baseline packages
      ansible.builtin.apt:
        name: [htop, tree, curl]
        state: present
        update_cache: true
        cache_valid_time: 3600

    - name: Write the login banner
      ansible.builtin.template:
        src: ../templates/motd.j2
        dest: /etc/motd
        mode: "0644"

# templates/motd.j2
{{ inventory_hostname }} ({{ env_name | default("unset") }})
{{ motd_message }}

Semaphore reaches the managed hosts by SSH, so make a key pair for it and put the public half in authorized_keys on each host (the lab's hosts allow the ubuntu user with sudo):

$ ssh-keygen -q -t ed25519 -N "" -C semaphore-lab -f /tmp/semkey
$ ssh-copy-id -i /tmp/semkey.pub ubuntu@HOST          (or append the public key to ~/.ssh/authorized_keys)

Create the project, and store the private key in its key store:

$ curl -s -H "$H" -H "$C" -X POST $G/projects -d '{"name":"Lab ops","alert":false,"max_parallel_tasks":0}' | jq -c '{id,name}'
{"id":1,"name":"Lab ops"}
$ K=$(jq -Rs . /tmp/semkey)
$ curl -s -H "$H" -H "$C" -X POST $G/project/1/keys -d "{\"name\":\"lab-ssh\",\"type\":\"ssh\",\"project_id\":1,\"ssh\":{\"login\":\"ubuntu\",\"private_key\":$K}}" | jq -c '{id,name,type}'
{"id":2,"name":"lab-ssh","type":"ssh"}
$ curl -s -H "$H" $G/project/1/keys | jq -c '.[] | {id,name,type}'
{"id":1,"name":"None","type":"none"}
{"id":2,"name":"lab-ssh","type":"ssh"}

Every project starts with a key named None (id 1), for repositories and inventories that need no key. Key types are ssh, login_password, string (a single secret) and none. Once saved, the secret is not returned by the API, not even to the owner (the lab read key 2 back as admin and got an empty login and private_key), and in the database it is encrypted (the lab's SQLite file contains no "BEGIN OPENSSH" text).

The repository, the inventory (here typed in, using ansible_host to point at the container names) and the environment (extra variables as JSON):

$ curl -s -H "$H" -H "$C" -X POST $G/project/1/repositories -d '{"name":"ops-file","project_id":1,"git_url":"file:///srv/git/ops.git","git_branch":"main","ssh_key_id":1}' | jq -c '{id,name,git_url}'
{"id":2,"name":"ops-file","git_url":"file:///srv/git/ops.git"}
$ INV=$(printf "[web]\nweb1 ansible_host=sweb1\nweb2 ansible_host=sweb2\n" | jq -Rs .)
$ curl -s -H "$H" -H "$C" -X POST $G/project/1/inventory -d "{\"name\":\"web servers\",\"project_id\":1,\"type\":\"static\",\"inventory\":$INV,\"ssh_key_id\":2}" | jq -c '{id,name,type}'
{"id":1,"name":"web servers","type":"static"}
$ curl -s -H "$H" -H "$C" -X POST $G/project/1/environment -d '{"name":"lab","project_id":1,"json":"{\"env_name\":\"lab\"}","env":"{}"}' | jq -c '{id,name}'
{"id":1,"name":"lab"}

A repository gotcha from the lab. The first repository used the plain path /srv/git/ops.git as its URL. Semaphore treats a URL that is a path as a local folder and runs the playbook from that folder as it is. A bare repository has no files in it, so the run failed with
  ERROR! the playbook: playbooks/ping.yml could not be found
Use a real clone URL (file:///..., https://..., ssh), as in the example; a plain path is only for a folder that already holds the playbooks.

Then the template, tying the pieces together:

$ curl -s -H "$H" -H "$C" -X POST $G/project/1/templates -d '{"project_id":1,"inventory_id":1,"repository_id":2,"environment_id":1,"name":"ping","playbook":"playbooks/ping.yml","app":"ansible","type":"","arguments":"[]","allow_override_args_in_task":true}' | jq -c '{id,name,playbook}'
{"id":1,"name":"ping","playbook":"playbooks/ping.yml"}

(The lab made three templates this way: ping (1), baseline (2) and maintenance (3).) The git protocol for a real Git server over SSH needs a key too: add a key of type ssh for the repository (the lab's repository needed none).

Not run in the lab: a repository on a hosted Git service, a repository that needs a key, inventories in a file in the repository or from Terraform, and Vault passwords.

A task is one run of a template. Start it with the API (or the Run button) and read its output:

$ semrun 1
task 2 started
status: success
Task ping added to queue
Started task #2 of template 'ping'

Preparing: 2
Cloning Repository file:///srv/git/ops.git
Cloning into 'repository_2_template_1'...
Get current commit hash
Get current commit message
installing static inventory
No .../requirements.yml file found. Skip galaxy install process.

PLAY [Check that Ansible can reach the hosts] **********************************
TASK [Gathering Facts] *********************************************************
ok: [web1]
ok: [web2]
TASK [Ping] ********************************************************************
ok: [web2]
ok: [web1]
TASK [Show what we found] ******************************************************
ok: [web1] => { "msg": "web1 runs Ubuntu 24.04" }
ok: [web2] => { "msg": "web2 runs Ubuntu 24.04" }
PLAY RECAP *********************************************************************
web1   : ok=3  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0
web2   : ok=3  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

What happened: Semaphore cloned the repository into its tmp_path (project_1/repository_2_template_1), looked for requirements files (the long list of "No ... found" lines), wrote the inventory and the key to a private folder, and ran ansible-playbook. The output is stored, so anyone with access can read it later. On the next run of the same template the log says "Updating Repository" instead of "Cloning". The task record has the commit that ran:

$ curl -s -H "$H" $G/project/1/tasks/8 | jq -c '{id,status,params,commit_hash:(.commit_hash|.[0:8]),commit_message,user_id}'
{"id":8,"status":"success","params":{"limit":["web1"]},"commit_hash":"8bc6fc38","commit_message":"First playbooks","user_id":1}

Every task is tied to a commit and a user, which is the audit trail you did not have with a shell.

The lab did not need to accept host keys: its hosts were in nobody's known_hosts file and the runs went through without a prompt. If your policy is to verify host keys, check how Semaphore and your ansible.cfg handle that on your side (not run in the lab).

Run the baseline template, then run it again. The first run changes things, the second changes nothing (this is Ansible's idempotence, seen through Semaphore):

$ semrun 2        (first run)
TASK [Install the baseline packages] ... changed: [web2] changed: [web1]
TASK [Write the login banner]        ... changed: [web1] changed: [web2]
web1 : ok=3 changed=2 ...            web2 : ok=3 changed=2 ...
$ semrun 2        (second run)
web1 : ok=3 changed=0 ...            web2 : ok=3 changed=0 ...

A run that fails is a task with status error. The lab's maintenance playbook fails on web2 on purpose:

$ semrun 3
status: error
TASK [Refuse to continue on web2] **********************************************
skipping: [web1]
fatal: [web2]: FAILED! => {"changed": false, "msg": "web2 is not ready for maintenance"}
PLAY RECAP *********************************************************************
web1   : ok=1  changed=0  unreachable=0  failed=0  skipped=1 ...
web2   : ok=1  changed=0  unreachable=0  failed=1  skipped=0 ...
Failed to run task: exit status 2

The exit status of ansible-playbook is the task's status (2 means a host failed).

A template can let the person who runs it change a few things for one run: limit it to some hosts, run it as a dry run (Ansible's check mode), show a diff, add variables. The API takes these in a params object on the task, and the template has to allow the ones that change what runs.

Allow overriding the limit on the template (the lab updated all three templates with the same extra field):

$ curl -s -H "$H" -H "$C" -X PUT $G/project/1/templates/1 -d '{"id":1,"project_id":1,"inventory_id":1,"repository_id":2,"environment_id":1,"name":"ping","playbook":"playbooks/ping.yml","app":"ansible","type":"","arguments":"[]","allow_override_args_in_task":true,"task_params":{"allow_debug":true,"allow_override_limit":true,"allow_override_tags":true,"allow_override_skip_tags":true}}'
$ curl -s -H "$H" $G/project/1/templates/1 | jq -c .task_params
{"allow_debug":true,"allow_override_limit":true,"allow_override_skip_tags":true,"allow_override_tags":true}

Then run with a limit. Before the template allowed it, the lab's limit was ignored and the playbook ran on both hosts. After it, only web1 ran:

$ semrun 1 '"params":{"limit":["web1"]}'
TASK [Ping]  ok: [web1]
PLAY RECAP  web1 : ok=3 changed=0 ...

A dry run with a diff. The lab changed the banner text in git, then asked for a dry run with a diff. The playbook showed what it would change, and the file on the host stayed as it was:

$ semrun 2 '"params":{"dry_run":true,"diff":true}'
TASK [Write the login banner] **************************************************
--- before: /etc/motd
+++ after: .../motd.j2
@@ -1,2 +1,2 @@
 web1 (lab)
-Managed by Ansible - maintained by the ops team
+Managed by Ansible - maintained by the platform team
changed: [web1]
$ ssh ubuntu@web1 cat /etc/motd
web1 (lab)
Managed by Ansible - maintained by the ops team

THE TRAP. The first dry run the lab tried put the options at the top level of the task: {"template_id":2,"dry_run":true,"debug":false}. Semaphore accepted it (HTTP 201), stored dry_run as null, and ran the playbook for real. The banner on both hosts changed. Nothing in the answer said the flag was ignored. The fields belong inside "params". Check any new automation with a harmless playbook first, and read the task record: if params is missing, the options were not applied.

Extra variables. The task's environment field is a JSON string of variables for this one run (the web interface's survey and the "extra variables" box end up here). They override the environment saved on the template:

$ semrun 2 '"environment":"{\"motd_message\":\"Patch window tonight\"}"'
$ ssh ubuntu@web1 cat /etc/motd
web1 (lab)
Patch window tonight - maintained by the platform team

Put variables with fixed values in the template's environment; use the task's environment (or a survey on the template, not run in the lab) for values a person chooses each time.

Semaphore can email the people on a project when a task fails. Turn it on in three places. First the mail server in config.json (the lab used a local postfix; use your relay and its credentials in real life), and the public URL, which is used in the links inside the email:

  "email_alert": true,
  "email_sender": "semaphore@sem.example.com",
  "email_host": "localhost",
  "email_port": "25",
  "email_secure": false,
  "web_host": "http://sem.example.com"

Restart the server after changing config.json (it is read at start-up). Then switch alerts on for the project and for the user who should get them (the user's email must be right):

$ curl -s -H "$H" -H "$C" -X PUT $G/project/1 -d '{"id":1,"name":"Lab ops","alert":true,"alert_chat":"","max_parallel_tasks":0}' -w "project alert: %{http_code}\n"
project alert: 204
$ curl -s -H "$H" -H "$C" -X PUT $G/users/1 -d '{"name":"Admin","username":"admin","email":"opsmail@sem.example.com","alert":true,"admin":true}' -w "user alert: %{http_code}\n"
user alert: 204

(The first attempt used /api/projects/1 and got "Not Found". The project is at /api/project/1, singular, for this call.) After that, the next failed run:

$ semrun 3
...
Failed to run task: exit status 2
Attempting to send email alert to opsmail@sem.example.com
Sent successfully email alert to opsmail@sem.example.com

and the mailbox had:

From: semaphore@sem.example.com
To: opsmail@sem.example.com
Subject: Task 'maintenance' failed

<p>Task 18 with template 'maintenance' has failed!</p>
<p>Task Log: <a href="http://sem.example.com/project/1/templates/3?t=18">Link</a></p>

A run that failed before the project's alert was on (task 17) sent nothing, and a run that succeeded sent nothing: the lab's mailbox held one message after both. There was no "recovered" email either: a successful run after the failure sent nothing, so Semaphore tells you about failures only. In the lab it went to the admin user, who had alerts on. Chat alerts (Telegram, Slack, Rocket.Chat, Microsoft Teams) are other settings in the same file, not run in the lab.

The same output is in the web interface, in the task list. If you also run Prometheus or Grafana (see those guides), a failing scheduled run is a reason for a service-level alert: Semaphore itself is just a web server, and a task that never starts is not something it will email you about.

A schedule starts a template on a cron expression (five fields, minute first). The lab used every minute to see it work, which you should never do for real:

$ curl -s -H "$H" -H "$C" -X POST $G/project/1/schedules -d '{"project_id":1,"template_id":1,"name":"ping every minute","cron_format":"* * * * *","active":true}' | jq -c '{id,name,cron_format,active}'
{"id":1,"name":"ping every minute","cron_format":"* * * * *","active":true}
$ sleep 75; curl -s -H "$H" $G/project/1/tasks | jq -c '.[0] | {id,status,template_id}'
{"id":15,"status":"success","template_id":1}
$ curl -s -H "$H" $G/project/1/schedules | jq -c '.[] | {id,name,cron_format,active,tpl_name}'
{"id":1,"name":"ping every minute","cron_format":"* * * * *","active":true,"tpl_name":"ping"}

The task appeared on its own, one minute after the schedule was created. Remove a schedule with DELETE /api/project/1/schedules/ID (204). A bad expression is refused with the reason:

$ curl -s -H "$H" -H "$C" -X POST $G/project/1/schedules -d '{"project_id":1,"template_id":1,"name":"bad","cron_format":"every day","active":true}'
{"error":"Cron: expected exactly 5 fields, found 2: [every day]"}      (HTTP 400)

Good uses: a nightly baseline run in check mode that only reports drift, a weekly patching run, a daily run of a playbook that cleans up. Schedule only templates that are safe to run unattended, and make them report: a failed scheduled task is the email above. Not run in the lab: the timezone of schedules (check the setting for your version), and schedules that delete themselves after one run.

An integration is a URL that starts a template when somebody calls it. The usual use is a webhook from git: push a change, the playbook runs. Create the integration for a template, give it an alias (the secret-looking part of the URL), and read the URL back:

$ curl -s -H "$H" -H "$C" -X POST $G/project/1/integrations/2/aliases | jq -c .
{"id":2,"url":"http://sem.example.com/api/integrations/kep7khqiwdw2qsex"}

The URL is built from web_host in config.json, so set that to the address people really use. The call to the URL is not made with a login, which is why it needs its own check. The check is the auth_method of the integration. The lab found that the obvious choice does not exist, and one wrong choice fails open.

1. auth_method "none" is not a method. The lab created an integration with it. Calling it returned HTTP 204 and nothing ran; the server log said:
   level=error msg="Unknown verification method: none" context=integrations
   Use token, hmac, github or bitbucket (the lab ran token).

2. A token check needs a secret: create a key and point the integration at it with auth_secret_id, and name the header the caller must send with auth_header. The lab's first try used a key of type "string" for the secret. The result was the worst one: a call with no token at all started a task (the secret came out empty, and an absent header matched it), while a call with the right token was refused ("Invalid verification token"). A key of type login_password, with the secret in its password field, worked:

$ curl -s -H "$H" -H "$C" -X POST $G/project/1/keys -d '{"name":"webhook-login","type":"login_password","project_id":1,"login_password":{"login":"hook","password":"THE-SECRET"}}' | jq -c '{id,name,type}'
{"id":5,"name":"webhook-login","type":"login_password"}
$ curl -s -H "$H" -H "$C" -X PUT $G/project/1/integrations/2 -d '{"id":2,"name":"git push","project_id":1,"template_id":1,"auth_method":"token","auth_header":"X-Hook-Token","auth_secret_id":5,"task_params":{}}'

Then the lab tested the three cases against the alias URL, noting the newest task id after each call:

  no header            204   no task started
  wrong header value   204   no task started   (log: Invalid verification token)
  right header value   204   task 21 started   (integration_id 2)

The answer is 204 in every case. The only sign of a refused call is the server log, and a call to an alias that does not exist also answered 200. So test an integration by looking for the task, not at the status code. And test the refused cases, as above, to make sure a wrong secret really is refused.

Hook it to git. The lab's bare repository got a post-receive hook that calls the URL with the token; a push then started the template:

# /srv/git/ops.git/hooks/post-receive   (mode 755, owned by the semaphore user)
#!/bin/sh
curl -s -o /dev/null -X POST -H X-Hook-Token:THE-SECRET -H Content-Type:application/json -d {} http://localhost:3000/api/integrations/kep7khqiwdw2qsex

$ git commit -qam "README third" && git push -q origin main
$ curl -s -H "$H" $G/project/1/tasks | jq -c '.[0] | {id,status,template_id,commit_message,integration_id}'
{"id":22,"status":"success","template_id":1,"commit_message":"README third","integration_id":2}

The task ran the commit that was just pushed. On GitHub or GitLab you would use their webhook settings with the hmac, github or bitbucket method instead of a hook script (not run in the lab). Never leave an integration open: anyone who knows the URL could start your playbooks.

Playbooks that use modules outside ansible-core need the collections installed. Semaphore does it for you: before each run it looks in the repository for a requirements file and installs what it lists. It looks in these places, in this order (the log lists each one):

  playbooks/collections/requirements.yml     playbooks/requirements.yml     collections/requirements.yml
  requirements.yml                           playbooks/roles/requirements.yml     roles/requirements.yml

The lab added collections/requirements.yml and a play that uses a module from community.general:

# collections/requirements.yml
---
collections:
  - name: community.general

# playbooks/ini.yml
---
- name: Use a collection module
  hosts: all
  gather_facts: false
  tasks:
    - name: Set a value in an ini file
      community.general.ini_file:
        path: /tmp/lab.ini
        section: lab
        option: owner
        value: ops

$ semrun 5
Starting galaxy collection install process
Downloading https://galaxy.ansible.com/.../community-general-13.5.0.tar.gz to .../repository_2_template_5_home/.ansible/...
community.general:13.5.0 was installed successfully
[WARNING]: Collection community.general does not support Ansible version 2.16.3
TASK [Set a value in an ini file]  changed: [web1] changed: [web2]

The next run said "collections/requirements.yml has no changes. Skip galaxy install process." so the download is done again only when the file changes.

Two things that went wrong:

1. An empty requirements file fails the task. The lab's first attempt wrote an empty collections/requirements.yml (a shell mistake) and the task failed with
   ERROR! No requirements found in file '.../collections/requirements.yml'
   Failed to install requirements: exit status 1
   Remove the file or give it content.

2. The newest collection wanted a newer Ansible than Ubuntu's. The warning above says community.general 13.5.0 "does not support Ansible version 2.16.3" and the module happened to work. That is luck. Pin the collection to a version made for your ansible-core (version: ">=10.0.0,<11.0.0" style ranges in requirements.yml), or install a newer ansible-core for Semaphore (see the install section). Roles install the same way from roles/requirements.yml (not run in the lab).

A project has people, and each has a role in it. The lab made two users and gave them two different roles:

$ curl -s -H "$H" -H "$C" -X POST $G/users -d '{"name":"Ria Runner","username":"ria","email":"ria@example.com","password":"PASSWORD","admin":false,"alert":false}' | jq -c '{id,username,admin}'
{"id":2,"username":"ria","admin":false}
$ curl -s -H "$H" -H "$C" -X POST $G/project/1/users -d '{"user_id":2,"role":"task_runner"}'
$ curl -s -H "$H" -H "$C" -X POST $G/project/1/users -d '{"user_id":3,"role":"guest"}'
$ curl -s -H "$H" $G/project/1/users | jq -c '.[] | {username,role}'
{"username":"admin","role":"owner"}
{"username":"gus","role":"guest"}
{"username":"ria","role":"task_runner"}

The roles are owner, manager, task_runner and guest. What the lab saw for each person (as that user, over the API):

  ria (task_runner)   sees the project; lists the keys; starts a task (201); cannot create a template (403)
  gus (guest)         sees the project; lists the keys; cannot start a task (403); cannot create a template (403)

The key list does not leak secrets: the entries show names and types, and the private key field is empty even for the owner. A task_runner can start any template in the project, so a template that does something dangerous is dangerous for every task_runner. Keep dangerous playbooks in a project of their own, with a smaller group.

User accounts can be created in the interface, with the API as above, or with the command line on the server, which is also your way back in if you lose the admin password. It reads the same configuration file as the server:

$ cd /var/lib/semaphore && sudo -u semaphore semaphore user get --login ria --config /etc/semaphore/config.json
Name: Ria Runner
Email: ria@example.com
Admin: false
$ sudo -u semaphore semaphore user change-by-login --login ria --password NEWPASSWORD --config /etc/semaphore/config.json
User ria <ria@example.com> changed!
$ sudo -u semaphore semaphore user add --admin --login boss --name Boss --email boss@example.com --password PASSWORD --config /etc/semaphore/config.json

(After the change the old password gave HTTP 401 and the new one 204.) Also in the same command: list, delete, token (API tokens) and totp (two-step login). LDAP and OIDC logins, and TOTP for users, are settings in config.json (not run in the lab). Give each person their own login, so that "who ran this" in the task list means something.

By default the server runs the playbooks itself, so a Semaphore server needs network access to every host and holds every secret. A runner is a second machine that polls the server for work and runs the playbook there. Use runners to reach a network the server cannot, to spread the load, or to keep the SSH access in one place.

1. Tell the server a runner may register, by setting a registration token in its config.json (any long random string; restart the server):

  "runner_registration_token": "reg-labtoken-0123456789"

2. On the runner machine (the lab used one of the managed hosts), install the same package and Ansible, make a user, and write the runner's configuration with setup. The questions are the server URL, whether the token is in a file, and whether to register:

$ sudo apt-get install -y ./semaphore_2.19.16_linux_amd64.deb ansible-core git
$ sudo useradd --system --create-home --home-dir /var/lib/semaphore-runner --shell /usr/sbin/nologin semrunner
$ sudo semaphore runner setup --config /etc/semaphore-runner/config.json
Semaphore server URL: http://sem
...
Configuration written to /etc/semaphore-runner/config.json..
$ echo "reg-labtoken-0123456789" | sudo semaphore runner register --config /etc/semaphore-runner/config.json --stdin-registration-token --name sweb2 --tags edge
level=info msg="Registering a new runner"

The register step puts the runner's own token into its config file. That file is now a secret, so restrict it (chown root:semrunner, chmod 640), then start the runner as the unprivileged user:

$ sudo chown -R semrunner:semrunner /var/lib/semaphore-runner
$ cd /var/lib/semaphore-runner && sudo -u semrunner semaphore runner start --config /etc/semaphore-runner/config.json
Runner connected

3. On the server the runner is listed, but it is not active yet. The lab's list showed registered true and active false, and the single-runner call showed the tag and "status":"online". Enable it (the update call needs the name, the tags and the flag):

$ curl -s -H "$H" -H "$C" -X PUT $G/runners/3 -d '{"id":3,"name":"sweb2","active":true,"tags":["edge"],"max_parallel_tasks":2}' -w "enable runner: %{http_code}\n"
enable runner: 204
$ curl -s -H "$H" $G/runners/3 | jq -c '{name,active,status,tags}'
{"name":"sweb2","active":true,"status":"online","tags":["edge"]}

4. A template chooses a runner with a tag. Give the template runner_tag "edge" and the task is handed to the runner:

$ semrun 4
status: success
Task ping on runner added to queue
Task #24 is assigned to runner #3
Cloning Repository git://sem/ops.git
PLAY RECAP ... web1 : ok=3 ... web2 : ok=3 ...

Things the lab ran into:

  The runner clones the repository itself, so the repository URL must work from the runner. The file:// repository of the server's own tasks is not reachable from another machine. The lab shared the repository with git daemon (git://sem/ops.git, read-only), and the first run failed with "Could not read from remote repository". The daemon had been started as root, and git refused a repository owned by another user: "fatal: detected dubious ownership in repository at '/srv/git/ops.git'" (git's safe.directory protection). Running the daemon as the repository's owner (the semaphore user) fixed it. In real life use your Git server over https or ssh instead.

  The runner has to be able to reach the server's URL. After the lab moved the server to the loopback address (the proxy section), the runner log filled with "failed to fetch new jobs from the server ... http://sem:3000/...". Pointing the runner's web_host at the proxy address fixed it ("Runner connected").

  A running task shows "Task #N is assigned to runner #R" in its output, which is how you tell the two apart.

The server sends the SSH key and the inventory to the runner for each task. Not run in the lab: runners that belong to one project only (the register command's --project-id), the per-runner webhook, and unregistering.

Semaphore speaks plain HTTP and has no TLS of its own in this setup. Put nginx in front for TLS, and bind Semaphore to the loopback address so that the only way in is through the proxy. In config.json:

  "interface": "127.0.0.1",
  "port": ":3000",
  "web_host": "http://sem.example.com"

(Use your real https address in web_host.) The log confirms what it bound to: Interface 127.0.0.1, Port :3000. The page uses a WebSocket to stream a running task's output, so the proxy has to allow the upgrade. The lab's nginx file:

# /etc/nginx/conf.d/semaphore.conf
map $http_upgrade $connection_upgrade { default upgrade; "" close; }
server {
    listen 80;
    server_name sem.example.com;

    location / {
        proxy_pass http://127.0.0.1:3000/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    location /api/ws {
        proxy_pass http://127.0.0.1:3000/api/ws;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_read_timeout 3600s;
    }
}

Test the three things that can go wrong, from the server and from another machine:

$ curl -s -c /tmp/cj2 -H "Host: sem.example.com" -X POST -H "Content-Type: application/json" -d '{"auth":"admin","password":"PASSWORD"}' -w "%{http_code}\n" http://127.0.0.1/api/auth/login
204
$ curl -s -b /tmp/cj2 -H "Host: sem.example.com" http://127.0.0.1/api/user | jq -c .username
"admin"
$ curl -s -i -m 3 -b /tmp/cj2 -H "Host: sem.example.com" -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" -H "Origin: http://sem.example.com" http://127.0.0.1/api/ws | head -1
HTTP/1.1 101 Switching Protocols

101 means the WebSocket upgrade works through nginx. From another container, port 3000 gave no answer (loopback only) and port 80 through nginx returned 200. Add listen 443 ssl and a certificate for real use (the lab ran plain HTTP; TLS not run). If the live log in the browser stays empty while tasks run, the WebSocket location is the usual cause (not seen in the lab, which had no browser).

PostgreSQL. SQLite is one file and is fine for a small team. For several people running many tasks, or for a server you want to back up with the usual database tools, use PostgreSQL. The lab created a database and a user, wrote a configuration with a postgres section, created the tables, added an admin and started a second Semaphore on it:

$ sudo -u postgres psql -c "CREATE USER semaphore WITH PASSWORD 'PASSWORD';" -c "CREATE DATABASE semaphore OWNER semaphore;"
# /tmp/pg-config.json  (the same cookie and key strings as any config; new ones come from setup)
{
  "postgres": {"host": "127.0.0.1:5432", "user": "semaphore", "pass": "PASSWORD", "name": "semaphore", "options": {"sslmode": "disable"}},
  "dialect": "postgres",
  "port": ":3002",
  "interface": "127.0.0.1",
  "tmp_path": "/tmp/pg-tmp",
  "cookie_hash": "...", "cookie_encryption": "...", "access_key_encryption": "..."
}
$ sudo -u semaphore semaphore migrate --config /tmp/pg-config.json
Postgres semaphore@127.0.0.1:5432 semaphore
 [1/0]
Migrations Finished
$ sudo -u semaphore semaphore user add --admin --login admin --name Admin --email admin@example.com --password PASSWORD --config /tmp/pg-config.json
User admin <admin@example.com> added!
$ sudo -u semaphore semaphore server --config /tmp/pg-config.json
Postgres semaphore@127.0.0.1:5432 semaphore
Port :3002
Server is running

It started, answered /api/ping with pong, and the admin was in the database. The lab did not move data from SQLite to PostgreSQL: that is a separate job (export and re-create, or start fresh and re-create the project from files; not run). Keep sslmode on for a database on another machine. MySQL is set up the same way (not run in the lab).

What to back up: the database, the configuration file (it has the encryption keys) and your own scripts. The playbooks are in git, which is its own backup. Use SQLite's .backup, which is safe while the server is running, and copy the config with restricted permissions:

$ sudo mkdir -p /var/backups/semaphore
$ sudo sqlite3 /var/lib/semaphore/semaphore.db ".backup /var/backups/semaphore/semaphore-$(date +%F).db"
$ sudo sqlite3 /var/backups/semaphore/semaphore-$(date +%F).db "pragma integrity_check;"
ok
$ sudo cp -p /etc/semaphore/config.json /var/backups/semaphore/config-$(date +%F).json
$ sudo chmod 600 /var/backups/semaphore/*

The database and the configuration are a pair. Restored with a different configuration, the keys in the key store cannot be decrypted. The lab started a copy of the database with a made-up access_key_encryption value and ran a task:

Failed to install inventory: encryption key id "2xhjQ-zP_Jw" not found in keyset (the key encrypting this value is missing)
Failed to run task: encryption key id "2xhjQ-zP_Jw" not found in keyset (the key encrypting this value is missing)

The error names the missing key. It means the config.json is not the one the data was made with: find the right one rather than re-entering all the keys. (Logins worked on that copy, because those are not encrypted this way; only the key store is.)

A full restore test. The lab deleted a template through the API, stopped the server, copied the backup over the database (owner semaphore, mode 600, with any -wal and -shm files removed), started the server and listed the templates:

$ curl -s -H "$H" -X DELETE $G/project/1/templates/3 -w "delete template: %{http_code}\n"
delete template: 204
$ curl -s -H "$H" $G/project/1/templates | jq -c '[.[]|.name]'
["baseline","ping","ping on runner"]
   (stop the server, restore the file, start the server)
$ curl -s -H "$H" $G/project/1/templates | jq -c '[.[]|.name]'
["baseline","maintenance","ping","ping on runner"]

The deleted template came back, and the same API token still worked (tokens are in the database). For PostgreSQL use pg_dump and pg_restore (not run in the lab).

Upgrade: back up first, then install the new package over the old one and restart. The database tables are upgraded by the migration step (semaphore migrate, which setup also runs; whether the server does it by itself on start-up was not checked). You cannot go back without the old backup. Read the release notes before a big jump. The lab installed the current release and did not upgrade (not run). Runners must be upgraded to the same version as the server (not run in the lab).

Disk: every task keeps its full output in the database, and the clones in tmp_path are deleted or reused. Watch the size of the database and of tmp_path, and remove old tasks you do not need. How to prune them depends on your version (not run in the lab).

Install             download the .deb, sha256sum -c, apt install ./semaphore_VERSION_linux_amd64.deb ; apt install ansible-core git
Setup               sudo semaphore setup --config /etc/semaphore/config.json   (then chown root:semaphore, chmod 640 the file)
Run                 sudo -u semaphore semaphore server --config /etc/semaphore/config.json     (systemd unit: ExecStart=/usr/bin/semaphore server --config ...)
Health              curl -s localhost:3000/api/ping     -> pong
Login (API)         POST /api/auth/login {"auth":"USER","password":"PW"} -> cookie ; POST /api/user/tokens -> Bearer token
Project objects     POST /api/projects ; /api/project/ID/keys | repositories | inventory | environment | templates
Start a run         POST /api/project/ID/tasks {"template_id":N}
Run options         add  "params":{"limit":["web1"],"dry_run":true,"diff":true}   (inside params, never at the top level)
Extra variables     add  "environment":"{\"name\":\"value\"}"   to the task
Read a run          GET /api/project/ID/tasks/TASK ; GET /api/project/ID/tasks/TASK/output
Schedule            POST /api/project/ID/schedules {"template_id":N,"cron_format":"0 3 * * *","active":true}
Webhook             POST /api/project/ID/integrations (auth_method token + auth_header + auth_secret_id of a login_password key), then POST .../integrations/ID/aliases
Alerts              config.json email_alert/email_sender/email_host/email_port, PUT /api/project/ID {"alert":true}, user alert true
People              POST /api/users ; POST /api/project/ID/users {"user_id":N,"role":"task_runner"} ; roles: owner manager task_runner guest
CLI users           semaphore user add|get|change-by-login|list|delete --config FILE
Runner              server: runner_registration_token ; runner: setup, register --stdin-registration-token --name N --tags T, start ; enable with PUT /api/runners/ID
Collections         collections/requirements.yml (not empty!) ; pin versions for your ansible-core
Proxy               interface 127.0.0.1, web_host, nginx with Upgrade headers on /api/ws
PostgreSQL          config "postgres":{host,user,pass,name,options}, "dialect":"postgres"; semaphore migrate ; semaphore user add
Backup              sqlite3 DB ".backup FILE" + cp -p config.json (keep them together, mode 600)

Setup panics "open /etc/semaphore/...: permission denied"   run setup as root, then chown root:semaphore and chmod 640 the config
Anyone on the machine can read the config                  setup writes it world-readable (644) and it holds the encryption keys: chmod 640 and chown root:semaphore
BoltDB is "NOT SUPPORTED"                                  choose SQLite, PostgreSQL or MySQL
"the playbook: playbooks/x.yml could not be found"         a repository with a plain path as its URL is a local folder, not a clone; use file://, https:// or ssh. Also check the playbook path in the template
Run options seem ignored, and the run was real             dry_run, diff, limit, debug must be inside "params", and overrides must be allowed in the template's task_params. Check the task record
Limit has no effect                                        allow_override_limit is not set on the template
"No requirements found in file ..."                        collections/requirements.yml is empty: give it content or remove it
Warning "Collection X does not support Ansible version"    the collection is newer than ansible-core: pin its version or install a newer ansible-core
Webhook returns 204 but nothing runs                       look at the Semaphore log: "Unknown verification method" (auth_method none is not valid) or "Invalid verification token"
A webhook with a token started without one                 the secret key was of type string and came out empty; use a login_password key and test the refused cases
Email alert never arrives                                  email_alert off in config.json, project alert or user alert off, the user's address wrong, or the mail server is refusing: read its log. Only failures send mail
Template cannot be created / run: 403                      the user's project role is task_runner or guest
Runner shows registered but does not run tasks             it is not active (PUT /api/runners/ID with active true), or the template has no runner_tag, or the tags differ
Runner log "failed to fetch new jobs"                      the runner's web_host does not reach the server (the server moved to loopback, or the proxy address changed)
Runner: "Could not read from remote repository"            the repository URL is not reachable from the runner (file:// is local to the server); and git daemon as another user hits "dubious ownership"
Live log stays empty behind a proxy                        the proxy does not pass the WebSocket upgrade for /api/ws
"encryption key id ... not found in keyset"                config.json is not the one that goes with this database: restore the right access_key_encryption
Locked out                                                 sudo -u semaphore semaphore user change-by-login --login admin --password NEW --config FILE
Tasks stay "waiting" (not seen in the lab)                 check max_parallel_tasks on the project, and that the runner for the template's tag is active

Keep every playbook in git, and let Semaphore be the thing that runs them. A change is a commit, a run is a task with the commit recorded.
Start with a ping template and a baseline template in check mode. Run the real thing only after the dry run's diff looks right.
Put the options in "params", and test any new automation with a playbook that does no harm first.
Give each person a login and the smallest role that does the job. Treat task_runner as "may run every template in this project".
Keep the keys in the key store and the config.json out of the repository. Copy config.json with the database backup, in a safe place with restricted access.
Use a token or HMAC check on every webhook and test that a wrong secret is refused, by looking for the task rather than the status code.
Schedule drift checks (dry run with a diff) rather than silent fixes, and make a failed scheduled task send mail to somebody who will read it.
Pin collection versions in requirements.yml to what your ansible-core supports.
Run the server on the loopback address behind a TLS proxy, and give runners a reachable URL. Never open port 3000 to the network.
Use PostgreSQL once several people depend on it, and rehearse the restore on a spare machine at least once.
Read the failed task's output first. It is plain ansible-playbook output, and the Ansible guides on this site explain every line of it.