📦 Ansible Roles and Projects Guide

Ansible roles and projects: from one playbook to something you can maintain
ansible-playbook site.yml --check --diff

The Ansible sheet on this site explains the pieces and the One-Liners page is the quick, ad-hoc side. This page is the middle: what happens when a playbook outgrows one file. It builds one small but real project (a common role, a web server role and a database role, three machines) and uses it to show how roles are laid out, where variables live and which one wins, how templates and handlers really behave, how to preview and roll out a change safely, how to keep secrets, and how to lint the result. It also shows the traps, because most of what you learn about Ansible you learn from them: a handler that never runs, a check that cannot see a variable, a command that always says "changed".
Every output below ran for real, on 9 October 2026: ansible-core 2.21.5 and ansible-lint 26.9.0 (installed with pip in a virtualenv), with one controller and three managed machines (web1, web2, db1), all Ubuntu 24.04 containers with Python 3.12 and an SSH server, reached over SSH as a normal user (deploy) that has passwordless sudo. nginx and PostgreSQL 16 came from the Ubuntu repositories. A container is not a server: there is no systemd inside, so the service module used the old init scripts. Where that matters, or where something was not run, this page says so. Anything marked "not run in the lab" was not run.

Four containers on a private network, built from one image: ubuntu:24.04 plus openssh-server, python3, sudo and curl, a user deploy with a key and passwordless sudo.
  ctl    the controller. Ansible lives here, in a virtualenv: python3 -m venv ~/venv; ~/venv/bin/pip install ansible-core ansible-lint
  web1   managed machine, group web
  web2   managed machine, group web
  db1    managed machine, group db
$ ansible --version | head -1
ansible [core 2.21.5]
$ ansible all -m ping | grep -c pong
3
The project directory on the controller is ~/proj, and every command below runs from inside it.

$ find . -type f | sort
./ansible.cfg
./group_vars/all.yml
./group_vars/web.yml
./host_vars/web2.yml
./inventory/hosts.ini
./roles/common/defaults/main.yml
./roles/common/tasks/main.yml
./roles/common/templates/motd.j2
./roles/dbserver/defaults/main.yml
./roles/dbserver/handlers/main.yml
./roles/dbserver/tasks/main.yml
./roles/dbserver/templates/ansible.conf.j2
./roles/webserver/defaults/main.yml
./roles/webserver/handlers/main.yml
./roles/webserver/meta/main.yml
./roles/webserver/tasks/main.yml
./roles/webserver/templates/index.html.j2
./roles/webserver/templates/nginx.conf.j2
./roles/webserver/templates/site.conf.j2
./rolling.yml
./site.yml
The idea: site.yml says WHICH roles go on WHICH machines, and nothing else. Everything about HOW lives in the role. A role is a directory with fixed names, and Ansible looks for each file by name, so you never write an include:
  tasks/main.yml       what the role does (required)
  handlers/main.yml    actions that run on notification (restart, reload)
  defaults/main.yml    the role's variables with the LOWEST priority: safe defaults that anybody can override
  templates/           Jinja2 files, found by name from template: src=
  files/               plain files for copy: (not used in this project)
  meta/main.yml        dependencies and metadata
  vars/main.yml        variables with HIGH priority, hard to override. Use sparingly (not used here)
The configuration file in the project directory is found automatically when you run ansible from there:
$ cat ansible.cfg
[defaults]
inventory = inventory/hosts.ini
remote_user = deploy
roles_path = roles
result_format = yaml
retry_files_enabled = False
interpreter_python = auto_silent

[privilege_escalation]
become = True
(I set result_format = yaml hoping for YAML-style results. In 2.21.5 the output below is still JSON-style, so it had no visible effect here, and I do not rely on it.)
The inventory has two groups and a parent group made of them:
$ cat inventory/hosts.ini
[web]
web1
web2

[db]
db1

[prod:children]
web
db
$ ansible-inventory --graph
@all:
  |--@ungrouped:
  |--@prod:
  |  |--@web:
  |  |  |--web1
  |  |  |--web2
  |  |--@db:
  |  |  |--db1
and the whole playbook is:
$ cat site.yml
---
- name: Things every server gets
  hosts: all
  roles:
    - common

- name: Web servers
  hosts: web
  roles:
    - role: webserver
      tags: web

- name: Database servers
  hosts: db
  roles:
    - role: dbserver
      tags: db
Three plays, five lines each. That is the point of roles.

In this project a variable comes from one of four places. From weakest to strongest:
  1. role defaults       roles/webserver/defaults/main.yml      webserver_worker_connections: 768
  2. group_vars          group_vars/web.yml, group_vars/all.yml  the list of sites, the contact address
  3. host_vars           host_vars/web2.yml                      webserver_worker_connections: 1024
  4. extra vars          -e name=value on the command line       beats everything
(Ansible has more levels than these four; only these were tested here.)
Ask the inventory what a host will see:
$ ansible-inventory --host web2 --yaml
admin_contact: ops@example.org
environment_name: lab
webserver_sites:
- message: Welcome to the shop
  name: shop
  server_name: shop.example.org
- message: Read the blog
  name: blog
  server_name: blog.example.org
webserver_worker_connections: 1024
Notice what is missing for web1: it has no webserver_worker_connections at all. The role default (768) is NOT in this output, because ansible-inventory knows nothing about roles. An ad-hoc command has the same blind spot:
$ ansible web1,web2 -m ansible.builtin.debug -a var=webserver_worker_connections
web1 | SUCCESS => {
    "webserver_worker_connections": "<< error 1 - 'webserver_worker_connections' is undefined >>"
}
web2 | SUCCESS => {
    "webserver_worker_connections": 1024
}
So a role default only exists once the role is running. To see the value a host will really use, print it from inside a play that includes the role (a one-task role was used for this test). Without and with an extra variable:
$ ansible-playbook show.yml | grep msg
    "msg": "web1 worker_connections = 768"
    "msg": "web2 worker_connections = 1024"
$ ansible-playbook show.yml -e webserver_worker_connections=4096 | grep msg
    "msg": "web1 worker_connections = 4096"
    "msg": "web2 worker_connections = 4096"
The default (web1: 768), the host variable (web2: 1024) and then -e beating both. (This test ran while the host file said 2048; the page shows the final 1024 to match the project listing. A note on names: the experiments in this page ran under the short names web_port, web_sites and so on, which the lint section renames at the end. The page shows every output under the final names.)
A naming rule worth learning early, and the lint section shows why: start every variable of a role with the role's name (webserver_port, dbserver_max_connections). Variables from all roles share ONE namespace, and a generic name like port or max_connections will collide with another role's.

The defaults file lists everything a user of the role may change, with a safe value:
$ cat roles/webserver/defaults/main.yml
webserver_port: 80
webserver_worker_connections: 768
webserver_sites: []
The tasks file is the work. The whole thing is long, so here is the heart of it. Every module name is written in full (ansible.builtin.template, not template): that is the "fully qualified collection name", it avoids a clash with a module of the same name from another collection, and the linter asks for it.
$ cat roles/webserver/tasks/main.yml        (excerpt)
- name: Write the main nginx config (checked with nginx -t before it is installed)
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: "0644"
    validate: nginx -t -c %s
  notify: Restart nginx

- name: Write one config per site
  ansible.builtin.template:
    src: site.conf.j2
    dest: "/etc/nginx/conf.d/{{ item.name }}.conf"
    owner: root
    group: root
    mode: "0644"
  loop: "{{ webserver_sites }}"
  loop_control:
    label: "{{ item.name }}"
  notify: Reload nginx
Three details that matter:
  validate: nginx -t -c %s   Ansible writes the new file to a temporary place and runs this command on it BEFORE moving it into position. %s is the temporary file. If the check fails, the real file is never touched (shown under "Error messages").
  loop and loop_control   one task, one pass per item of the list. label keeps the output readable: (item=shop) instead of the whole dictionary.
  notify   names a handler. The handler is asked to run, not run on the spot.
The handlers file:
$ cat roles/webserver/handlers/main.yml
---
- name: Restart nginx
  ansible.builtin.service:
    name: nginx
    state: restarted

- name: Reload nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded
The other two roles follow the same pattern: common installs a few packages and writes a login banner; dbserver installs PostgreSQL and drops one settings file into its conf.d directory, with a handler that restarts it.

A template is a text file with {{ variables }} and {% statements %}. Ansible renders it on the controller, with the variables and the facts of the target host, and copies the result. This is the banner template of the common role:
$ cat roles/common/templates/motd.j2
{{ ansible_managed }}
Host:        {{ inventory_hostname }} ({{ ansible_facts['default_ipv4']['address'] }})
Environment: {{ environment_name }}
Groups:      {{ group_names | join(', ') }}
Contact:     {{ admin_contact }}
and this is what arrived on db1:
$ ssh db1 cat /etc/motd
Ansible managed
Host:        db1 (172.21.0.4)
Environment: lab
Groups:      db, prod
Contact:     ops@example.org
{{ ansible_managed }} is a built-in variable holding a short "do not edit by hand" text (its default is just "Ansible managed", as shown). In the nginx templates it is piped through the comment filter (ansible_managed | comment) so that it becomes a comment in the target file's syntax.
Loops, conditions and filters, all in one template. This one is only rendered to the screen (on the controller, with web1's facts), not installed:
$ cat /tmp/report.j2
{% for s in webserver_sites %}
{{ loop.index }}/{{ loop.length }} {{ s.name | upper }} -> {{ s.server_name }}{% if loop.last %}   (last one){% endif %}

{% endfor %}
Port: {{ webserver_port | default(80) }}   Hosts in group web: {{ groups['web'] | join(', ') }}
Names as JSON: {{ webserver_sites | map(attribute='name') | list | to_json }}
Memory: {{ ansible_facts['memtotal_mb'] }} MB, python {{ ansible_facts['python_version'] }}
{% if 'db' in group_names %}This is a database server.{% else %}This is not a database server.{% endif %}
$ ansible-playbook show.yml        (msg: "{{ lookup('ansible.builtin.template', '/tmp/report.j2') | split('\n') }}")
    "msg": [
        "1/2 SHOP -> shop.example.org",
        "2/2 BLOG -> blog.example.org   (last one)",
        "Port: 80   Hosts in group web: web1, web2",
        "Names as JSON: [\"shop\", \"blog\"]",
        "Memory: 31778 MB, python 3.12.3",
        "This is not a database server.",
        ""
    ]
(The memory figure is the host machine's, because a container sees the whole host's memory.) Things to notice: loop.index, loop.length and loop.last inside a loop; default(80) as a safety net for an undefined variable; map(attribute=...) plus list and to_json to turn a list of dictionaries into a list of names; groups['web'] for the members of another group; and group_names for the groups the current host belongs to. Use the default filter freely, but not to hide a variable that should always exist: an undefined variable is better as an error (see below).

--check runs the play without changing anything, and --diff shows what each file change would look like. On a fresh machine this stopped immediately:
$ ansible-playbook site.yml --check --diff
TASK [common : Install the common packages] ************************************
[ERROR]: Task failed: Module failed: python3-apt must be installed to use check mode. If run normally this module can auto-install it, see the auto_install_module_deps option.
fatal: [web1]: FAILED! => {"changed": false, "msg": "python3-apt must be installed to use check mode. If run normally this module can auto-install it, see the auto_install_module_deps option."}
The apt module needs the python3-apt library, installs it for itself in a normal run, but will not do that in check mode. So the very first preview of a brand-new Ubuntu machine fails. Install it once with a real, ad-hoc run (the become setting in ansible.cfg makes it root):
$ ansible all -m ansible.builtin.apt -a "name=python3-apt update_cache=true" | grep -E "SUCCESS|changed"
db1 | SUCCESS => {
    "changed": false
web1 | SUCCESS => {
    "changed": false
web2 | SUCCESS => {
    "changed": false
(changed: false because the lab image already had it. On a machine without it, the same command would install it.) Then the preview works, and shows the files it would write:
$ ansible-playbook site.yml --check --diff | sed -n '/Write the login banner/,/changed: \[db1\]/p'
TASK [common : Write the login banner] *****************************************
--- before
+++ after: /home/deploy/.ansible/tmp/ansible-local-232kcmn_7p0/tmpvzr5qjmu/motd.j2
@@ -0,0 +1,5 @@
+Ansible managed
+Host:        db1 (172.21.0.4)
+Environment: lab
+Groups:      db, prod
+Contact:     ops@example.org

changed: [db1]
Limits of check mode, seen in this run: the nginx package is not installed yet, so the install is reported "changed" but does not happen, and the tasks that depend on it are judged against a machine without nginx ("Remove the default site" says ok because there is no default site yet, and the nginx.conf diff is shown against an empty file). On a machine that is already configured, the preview matched what the real run then did (shown in "Change one thing" below).

I added the package tree to the common role's list (so that something would really be installed), and ran it:
$ time ansible-playbook site.yml
PLAY RECAP *********************************************************************
db1                        : ok=8    changed=6    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web1                       : ok=13   changed=11   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2                       : ok=13   changed=11   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
real	1m8.247s
About a minute, for three machines from nothing to nginx with two sites each and PostgreSQL. The recap reads: ok = tasks that ran fine (including the ones that changed something), changed = tasks that changed something. Then the same command again:
$ ansible-playbook site.yml | grep -E "RECAP|: ok="
PLAY RECAP *********************************************************************
db1                        : ok=7    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web1                       : ok=11   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2                       : ok=11   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
changed=0 on every machine. That is idempotence: the playbook describes the DESIRED STATE, not a list of actions, so running it again does nothing. It is the property you test for. A second run that still says changed is a task that is not idempotent, and it will cause trouble later (see "Commands and idempotence").
Finally, does it really work? The web servers answer for each site name, from the controller:
$ for h in web1 web2; do for s in shop blog; do echo "== $h $s.example.org"; curl -s -H "Host: $s.example.org" http://$h/; done; done
== web1 shop.example.org
<h1>Welcome to the shop</h1>
<p>Served by web1 for shop.example.org.</p>
== web1 blog.example.org
<h1>Read the blog</h1>
<p>Served by web1 for blog.example.org.</p>
== web2 shop.example.org
<h1>Welcome to the shop</h1>
<p>Served by web2 for shop.example.org.</p>
== web2 blog.example.org
<h1>Read the blog</h1>
<p>Served by web2 for blog.example.org.</p>
$ ssh db1 "sudo -u postgres psql -Atc 'show max_connections'"
100
On the first run the handlers ran in this order on each web machine, after all the tasks: Restart nginx, then Reload nginx (a pointless double, harmless on a first install; see the handlers section for why they both ran), and on db1: Restart postgresql.

Change one host variable (web2: 1024 to 2048) and preview just the web part:
$ ansible-playbook site.yml --tags web --check --diff
TASK [webserver : Write the main nginx config (checked with nginx -t before it is installed)] ***
--- before: /etc/nginx/nginx.conf
+++ after: /home/deploy/.ansible/tmp/ansible-local-1364j5ygfcxs/tmpk24jiwjs/nginx.conf.j2
@@ -6,7 +6,7 @@
 pid /run/nginx.pid;

 events {
-    worker_connections 1024;
+    worker_connections 2048;
 }

 http {

changed: [web2]
RUNNING HANDLER [webserver : Restart nginx] ************************************
changed: [web2]
That is the whole preview: one line of one file, on one machine, and one handler (the preview says what it WOULD do). On a machine that is already configured, a check is exact. Apply it:
$ ansible-playbook site.yml --tags web | grep -E "^(changed|RUNNING HANDLER)|RECAP|: ok="
changed: [web2]
RUNNING HANDLER [webserver : Restart nginx] ************************************
changed: [web2]
PLAY RECAP *********************************************************************
db1                        : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web1                       : ok=9    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2                       : ok=10   changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web1 and db1 were not touched (db1 shows ok=2 because the first play's fact gathering still runs on every host). Two changes on web2: the file, and the restart.
Now change a page message (in group_vars/web.yml, "Welcome to the shop" to "Welcome to the NEW shop") and apply:
$ ansible-playbook site.yml --tags web | grep -E "^(changed|RUNNING HANDLER)|RECAP|: ok="
changed: [web1] => (item=shop)
changed: [web2] => (item=shop)
...
No handler ran, and nginx was not touched. The task that writes the page has no notify, because nginx serves a changed file straight from disk. Decide which files need which handler and put notify only on those. A handler for everything is a restart at every change; a handler for nothing is a server that serves the old config.

Four rules cover almost everything:
  - A handler runs only if a task that notifies it reported CHANGED.
  - It runs once per host, however many tasks notified it, and at the END of the play.
  - Handlers run in the order they are DEFINED in handlers/main.yml, not in the order they were notified. In this role Restart comes before Reload, so on the first run both fired.
  - If a task fails on a host, the pending handlers of that host do NOT run.
The last rule produces the worst trap in Ansible. Here is a real run. I changed the port (webserver_port: 8080 in group_vars/web.yml) and added a check after the config tasks, to prove that the shop answers on the new port:
- name: Check that the shop answers on the new port
  ansible.builtin.uri:
    url: "http://localhost:{{ webserver_port }}/"
    headers:
      Host: shop.example.org
The first run:
$ ansible-playbook site.yml --tags web --limit web1
changed: [web1] => (item=shop)
changed: [web1] => (item=blog)
fatal: [web1]: FAILED! => {"changed": false, "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", ...
PLAY RECAP *********************************************************************
web1                       : ok=9    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
$ ssh web1 'grep listen /etc/nginx/conf.d/shop.conf; curl ... :8080 ; curl ... :80'
    listen 8080;
port 8080: 000
port 80:   200
The config file says 8080. The running nginx still listens on 80, because the Reload handler was waiting for the end of the play, and the check, which comes BEFORE the end, failed. Now run it again:
$ ansible-playbook site.yml --tags web --limit web1
fatal: [web1]: FAILED! => {"changed": false, "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", ...
PLAY RECAP *********************************************************************
web1                       : ok=9    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
changed=0. The file is already correct, so no task changes, so nothing notifies the handler, so nginx is NEVER reloaded, and the check fails for ever. On a playbook without such a check, this is worse: it ends GREEN, and the server runs a configuration other than the file. You find out when someone restarts the service months later.
Two cures, both tested. The first is a flag for the run that fails: --force-handlers runs the pending handlers even after a failure. I changed the port again (7070) and:
$ ansible-playbook site.yml --tags web --limit web1 --force-handlers
changed: [web1] => (item=shop)
changed: [web1] => (item=blog)
fatal: [web1]: FAILED! => {...Connection refused...}
RUNNING HANDLER [webserver : Reload nginx] *************************************
changed: [web1]
$ ssh web1 ... for p in 80 8080 7070 9090 ...
port 80: 000
port 8080: 000
port 7070: 200
port 9090: 000
The check task still failed (it ran before the handler), but the handler ran anyway and the server is now on 7070. Note it only helps the run where the file CHANGED; there is no flag that repairs the run after it. (The play keyword force_handlers: true should do the same inside the playbook; that was not run in the lab.)
The second cure is the better one: run the handlers yourself, at the right moment, with a meta task placed between the change and the check:
- name: Run the pending handlers NOW, before the check
  ansible.builtin.meta: flush_handlers
I put that in the role, changed the port to 9090 and ran again:
$ ansible-playbook site.yml --tags web --limit web1
changed: [web1] => (item=shop)
changed: [web1] => (item=blog)
RUNNING HANDLER [webserver : Reload nginx] *************************************
changed: [web1]
PLAY RECAP *********************************************************************
web1                       : ok=11   changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
port 9090: 200
The handler ran in the middle of the role, before the check, and the check passed in the same run. The rule that follows: whenever a task AFTER a change depends on the change having taken effect (a health check, a database migration that needs the new config, a task that uses the new service), put flush_handlers between them.
(After these tests the port variable was removed again, and all machines went back to port 80 in the next run.)

Before you run something, ask Ansible what it would do:
$ ansible-playbook site.yml --list-tags
  play #2 (web): Web servers	TAGS: []
      TASK TAGS: [web]
  play #3 (db): Database servers	TAGS: []
      TASK TAGS: [db]
$ ansible-playbook site.yml --list-tasks --tags db
  play #3 (db): Database servers	TAGS: []
    tasks:
      dbserver : Install PostgreSQL	TAGS: [db]
      dbserver : Drop in our settings	TAGS: [db]
      dbserver : Make sure PostgreSQL is running	TAGS: [db]
$ ansible-playbook site.yml --list-hosts
  play #1 (all): Things every server gets	TAGS: []
    pattern: ['all']
    hosts (3):
      web1
      web2
      db1
  ...
$ ansible-playbook site.yml --tags web --limit web1 | grep -E "RECAP|: ok="
web1                       : ok=9    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
--tags selects tasks, --limit selects machines, and they combine. The tag was put on the ROLE in site.yml (role: webserver, tags: web), which tags every task of the role at once. Note that the first play, the common role, has no tags at all, so with --tags web it does not run (only its fact gathering).

Changing every web server at the same time is how a bad change becomes an outage. serial: 1 runs the play on one host, completely, then the next. The play below applies the role, runs the handlers, and then checks that the site really answers. If anything in the block fails, the rescue part runs, and a fail task stops the rollout:
$ cat rolling.yml
---
- name: Change the web servers one at a time
  hosts: web
  serial: 1
  tasks:
    - name: Apply the change and prove it works
      block:
        - name: Apply the webserver role
          ansible.builtin.include_role:
            name: webserver
            public: true

        - name: Run the pending handlers now
          ansible.builtin.meta: flush_handlers

        - name: The shop must answer
          ansible.builtin.uri:
            url: "http://localhost:{{ webserver_port }}/"
            headers:
              Host: shop.example.org
      rescue:
        - name: Say what went wrong
          ansible.builtin.debug:
            msg: "{{ inventory_hostname }} failed at: {{ ansible_failed_task.name }}"

        - name: Stop the rollout here
          ansible.builtin.fail:
            msg: "Rolling change aborted on {{ inventory_hostname }}"
      always:
        - name: Note that this batch is over
          ansible.builtin.debug:
            msg: "finished {{ inventory_hostname }}"
block groups tasks; rescue runs when something in the block fails (like catch); always runs either way (like finally); ansible_failed_task is filled in for the rescue.
First a BAD change: an invalid value for the worker connections, given as an extra variable. Before this, web1 had 768 and web2 had 2048:
$ ansible-playbook rolling.yml -e webserver_worker_connections=abc
TASK [webserver : Write the main nginx config (checked with nginx -t before it is installed)] ***
fatal: [web1]: FAILED! => {"changed": false, "checksum": "f1b77eba...", "exit_status": 1, "msg": "failed to validate", "stderr": "...[emerg]...
TASK [Say what went wrong] *****************************************************
    "msg": "web1 failed at: Write the main nginx config (checked with nginx -t before it is installed)"
TASK [Stop the rollout here] ****************************************************
fatal: [web1]: FAILED! => {"changed": false, "msg": "Rolling change aborted on web1"}
TASK [Note that this batch is over] ********************************************
    "msg": "finished web1"
PLAY RECAP *********************************************************************
web1                       : ok=6    changed=0    unreachable=0    failed=1    skipped=0    rescued=1    ignored=0
$ ssh web2 "grep worker_connections /etc/nginx/nginx.conf"; ssh web1 "grep worker_connections /etc/nginx/nginx.conf"
    worker_connections 2048;
    worker_connections 768;
web1 failed, the validate check kept the broken file out, the rescue reported it, and web2 was NEVER TOUCHED: it does not appear in the recap and still has its old value. That is what serial buys you: one machine as a canary instead of all of them. (The play stopped there and did not go on to web2. I did not test the settings that change this, such as max_fail_percentage.)
Then a GOOD change. The first attempt failed for a different reason, and the reason is instructive:
$ ansible-playbook rolling.yml -e webserver_worker_connections=1500 --limit web1
fatal: [web1]: FAILED! => {"changed": false, "msg": "Task failed: Finalization of task args for 'ansible.builtin.uri' failed: Error while resolving value for 'url': 'webserver_port' is undefined"}
The role was applied, the file was changed, nginx was restarted. But the check task is OUTSIDE the role, and webserver_port is a role default: with include_role, the role's variables are not visible to the tasks after it. (Whether plain roles: or import_role behave differently here was not tested.) The cure is one line on the include, public: true, which is already in the file listing above:
$ ansible-playbook rolling.yml -e webserver_worker_connections=1600
PLAY [Change the web servers one at a time] ************************************
changed: [web1]
RUNNING HANDLER [webserver : Restart nginx] ************************************
changed: [web1]
    "msg": "finished web1"
PLAY [Change the web servers one at a time] ************************************
changed: [web2]
RUNNING HANDLER [webserver : Restart nginx] ************************************
changed: [web2]
    "msg": "finished web2"
PLAY RECAP *********************************************************************
web1                       : ok=13   changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2                       : ok=13   changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
The play header printed TWICE: one batch per machine. web1 finished, with its check passed, before web2 started.

The command and shell modules cannot know whether they changed anything, so they always say "changed". That breaks the "second run says changed=0" test, and it makes handlers fire for no reason. Four tasks, run twice:
- name: Naive command
  ansible.builtin.command: touch /tmp/marker
- name: Command with creates
  ansible.builtin.command: touch /tmp/marker2
  args:
    creates: /tmp/marker2
- name: Read-only command, never a change
  ansible.builtin.command: cat /etc/hostname
  changed_when: false
- name: Only change when the output says so
  ansible.builtin.command: grep -c max_connections /etc/postgresql/16/main/conf.d/ansible.conf
  register: grepped
  changed_when: grepped.stdout | int > 1
  failed_when: grepped.rc > 1
First run:                          Second run:
  Naive command         changed       Naive command         changed
  Command with creates  changed       Command with creates  ok
  Read-only command     ok            Read-only command     ok
  Only change when...   ok            Only change when...   ok
creates: skips the command when the file exists. changed_when: false is for commands that only look. changed_when with an expression is for commands whose output tells you. failed_when decides what counts as failure: grep returns 1 when nothing matches, which is not an error here, so only a code above 1 is. The first rule is: use a real module (apt, file, copy, template, service) whenever one exists. They are idempotent by design, they support --check and --diff, and they produce readable errors.

Both apply a role from inside a task list, and they behave differently in ways that matter:
  import_role    STATIC. Read when the playbook is loaded. Its tasks are visible to --list-tasks, and tags on the import apply to every task inside.
  include_role   DYNAMIC. Read when the play reaches it. Its tasks are not listed in advance, and tags apply to the include itself, not to the tasks inside it. It can be used in a loop or with a condition computed at run time, and it has the public: option.
A test with a one-task role, imported once and included once, each with its own tag:
$ ansible-playbook p.yml --list-tasks
      demo : Say hello from the role	TAGS: [imported]
      Include the role	TAGS: [included]
$ ansible-playbook p.yml --tags imported | grep -E "TASK|msg"
TASK [demo : Say hello from the role] ******************************************
    "msg": "hello"
$ ansible-playbook p.yml --tags included | grep -E "TASK|msg"
TASK [Include the role] ********************************************************
The imported role is listed and runs. The included one is only a single line, and with --tags included the include ran but the role's task did NOT print its message: the tag was not inherited by the role's tasks. The include has an apply: option for passing tags down, which was not run in the lab. The plain roles: in site.yml were listed by --list-tasks too (the dbserver tasks above).

Make an empty role with the standard layout:
$ ansible-galaxy role init demo --init-path roles
- Role demo was created successfully
$ find roles -type f | sort
roles/demo/README.md
roles/demo/defaults/main.yml
roles/demo/handlers/main.yml
roles/demo/meta/main.yml
roles/demo/tasks/main.yml
roles/demo/tests/inventory
roles/demo/tests/test.yml
roles/demo/vars/main.yml
Modules beyond the core set come in collections. Pin the ones you need in a file, so everybody installs the same:
$ cat requirements.yml
---
collections:
  - name: community.general
    version: ">=9.0.0"
$ ansible-galaxy collection install -r requirements.yml -p ./collections
Installing 'community.general:13.5.0' to '/home/deploy/proj/collections/ansible_collections/community/general'
community.general:13.5.0 was installed successfully
Installing 'community.library_inventory_filtering_v1:1.1.5' to '/home/deploy/proj/collections/ansible_collections/community/library_inventory_filtering_v1'
community.library_inventory_filtering_v1 was installed successfully
$ ANSIBLE_COLLECTIONS_PATH=./collections ansible-doc -t module community.general.ufw | head -3
> MODULE community.general.ufw (/home/deploy/proj/collections/ansible_collections/community/general/plugins/modules/ufw.py)
  Manage firewall with UFW.
It downloaded the collection (and a collection it depends on) from galaxy.ansible.com, into the project (-p ./collections), so nothing is installed system-wide. Keep requirements.yml in version control. Installing roles from Galaxy (ansible-galaxy role install) was not run in the lab; collections were.

Never put passwords in plain text in group_vars. encrypt_string encrypts ONE value, and you paste the result into the vars file:
$ ansible-vault encrypt_string --vault-password-file /tmp/vp --name dbserver_admin_password "S3cret-Pass-123"
dbserver_admin_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          32353364396134393139643735306632323865336335376131336564313333653333633133613933
          3637303935343362633338313737323739343638346633360a313039656539643336343933623239
          ...
The rest of the file stays readable, so version control diffs still work and only the secret is hidden. A task that uses the variable, writing it to a root-only file:
- name: Store the admin password for the backup script
  ansible.builtin.copy:
    content: "{{ dbserver_admin_password }}\n"
    dest: /etc/postgresql/16/main/admin.pass
    owner: postgres
    group: postgres
    mode: "0600"
  no_log: true
Without the vault password, it stops at once:
$ ansible-playbook vault-demo.yml
TASK [Store the admin password for the backup script] **************************
[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.copy' failed: Error while resolving value for 'content': Error rendering template: Attempt to use undecryptable ...
With it:
$ ansible-playbook vault-demo.yml --vault-password-file /tmp/vp | grep -E "^(changed|ok|fatal)|RECAP|: ok="
ok: [db1]
changed: [db1]
db1                        : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
$ ssh db1 'sudo ls -l /etc/postgresql/16/main/admin.pass'
-rw------- 1 postgres postgres 16 Oct  9 12:32 /etc/postgresql/16/main/admin.pass
Two surprises. First, ansible-vault view only works on a file that is encrypted WHOLE; on a vars file with an inline encrypted value it says:
$ ansible-vault view group_vars/db.yml --vault-password-file /tmp/vp
[ERROR]: Failed to view 'group_vars/db.yml': Input is not vault encrypted data.
(There is nothing wrong with the file: it is a normal YAML file with one encrypted string in it. Use ansible-vault encrypt on a whole file, or decrypt the single value with a debug task.) Second, no_log: true is meant to keep a task's arguments and results out of the output. I also tried the same task WITHOUT no_log and with a deliberately bad destination, hoping to see the secret in the error: it did NOT appear in that failure message. So I cannot show you a leak here; look at your own output, and keep no_log on tasks that handle secrets anyway. The password file (/tmp/vp here) is itself a secret and must stay out of the repository. Multiple vault ids, rekeying and external secret managers were not run in the lab.

ansible-lint reads your playbooks and roles and complains about bad habits. First a sloppy playbook, to see the kind of thing it finds:
$ cat bad.yml
- hosts: all
  tasks:
    - shell: apt-get install -y nginx
    - command: cat /etc/hostname
    - name: copy a file
      copy: src=a dest=/tmp/a mode=777
    - name: start nginx
      service: name=nginx state=started
      ignore_errors: yes
$ ansible-lint bad.yml
Failed: 17 failure(s), 0 warning(s) in 1 files processed of 1 encountered.
name[play]: All plays should be named.
command-instead-of-module: apt-get used in place of apt-get module
command-instead-of-shell: Use shell only when shell functionality is required.
fqcn[action-core]: Use FQCN for builtin module actions (shell).
name[missing]: All tasks should be named.
no-changed-when: Commands should not change things if nothing needs doing.
name[casing]: All names should start with an uppercase letter.
no-free-form: Avoid using free-form when calling module actions. (copy)
ignore-errors: Use failed_when and specify error conditions instead of using ignore_errors.
... (17 findings in total)
Each finding has a rule name. Now the real project, which I thought was fine:
$ ansible-lint site.yml rolling.yml
Failed: 5 failure(s), 0 warning(s) in 11 files processed of 11 encountered. Last profile that met the validation criteria was 'min'.
var-naming[no-role-prefix]: Variables names from within roles should use dbserver_ as a prefix. (vars: db_max_connections)
roles/dbserver/defaults/main.yml:1:1
var-naming[no-role-prefix]: Variables names from within roles should use webserver_ as a prefix. (vars: web_port)
roles/webserver/defaults/main.yml:1:1
(5 findings, all of this one kind: my variables were called web_port, web_sites, web_worker_connections, db_max_connections and db_listen_addresses.)
The linter was right; that is why this page uses the long names throughout (the experiments above that show web_... in the project were run before this rename, and the page shows them under the final names). I renamed the five variables everywhere (nine files) with sed, then CONVERGED first and checked afterwards, so that I knew the rename changed nothing on the machines:
$ ansible-playbook site.yml --check --diff | grep -E "RECAP|: ok="
db1                        : ok=7    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web1                       : ok=11   changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
web2                       : ok=11   changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
$ ansible-lint site.yml rolling.yml
Passed: 0 failure(s), 0 warning(s) in 11 files processed of 11 encountered. Last profile that met the validation criteria was 'production'.
(skipped=1 in check mode is the uri check: "This action (ansible.builtin.uri) does not support check mode.") The linter reports a "profile" (min, basic and production appeared in this run's output); the project went from min to production. Put it in a CI job and in a pre-commit hook. Testing roles with Molecule (the usual next step) was not run in the lab.

  "python3-apt must be installed to use check mode"   run the apt module once for real (or install the package), then --check works
  "failed to validate" + [emerg] from nginx    the validate: command rejected the new file. The real file is unchanged. Read the stderr, fix the template:
      2026/10/09 12:25:31 [emerg] 3939#3939: unknown directive "worker_connectionz" in /home/deploy/.ansible/tmp/ansible-tmp-.../.source.conf:9
      nginx: configuration file /home/deploy/.ansible/tmp/ansible-tmp-.../.source.conf test failed
    (I put a typo in the template to get this; afterwards the machine still had its old config and nginx still ran with its 9 processes.)
  "'X' is undefined"   a variable that does not exist. In an ad-hoc command: $ ansible web1 -m ansible.builtin.debug -a "msg={{ nosuch_var }}"
      [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.debug' failed: Error while resolving value for 'msg': 'nosuch_var' is undefined
    It is an ERROR on purpose. Typos in variable names are caught here, instead of producing an empty string in a config file.
  "'webserver_port' is undefined" in a task outside the role   include_role keeps the role's variables private: add public: true
  "Input is not vault encrypted data"   you used ansible-vault view on a file that only contains an inline !vault value
  "Attempt to use undecryptable ..."   the vault password was not given (--vault-password-file or --ask-vault-pass)
  "Connection refused" in a uri check    the service is not listening yet. Is it the handler that did not run? See the handlers section

  - Real machines, systemd (the containers use init scripts: the service module is the same, the init system under it is not), real network latency and many hosts (forks, pipelining, ssh multiplexing)
  - Windows hosts, network devices, cloud and dynamic inventories, ansible-pull
  - Molecule (testing roles), execution environments (ansible-navigator), AWX / the automation controller
  - Galaxy roles (ansible-galaxy role install), role dependencies in meta/main.yml (the file exists but lists none), role_path and file lookups from other roles
  - include_role with apply: tags, force_handlers in the play, max_fail_percentage, any_errors_fatal, the free strategy, async and poll
  - Vault ids, rekeying, vault-encrypted whole files, external secret stores
  - Anything about nginx or PostgreSQL itself beyond what the roles above do (tuning, TLS, replication, backups)
  - Ansible versions other than ansible-core 2.21.5. Module behaviour and messages change between versions: check yours

Preview a change on one host, with the diff:
  ansible-playbook site.yml --check --diff --limit web1
Run only one role's tasks, on the web group:
  ansible-playbook site.yml --tags web
What would run, and where:
  ansible-playbook site.yml --list-tasks --list-hosts
Override a variable for a single run:
  ansible-playbook site.yml -e webserver_port=8080
See the variables a host really gets (but not role defaults):
  ansible-inventory --host web2 --yaml
Check the syntax and then lint, before a commit:
  ansible-playbook site.yml --syntax-check && ansible-lint
Install pinned collections into the project:
  ansible-galaxy collection install -r requirements.yml -p ./collections
Roll a change over a group, one machine at a time:
  ansible-playbook rolling.yml
Run a failed play's handlers anyway:
  ansible-playbook site.yml --force-handlers
Start a new role with the standard layout:
  ansible-galaxy role init NAME --init-path roles
Encrypt one value for a vars file:
  ansible-vault encrypt_string --vault-password-file FILE --name NAME 'value'
The rule of thumb for every change: --check --diff, --limit to one machine, apply, run it AGAIN and expect changed=0, then the rest.

Symptom                                         Usually
Second run still says changed                    a command or shell task without creates / changed_when, or a template with something that changes every run (a timestamp, {{ ansible_date_time }})
Config file is right but the service runs the old one   the handler did not run: a task after the change failed. flush_handlers before the check, or --force-handlers
"X is undefined" in a task outside the role      role variables are private with include_role: public: true. With roles: they are visible
Ad-hoc debug says a role variable is undefined   role defaults only exist while the role runs. Test inside a play
--check fails on a new Ubuntu machine            python3-apt is missing: run the apt module once for real
The host variable is ignored                     a variable with higher priority wins (-e, or a task-level vars). Check with -e off, and with ansible-inventory --host
A tag on include_role runs nothing inside        the tag is not inherited by the included tasks: import_role, or apply: with tags
A rolling update touched every host              serial: was missing, or on the wrong play. Use --limit for the first test
Variables clash between two roles                generic names (port, user). Prefix every role variable with the role name; ansible-lint var-naming says so

Give every play and every task a name. The names are your log, and ansible-lint will ask for them anyway.
Use fully qualified module names (ansible.builtin.copy). It is longer, but never ambiguous.
Prefer a real module to command or shell. If you must run a command, give it creates, or changed_when, or both.
Put defaults in defaults/main.yml, and keep vars/main.yml for values that must never be overridden. Prefix every variable with the role name.
Put the values that differ per environment or per host in group_vars and host_vars, and keep the roles free of any site-specific value.
One role, one job. A role that installs nginx and also creates user accounts and also tunes the kernel is three roles.
validate: on every config file that has a checker (nginx -t, visudo -cf, named-checkconf, sshd -t). It is the cheapest safety net there is.
After a change that matters, always flush the handlers before you test it, or the test tests the old configuration.
Run a second time and look for changed=0. Run with --check --diff before the real thing, and with serial: and a health check when the machines serve users.
Never put a secret in plain text in a repository, and do not forget that the vault password is also a secret.
Lint in CI, test with Molecule if you can, and read the diff.