Posts

Ansible Enterprise Guide: Automation, Playbooks, Roles and Production Architecture

Automation

Ansible Enterprise Guide: Automation, Playbooks, Roles and Production Architecture

A production-focused Ansible reference with architecture, inventories, playbooks, roles, variables, templates, Vault, AWX/AAP, CI/CD, security, troubleshooting and real code examples.

🧱 → ⚙️ → 🚀

Architecture → Automation → Production

Enterprise Automation

Ansible Overview

Ansible is an automation tool used for configuration management, application deployment, orchestration, provisioning, patching, security hardening, and repeatable operations. It is popular because it is agentless for Linux over SSH, readable because it uses YAML, and practical because the same playbook can be stored in Git, reviewed, tested, and executed again.

The most important idea in Ansible is desired state. You describe what the system should look like. Ansible checks the remote system and changes only what is needed. Good playbooks are idempotent, meaning they can run many times without creating unnecessary changes.

Control nodeWhere Ansible runs
InventoryList of managed hosts
PlaybookYAML automation workflow
ModuleReusable task unit
Git → Review → Ansible Controller → Inventory → Managed Nodes → Logs

Core Design

Ansible Architecture

A normal Ansible environment has three main parts: the control node, the inventory, and managed nodes. The control node runs Ansible commands. The inventory defines target hosts and groups. Managed nodes are the servers, network devices, or systems being automated.

Ansible normally connects to Linux and Unix hosts over SSH. For Windows hosts, it commonly uses WinRM. In enterprise environments, Ansible may be executed from AWX, Red Hat Ansible Automation Platform, a CI/CD runner, or a locked automation server.

Developer laptop / AWX / AAP
        │
        ├── Git repository
        ├── ansible.cfg
        ├── inventories/
        ├── playbooks/
        └── roles/
             │
             └── SSH / WinRM
                    │
                    ├── web01
                    ├── app01
                    └── db01

Production automation should not depend on manual commands from random laptops. Use a controlled runner, Git-based change process, RBAC, secrets management, logs, and repeatable execution environments.

Setup

Installation and Basic Configuration

On a Linux control node, install Ansible from your approved package source. In enterprises, avoid installing random packages directly from the internet on production automation servers. Use approved repositories, mirrors, or execution environments.

# RHEL / CentOS / Fedora style
sudo dnf install ansible-core

# Debian / Ubuntu style
sudo apt update
sudo apt install ansible

# Python virtual environment style
python3 -m venv .venv
source .venv/bin/activate
pip install ansible-core ansible-lint

A simple ansible.cfg keeps behavior predictable.

[defaults]
inventory = inventories/dev/hosts.yml
roles_path = roles
collections_path = collections
host_key_checking = True
retry_files_enabled = False
stdout_callback = yaml
forks = 20
interpreter_python = auto_silent

[privilege_escalation]
become = True
become_method = sudo
become_ask_pass = False

Keep ansible.cfg in the repository root so every user and pipeline gets the same behavior.

Repository Design

Recommended Project Layout

A clean folder structure is critical. Without structure, Ansible becomes hard to review, test, and operate. Separate inventories by environment. Put reusable logic in roles. Keep secrets encrypted. Keep templates version-controlled.

ansible-platform/
├── ansible.cfg
├── requirements.yml
├── inventories/
│   ├── dev/
│   │   ├── hosts.yml
│   │   ├── group_vars/
│   │   └── host_vars/
│   ├── test/
│   └── prod/
├── playbooks/
│   ├── site.yml
│   ├── patching.yml
│   ├── deploy_app.yml
│   └── hardening.yml
├── roles/
│   ├── common/
│   ├── nginx/
│   ├── users/
│   └── monitoring/
├── templates/
├── files/
└── README.md

This structure supports Git review, CI tests, environment separation, and safer production releases.

Targets

Inventory

The inventory tells Ansible which systems exist and how they are grouped. Groups should reflect business or technical purpose: webservers, databases, kafka, elasticsearch, linux, windows, production, test, and so on.

all:
  children:
    prod:
      children:
        webservers:
          hosts:
            web01.example.com:
            web02.example.com:
        databases:
          hosts:
            db01.example.com:
    test:
      hosts:
        test01.example.com:

Inventory can also define connection settings.

all:
  vars:
    ansible_user: automation
    ansible_port: 22
    ansible_ssh_private_key_file: ~/.ssh/ansible_id_rsa

linux:
  hosts:
    app01.example.com:
      ansible_host: 10.10.10.21

For large environments, use dynamic inventory from cloud, CMDB, VMware, NetBox, or another source of truth. Static files are fine for small and controlled environments, but they become risky when they drift from reality.

Configuration Data

Variables and Precedence

Variables make playbooks reusable. Instead of hardcoding ports, package names, paths, or users, define variables in group_vars, host_vars, role defaults, role vars, or extra vars.

# inventories/prod/group_vars/webservers.yml
nginx_port: 443
nginx_worker_processes: auto
app_environment: production
allowed_admin_groups:
  - linux-admins
  - platform-ops

Host-specific variables belong in host_vars.

# inventories/prod/host_vars/web01.example.com.yml
backup_enabled: true
maintenance_window: sunday_02_00

Be careful with variable precedence. Extra vars passed with -e are very strong and can override many other values. In production, uncontrolled extra vars can create surprising changes.

ansible-playbook playbooks/deploy_app.yml -e "app_version=1.4.2"

Workflow

Playbooks

A playbook is a YAML file containing one or more plays. Each play targets hosts and runs tasks. Each task calls a module. Playbooks should be readable, small enough to review, and safe to run repeatedly.

---
- name: Configure web servers
  hosts: webservers
  become: true
  gather_facts: true

  tasks:
    - name: Install nginx
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Deploy nginx config
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        group: root
        mode: '0644'
      notify: Restart nginx

    - name: Ensure nginx is running
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

  handlers:
    - name: Restart nginx
      ansible.builtin.service:
        name: nginx
        state: restarted

Use check mode before production runs where possible.

ansible-playbook playbooks/site.yml --check --diff
ansible-playbook playbooks/site.yml --syntax-check
ansible-playbook playbooks/site.yml --list-hosts
ansible-playbook playbooks/site.yml --list-tasks

Task Units

Modules

Modules do the actual work. Use modules instead of shell commands whenever possible. Modules understand state and are usually safer and more idempotent.

packageInstall packages
serviceManage services
copyCopy files
templateRender Jinja2
userManage users
filePaths and permissions
- name: Create application user
  ansible.builtin.user:
    name: appuser
    shell: /sbin/nologin
    system: true

- name: Create application directory
  ansible.builtin.file:
    path: /opt/myapp
    state: directory
    owner: appuser
    group: appuser
    mode: '0750'

Use fully qualified collection names like ansible.builtin.package. This avoids ambiguity when different collections have modules with similar names.

Controlled Restarts

Handlers

Handlers run only when notified by a changed task. They are useful for restarting services only when configuration changes. This avoids unnecessary restarts and reduces production noise.

tasks:
  - name: Update sshd configuration
    ansible.builtin.template:
      src: sshd_config.j2
      dest: /etc/ssh/sshd_config
      validate: /usr/sbin/sshd -t -f %s
    notify: Restart sshd

handlers:
  - name: Restart sshd
    ansible.builtin.service:
      name: sshd
      state: restarted

The validate option is powerful. It checks the generated file before replacing the real configuration. Use it for SSH, Nginx, Apache, sudoers, and other critical files.

Jinja2

Templates

Templates generate configuration files from variables. They are useful when every environment needs similar files with different values.

# templates/app.conf.j2
[server]
port = {{ app_port }}
environment = {{ app_environment }}

[logging]
level = {{ log_level | default('INFO') }}

{% for backend in backend_servers %}
backend = {{ backend }}
{% endfor %}
- name: Deploy application configuration
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/myapp/app.conf
    owner: root
    group: myapp
    mode: '0640'
  notify: Restart myapp

Do not put secrets directly in templates unless the variable source is encrypted and access-controlled.

Reusable Automation

Roles

Roles are reusable automation units. They organize tasks, handlers, templates, files, variables, defaults, and metadata into a standard structure.

roles/nginx/
├── defaults/main.yml
├── vars/main.yml
├── tasks/main.yml
├── handlers/main.yml
├── templates/nginx.conf.j2
├── files/
└── meta/main.yml

Example role task:

# roles/nginx/tasks/main.yml
---
- name: Install nginx
  ansible.builtin.package:
    name: nginx
    state: present

- name: Configure nginx
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    mode: '0644'
  notify: Restart nginx

- name: Start nginx
  ansible.builtin.service:
    name: nginx
    state: started
    enabled: true

Use defaults for values that users may override. Use vars only for values that should be difficult to override.

Secrets

Ansible Vault

Ansible Vault encrypts sensitive files and variables. Use it for passwords, API tokens, private values, database credentials, and service secrets. Do not store cleartext secrets in Git.

ansible-vault create inventories/prod/group_vars/all/vault.yml
ansible-vault edit inventories/prod/group_vars/all/vault.yml
ansible-vault encrypt secrets.yml
ansible-vault decrypt secrets.yml
ansible-vault view secrets.yml

Example encrypted variable pattern:

# group_vars/all/vars.yml
db_user: appuser
db_password: "{{ vault_db_password }}"

# group_vars/all/vault.yml
vault_db_password: "super-secret-password"

Run with a vault password file only when the file itself is protected by your enterprise secrets process.

ansible-playbook playbooks/site.yml --ask-vault-pass
ansible-playbook playbooks/site.yml --vault-password-file ~/.vault_pass

In production, integrate Vault usage with CyberArk, HashiCorp Vault, AAP credentials, or another approved secrets manager where possible.

System Data

Facts

Facts are system details gathered from managed nodes. They include OS family, distribution, IP addresses, memory, CPU, mount points, and more.

ansible webservers -m ansible.builtin.setup
ansible webservers -m ansible.builtin.setup -a 'filter=ansible_distribution*'

Example usage:

- name: Install package on Red Hat family
  ansible.builtin.package:
    name: httpd
    state: present
  when: ansible_facts['os_family'] == 'RedHat'

Fact gathering is useful but adds runtime. For simple tasks that do not need facts, set gather_facts: false.

Logic

Conditionals

Conditionals decide whether a task should run. They are useful for OS-specific logic, environment-specific behavior, and optional features.

- name: Install Apache on Red Hat
  ansible.builtin.package:
    name: httpd
    state: present
  when: ansible_facts['os_family'] == 'RedHat'

- name: Install Apache on Debian
  ansible.builtin.package:
    name: apache2
    state: present
  when: ansible_facts['os_family'] == 'Debian'

Keep conditionals readable. If logic becomes complex, split it into roles or separate task files.

Repeat Tasks

Loops

Loops repeat a task for multiple items. They are useful for packages, users, directories, firewall rules, and configuration entries.

- name: Install common packages
  ansible.builtin.package:
    name: "{{ item }}"
    state: present
  loop:
    - vim
    - curl
    - git
    - unzip
- name: Create Linux admin users
  ansible.builtin.user:
    name: "{{ item.name }}"
    groups: "{{ item.groups }}"
    shell: /bin/bash
    state: present
  loop:
    - { name: 'alice', groups: 'wheel' }
    - { name: 'bob', groups: 'wheel' }

Selective Runs

Tags

Tags let you run part of a playbook. This is useful in large repositories, but overusing tags can make behavior confusing.

- name: Configure monitoring agent
  hosts: linux
  become: true
  roles:
    - role: monitoring
      tags: monitoring
    - role: hardening
      tags: security
ansible-playbook playbooks/site.yml --tags monitoring
ansible-playbook playbooks/site.yml --skip-tags security

Use tags for clear operational boundaries, not as a replacement for good playbook design.

Safe Failure

Error Handling

Production playbooks need safe failure behavior. Use blocks, rescue, always, failed_when, changed_when, serial, and max_fail_percentage carefully.

- name: Safe application deployment
  hosts: appservers
  become: true
  serial: 2

  tasks:
    - block:
        - name: Stop application
          ansible.builtin.service:
            name: myapp
            state: stopped

        - name: Deploy package
          ansible.builtin.copy:
            src: files/myapp.rpm
            dest: /tmp/myapp.rpm

        - name: Install package
          ansible.builtin.dnf:
            name: /tmp/myapp.rpm
            state: present

      rescue:
        - name: Start application after failed deploy
          ansible.builtin.service:
            name: myapp
            state: started

      always:
        - name: Remove temporary package
          ansible.builtin.file:
            path: /tmp/myapp.rpm
            state: absent

For production, use serial rolling deployments. Do not update every critical server at once unless the system is designed for it.

Source of Truth

Dynamic Inventory

Dynamic inventory reads hosts from an external system such as AWS, Azure, GCP, VMware, NetBox, ServiceNow, or an internal CMDB. This reduces manual inventory drift.

ansible-inventory -i inventories/prod/hosts.yml --list
ansible-inventory -i inventory.aws_ec2.yml --graph

Use dynamic inventory when infrastructure changes often. Use static inventory when the environment is small, stable, and controlled.

Reusable Content

Collections

Collections package modules, plugins, roles, and documentation. They allow teams to use vendor-supported automation content.

# requirements.yml
collections:
  - name: ansible.posix
  - name: community.general
  - name: community.crypto
  - name: kubernetes.core
ansible-galaxy collection install -r requirements.yml

Pin versions in enterprise environments so playbooks do not change unexpectedly.

collections:
  - name: community.general
    version: '>=9.0.0,<10.0.0'

Enterprise Platform

AWX and Red Hat Ansible Automation Platform

AWX is the upstream project. Red Hat Ansible Automation Platform is the supported enterprise product. They provide web-based job execution, RBAC, inventories, credentials, schedules, workflow templates, notifications, surveys, and audit logs.

ProjectsGit repositories
InventoriesHosts and groups
CredentialsSSH, Vault, cloud secrets
TemplatesReusable job definitions

A normal enterprise workflow looks like this:

Developer opens PR
        ↓
Code review and lint
        ↓
Merge to main
        ↓
AWX/AAP syncs project
        ↓
Operator launches approved job template
        ↓
Logs and result are stored

Use AWX/AAP when you need controlled access, auditability, scheduled jobs, workflows, and team execution. Use command line Ansible for development, testing, and small controlled operations.

GitOps

CI/CD Integration

Ansible works well with Git-based pipelines. A safe pipeline checks syntax, runs lint, validates YAML, runs molecule tests where possible, and only then allows deployment.

# Example GitHub Actions structure
name: ansible-checks
on: [pull_request]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install tools
        run: pip install ansible-core ansible-lint yamllint
      - name: Syntax check
        run: ansible-playbook playbooks/site.yml --syntax-check
      - name: Ansible lint
        run: ansible-lint
      - name: YAML lint
        run: yamllint .

For production, the CI/CD system should not expose secrets in logs. Use protected variables, approved credentials, and environment approvals.

Hardening

Security Best Practices

  • Use dedicated automation accounts.
  • Use SSH keys or approved credential systems.
  • Do not store cleartext passwords in Git.
  • Encrypt sensitive variables with Vault or external secret managers.
  • Use least privilege for sudo.
  • Separate dev, test, and production inventories.
  • Review all production changes through pull requests.
  • Use check mode and diff mode before risky changes.
  • Log production automation runs.
  • Pin collection versions.
  • Validate critical config files before replacing them.
- name: Deploy sudoers file safely
  ansible.builtin.template:
    src: sudoers_app.j2
    dest: /etc/sudoers.d/app
    owner: root
    group: root
    mode: '0440'
    validate: /usr/sbin/visudo -cf %s

Never give broad root automation access without audit, approval, and change control.

Speed

Performance Tuning

Ansible performance depends on SSH connection setup, facts, forks, task count, module behavior, and target system speed. Tune carefully instead of blindly increasing forks.

[defaults]
forks = 30
gathering = smart
fact_caching = jsonfile
fact_caching_connection = .ansible_facts
fact_caching_timeout = 86400

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s

Performance tips:

  • Disable gather_facts when not needed.
  • Use package modules with lists instead of loops where possible.
  • Use serial for safe rolling production changes.
  • Use async for long-running independent jobs.
  • Cache facts for large inventories.
  • Avoid unnecessary shell commands.
- name: Install packages efficiently
  ansible.builtin.package:
    name:
      - nginx
      - curl
      - unzip
    state: present

Operations

Troubleshooting

Most Ansible failures are caused by SSH access, sudo permissions, wrong inventory variables, missing Python, YAML syntax, wrong module arguments, package repository problems, or secrets access.

# Test connectivity
ansible all -m ping -i inventories/prod/hosts.yml

# Show inventory as Ansible sees it
ansible-inventory -i inventories/prod/hosts.yml --graph
ansible-inventory -i inventories/prod/hosts.yml --host web01.example.com

# Run with more detail
ansible-playbook playbooks/site.yml -vvv

# Check without changing systems
ansible-playbook playbooks/site.yml --check --diff

# Limit to one host
ansible-playbook playbooks/site.yml --limit web01.example.com

Common errors and fixes:

UNREACHABLECheck SSH, DNS, firewall, user, key
Permission deniedCheck sudo and become settings
YAML errorCheck indentation and quotes
Undefined variableCheck group_vars and host_vars

For production debugging, avoid changing many hosts at once. Use --limit and one test host first.

Cheat Sheet

Useful Commands

# Version and config
ansible --version
ansible-config dump --only-changed

# Inventory
ansible-inventory -i inventories/prod/hosts.yml --list
ansible-inventory -i inventories/prod/hosts.yml --graph

# Ad-hoc commands
ansible all -m ping
ansible webservers -m ansible.builtin.command -a 'uptime'
ansible linux -m ansible.builtin.setup

# Playbooks
ansible-playbook playbooks/site.yml
ansible-playbook playbooks/site.yml --check
ansible-playbook playbooks/site.yml --check --diff
ansible-playbook playbooks/site.yml --syntax-check
ansible-playbook playbooks/site.yml --list-hosts
ansible-playbook playbooks/site.yml --list-tasks
ansible-playbook playbooks/site.yml --limit web01.example.com
ansible-playbook playbooks/site.yml --tags security
ansible-playbook playbooks/site.yml --skip-tags monitoring
ansible-playbook playbooks/site.yml -e 'app_version=1.4.2'

# Vault
ansible-vault create vault.yml
ansible-vault edit vault.yml
ansible-vault view vault.yml
ansible-vault encrypt vars.yml
ansible-vault decrypt vars.yml
ansible-vault rekey vault.yml

# Galaxy and collections
ansible-galaxy collection install -r requirements.yml
ansible-galaxy role init roles/myrole
ansible-galaxy collection list

# Validation
ansible-lint
yamllint .

Summary

Key Takeaways

Ansible is strongest when it is treated as production code. Store it in Git. Review changes. Use inventories correctly. Use roles for reuse. Use Vault or a secret manager for sensitive data. Test with syntax-check, lint, check mode, and limited hosts before production. Use AWX or Ansible Automation Platform when you need RBAC, audit logs, scheduled jobs, workflows, and controlled enterprise execution.

Write clearly → Test safely → Review in Git → Run through controller → Audit results