Post cover image

October 14, 2026

Four QA Servers, One AWX Workflow, and the Systemd Dependencies I Almost Missed

Our QA environment had four application servers. Each one was quietly doing very little, and each one cost money every hour of every day…

By Gabriel Okiri

4 min read

Our QA environment had four application servers. Each one was quietly doing very little, and each one cost money every hour of every day. I'd looked at them for months without doing anything about it. Eventually I decided to find out what they were really doing and whether they could share one machine.

I didn't want to run ad-hoc commands from a laptop and hope I remembered what I'd done. We already manage our infrastructure through AWX, so I wanted the whole job to run there: every server checked the same way, every change logged, and a way back if something went wrong.

Step 1: Build the Inventory From AWS

Rather than maintaining a static list of hosts, I pointed AWX at our AWS account using the amazon.aws.aws_ec2 dynamic inventory plugin, filtered on the QA environment tag. That gave me one inventory group with the four servers in it, and it picks up any new QA instance automatically.

Step 2: Audit Before Touching Anything

The first job template I ran was the audit. It gathered facts from every host, then checked what each one was actually running. Some services were in Docker, some were Node apps managed by PM2, and some were plain systemd units. The process list doesn't tell you how something is run, so the playbook asked each host directly:

yaml

- name: Audit QA application servers
  hosts: qa_servers
  gather_facts: true
  tasks:
    - name: Record CPU, memory, and OS
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }}: {{ ansible_processor_vcpus }} vCPU, {{ ansible_memtotal_mb }} MB RAM, {{ ansible_distribution }} {{ ansible_distribution_version }}"
    - name: List running systemd services
      ansible.builtin.shell: systemctl list-units --type=service --state=running --no-pager | grep -v systemd
      changed_when: false
    - name: List Docker containers
      ansible.builtin.shell: "docker ps --format '{% raw %}{{.Names}} {{.Image}} {{.Status}}{% endraw %}' || true"
      changed_when: false
    - name: List PM2 processes
      ansible.builtin.shell: pm2 list || true
      changed_when: false
    - name: Check real utilisation
      ansible.builtin.shell: uptime && free -m && df -h /
      changed_when: false

The || true on the Docker and PM2 checks was deliberate. Not every server ran both, and I didn't want a missing tool to fail the audit. AWX stored every host's output side by side, which made the comparison easy.

I also pulled the instance types and attached volumes from AWS so I could compare what we were paying for against what was actually used.

Step 3: Map What Each App Actually Needs

This is the part I'd recommend to anyone. For each server, I wrote down:

  • How the app runs: Docker container, PM2 process, or systemd unit
  • Its resource footprint: CPU and memory under real load
  • Its files and dependencies: config directories, data folders, cron jobs, and systemd unit relationships
  • Its ports: what it listens on and what talks to it

The mapping showed that two services wanted the same default port. The bigger find was in systemd. One service's unit file declared dependencies on other units that only existed on its original host. I checked them with:

bash

systemctl show qa-service -p Requires,Wants,After,BindsTo
systemctl list-dependencies qa-service

Those commands showed that the unit wouldn't start cleanly anywhere else. Systemd would either refuse to start it or start it in the wrong order. Nothing had failed yet, because each service was still running on its own box, but it would have broken on the new machine.

Step 4: Start With the Smallest Server

I didn't start with the biggest or most important server. I started with the smallest, the one with the least memory and the simplest setup. If something broke, I wanted the blast radius to be small and the lesson to be cheap.

Step 5: Snapshot Before Any Change

Before I changed a single file, I ran a job template that took a snapshot of that server's root volume, tagged for rollback and set to wait for completion:

yaml

- name: Snapshot QA app server root volume before consolidation
  hosts: localhost
  tasks:
    - name: Create tagged snapshot
      amazon.aws.ec2_snapshot:
        region: eu-west-1
        volume_id: vol-0abc123def456789
        description: "Pre-consolidation snapshot: qa-app-small {{ ansible_date_time.date | default('today') }}"
        snapshot_tags:
          Purpose: consolidation-rollback
        wait: true

The wait: true matters. A snapshot that's still pending isn't a rollback plan yet. Once it completed, I knew I could return that server to exactly how it was. Running it as a job template also meant the snapshot ID was recorded in the job output, not in my shell history.

Step 6: Move, Then Verify

With the snapshot in place, I moved the first service over using a playbook, so the steps were identical every time. It copied the config and data directories, recreated the systemd units or Docker Compose files, and started the service on the new host.

The systemd dependencies caused the first failure in the test run. The unit referenced a sibling service that didn't exist on the new host, so I had to remove the dependency, fix the start order, and make the remaining services explicit about what they needed. I made those changes in the playbook and the unit templates, not by hand on the server, so they would survive the next run. That's the real advantage of AWX here: every change goes through version control and every run is logged.

After each move, a verification job checked the service directly, and I repeated the process for the remaining three services before retiring anything.

The Result

The saving came in at 5% of our total AWS cost. It isn't a headline number, but it's real, and the old instances no longer need patching or monitoring. The bigger win, for me, was that the environment is now described in code and run from AWX. The next time someone asks what runs in QA, the answer is in the inventory, the playbooks, and the job history, not in someone's memory.

What I'd Tell Anyone Doing This

Use one controller for everything. Running the audit, the snapshot, and the migration through AWX means every step has a record.

Build inventory from the cloud, not from memory. A dynamic inventory keeps the host list honest.

Check how apps are run, not just what is running. Docker, PM2, and systemd each hide different things, and systemd dependencies are easy to overlook because they only break on a new host.

Take the snapshot, and wait for it to finish. A rollback you haven't verified isn't a rollback.

Fix things in the playbook. Hand fixes on a server are the ones you forget about.

Start small. The smallest server is the cheapest place to learn what your process is missing.