Podman

I have been using Docker since 2013, and I’ve always felt containers are a great packaging mechanism for services and their dependencies. While containers have been a constant, the tooling I use to orchestrate them has continued to evolve. I’ve used everything from Jenkins + Docker CLI, Docker Swarm, Docker Compose, Podmand + Podman Compose, Terraform, etc.

Nothing ever felt like a perfect fit, but the combination of systemd and compose.yaml files came the closest. While this approach was much better than running docker run ... commands with a boatload of arguments, it was always an awkward integration for typical “service” containers that always run in the background. Quadlets feel like a much more natural replacement that integrates more closely with standard systemd style services.

The Old Compose Approach

Using compose.yml files has actually worked reasonably well for almost a decade. The trick for me has been to standardize on a simple systemd service unit file which runs compose commands to start, stop, and update my services.

A standard Ansible template was used to generate a systemd service unit file for each service.

[Unit]
Description={{ service }} service
Requires=network-online.target
After=network-online.target
StartLimitIntervalSec=0                         # don't limit number of restarts in a specific window

[Service]
WorkingDirectory=/opt/{{ service }}
Type=simple
TimeoutStartSec=5min                            # need a pretty long startup timeout for image pulls
Restart=always
RestartSec=10s                                  # restart after a 10s delay

ExecStartPre=sh -c 'until ping -c 1 google.com; do sleep 1; done'
ExecStartPre=docker-compose pull
ExecStart=docker-compose up
ExecStop=docker-compose down
ExecReload=docker-compose pull

[Install]
WantedBy=multi-user.target

A few notes:

  • each service has a directory with a compose.yml file and any necessary dependencies in /opt/<service>
  • sometimes after a power outage, containers would not start correctly
    • this appeared to be caused by the containers starting before the whole network (including network firewall/switches) were up a running
    • the until ping ... ensured the pull doesn’t happen before the network is up and running
  • the compose file can contain multiple dependent containers for a service
  • this approach also continued to work well after migrating from Docker to Podman a few years ago - just replace docker-compose with podman-compose and make a few tweaks to the compose.yaml files

Managing Container Image Updates

Since I run Ansible to configure and update my infrastructure, I wanted an approach that updated containers as part of an ansible-playbook run. Initially I used Watchtower but I found that it would update on a scheduled cadence and I often wasn’t around to fix issues. I settled on a script that could be run via Ansible, and compare the most recent image hashes for containers in my compose.yml files with the currently running containers and trigger an Ansible handler when they were out of date.

This script was installed as check-compose-updates.py.

#!/usr/bin/env python3
#
# Script to check if a running container has an updated image. We look in the compose.yml file to find a list
# of containers to check. Then we pull the lastest images and compare the digest of each image to the running container's
# image digest.

import subprocess
import sys
import os
import yaml

def docker_run(command, capture_output=True, check=True, input_data=None):
    """
    Executes a docker command and returns the output.

    Args:
        command (list): A list of strings representing the docker command and its arguments.
        capture_output (bool, optional): Whether to capture the standard output and standard error. Defaults to True.
        check (bool, optional): Whether to raise a CalledProcessError if the command returns a non-zero exit code. Defaults to True.
        input_data (str, bytes, optional): data to be passed to the stdin of the subprocess.

    Returns:
        subprocess.CompletedProcess: A CompletedProcess instance containing information about the executed command.

    Raises:
        subprocess.CalledProcessError: If the command returns a non-zero exit code and check is True.
    """
    try:
        result = subprocess.run(
            ["docker"] + command,
            capture_output=capture_output,
            text=True,
            check=check,
            input=input_data,
            )
        return result
    except subprocess.CalledProcessError as e:
        print(f"Error executing docker command: {e}")
        raise

def get_container_image_digest(container_name):
    """Retrieves the image hash for a running docker container."""
    try:
        result = docker_run(["inspect", "--format", "{{.ImageDigest}}", container_name])
        if result.stdout:
            return result.stdout.strip()
        else:
            return []
    except subprocess.CalledProcessError as e:
        print(f"Error during docker inspect: {e}")
        return None

def get_image_digest(image_name):
    """Retrieves the hash for a docker image."""
    try:
        result = docker_run(["inspect", "--format", "{{.Digest}}", image_name])
        if result.stdout:
            return result.stdout.strip()
        else:
            return []
    except subprocess.CalledProcessError as e:
        print(f"Error during docker inspect: {e}")
        return None

def pull_images(compose_file):
    """Pulls images using docker-compose."""
    try:
        subprocess.run(
            ["docker-compose", "-f", compose_file, "pull"],
            check=True
        )
        return True
    except subprocess.CalledProcessError:
        return False

def get_images(compose_file):
    """Gets a dictionary of service names and image names from the compose file."""
    try:
        with open(compose_file, 'r') as f:
            compose_data = yaml.safe_load(f)
            services = {}
            for service_name, service_data in compose_data['services'].items():
                if 'container_name' in service_data:
                    if 'image' in service_data:
                        services[service_data['container_name']] = service_data['image']
            return services
    except FileNotFoundError:
        print(f"Error: Compose file '{compose_file}' not found.")
        return None
    except KeyError:
        print(f"Error: 'services' or 'image' key not found in compose file.")
        return None
    except yaml.YAMLError as e:
        print(f"Error: Invalid YAML in compose file: {e}")
        return None

def main():
    if len(sys.argv) != 2:
        print("Usage: python script.py <compose_file>")
        return

    compose_file = sys.argv[1]
    images = get_images(compose_file)
    if images is None:
        print(f"No images found in {compose_file} - be sure your services have a container_name and image defined.")
        return

    if pull_images(compose_file):
        # Fetch the current image digests
        image_digests = {}
        for container_name, image_name in images.items():
            image_digests[container_name] = get_image_digest(image_name)

        # Get the running container's digest so we can compare to the current image
        running_digests = {}
        for container_name, image_name in images.items():
            running_digests[container_name] = get_container_image_digest(container_name)

        image_updated = False
        for container_name, image_name in images.items():
            running_digest = running_digests[container_name]
            image_digest = image_digests[container_name]

            if image_digest and running_digest and image_digest != running_digest:
                print(f"Service '{container_name}' image '{image_name}' was updated. Digest changed from '{running_digest}' to '{image_digest}'.")
                image_updated = True
            elif running_digest and image_digest and running_digest == image_digest:
                print(f"Service '{container_name}' image '{image_name}' was not updated. Digest remained the same ('{running_digest}').")
            elif not image_digest:
                print(f"Service '{container_name}' image '{image_name}' not found after pull. An unexpected error occurred.")
            else:
                print(f"Service '{container_name}' image '{image_name}' error determining update status.")
        print(f"Image Updated: {image_updated}")
    else:
        print("Error pulling images.")

if __name__ == "__main__":
    main()

Then my Ansible playbooks would call the script with the right compose file and trigger the relevant handler.

- name: restart service-foo when new image exists
  become: yes
  command: "check-compose-updates /opt/service-foo/compose.yml"
  register: check_updates
  changed_when: "'Image Updated: True' in check_updates.stdout"
  notify: restart service-foo

As you can see, this approach, while functional, required a fair amount of custom scripting and manual configuration for each service. It was a classic case of gluing together different tools to achieve a single goal.

Moving to Quadlets

Now that the upgrade to Debian 13 has been completed and I am running a recent podman version I can stop using compose and move entirely to Quadlets. This will allow each network, volume, and container to be defined in their own file and then systemd will generate service units for them when running systemctl daemon-reload.

Why Quadlets are a Better Fit

  • native systemd integration means you get all the benefits of systemd directly
  • simplified dependency management
  • built-in AutoUpdate functionality replaces the need for custom scripts like check-compose-updates.py

Miniflux Example

I’ll show a full example using Miniflux (which is a simple RSS reader) with a DB and service container. Here is my configuration for running it via Quadlets.

Set up a network with miniflux.network:

[Unit]
Description=Miniflux Network

[Network]
NetworkName=miniflux

[Install]
WantedBy=default.target

Create database with minifluxdb.container:

[Unit]
Description=Miniflux DB Container
Requires=miniflux-network.service
After=miniflux-network.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Container]
AutoUpdate=registry
ContainerName=minifluxdb
Image=docker.io/library/postgres:17-alpine
Network=miniflux.network
Volume=/opt/miniflux/data/db:/var/lib/postgresql/data
Environment=POSTGRES_USER=miniflux
Environment=POSTGRES_DB=miniflux
Secret=miniflux_db_pwd,type=env,target=POSTGRES_PASSWORD
HealthCmd=pg_isready -U miniflux
HealthInterval=10s
HealthStartPeriod=30s

[Service]
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

And finally the service with miniflux.container:

[Unit]
Description=Miniflux Container
Requires=minifluxdb.service
After=minifluxdb.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Container]
AutoUpdate=registry
ContainerName=miniflux
Image=ghcr.io/miniflux/miniflux:latest
Network=miniflux.network
Network=traefik.network
Environment=ADMIN_USERNAME=fuzz
Environment=ADMIN_PASSWORD=
Environment=AUTH_PROXY_HEADER=X-Forwarded-User
Environment=AUTH_PROXY_USER_CREATION=true
Environment=BASE_URL=https://rss.example.com
Environment=HTTPS=true
Environment=LOG_DATE_TIME=true
Environment=RUN_MIGRATIONS=1
Secret=miniflux_db_url,type=env,target=DATABASE_URL
Label=traefik.enable=true
Label=traefik.http.routers.miniflux.rule="Host(`rss.example.com`)"
Label=traefik.http.routers.miniflux.entrypoints="websecure"
Label=traefik.http.routers.miniflux.middlewares="oidc-auth@file"

[Service]
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

These files are all dropped in /etc/containers/systemd. When sudo systemctl daemon-reload runs, it will generate service units which can be viewed in /run/systemd/generator/*.service files.

➜ ls -al /run/systemd/generator/miniflux*.service
-rw-r--r-- 1 root root 1528 Aug 25 16:42 /run/systemd/generator/minifluxdb.service
-rw-r--r-- 1 root root  459 Aug 25 16:42 /run/systemd/generator/miniflux-network.service
-rw-r--r-- 1 root root 2222 Aug 25 16:42 /run/systemd/generator/miniflux.service

Tips and Tricks

  • each network, volume, and container becomes an independent service file which can then have dependencies on each other
    • this ensures they startup and shutdown in the correct order
    • if you stop the miniflux-network.service, systemd will first stop the miniflux.service and minifluxdb.service
  • pay attention to the Podman version you’re running and use the right documentation
    • for example the conventions for dependencies has changed in recent releases
    • in 5.4.2 (Debian 13) the Requires=, After=, and Network= do not point to the same file - the systemd dependencies point to the miniflux-network.service generated file while the container network points to the miniflux.network
    • more recent versions of Podman appear to have fixed this oddity, but Debian 13 bundles v5.4.2 which calls this out in the documentation
  • if you can’t find configuration in the docs for a Podman command line arg, use the PodmanArgs=... generic command line arg
    • use this to pass flags like PodmanArgs=--privileged or PodmanArgs=--no-hosts
  • when something is wrong with your unit file, the generator fails silently
    • manually running the podman-system-generator will allow you to see the issue
    sudo /usr/lib/systemd/system-generators/podman-system-generator --dryrun
    ...
    quadlet-generator[3466310]: converting "miniflux.container": unsupported key 'FooArg' in group 'Container' in /etc/containers/systemd/miniflux.container
    
  • Podman secrets is a clean way to manage secure credentials, API keys, etc
    • once a Podman secret is created, you can pass it to a container as a file or ENV var
    • for example, the following takes the miniflux_db_pwd secret and injects it into the container as an environment variable called POSTGRES_PASSWORD
    Secret=miniflux_db_pwd,type=env,target=POSTGRES_PASSWORD
    
  • use systemd restart policies to restart services on failures but prevent misbehaving services from continuous restart loops
    • Restart=always and RestartSec=10 will ensure the service is always restarted (even after reboot) waiting 10s between restart attempts
    • StartLimitIntervalSec=300 and StartLimitBurst=5 will allow the service to restart up to 5 times in a 5 minute window - after that it will mark the service as failed and require manual intervention

Managing Container Image Updates

Each container unit file has been marked with AutoUpdate=registry. This ensures they are updated when podman auto-update is run. By default a systemd timer is run daily at midnight and can be customized as needed.

Instead of updating daily overnight, I prefer to update manually as part of my regular Ansible playbook run. If anything breaks during the update, I know I’m available to immediately investigate. The following Ansible playbook will disable the regular daily timer and instead manually run auto-update.

- name: Stop and disable podman-auto-update.timer so we can manually run auto-update
  become: yes
  ansible.builtin.systemd:
    name: podman-auto-update.timer
    state: stopped
    enabled: false

- name: Manually run the podman-auto-update service to update any out of date containers
  become: yes
  ansible.builtin.systemd:
    name: podman-auto-update.service
    state: started

Conclusion

Migrating to Quadlets has been rather straightforward and feel like a simple and convenient mechanisms to orchestrate my self-hosted services. I’d certainly consider it if you are looking for the same.