How to Install Apache Airflow on Ubuntu 24.04 with Docker

Apache Airflow is an open source platform for scheduling and monitoring workflows written in Python. Each workflow is a DAG (directed acyclic graph) of tasks, and Airflow takes care of scheduling runs, retrying failed tasks, passing data between them and showing the whole history in a web UI. In this tutorial you will deploy Airflow 3 on Ubuntu 24.04 using the official Docker Compose stack from the Airflow project, write and run a first DAG, add a connection to an external database and publish the UI over HTTPS behind Nginx.

Prerequisites

To follow this guide you need:

• A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 4 GB of RAM (8 GB recommended) and 2 vCPUs. The stack runs PostgreSQL, Redis and five Airflow components.
• A non-root user with sudo privileges.
• Docker Engine and the Docker Compose plugin installed from Docker's official repository.
• A domain or subdomain (this guide uses airflow.your_domain) with a DNS A record pointing to your_server_ip.
• Ports 22, 80 and 443 open in your firewall.

Check the available memory before you start:

free -h

Step 1 - Downloading the official Compose file

The Airflow project publishes a Compose file that runs a complete deployment with the CeleryExecutor:

Create a project directory owned by your user, so the files that Airflow writes in dags/ and logs/ belong to you:

sudo mkdir -p /opt/airflow
sudo chown "$USER":"$USER" /opt/airflow
cd /opt/airflow

Download the Compose file for the current stable release:

curl -fLO https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml

Tip

To pin a specific release, replace stable in the URL with a version number such as 3.1.0. The file references the image of that same version.

Confirm which Airflow image the file uses:

grep -m1 'image:' docker-compose.yaml

  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.1.0}

Your version may be newer.

Step 2 - Preparing directories and settings

The containers mount four directories from the project folder. Create them:

mkdir -p ./dags ./logs ./plugins ./config

Airflow runs inside the containers with the UID stored in .env. Setting it to your own UID avoids permission problems on the mounted directories. The same file sets the administrator credentials that airflow-init creates; replace your_strong_password with a real password:

cat > .env <<EOF
AIRFLOW_UID=$(id -u)
_AIRFLOW_WWW_USER_USERNAME=admin
_AIRFLOW_WWW_USER_PASSWORD=your_strong_password
EOF
chmod 600 .env

Next, generate a Fernet key. Airflow uses it to encrypt connection passwords and variables in the database, and the default Compose file leaves it empty:

python3 -c "import base64, os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"

q3k8m1Z0yJ4c0xw5b3V9oQ2hZr7mN6tP1sL8dK4eF0A=

Open the Compose file:

nano docker-compose.yaml

Make three changes:

1. In the x-airflow-common environment block, paste your key into AIRFLOW__CORE__FERNET_KEY.
2. In the same block, set AIRFLOW__CORE__LOAD_EXAMPLES to 'false' so the UI only shows your own DAGs.
3. In the airflow-apiserver service, bind the port to the loopback interface. Ports published by Docker bypass UFW, so this keeps the UI private until Nginx is in front of it.

The edited lines should look like this:

    AIRFLOW__CORE__FERNET_KEY: 'q3k8m1Z0yJ4c0xw5b3V9oQ2hZr7mN6tP1sL8dK4eF0A='
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'

  airflow-apiserver:
    <<: *airflow-common
    command: api-server
    ports:
      - "127.0.0.1:8080:8080"

Validate the file after editing it:

sudo docker compose config --quiet && echo OK

OK

Step 3 - Initializing and starting Airflow

Run the initialization container. It checks resources, migrates the metadata database and creates the admin user:

sudo docker compose up airflow-init

When it finishes you should see the container exit with code 0:

airflow-init-1 exited with code 0

Start the rest of the stack in the background:

sudo docker compose up -d

After a minute or two, every service should report as healthy:

sudo docker compose ps --format 'table {{.Service}}\t{{.Status}}'

SERVICE                 STATUS
airflow-apiserver       Up 2 minutes (healthy)
airflow-dag-processor   Up 2 minutes (healthy)
airflow-scheduler       Up 2 minutes (healthy)
airflow-triggerer       Up 2 minutes (healthy)
airflow-worker          Up 2 minutes (healthy)
postgres                Up 3 minutes (healthy)
redis                   Up 3 minutes (healthy)

Check the Airflow version through the CLI inside a running container:

sudo docker compose exec airflow-scheduler airflow version

Step 4 - Writing your first DAG

DAGs are Python files placed in the dags/ directory, which is mounted into every Airflow container. Create one:

nano /opt/airflow/dags/hello_pipeline.py

This example uses the TaskFlow API from the Airflow 3 Task SDK. Three tasks pass data to each other: the return value of one task becomes the argument of the next, and Airflow stores it between tasks automatically:

import pendulum

from airflow.sdk import dag, task


@dag(
    schedule="0 6 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],
    default_args={"retries": 2},
)
def hello_pipeline():
    @task
    def extract() -> list[int]:
        return [12, 7, 30]

    @task
    def transform(values: list[int]) -> int:
        return sum(values)

    @task
    def load(total: int) -> None:
        print(f"Total processed: {total}")

    load(transform(extract()))


hello_pipeline()

The DAG runs every day at 06:00 UTC, does not backfill past dates (catchup=False) and retries each failed task twice.

The DAG processor scans the folder periodically, so a new file can take a few minutes to appear. Check for syntax or import problems first:

sudo docker compose exec airflow-scheduler airflow dags list-import-errors

An empty result means every file in dags/ imported cleanly. Then confirm that Airflow registered the DAG:

sudo docker compose exec airflow-scheduler airflow dags list

hello_pipeline should be listed with its file location /opt/airflow/dags/hello_pipeline.py and is_paused set to True.

Step 5 - Running the DAG

New DAGs start paused. Unpause it and trigger a run:

sudo docker compose exec airflow-scheduler airflow dags unpause hello_pipeline
sudo docker compose exec airflow-scheduler airflow dags trigger hello_pipeline

The run is queued, picked up by the Celery worker and executed task by task. For quick debugging while you write DAGs, you can also execute a whole DAG in a single process, without the scheduler, and see every task's output in your terminal:

sudo docker compose exec airflow-scheduler airflow dags test hello_pipeline

Among the task logs printed to the terminal you should see the output of the load task:

Total processed: 49

You will check the triggered run in the web UI once it is published in Step 7.

Step 6 - Storing a connection

Connections hold the host, credentials and options that tasks need to reach external systems. Keeping them in Airflow instead of in the DAG code means credentials are encrypted with the Fernet key and can be changed without editing code.

Add a PostgreSQL connection with a URI. Replace the placeholders with the details of a database your DAGs will use:

sudo docker compose exec airflow-scheduler airflow connections add analytics_db \
  --conn-uri 'postgres://db_user:[email protected]_domain:5432/analytics'

Verify it was stored:

sudo docker compose exec airflow-scheduler airflow connections get analytics_db -o yaml

In a DAG you reference the connection by its ID, for example with a hook from the PostgreSQL provider (apache-airflow-providers-postgres, included in the official image):

from airflow.providers.postgres.hooks.postgres import PostgresHook


def count_events() -> int:
    hook = PostgresHook(postgres_conn_id="analytics_db")
    return hook.get_first("SELECT count(*) FROM events")[0]

Step 7 - Publishing the UI with Nginx and HTTPS

Install Nginx and Certbot:

sudo apt update
sudo apt install nginx certbot python3-certbot-nginx

Create a server block:

sudo nano /etc/nginx/sites-available/airflow

Add the following configuration, replacing airflow.your_domain with your domain:

server {
    listen 80;
    listen [::]:80;
    server_name airflow.your_domain;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable it, allow web traffic and request a certificate:

sudo ln -s /etc/nginx/sites-available/airflow /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
sudo ufw allow 'Nginx Full'
sudo certbot --nginx -d airflow.your_domain

Open https://airflow.your_domain and sign in as admin with the password from Step 2. Open Dags, select hello_pipeline and check that the triggered run finished with all three tasks green. Click the load task and open its logs to see the Total processed: 49 line.

Step 8 - Operating the stack

A few commands cover most day-to-day work. Run them from /opt/airflow:

sudo docker compose logs -f airflow-scheduler
sudo docker compose restart airflow-worker
sudo docker compose down
sudo docker compose up -d

docker compose down stops the stack but keeps the PostgreSQL volume, so your metadata and run history survive. Do not add -v unless you want to delete it.

Back up the metadata database regularly with pg_dump from the postgres container (the default database, user and password in the official file are all airflow):

sudo mkdir -p /var/backups/airflow
sudo docker compose exec -T postgres pg_dump -U airflow airflow | gzip | sudo tee /var/backups/airflow/airflow-$(date +%F).sql.gz > /dev/null

To upgrade, download the Compose file of the new version, reapply the edits from Step 2, then run sudo docker compose up airflow-init followed by sudo docker compose up -d. The init container migrates the database schema.

Troubleshooting

A DAG does not appear in the UI. Run airflow dags list-import-errors as shown in Step 4. Import errors show the file and the Python traceback. If there are none, wait for the next DAG processor scan or check its logs with sudo docker compose logs airflow-dag-processor.

airflow-init fails with permission errors on logs/. The UID in .env does not match the owner of the project directory. Check both with id -u and ls -ln /opt/airflow, fix .env and run airflow-init again.

Tasks stay in the queued state. The worker is not consuming from Redis. Check sudo docker compose ps for an unhealthy airflow-worker or redis, and read sudo docker compose logs airflow-worker. On servers with too little RAM, the kernel may be killing the worker; sudo dmesg | grep -i oom confirms it.

Conclusion

You now have Apache Airflow 3 running on Ubuntu 24.04 with the official Docker Compose stack, an encrypted metadata store, a first DAG running on a schedule and the UI available over HTTPS. As next steps, keep your dags/ folder in a Git repository and deploy it with a pull or CI job, install additional providers by building a custom image from apache/airflow, and move the metadata database to a managed or dedicated PostgreSQL server as your workload grows.