--- title: "How to Install Apache Airflow on Ubuntu 24.04 with Docker" description: "Install Apache Airflow 3 on Ubuntu 24.04 with the official Docker Compose stack, write and run your first DAG, store a connection and serve the UI over HTTPS." category: "Data Engineering" language: en source: https://cubepath.com/docs/data-engineering/apache-airflow-installation-and-configuration updated: 2026-09-25 --- # How to Install Apache Airflow on Ubuntu 24.04 with Docker Apache Airflow is an open source platform for scheduling and monitoring workflows written in Python. Each workflow is a DAG (directed acyclic graph) of tasks, and Airflow takes care of scheduling runs, retrying failed tasks, passing data between them and showing the whole history in a web UI. In this tutorial you will deploy Airflow 3 on Ubuntu 24.04 using the official Docker Compose stack from the Airflow project, write and run a first DAG, add a connection to an external database and publish the UI over HTTPS behind Nginx. ## Prerequisites To follow this guide you need: - A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 4 GB of RAM (8 GB recommended) and 2 vCPUs. The stack runs PostgreSQL, Redis and five Airflow components. - A non-root user with `sudo` privileges. - Docker Engine and the Docker Compose plugin installed from Docker's official repository. - A domain or subdomain (this guide uses **`airflow.your_domain`**) with a DNS `A` record pointing to **`your_server_ip`**. - Ports 22, 80 and 443 open in your firewall. Check the available memory before you start: ```bash free -h ``` ## Step 1 - Downloading the official Compose file The Airflow project publishes a Compose file that runs a complete deployment with the CeleryExecutor: | Service | Role | |---|---| | `postgres` | Metadata database | | `redis` | Message broker between the scheduler and the workers | | `airflow-apiserver` | Web UI and REST API on port 8080 | | `airflow-scheduler` | Decides which tasks run and when | | `airflow-dag-processor` | Parses the DAG files | | `airflow-worker` | Celery worker that executes the tasks | | `airflow-triggerer` | Runs deferrable tasks | | `airflow-init` | One-shot container that migrates the database and creates the admin user | Create a project directory owned by your user, so the files that Airflow writes in `dags/` and `logs/` belong to you: ```bash sudo mkdir -p /opt/airflow sudo chown "$USER":"$USER" /opt/airflow cd /opt/airflow ``` Download the Compose file for the current stable release: ```bash curl -fLO https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml ``` > Tip: To pin a specific release, replace `stable` in the URL with a version number such as `3.1.0`. The file references the image of that same version. Confirm which Airflow image the file uses: ```bash grep -m1 'image:' docker-compose.yaml ``` ```output image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.1.0} ``` Your version may be newer. ## Step 2 - Preparing directories and settings The containers mount four directories from the project folder. Create them: ```bash mkdir -p ./dags ./logs ./plugins ./config ``` Airflow runs inside the containers with the UID stored in `.env`. Setting it to your own UID avoids permission problems on the mounted directories. The same file sets the administrator credentials that `airflow-init` creates; replace **`your_strong_password`** with a real password: ```bash cat > .env < list[int]: return [12, 7, 30] @task def transform(values: list[int]) -> int: return sum(values) @task def load(total: int) -> None: print(f"Total processed: {total}") load(transform(extract())) hello_pipeline() ``` The DAG runs every day at 06:00 UTC, does not backfill past dates (`catchup=False`) and retries each failed task twice. The DAG processor scans the folder periodically, so a new file can take a few minutes to appear. Check for syntax or import problems first: ```bash sudo docker compose exec airflow-scheduler airflow dags list-import-errors ``` An empty result means every file in `dags/` imported cleanly. Then confirm that Airflow registered the DAG: ```bash sudo docker compose exec airflow-scheduler airflow dags list ``` `hello_pipeline` should be listed with its file location `/opt/airflow/dags/hello_pipeline.py` and `is_paused` set to `True`. ## Step 5 - Running the DAG New DAGs start paused. Unpause it and trigger a run: ```bash sudo docker compose exec airflow-scheduler airflow dags unpause hello_pipeline sudo docker compose exec airflow-scheduler airflow dags trigger hello_pipeline ``` The run is queued, picked up by the Celery worker and executed task by task. For quick debugging while you write DAGs, you can also execute a whole DAG in a single process, without the scheduler, and see every task's output in your terminal: ```bash sudo docker compose exec airflow-scheduler airflow dags test hello_pipeline ``` Among the task logs printed to the terminal you should see the output of the `load` task: ```output Total processed: 49 ``` You will check the triggered run in the web UI once it is published in Step 7. ## Step 6 - Storing a connection Connections hold the host, credentials and options that tasks need to reach external systems. Keeping them in Airflow instead of in the DAG code means credentials are encrypted with the Fernet key and can be changed without editing code. Add a PostgreSQL connection with a URI. Replace the placeholders with the details of a database your DAGs will use: ```bash sudo docker compose exec airflow-scheduler airflow connections add analytics_db \ --conn-uri 'postgres://db_user:db_password@db.your_domain:5432/analytics' ``` Verify it was stored: ```bash sudo docker compose exec airflow-scheduler airflow connections get analytics_db -o yaml ``` In a DAG you reference the connection by its ID, for example with a hook from the PostgreSQL provider (`apache-airflow-providers-postgres`, included in the official image): ```python from airflow.providers.postgres.hooks.postgres import PostgresHook def count_events() -> int: hook = PostgresHook(postgres_conn_id="analytics_db") return hook.get_first("SELECT count(*) FROM events")[0] ``` ## Step 7 - Publishing the UI with Nginx and HTTPS Install Nginx and Certbot: ```bash sudo apt update sudo apt install nginx certbot python3-certbot-nginx ``` Create a server block: ```bash sudo nano /etc/nginx/sites-available/airflow ``` Add the following configuration, replacing `airflow.your_domain` with your domain: ```nginx server { listen 80; listen [::]:80; server_name airflow.your_domain; location / { proxy_pass http://127.0.0.1:8080; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } ``` Enable it, allow web traffic and request a certificate: ```bash sudo ln -s /etc/nginx/sites-available/airflow /etc/nginx/sites-enabled/ sudo nginx -t sudo systemctl reload nginx sudo ufw allow 'Nginx Full' sudo certbot --nginx -d airflow.your_domain ``` Open `https://airflow.your_domain` and sign in as `admin` with the password from Step 2. Open **Dags**, select `hello_pipeline` and check that the triggered run finished with all three tasks green. Click the `load` task and open its logs to see the `Total processed: 49` line. ## Step 8 - Operating the stack A few commands cover most day-to-day work. Run them from `/opt/airflow`: ```bash sudo docker compose logs -f airflow-scheduler sudo docker compose restart airflow-worker sudo docker compose down sudo docker compose up -d ``` `docker compose down` stops the stack but keeps the PostgreSQL volume, so your metadata and run history survive. Do not add `-v` unless you want to delete it. Back up the metadata database regularly with `pg_dump` from the `postgres` container (the default database, user and password in the official file are all `airflow`): ```bash sudo mkdir -p /var/backups/airflow sudo docker compose exec -T postgres pg_dump -U airflow airflow | gzip | sudo tee /var/backups/airflow/airflow-$(date +%F).sql.gz > /dev/null ``` To upgrade, download the Compose file of the new version, reapply the edits from Step 2, then run `sudo docker compose up airflow-init` followed by `sudo docker compose up -d`. The init container migrates the database schema. ## Troubleshooting **A DAG does not appear in the UI.** Run `airflow dags list-import-errors` as shown in Step 4. Import errors show the file and the Python traceback. If there are none, wait for the next DAG processor scan or check its logs with `sudo docker compose logs airflow-dag-processor`. **`airflow-init` fails with permission errors on `logs/`.** The UID in `.env` does not match the owner of the project directory. Check both with `id -u` and `ls -ln /opt/airflow`, fix `.env` and run `airflow-init` again. **Tasks stay in the queued state.** The worker is not consuming from Redis. Check `sudo docker compose ps` for an unhealthy `airflow-worker` or `redis`, and read `sudo docker compose logs airflow-worker`. On servers with too little RAM, the kernel may be killing the worker; `sudo dmesg | grep -i oom` confirms it. ## Conclusion You now have Apache Airflow 3 running on Ubuntu 24.04 with the official Docker Compose stack, an encrypted metadata store, a first DAG running on a schedule and the UI available over HTTPS. As next steps, keep your `dags/` folder in a Git repository and deploy it with a pull or CI job, install additional providers by building a custom image from `apache/airflow`, and move the metadata database to a managed or dedicated PostgreSQL server as your workload grows.