---
title: "Canary Deployments with Nginx and Docker on Ubuntu 24.04"
description: "Send a small share of users to a new release with Nginx split_clients, compare error rates in the logs, then raise the share gradually or roll back in seconds."
category: "CI/CD DevOps"
language: en
source: https://cubepath.com/docs/ci-cd-devops/canary-deployments-strategy-on-linux
updated: 2026-09-25
---

# Canary Deployments with Nginx and Docker on Ubuntu 24.04

A canary deployment runs the new version of an application next to the current one and sends only a small share of real users to it. If error rates and latency stay normal you raise the share step by step until the new version serves everyone; if not, you send everyone back to the stable version before most users notice. In this tutorial you will implement this on Ubuntu 24.04 with two Docker containers and Nginx's `split_clients` module, add a cookie to force either version for testing, log which version served each request and use those logs to decide whether to continue.

## Prerequisites

To follow this tutorial you need:

- A server running Ubuntu 24.04 LTS, for example a CubePath VPS, and a non-root user with `sudo` privileges.
- Docker Engine installed from Docker's official repository.
- Nginx installed (`sudo apt install nginx`), and UFW allowing `Nginx Full` if the firewall is enabled.
- Two versions of your application as container images, each listening on one port with a health endpoint. The examples use two releases of the public `traefik/whoami` image (listening on port 80, with a `/health` endpoint) so you can follow along first.

## How traffic is split

`split_clients` hashes a key for each request and assigns it to a bucket by percentage. The key used here is the client IP plus the User-Agent, so the same user keeps landing on the same version instead of switching back and forth between requests. Because the buckets are cumulative, raising the canary from 10% to 25% keeps the users who were already on the canary and adds new ones.

| Component | Stable | Canary |
|---|---|---|
| Container | `app-stable` | `app-canary` |
| Port (localhost only) | `127.0.0.1:8081` | `127.0.0.1:8082` |
| Nginx upstream | `app_stable` | `app_canary` |

## Step 1 - Running the stable and canary containers

Start the current release as the stable container. `--hostname` makes `whoami` print which container answered:

```bash
docker run -d --name app-stable --hostname app-stable --restart unless-stopped \
  -p 127.0.0.1:8081:80 traefik/whoami:v1.10.3
```

Start the new release as the canary:

```bash
docker run -d --name app-canary --hostname app-canary --restart unless-stopped \
  -p 127.0.0.1:8082:80 traefik/whoami:v1.11.0
```

Check that both are healthy before Nginx sends them any traffic:

```bash
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8081/health
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8082/health
```

```output
200
200
```

## Step 2 - Configuring the traffic split in Nginx

The configuration has two parts: a file with the upstreams, routing logic and log format, which you write once, and a small file with the percentage, which changes during each rollout.

Create the percentage file, starting at 0% so that nothing reaches the canary yet:

```bash
sudo nano /etc/nginx/canary-split.conf
```

```nginx
split_clients "${remote_addr}${http_user_agent}" $canary_variant {
    *    app_stable;
}
```

Create the main file in `conf.d`, which Ubuntu's `nginx.conf` includes inside the `http` block:

```bash
sudo nano /etc/nginx/conf.d/canary.conf
```

```nginx
upstream app_stable {
    server 127.0.0.1:8081;
    keepalive 16;
}

upstream app_canary {
    server 127.0.0.1:8082;
    keepalive 16;
}

include /etc/nginx/canary-split.conf;

# A "canary" cookie overrides the random split: always, never, or unset
map $cookie_canary $app_upstream {
    always   app_canary;
    never    app_stable;
    default  $canary_variant;
}

log_format canary '$time_iso8601 $app_upstream $status $request_time "$request"';
```

The `map` lets you and your testers pin a version with a cookie, while everyone else follows the percentage. The `canary` log format puts the upstream, status code and response time in fixed positions so they are easy to analyze.

Create the site:

```bash
sudo nano /etc/nginx/sites-available/app
```

```nginx
server {
    listen 80;
    server_name your_domain;

    access_log /var/log/nginx/canary.log canary;

    location / {
        proxy_pass http://$app_upstream;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header X-Canary $app_upstream always;
    }
}
```

When `proxy_pass` contains a variable whose value is the name of an upstream block, Nginx sends the request to that upstream. The `X-Canary` header tells you which version served a response.

Enable the site, disable the default one, test and reload:

```bash
sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled/app
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx
```

Check that normal traffic goes to stable and that the cookie reaches the canary:

```bash
curl -s http://localhost/ | grep Hostname
curl -s --cookie "canary=always" http://localhost/ | grep Hostname
```

```output
Hostname: app-stable
Hostname: app-canary
```

## Step 3 - Creating a script to set the canary percentage

Editing the percentage file by hand during a rollout is error prone, especially the 0% and 100% cases. A short script writes the file, validates the whole configuration and reloads Nginx, or restores the previous file if the test fails:

```bash
sudo nano /usr/local/sbin/canary-weight
```

```bash
#!/usr/bin/env bash
set -euo pipefail

SPLIT_CONF=/etc/nginx/canary-split.conf
percent=${1:-}

if ! [[ "$percent" =~ ^[0-9]+$ ]] || (( percent > 100 )); then
    echo "Usage: $0 PERCENT (0-100)" >&2
    exit 2
fi

if (( percent == 0 )); then
    rules='    *    app_stable;'
elif (( percent == 100 )); then
    rules='    *    app_canary;'
else
    rules=$(printf '    %s%%  app_canary;\n    *    app_stable;' "$percent")
fi

cp "$SPLIT_CONF" "${SPLIT_CONF}.bak"
printf 'split_clients "${remote_addr}${http_user_agent}" $canary_variant {\n%s\n}\n' "$rules" > "$SPLIT_CONF"

if ! nginx -t -q; then
    mv "${SPLIT_CONF}.bak" "$SPLIT_CONF"
    echo "nginx -t failed, previous split restored" >&2
    exit 1
fi

systemctl reload nginx
echo "Canary now receives ${percent}% of traffic"
```

The `printf` format is in single quotes, so `${remote_addr}` and `$canary_variant` are written literally for Nginx instead of being expanded by Bash.

Make it executable and send 10% of users to the canary:

```bash
sudo chmod 755 /usr/local/sbin/canary-weight
sudo canary-weight 10
cat /etc/nginx/canary-split.conf
```

```output
Canary now receives 10% of traffic
split_clients "${remote_addr}${http_user_agent}" $canary_variant {
    10%  app_canary;
    *    app_stable;
}
```

## Step 4 - Verifying the split

Requests from one machine always carry the same IP and User-Agent, so they all land on the same version. To see the split, send 1,000 requests with a different User-Agent each and count the `X-Canary` headers:

```bash
for i in $(seq 1 1000); do
  curl -s -o /dev/null -D - -A "client-$i" http://localhost/ | grep -i '^x-canary'
done | sort | uniq -c
```

```output
    103 X-Canary: app_canary
    897 X-Canary: app_stable
```

The numbers will not be exactly 100 and 900, since the split is based on a hash, but they should be close.

## Step 5 - Comparing stable and canary in the logs

The point of a canary is to compare the two versions under the same real traffic. Each line of `/var/log/nginx/canary.log` contains the upstream in field 2, the status code in field 3 and the response time in field 4:

```bash
sudo tail -n 3 /var/log/nginx/canary.log
```

```output
2026-09-25T10:41:12+00:00 app_stable 200 0.002 "GET / HTTP/1.1"
2026-09-25T10:41:12+00:00 app_canary 200 0.003 "GET / HTTP/1.1"
2026-09-25T10:41:12+00:00 app_stable 200 0.002 "GET / HTTP/1.1"
```

Summarize requests, 5xx errors and average response time per version for the last 10,000 requests:

```bash
sudo tail -n 10000 /var/log/nginx/canary.log | awk '
  { n[$2]++; t[$2] += $4; if ($3 >= 500) e[$2]++ }
  END { for (u in n) printf "%-11s requests=%-6d 5xx=%-4d (%.2f%%) avg=%.3fs\n", u, n[u], e[u], 100 * e[u] / n[u], t[u] / n[u] }'
```

```output
app_stable  requests=8962   5xx=3    (0.03%) avg=0.041s
app_canary  requests=1038   5xx=1    (0.10%) avg=0.044s
```

Decide your thresholds before the rollout, for example: the canary's 5xx rate stays below 1% and its average response time within 20% of stable. With low traffic, wait until the canary has served enough requests for the numbers to mean something. If you already have Prometheus, Grafana or a similar tool, compare the same metrics there, grouped by the `X-Canary` value or by container.

## Step 6 - Increasing traffic and promoting the release

If the numbers look good, raise the percentage in steps and check the logs again after each one:

```bash
sudo canary-weight 25
sudo canary-weight 50
sudo canary-weight 100
```

At 100% all users are on the new version and the stable container is idle. Now replace the stable container with the new release, check it and move traffic back to it:

```bash
docker rm -f app-stable
docker run -d --name app-stable --hostname app-stable --restart unless-stopped \
  -p 127.0.0.1:8081:80 traefik/whoami:v1.11.0
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8081/health
sudo canary-weight 0
```

```output
200
Canary now receives 0% of traffic
```

The rollout is complete: `app-stable` runs the new version and the canary slot is free for the next release, which you start with `docker rm -f app-canary` and a new `docker run` on port 8082.

## Step 7 - Rolling back

If the canary misbehaves at any stage, send all traffic back to the stable version:

```bash
sudo canary-weight 0
```

Nginx reloads gracefully, so requests in progress on the canary finish and new ones go to stable. Leave the canary container running while you investigate with `docker logs app-canary`, and test fixes against it using the `canary=always` cookie.

## Things to keep in mind

- **Shared IP addresses:** users behind the same NAT with the same browser share a hash and are routed together. With enough users this averages out.
- **Database changes:** both versions use the same database during the rollout, so migrations must work with both the old and the new code.
- **Several servers:** if you run more than one Nginx, apply the same percentage on all of them, for example with a configuration management tool, or users will be routed differently depending on the server they hit.

## Troubleshooting

- **All requests from your test loop go to one version:** you did not vary the User-Agent (`-A`), or the percentage is 0 or 100. Check `/etc/nginx/canary-split.conf`.
- **`nginx -t` reports `unknown "app_upstream" variable`:** `/etc/nginx/conf.d/canary.conf` is not being loaded. Make sure `/etc/nginx/nginx.conf` contains `include /etc/nginx/conf.d/*.conf;` in the `http` block.
- **502 Bad Gateway for part of the users:** one of the containers is down. Check both with `docker ps` and the health endpoints, and set the canary to 0 if it is the canary.

## Conclusion

You now route a controlled share of users to a new release with Nginx, can pin either version with a cookie, compare both versions from the access log and roll back with a single command. Next, run the log summary from a monitoring system so it alerts you when the canary's error rate rises, trigger the canary steps from your CI pipeline, or use a blue-green switch for releases that do not need a gradual rollout.