A canary deployment runs the new version of an application next to the current one and sends only a small share of real users to it. If error rates and latency stay normal you raise the share step by step until the new version serves everyone; if not, you send everyone back to the stable version before most users notice. In this tutorial you will implement this on Ubuntu 24.04 with two Docker containers and Nginx's split_clients module, add a cookie to force either version for testing, log which version served each request and use those logs to decide whether to continue.

Prerequisites

To follow this tutorial you need:

  • A server running Ubuntu 24.04 LTS, for example a CubePath VPS, and a non-root user with sudo privileges.
  • Docker Engine installed from Docker's official repository.
  • Nginx installed (sudo apt install nginx), and UFW allowing Nginx Full if the firewall is enabled.
  • Two versions of your application as container images, each listening on one port with a health endpoint. The examples use two releases of the public traefik/whoami image (listening on port 80, with a /health endpoint) so you can follow along first.

How traffic is split

split_clients hashes a key for each request and assigns it to a bucket by percentage. The key used here is the client IP plus the User-Agent, so the same user keeps landing on the same version instead of switching back and forth between requests. Because the buckets are cumulative, raising the canary from 10% to 25% keeps the users who were already on the canary and adds new ones.

ComponentStableCanary
Containerapp-stableapp-canary
Port (localhost only)127.0.0.1:8081127.0.0.1:8082
Nginx upstreamapp_stableapp_canary

Step 1 - Running the stable and canary containers

Start the current release as the stable container. --hostname makes whoami print which container answered:

docker run -d --name app-stable --hostname app-stable --restart unless-stopped \
  -p 127.0.0.1:8081:80 traefik/whoami:v1.10.3

Start the new release as the canary:

docker run -d --name app-canary --hostname app-canary --restart unless-stopped \
  -p 127.0.0.1:8082:80 traefik/whoami:v1.11.0

Check that both are healthy before Nginx sends them any traffic:

curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8081/health
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8082/health

200
200

Step 2 - Configuring the traffic split in Nginx

The configuration has two parts: a file with the upstreams, routing logic and log format, which you write once, and a small file with the percentage, which changes during each rollout.

Create the percentage file, starting at 0% so that nothing reaches the canary yet:

sudo nano /etc/nginx/canary-split.conf

split_clients "${remote_addr}${http_user_agent}" $canary_variant {
    *    app_stable;
}

Create the main file in conf.d, which Ubuntu's nginx.conf includes inside the http block:

sudo nano /etc/nginx/conf.d/canary.conf

upstream app_stable {
    server 127.0.0.1:8081;
    keepalive 16;
}

upstream app_canary {
    server 127.0.0.1:8082;
    keepalive 16;
}

include /etc/nginx/canary-split.conf;

# A "canary" cookie overrides the random split: always, never, or unset
map $cookie_canary $app_upstream {
    always   app_canary;
    never    app_stable;
    default  $canary_variant;
}

log_format canary '$time_iso8601 $app_upstream $status $request_time "$request"';

The map lets you and your testers pin a version with a cookie, while everyone else follows the percentage. The canary log format puts the upstream, status code and response time in fixed positions so they are easy to analyze.

Create the site:

sudo nano /etc/nginx/sites-available/app

server {
    listen 80;
    server_name your_domain;

    access_log /var/log/nginx/canary.log canary;

    location / {
        proxy_pass http://$app_upstream;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header X-Canary $app_upstream always;
    }
}

When proxy_pass contains a variable whose value is the name of an upstream block, Nginx sends the request to that upstream. The X-Canary header tells you which version served a response.

Enable the site, disable the default one, test and reload:

sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled/app
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

Check that normal traffic goes to stable and that the cookie reaches the canary:

curl -s http://localhost/ | grep Hostname
curl -s --cookie "canary=always" http://localhost/ | grep Hostname

Hostname: app-stable
Hostname: app-canary

Step 3 - Creating a script to set the canary percentage

Editing the percentage file by hand during a rollout is error prone, especially the 0% and 100% cases. A short script writes the file, validates the whole configuration and reloads Nginx, or restores the previous file if the test fails:

sudo nano /usr/local/sbin/canary-weight

#!/usr/bin/env bash
set -euo pipefail

SPLIT_CONF=/etc/nginx/canary-split.conf
percent=${1:-}

if ! [[ "$percent" =~ ^[0-9]+$ ]] || (( percent > 100 )); then
    echo "Usage: $0 PERCENT (0-100)" >&2
    exit 2
fi

if (( percent == 0 )); then
    rules='    *    app_stable;'
elif (( percent == 100 )); then
    rules='    *    app_canary;'
else
    rules=$(printf '    %s%%  app_canary;\n    *    app_stable;' "$percent")
fi

cp "$SPLIT_CONF" "${SPLIT_CONF}.bak"
printf 'split_clients "${remote_addr}${http_user_agent}" $canary_variant {\n%s\n}\n' "$rules" > "$SPLIT_CONF"

if ! nginx -t -q; then
    mv "${SPLIT_CONF}.bak" "$SPLIT_CONF"
    echo "nginx -t failed, previous split restored" >&2
    exit 1
fi

systemctl reload nginx
echo "Canary now receives ${percent}% of traffic"

The printf format is in single quotes, so ${remote_addr} and $canary_variant are written literally for Nginx instead of being expanded by Bash.

Make it executable and send 10% of users to the canary:

sudo chmod 755 /usr/local/sbin/canary-weight
sudo canary-weight 10
cat /etc/nginx/canary-split.conf

Canary now receives 10% of traffic
split_clients "${remote_addr}${http_user_agent}" $canary_variant {
    10%  app_canary;
    *    app_stable;
}

Step 4 - Verifying the split

Requests from one machine always carry the same IP and User-Agent, so they all land on the same version. To see the split, send 1,000 requests with a different User-Agent each and count the X-Canary headers:

for i in $(seq 1 1000); do
  curl -s -o /dev/null -D - -A "client-$i" http://localhost/ | grep -i '^x-canary'
done | sort | uniq -c

    103 X-Canary: app_canary
    897 X-Canary: app_stable

The numbers will not be exactly 100 and 900, since the split is based on a hash, but they should be close.

Step 5 - Comparing stable and canary in the logs

The point of a canary is to compare the two versions under the same real traffic. Each line of /var/log/nginx/canary.log contains the upstream in field 2, the status code in field 3 and the response time in field 4:

sudo tail -n 3 /var/log/nginx/canary.log

2026-09-25T10:41:12+00:00 app_stable 200 0.002 "GET / HTTP/1.1"
2026-09-25T10:41:12+00:00 app_canary 200 0.003 "GET / HTTP/1.1"
2026-09-25T10:41:12+00:00 app_stable 200 0.002 "GET / HTTP/1.1"

Summarize requests, 5xx errors and average response time per version for the last 10,000 requests:

sudo tail -n 10000 /var/log/nginx/canary.log | awk '
  { n[$2]++; t[$2] += $4; if ($3 >= 500) e[$2]++ }
  END { for (u in n) printf "%-11s requests=%-6d 5xx=%-4d (%.2f%%) avg=%.3fs\n", u, n[u], e[u], 100 * e[u] / n[u], t[u] / n[u] }'

app_stable  requests=8962   5xx=3    (0.03%) avg=0.041s
app_canary  requests=1038   5xx=1    (0.10%) avg=0.044s

Decide your thresholds before the rollout, for example: the canary's 5xx rate stays below 1% and its average response time within 20% of stable. With low traffic, wait until the canary has served enough requests for the numbers to mean something. If you already have Prometheus, Grafana or a similar tool, compare the same metrics there, grouped by the X-Canary value or by container.

Step 6 - Increasing traffic and promoting the release

If the numbers look good, raise the percentage in steps and check the logs again after each one:

sudo canary-weight 25
sudo canary-weight 50
sudo canary-weight 100

At 100% all users are on the new version and the stable container is idle. Now replace the stable container with the new release, check it and move traffic back to it:

docker rm -f app-stable
docker run -d --name app-stable --hostname app-stable --restart unless-stopped \
  -p 127.0.0.1:8081:80 traefik/whoami:v1.11.0
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8081/health
sudo canary-weight 0

200
Canary now receives 0% of traffic

The rollout is complete: app-stable runs the new version and the canary slot is free for the next release, which you start with docker rm -f app-canary and a new docker run on port 8082.

Step 7 - Rolling back

If the canary misbehaves at any stage, send all traffic back to the stable version:

sudo canary-weight 0

Nginx reloads gracefully, so requests in progress on the canary finish and new ones go to stable. Leave the canary container running while you investigate with docker logs app-canary, and test fixes against it using the canary=always cookie.

Things to keep in mind

  • Shared IP addresses: users behind the same NAT with the same browser share a hash and are routed together. With enough users this averages out.
  • Database changes: both versions use the same database during the rollout, so migrations must work with both the old and the new code.
  • Several servers: if you run more than one Nginx, apply the same percentage on all of them, for example with a configuration management tool, or users will be routed differently depending on the server they hit.

Troubleshooting

  • All requests from your test loop go to one version: you did not vary the User-Agent (-A), or the percentage is 0 or 100. Check /etc/nginx/canary-split.conf.
  • nginx -t reports unknown "app_upstream" variable: /etc/nginx/conf.d/canary.conf is not being loaded. Make sure /etc/nginx/nginx.conf contains include /etc/nginx/conf.d/*.conf; in the http block.
  • 502 Bad Gateway for part of the users: one of the containers is down. Check both with docker ps and the health endpoints, and set the canary to 0 if it is the canary.

Conclusion

You now route a controlled share of users to a new release with Nginx, can pin either version with a cookie, compare both versions from the access log and roll back with a single command. Next, run the log summary from a monitoring system so it alerts you when the canary's error rate rises, trigger the canary steps from your CI pipeline, or use a blue-green switch for releases that do not need a gradual rollout.