A Docker overlay network is a virtual network that spans several Docker hosts. Containers attached to it get addresses on the same subnet and reach each other by name, even when they run on different servers, because Docker encapsulates their traffic in VXLAN between the hosts. In this tutorial you will join three Ubuntu 24.04 servers into a Docker Swarm, open the ports the overlay needs, create an encrypted overlay network and verify that services and standalone containers on different hosts can talk to each other.
Prerequisites
To follow this tutorial you need:
- Three servers running Ubuntu 24.04 LTS, for example three CubePath VPS, that can reach each other over the network. A private network between them is strongly recommended so cluster traffic does not travel over the public internet.
- A non-root user with
sudoprivileges on each server. - Docker Engine installed from Docker's official repository on all three servers, with the same major version.
- UFW enabled on each server, with SSH already allowed.
This guide uses the following names and private addresses. Replace them with your own:
| Host | Role | Private IP |
|---|---|---|
| node1 | Manager | 10.10.0.11 |
| node2 | Worker | 10.10.0.12 |
| node3 | Worker | 10.10.0.13 |
Step 1 - Opening the Swarm ports
Swarm nodes need three ports between each other, plus ESP (IP protocol 50) if you encrypt overlay traffic as you will in Step 4:
| Port | Protocol | Purpose |
|---|---|---|
| 2377 | TCP | Cluster management, only needed on managers |
| 7946 | TCP and UDP | Node discovery and gossip |
| 4789 | UDP | VXLAN data plane of overlay networks |
| ESP (50) | IP protocol | IPsec for encrypted overlay networks |
Allow them only from the private subnet of your nodes. Run these commands on all three servers:
sudo ufw allow from 10.10.0.0/24 to any port 2377 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 7946 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 7946 proto udp
sudo ufw allow from 10.10.0.0/24 to any port 4789 proto udp
sudo ufw allow proto esp from 10.10.0.0/24
Check the rules:
sudo ufw status
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
2377/tcp ALLOW 10.10.0.0/24
7946/tcp ALLOW 10.10.0.0/24
7946/udp ALLOW 10.10.0.0/24
4789/udp ALLOW 10.10.0.0/24
Anywhere/esp ALLOW 10.10.0.0/24
WarningNever expose port 2377 or 4789 to the internet. VXLAN traffic on 4789 is not authenticated, so anyone who can reach it could inject packets into unencrypted overlay networks.
Step 2 - Creating the Swarm
Overlay networks for services require Swarm mode. On node1, initialize the Swarm and tell it to advertise its private address, so the other nodes and the VXLAN tunnels use the private network:
docker swarm init --advertise-addr 10.10.0.11
Swarm initialized: current node (k2n5x8...) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-3x0...-9ab... 10.10.0.11:2377
Copy the docker swarm join line. If you lose it, print it again on the manager with docker swarm join-token worker.
On node2 and node3, run the join command, adding --advertise-addr with each node's own private IP:
docker swarm join --advertise-addr 10.10.0.12 --token SWMTKN-1-3x0...-9ab... 10.10.0.11:2377
This node joined a swarm as a worker.
Back on node1, list the nodes:
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
k2n5x8... * node1 Ready Active Leader 28.x
p7d1q3... node2 Ready Active 28.x
w9c4m6... node3 Ready Active 28.x
All three nodes must show Ready. Run the remaining commands on node1 unless stated otherwise.
Step 3 - Understanding the default networks
Swarm mode creates two networks on every node. List the overlay ones:
docker network ls --filter driver=overlay
NETWORK ID NAME DRIVER SCOPE
t1v8y0... ingress overlay swarm
The ingress network carries the routing mesh: a port published by a service is reachable on every node and forwarded to a healthy task. Do not attach your application to ingress. Instead, create your own overlay networks so each application has its own isolated subnet and DNS.
Step 4 - Creating an encrypted, attachable overlay network
Create a network named appnet with an explicit subnet that does not overlap your private network, IPsec encryption between nodes and the ability to attach standalone containers:
docker network create \
--driver overlay \
--subnet 10.20.0.0/24 \
--opt encrypted \
--attachable \
appnet
What each option does:
--subnet: fixes the address range, so it cannot collide with networks Docker would otherwise pick automatically.--opt encrypted: encrypts the VXLAN traffic between nodes with IPsec (AES-GCM). Traffic between two containers on the same host is not affected. Expect some CPU cost and a smaller effective MTU.--attachable: lets you attach containers started withdocker run, not only Swarm services. It is handy for debugging and for workloads that are not services.
Inspect the result:
docker network inspect appnet --format 'driver={{.Driver}} scope={{.Scope}} attachable={{.Attachable}} subnet={{(index .IPAM.Config 0).Subnet}}'
driver=overlay scope=swarm attachable=true subnet=10.20.0.0/24
The encrypted key also appears under Options in the full docker network inspect appnet output.
The network only appears on worker nodes once a task that uses it is scheduled there. That is expected.
Step 5 - Deploying a service across the hosts
Deploy three replicas of traefik/whoami, a tiny web server that answers with the hostname and IP address of the container that handled the request:
docker service create --name whoami --network appnet --replicas 3 traefik/whoami
Check where Swarm placed the tasks:
docker service ps whoami --format 'table {{.Name}}\t{{.Node}}\t{{.CurrentState}}'
NAME NODE CURRENT STATE
whoami.1 node2 Running 20 seconds ago
whoami.2 node3 Running 20 seconds ago
whoami.3 node1 Running 20 seconds ago
The replicas run on different hosts but share the appnet subnet.
Step 6 - Testing cross-host communication and service discovery
Docker runs an internal DNS server at 127.0.0.11 inside every container on a user-defined network. For a Swarm service it answers two names:
whoamiresolves to a single virtual IP (VIP). Connections to it are load balanced across all healthy tasks.tasks.whoamiresolves to the individual IP address of each task.
Because appnet is attachable, you can start a throwaway container on it from any node. Resolve both names:
docker run --rm --network appnet busybox nslookup whoami
docker run --rm --network appnet busybox nslookup tasks.whoami
Name: whoami
Address: 10.20.0.2
Name: tasks.whoami
Address: 10.20.0.5
Name: tasks.whoami
Address: 10.20.0.4
Name: tasks.whoami
Address: 10.20.0.3
Now send several requests to the service name and look at which container answered:
docker run --rm --network appnet curlimages/curl -s http://whoami | grep -E '^(Hostname|IP)'
Run it three or four times. The Hostname changes between requests, and most of them are answered by containers on other hosts, which proves the overlay is carrying traffic between servers:
Hostname: 7c1e5b2f0a93
IP: 127.0.0.1
IP: 10.20.0.4
Step 7 - Publishing the service to the outside
Services on appnet are reachable from each other but not from outside the cluster. To expose one, publish a port. It is then available on every node through the routing mesh:
docker service update --publish-add published=8080,target=80 whoami
From any machine that can reach the nodes, request port 8080 on any of them:
curl -s http://10.10.0.12:8080 | grep Hostname
Hostname: 7c1e5b2f0a93
NotePorts published by Docker are opened directly in the kernel firewall and are not filtered by UFW rules. If a service must stay private, do not publish it, or publish it only on a load balancer node reachable from your private network.
Troubleshooting
Containers resolve each other but connections time out. DNS works over the control plane, while the data travels over UDP 4789. Check that 4789/udp and, for encrypted networks, ESP are allowed between all nodes, and that any provider-level firewall in front of the servers allows them too.
Large requests hang while small ones work. This is an MTU problem. VXLAN adds 50 bytes and IPsec adds more. If your underlying network has an MTU below 1500, recreate the overlay with a lower value, for example --opt com.docker.network.driver.mtu=1400, and test with docker run --rm --network appnet busybox ping -c 3 -s 1300 tasks.whoami.
A node shows Down in docker node ls. Check that TCP and UDP 7946 are open in both directions and that the node advertised its private address. Leave and rejoin with docker swarm leave and the correct --advertise-addr if needed.
docker network inspect appnet on a worker says the network does not exist. Overlay networks are created on a worker only when a task that uses them runs there. Scale the service or start an attached container on that node.
To remove the test setup, run on the manager:
docker service rm whoami
docker network rm appnet
Conclusion
You built a three-node Swarm, opened only the ports it needs on the private network, created an encrypted, attachable overlay network and verified that services on different hosts discover and reach each other by name. From here you can describe multi-service applications in a Compose file and deploy them with docker stack deploy, add a second and third manager for high availability, and set CPU and memory limits on each service so one workload cannot starve a node.