VXLAN (Virtual Extensible LAN) carries Ethernet frames inside UDP packets, so machines on different servers, or even different subnets, can share one Layer 2 segment. Each server acts as a VXLAN Tunnel Endpoint (VTEP) that encapsulates and decapsulates the frames, and each segment is identified by a 24-bit VXLAN Network Identifier (VNI). In this tutorial you will build a VXLAN overlay between Ubuntu 24.04 servers with iproute2, plug a Linux bridge and a test namespace into it, extend it to a third server and make the configuration survive reboots with systemd-networkd.

Prerequisites

To follow this tutorial you need:

  • Two servers running Ubuntu 24.04 LTS (three for Step 5), for example CubePath VPS instances attached to the same private network.
  • A non-root user with sudo privileges on each server.
  • IP connectivity between the servers. This network is called the underlay. The guide uses these underlay addresses on interface eth1; replace them with your own:
ServerUnderlay IPOverlay IP (VNI 100)
host-a10.20.0.1110.100.0.1
host-b10.20.0.1210.100.0.2
host-c10.20.0.1310.100.0.3

VXLAN can discover remote VTEPs with IP multicast, but cloud and most data center networks do not route multicast. This guide uses unicast VXLAN, where you list the remote VTEPs yourself.

Step 1 - Checking the underlay and opening the firewall

The overlay can only work if the underlay does. From host-a, ping host-b's underlay address:

ping -c 3 10.20.0.12

64 bytes from 10.20.0.12: icmp_seq=1 ttl=64 time=0.402 ms

Check the MTU of the underlay interface, since you will need it to size the overlay:

ip link show eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000

VXLAN uses UDP port 4789. If UFW is active, allow it from the underlay subnet on every host:

sudo ufw allow from 10.20.0.0/24 to any port 4789 proto udp

Step 2 - Creating a point-to-point VXLAN interface

On host-a, create a VXLAN interface for VNI 100 that sends encapsulated frames to host-b. The local address is the underlay IP to send from and dstport 4789 is the IANA port (without it, Linux uses the legacy port 8472):

sudo ip link add vxlan100 type vxlan id 100 dstport 4789 local 10.20.0.11 remote 10.20.0.12 dev eth1

VXLAN headers add 50 bytes to every frame. With a 1500-byte underlay, set the overlay MTU to 1450, then assign the overlay address and bring the interface up:

sudo ip link set vxlan100 mtu 1450
sudo ip addr add 10.100.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

On host-b, create the mirror image:

sudo ip link add vxlan100 type vxlan id 100 dstport 4789 local 10.20.0.12 remote 10.20.0.11 dev eth1
sudo ip link set vxlan100 mtu 1450
sudo ip addr add 10.100.0.2/24 dev vxlan100
sudo ip link set vxlan100 up

Inspect the VXLAN parameters the kernel applied:

ip -d link show vxlan100

5: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 6e:8f:1a:22:c4:10 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
    vxlan id 100 remote 10.20.0.11 local 10.20.0.12 dev eth1 srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx

Step 3 - Testing the overlay

From host-a, ping host-b's overlay address:

ping -c 3 10.100.0.2

64 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=0.781 ms
64 bytes from 10.100.0.2: icmp_seq=2 ttl=64 time=0.512 ms
64 bytes from 10.100.0.2: icmp_seq=3 ttl=64 time=0.498 ms

To see the encapsulation, capture on the underlay interface of host-b while the ping runs:

sudo tcpdump -ni eth1 udp port 4789 -c 2

IP 10.20.0.11.45312 > 10.20.0.12.4789: VXLAN, flags [I] (0x08), vni 100
IP 10.100.0.1 > 10.100.0.2: ICMP echo request, id 7, seq 1, length 64
IP 10.20.0.12.45312 > 10.20.0.11.4789: VXLAN, flags [I] (0x08), vni 100
IP 10.100.0.2 > 10.100.0.1: ICMP echo reply, id 7, seq 1, length 64

Each line pair shows the outer UDP packet between the VTEPs and the inner ICMP packet of the overlay.

Step 4 - Attaching the overlay to a Linux bridge

To put virtual machines or containers on the overlay, attach the VXLAN interface to a bridge and plug the guests into the same bridge. The address moves from vxlan100 to the bridge. On host-a:

sudo ip addr del 10.100.0.1/24 dev vxlan100
sudo ip link add br100 type bridge
sudo ip link set vxlan100 master br100
sudo ip addr add 10.100.0.1/24 dev br100
sudo ip link set br100 up

Now simulate a guest with a network namespace connected to the bridge through a veth pair:

sudo ip netns add guest1
sudo ip link add guest1-br type veth peer name guest1-eth
sudo ip link set guest1-br master br100
sudo ip link set guest1-br up
sudo ip link set guest1-eth netns guest1
sudo ip -n guest1 link set guest1-eth mtu 1450
sudo ip -n guest1 addr add 10.100.0.101/24 dev guest1-eth
sudo ip -n guest1 link set guest1-eth up
sudo ip -n guest1 link set lo up

Ping host-b's overlay address from inside the guest. The frame goes guest to bridge to VTEP, across the underlay, and out on host-b:

sudo ip netns exec guest1 ping -c 3 10.100.0.2

64 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=0.911 ms

The bridge's forwarding database (FDB) shows which MAC addresses were learned on the VXLAN port and behind which remote VTEP:

bridge fdb show dev vxlan100

00:00:00:00:00:00 dst 10.20.0.12 self permanent
6e:8f:1a:22:c4:10 dst 10.20.0.12 self

The all-zeros entry is the default destination for broadcast and unknown traffic, created by the remote option.

Step 5 - Adding a third VTEP

The remote option only accepts one peer. For more than two hosts, create the VXLAN interface without remote and add one all-zeros FDB entry per remote VTEP. Broadcast frames such as ARP requests are then replicated to every listed VTEP (head-end replication), and the kernel learns where each MAC lives from the replies.

On host-c, create the interface and one flood entry for each other host:

sudo ip link add vxlan100 type vxlan id 100 dstport 4789 local 10.20.0.13 dev eth1
sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.20.0.11
sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.20.0.12
sudo ip link set vxlan100 mtu 1450
sudo ip addr add 10.100.0.3/24 dev vxlan100
sudo ip link set vxlan100 up

On host-a and host-b, add host-c as an extra flood destination next to the existing remote:

sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.20.0.13

From host-c, ping both other hosts:

ping -c 2 10.100.0.1
ping -c 2 10.100.0.2

Check the flood list on host-a:

bridge fdb show dev vxlan100 | grep 00:00:00:00:00:00

00:00:00:00:00:00 dst 10.20.0.12 self permanent
00:00:00:00:00:00 dst 10.20.0.13 self permanent

Every host must list every other host. Static lists work well for a handful of VTEPs; beyond that, a control plane such as BGP EVPN (for example with FRRouting) distributes VTEPs and MAC addresses automatically.

Step 6 - Making the configuration persistent

Everything created with ip and bridge disappears at reboot. Ubuntu 24.04 uses systemd-networkd underneath Netplan, and networkd also reads its own files from /etc/systemd/network/. Use unique file names so they do not collide with the ones Netplan generates. Before continuing, remove the temporary lab on host-a so networkd can create the interfaces itself:

sudo ip netns del guest1
sudo ip link del vxlan100
sudo ip link del br100

Create the VXLAN device on host-a. Independent=yes creates the VXLAN interface without tying it to a specific underlay .network file:

sudo nano /etc/systemd/network/30-vxlan100.netdev

[NetDev]
Name=vxlan100
Kind=vxlan
MTUBytes=1450

[VXLAN]
VNI=100
Local=10.20.0.11
DestinationPort=4789
Independent=yes

Attach it to the bridge and declare one flood entry per remote VTEP:

sudo nano /etc/systemd/network/30-vxlan100.network

[Match]
Name=vxlan100

[Network]
Bridge=br100

[BridgeFDB]
MACAddress=00:00:00:00:00:00
Destination=10.20.0.12

[BridgeFDB]
MACAddress=00:00:00:00:00:00
Destination=10.20.0.13

Define the bridge device:

sudo nano /etc/systemd/network/30-br100.netdev

[NetDev]
Name=br100
Kind=bridge

And give the bridge its overlay address:

sudo nano /etc/systemd/network/30-br100.network

[Match]
Name=br100

[Network]
Address=10.100.0.1/24
ConfigureWithoutCarrier=yes

Ask networkd to reload its configuration. reload creates new devices without restarting the service or touching your existing interfaces:

sudo networkctl reload
networkctl status vxlan100 --no-pager

● 6: vxlan100
                   Link File: /usr/lib/systemd/network/99-default.link
                Network File: /etc/systemd/network/30-vxlan100.network
                       State: carrier (configured)
                        Type: vxlan
                        Kind: vxlan
                         VNI: 100
                       Local: 10.20.0.11
            Destination Port: 4789

Confirm the flood entries and test the overlay again:

bridge fdb show dev vxlan100 | grep 00:00:00:00:00:00
ping -c 2 10.100.0.2

Repeat on host-b and host-c with their own Local=, Address= and the Destination= of the other two hosts, then reboot one host to confirm the overlay comes back on its own.

Troubleshooting

  • Underlay ping works but overlay ping does not: run sudo tcpdump -ni eth1 udp port 4789 on the receiver. No packets means a firewall or security group drops UDP 4789; packets arriving without replies usually means a VNI or port mismatch between hosts (ip -d link show vxlan100).
  • Small pings work but SSH or HTTP over the overlay hangs: the overlay MTU is too large. It must be at least 50 bytes below the underlay MTU on the VXLAN interface, the bridge and every guest.
  • A new host is only reachable from some hosts: a flood entry is missing. Every VTEP needs an all-zeros entry for every other VTEP.
  • networkctl shows vxlan100 as failed: check sudo journalctl -u systemd-networkd -n 50 for typos in the .netdev or .network files.

Conclusion

You created a unicast VXLAN overlay between Ubuntu 24.04 servers, attached it to a Linux bridge so guests can join it, extended it to a third VTEP with FDB flood entries and made it persistent with systemd-networkd. The same VTEP can carry more segments: create one VXLAN interface and bridge per VNI. As next steps, attach KVM guests to br100, encrypt the underlay with WireGuard if it crosses untrusted networks, or replace the static flood lists with BGP EVPN when the number of hosts grows.