How to Set Up a Consul Cluster on Ubuntu 24.04

HashiCorp Consul keeps a live catalog of the services running in your infrastructure, checks their health and answers "where is service X?" through DNS or an HTTP API. In this tutorial you will build a three-node Consul server cluster on Ubuntu 24.04 with encrypted gossip, join an application server as a client agent, register a service with a health check, resolve it through Consul DNS and store configuration in the key-value store.

Prerequisites

To follow this guide you need:

Three servers running Ubuntu 24.04 LTS for the Consul servers, and at least one more for your application (the client agent). CubePath VPS instances connected by a private network work well. Each needs 1 vCPU and 1 GB of RAM or more.
A non-root user with sudo privileges on every server.
Static private IP addresses. This guide uses:

Hostname	Private IP	Role
consul-01	10.0.0.11	Server
consul-02	10.0.0.12	Server
consul-03	10.0.0.13	Server
web-01	10.0.0.21	Client agent

Replace these addresses with your own. Consul relies on clocks being in sync, which Ubuntu handles by default with systemd-timesyncd.

Step 1 - Installing Consul on every node

Run this step on all four servers. HashiCorp publishes signed packages in its own APT repository. Add its key and the repository:

sudo apt update
sudo apt install -y gpg wget
wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/hashicorp.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/hashicorp.gpg] https://apt.releases.hashicorp.com $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee /etc/apt/sources.list.d/hashicorp.list > /dev/null

Install Consul:

sudo apt update
sudo apt install -y consul
consul version

Consul v1.21.x
Revision ...

The package creates a consul system user, the configuration directory /etc/consul.d/, the data directory /opt/consul and a consul.service systemd unit that loads every .hcl and .json file in /etc/consul.d/.

Step 2 - Opening the Consul ports on the private network

Consul nodes talk to each other on these ports:

Port	Protocol	Used for
8300	TCP	Server RPC (clients and servers to servers)
8301	TCP and UDP	LAN gossip between all agents
8500	TCP	HTTP API and web UI (kept on localhost in this guide)
8600	TCP and UDP	DNS interface (kept on localhost in this guide)

On every node, allow the cluster ports only from the private network:

sudo ufw allow OpenSSH
sudo ufw allow from 10.0.0.0/24 to any port 8300,8301 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 8301 proto udp
sudo ufw enable

Step 3 - Generating a gossip encryption key

Gossip traffic between agents should be encrypted with a shared symmetric key. Generate it once, on any node:

consul keygen

pUqJrVyVRj5jsiYEkM/tFQYfWyJIv4s3XkvDwy7Cu5s=

Copy the value. Every server and client uses the same key, referred to as your_gossip_key below.

Step 4 - Configuring and starting the servers

On each of the three servers, replace the default configuration file:

sudo nano /etc/consul.d/consul.hcl

Use this content, changing node_name and bind_addr on each server:

datacenter = "dc1"
data_dir   = "/opt/consul"
node_name  = "consul-01"

server           = true
bootstrap_expect = 3

bind_addr   = "10.0.0.11"
client_addr = "127.0.0.1"

retry_join = ["10.0.0.11", "10.0.0.12", "10.0.0.13"]
encrypt    = "your_gossip_key"

ui_config {
  enabled = true
}

What these settings do:

bootstrap_expect = 3 makes the servers wait until three of them have joined before electing a leader. Three servers tolerate the loss of one.
bind_addr is the private address used for cluster traffic. Set it explicitly when a server has more than one network interface.
client_addr = "127.0.0.1" keeps the HTTP API, UI and DNS on localhost so they are not reachable from other machines.
retry_join lists the servers to contact on startup, and keeps retrying until they respond.

Make sure the file belongs to the consul user and validate it:

sudo chown consul:consul /etc/consul.d/consul.hcl
sudo chmod 640 /etc/consul.d/consul.hcl
sudo consul validate /etc/consul.d/

Configuration is valid!

Enable and start the service on all three servers:

sudo systemctl enable --now consul

When the third server starts, the cluster elects a leader. Check the members and the Raft peers from any server:

consul members
consul operator raft list-peers

Node       Address         Status  Type    Build   Protocol  DC   Partition  Segment
consul-01  10.0.0.11:8301  alive   server  1.21.x  2         dc1  default    <all>
consul-02  10.0.0.12:8301  alive   server  1.21.x  2         dc1  default    <all>
consul-03  10.0.0.13:8301  alive   server  1.21.x  2         dc1  default    <all>

Node       ID                                    Address         State     Voter  RaftProtocol
consul-01  2b1f...                               10.0.0.11:8300  leader    true   3
consul-02  7c0e...                               10.0.0.12:8300  follower  true   3
consul-03  e41a...                               10.0.0.13:8300  follower  true   3

Warning

This setup does not enable Consul ACLs, so any process that can reach the HTTP API on a node can read and change the catalog and the KV store. Keeping client_addr on localhost limits that to local users. Before running production workloads, enable ACLs (acl { enabled = true, default_policy = "deny" }) and bootstrap them with consul acl bootstrap, following HashiCorp's ACL documentation.

Step 5 - Joining a client agent

Application servers run Consul in client mode. The local agent registers the services running on that machine, runs their health checks and forwards queries to the servers. On web-01, write the configuration:

sudo nano /etc/consul.d/consul.hcl

datacenter = "dc1"
data_dir   = "/opt/consul"
node_name  = "web-01"

server = false

bind_addr   = "10.0.0.21"
client_addr = "127.0.0.1"

retry_join = ["10.0.0.11", "10.0.0.12", "10.0.0.13"]
encrypt    = "your_gossip_key"

Set permissions, validate and start the agent:

sudo chown consul:consul /etc/consul.d/consul.hcl
sudo chmod 640 /etc/consul.d/consul.hcl
sudo consul validate /etc/consul.d/
sudo systemctl enable --now consul
consul members

Node       Address         Status  Type    Build   Protocol  DC   Partition  Segment
consul-01  10.0.0.11:8301  alive   server  1.21.x  2         dc1  default    <all>
consul-02  10.0.0.12:8301  alive   server  1.21.x  2         dc1  default    <all>
consul-03  10.0.0.13:8301  alive   server  1.21.x  2         dc1  default    <all>
web-01     10.0.0.21:8301  alive   client  1.21.x  2         dc1  default    <default>

Step 6 - Registering a service with a health check

As an example, register an Nginx web server running on web-01. Install it if it is not there yet:

sudo apt install -y nginx

Create a service definition next to the agent configuration:

sudo nano /etc/consul.d/web.hcl

service {
  name = "web"
  port = 80
  tags = ["nginx"]

  check {
    id       = "web-http"
    name     = "HTTP on port 80"
    http     = "http://127.0.0.1:80/"
    interval = "10s"
    timeout  = "2s"
  }
}

The agent requests the URL every 10 seconds. A 2xx response marks the service as passing, and any other result marks it as critical, which removes it from DNS answers. Apply the definition without restarting the agent:

sudo chown consul:consul /etc/consul.d/web.hcl
sudo consul validate /etc/consul.d/
consul reload

List the catalog and the nodes that provide the service:

consul catalog services
consul catalog nodes -service=web

consul
web
Node    ID        Address    DC
web-01  5f2c8a1e  10.0.0.21  dc1

Then check the state of the health check through the HTTP API:

curl -s http://127.0.0.1:8500/v1/health/checks/web | python3 -m json.tool | grep '"Status"'

        "Status": "passing",

Step 7 - Discovering services through DNS

Every agent answers DNS queries for the .consul domain on port 8600. Query it directly with dig:

dig @127.0.0.1 -p 8600 web.service.consul +short
dig @127.0.0.1 -p 8600 web.service.consul SRV +short

10.0.0.21
1 1 80 web-01.node.dc1.consul.

The A record returns the addresses of healthy instances, and the SRV record adds the port. You can also filter by tag with nginx.web.service.consul.

So that applications can resolve web.service.consul without a special port, forward the .consul domain from systemd-resolved to the local agent. Create a drop-in file:

sudo mkdir -p /etc/systemd/resolved.conf.d
sudo nano /etc/systemd/resolved.conf.d/consul.conf

[Resolve]
DNS=127.0.0.1:8600
DNSSEC=false
Domains=~consul

Domains=~consul sends only .consul queries to Consul; everything else keeps using your normal resolvers. Restart the resolver and test with the standard tools:

sudo systemctl restart systemd-resolved
resolvectl query web.service.consul

web.service.consul: 10.0.0.21                   -- link: ...

Step 8 - Storing configuration in the KV store

Consul includes a replicated key-value store, useful for small pieces of shared configuration and feature flags. Write, read and list keys:

consul kv put config/web/max_connections 200
consul kv put config/web/maintenance false
consul kv get config/web/max_connections
consul kv get -recurse config/web/

Success! Data written to: config/web/max_connections
Success! Data written to: config/web/maintenance
200
config/web/maintenance:false
config/web/max_connections:200

Applications read the same values over HTTP. The raw parameter returns only the value:

curl -s 'http://127.0.0.1:8500/v1/kv/config/web/max_connections?raw'

To reach the web UI, which listens on localhost only, open an SSH tunnel from your workstation to a server and browse to http://127.0.0.1:8500/ui/:

ssh -L 8500:127.0.0.1:8500 your_user@consul_server_ip

Troubleshooting

consul members shows only the local node. The agents cannot gossip. Check that ports 8301/tcp and 8301/udp are open between all nodes with sudo ufw status, that bind_addr is the private IP and that the encrypt key is identical everywhere. Read the logs with sudo journalctl -u consul -n 50 --no-pager.

No cluster leader errors. Fewer than bootstrap_expect servers have joined, or a majority of servers is down. Run consul operator raft list-peers and start the missing servers.

The service does not appear in DNS. Its health check is failing. Run curl -s http://127.0.0.1:8500/v1/health/checks/web | python3 -m json.tool and look at the Output field, which contains the error from the last check.

The consul service will not start. Run sudo consul validate /etc/consul.d/ to find syntax errors, and check that the files are readable by the consul user.

Conclusion

You now have a three-node Consul cluster on Ubuntu 24.04 with encrypted gossip, a client agent that registers a health-checked service, DNS-based discovery through systemd-resolved and a shared key-value store. Next, enable ACLs and TLS for production, use consul-template to regenerate an HAProxy or Nginx configuration whenever the healthy instances of a service change, or add more services and client agents.

On this page