HashiCorp Consul Installation and Service Discovery

HashiCorp Consul is a distributed service mesh and service discovery tool that provides health checking, a DNS interface, a key-value store, and multi-datacenter support. This guide covers installing Consul on Linux, bootstrapping a cluster, registering services, and integrating with load balancers.

Prerequisites

  • 3 or more Linux servers for the Consul server cluster (Ubuntu 22.04/Debian 12 or CentOS/Rocky 9)
  • Root or sudo access
  • All nodes can reach each other on TCP/UDP ports 8300, 8301, 8302, 8500, 8600
  • Consistent hostnames and static IPs
  • NTP synchronized across all nodes

Install Consul

Install Consul on all nodes (servers and clients):

# Ubuntu/Debian
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y consul

# CentOS/Rocky
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul

# Verify installation
consul version

Create the required directories:

sudo mkdir -p /etc/consul.d /opt/consul/data
sudo chown consul:consul /opt/consul/data

Configure a Consul Server Cluster

Generate a gossip encryption key (run once, use on all nodes):

consul keygen
# Save the output - you'll use it in every node's config
# Example: kYBv5XlABCDefGhIJk1234NoPqrSTuvwxYz56789abc=

Create the server configuration on each of your 3 server nodes:

# Replace node_name, bind_addr, and retry_join IPs for each server
sudo tee /etc/consul.d/consul.hcl << 'EOF'
datacenter          = "dc1"
data_dir            = "/opt/consul/data"
log_level           = "INFO"
node_name           = "consul-server-01"
server              = true
bootstrap_expect    = 3

bind_addr           = "192.168.1.10"   # This node's IP
client_addr         = "0.0.0.0"

retry_join          = ["192.168.1.10", "192.168.1.11", "192.168.1.12"]

ui_config {
  enabled = true
}

encrypt             = "kYBv5XlABCDefGhIJk1234NoPqrSTuvwxYz56789abc="

performance {
  raft_multiplier = 1
}
EOF

sudo chmod 640 /etc/consul.d/consul.hcl

Enable and start the Consul service:

sudo systemctl enable consul
sudo systemctl start consul

# Check cluster status after all 3 servers start
consul members
consul operator raft list-peers

Configure Consul Agents on Clients

Application servers run Consul in agent (client) mode:

sudo tee /etc/consul.d/consul.hcl << 'EOF'
datacenter  = "dc1"
data_dir    = "/opt/consul/data"
log_level   = "INFO"
node_name   = "web-server-01"
server      = false

bind_addr   = "192.168.1.50"    # This client's IP
client_addr = "127.0.0.1"       # Only listen locally on clients

retry_join  = ["192.168.1.10", "192.168.1.11", "192.168.1.12"]

encrypt     = "kYBv5XlABCDefGhIJk1234NoPqrSTuvwxYz56789abc="
EOF

sudo systemctl enable consul
sudo systemctl start consul
consul members

Service Registration and Health Checks

Register services so Consul tracks their health and location. Create service definition files in /etc/consul.d/:

# Register an Nginx web service
sudo tee /etc/consul.d/web.json << 'EOF'
{
  "service": {
    "name": "web",
    "tags": ["nginx", "frontend"],
    "port": 80,
    "check": {
      "http": "http://localhost:80/health",
      "interval": "10s",
      "timeout": "3s"
    }
  }
}
EOF

# Register a PostgreSQL database service with TCP check
sudo tee /etc/consul.d/postgres.json << 'EOF'
{
  "service": {
    "name": "postgres",
    "tags": ["primary"],
    "port": 5432,
    "check": {
      "tcp": "localhost:5432",
      "interval": "15s",
      "timeout": "5s"
    }
  }
}
EOF

# Reload Consul to pick up new services
consul reload

# Query service health via API
curl http://localhost:8500/v1/health/service/web?passing

Register services via the HTTP API (no config file needed):

curl -X PUT http://localhost:8500/v1/agent/service/register \
  -H "Content-Type: application/json" \
  -d '{
    "Name": "api",
    "Port": 8080,
    "Tags": ["v2"],
    "Check": {
      "HTTP": "http://localhost:8080/ready",
      "Interval": "10s"
    }
  }'

DNS Interface

Consul runs a DNS server on port 8600 for service discovery via DNS:

# Query a service via Consul DNS
dig @127.0.0.1 -p 8600 web.service.consul

# Query with tag filter
dig @127.0.0.1 -p 8600 nginx.web.service.consul

# Query across datacenters
dig @127.0.0.1 -p 8600 web.service.dc2.consul

# SRV records include port information
dig @127.0.0.1 -p 8600 web.service.consul SRV

Forward .consul DNS queries from the system resolver to Consul:

# Using systemd-resolved
sudo tee /etc/systemd/resolved.conf.d/consul.conf << 'EOF'
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul
EOF

sudo systemctl restart systemd-resolved

# Test
dig web.service.consul

Key-Value Store

Consul's KV store is useful for configuration sharing and service coordination:

# Write a key
consul kv put config/app/db_host "postgres.service.consul"
consul kv put config/app/db_port "5432"
consul kv put config/app/max_connections "100"

# Read a key
consul kv get config/app/db_host

# List all keys under a prefix
consul kv get -recurse config/app/

# Delete a key
consul kv delete config/app/max_connections

# Watch for changes (blocks until the key changes)
consul watch -type=key -key=config/app/db_host cat

Use the HTTP API for KV access from applications:

# Read via API
curl http://localhost:8500/v1/kv/config/app/db_host?raw

# Write via API
curl -X PUT http://localhost:8500/v1/kv/config/app/db_host \
  -d "new-postgres-host.example.com"

Integrating with Load Balancers

Use Consul with HAProxy via consul-template for dynamic backend updates:

# Install consul-template
sudo apt install -y consul-template   # or download from releases.hashicorp.com

# Create HAProxy template
sudo tee /etc/consul-template/haproxy.ctmpl << 'EOF'
global
    daemon
    maxconn 4096

defaults
    mode http
    timeout connect 5s
    timeout client  50s
    timeout server  50s

frontend web_frontend
    bind *:80
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
{{range service "web"}}
    server {{.Node}} {{.Address}}:{{.Port}} check
{{end}}
EOF

# Run consul-template to keep HAProxy config updated
consul-template \
  -template "/etc/consul-template/haproxy.ctmpl:/etc/haproxy/haproxy.cfg:systemctl reload haproxy" \
  -log-level info

Troubleshooting

Cluster fails to form:

# Check that all nodes can reach each other on port 8301
nc -zv 192.168.1.11 8301

# Verify the gossip key matches on all nodes
consul keyring -list

# Check for split-brain (should see all servers)
consul operator raft list-peers

Service health check failing:

# View agent logs
journalctl -u consul -n 100 --no-pager

# List all checks and their status
consul watch -type=checks -service=web cat

# Check health via API with verbose output
curl -s http://localhost:8500/v1/health/checks/web | python3 -m json.tool

DNS queries returning no results:

# Verify Consul DNS is running
dig @127.0.0.1 -p 8600 consul.service.consul

# Check Consul's DNS port binding
ss -tlnup | grep 8600

Conclusion

HashiCorp Consul provides a reliable foundation for service discovery and health checking in distributed environments, with built-in DNS resolution, a KV store, and multi-datacenter support. Running a 3-node server cluster ensures fault tolerance while client agents on application servers handle service registration and health checking locally. Use consul-template to keep load balancer configurations automatically updated as services come and go.