Redis Sentinel High Availability

Redis Sentinel is a distributed system that monitors Redis instances and automatically handles failover when the primary server becomes unavailable. It provides high availability without relying on external dependencies, using consensus algorithms to elect a new primary and redirect client connections. This comprehensive guide covers Sentinel deployment, configuration, monitoring, automatic failover behavior, client integration, and operational procedures for production Redis deployments.

Table of Contents

Architecture and Concepts

Redis Sentinel is a separate system running alongside Redis instances that monitors their health and coordinates failover. Multiple Sentinel instances form a consensus group that detects primary failure and promotes a replica without administrator intervention. Sentinel watches Redis instances continuously, detecting failures through a combination of health checks and heartbeat monitoring.

The Sentinel cluster itself requires a quorum (typically 3 instances) for voting on failover decisions. If the primary becomes unreachable and a quorum of Sentinels agrees, one of the replicas is promoted to primary and other Sentinels reconfigure the remaining replicas to replicate from the new primary.

Installation

Install Redis Server and Sentinel. First install Redis:

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y redis-server redis-tools

# CentOS/RHEL
sudo dnf install -y redis

# Verify installation
redis-server --version
redis-cli --version
redis-sentinel --version

Build from source for latest version:

# Download Redis
cd /tmp
wget http://download.redis.io/redis-stable.tar.gz
tar -xzf redis-stable.tar.gz
cd redis-stable

# Compile
make
make test

# Install
sudo make install PREFIX=/usr/local

# Create symlinks
sudo ln -s /usr/local/bin/redis-server /usr/bin/redis-server
sudo ln -s /usr/local/bin/redis-cli /usr/bin/redis-cli
sudo ln -s /usr/local/bin/redis-sentinel /usr/bin/redis-sentinel

# Verify
redis-server --version

Create dedicated user and directories:

# Create redis user
sudo useradd -r -s /bin/false redis 2>/dev/null || true

# Create directories
sudo mkdir -p /var/lib/redis /var/log/redis /etc/redis
sudo chown -R redis:redis /var/lib/redis /var/log/redis
sudo chmod 755 /var/lib/redis /var/log/redis

Sentinel Configuration

Configure Redis Sentinel. Create sentinel configuration file:

sudo nano /etc/redis/sentinel.conf

Add comprehensive Sentinel configuration:

# Listening port
port 26379

# Sentinel monitor configuration
# sentinel monitor <master-name> <master-ip> <master-port> <quorum>
sentinel monitor mymaster 192.168.1.10 6379 2

# Timeout for detecting master as down
sentinel down-after-milliseconds mymaster 30000

# Timeout for failover
sentinel failover-timeout mymaster 180000

# Number of replicas to reconfigure in parallel
sentinel parallel-syncs mymaster 1

# Notification script (optional)
# sentinel notification-script mymaster /path/to/notification.sh

# Client reconfiguration script (optional)
# sentinel client-reconfig-script mymaster /path/to/reconfig.sh

# Logging
logfile /var/log/redis/sentinel.log
loglevel notice
syslog-enabled yes
syslog-ident redis-sentinel

# Directory for state files
dir /var/lib/redis

# Sentinel authentication (if Redis requires password)
# sentinel auth-pass mymaster your-redis-password

# Quorum and failover policy
sentinel deny-scripts-reconfig yes
sentinel announce-ip 192.168.1.20
sentinel announce-port 26379

# Connection settings
sentinel tcp-backlog 511
sentinel tcp-keepalive 300

# Default replica priority (lower = higher priority for promotion)
sentinel replica-priority mymaster 100

Set permissions on configuration:

sudo chown redis:redis /etc/redis/sentinel.conf
sudo chmod 600 /etc/redis/sentinel.conf

Create systemd service for Sentinel:

sudo nano /etc/systemd/system/redis-sentinel.service

Add:

[Unit]
Description=Redis Sentinel
After=network.target

[Service]
Type=simple
User=redis
Group=redis

ExecStart=/usr/local/bin/redis-sentinel /etc/redis/sentinel.conf
ExecStop=/bin/kill -TERM $MAINPID
Restart=always
RestartSec=10

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Enable and start Sentinel:

sudo systemctl daemon-reload
sudo systemctl enable redis-sentinel
sudo systemctl start redis-sentinel
sudo systemctl status redis-sentinel

# Monitor logs
sudo journalctl -u redis-sentinel -f

Multi-Sentinel Deployment

Deploy multiple Sentinel instances for redundancy. Configure first Sentinel instance (on different server):

# SSH to sentinel1 (192.168.1.20)
ssh [email protected]

# Copy sentinel configuration from primary setup
sudo nano /etc/redis/sentinel.conf

Add configuration:

port 26379
sentinel monitor mymaster 192.168.1.10 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
logfile /var/log/redis/sentinel.log
dir /var/lib/redis
announce-ip 192.168.1.20
announce-port 26379

Configure second Sentinel instance (192.168.1.21):

# SSH to sentinel2
ssh [email protected]

# Configure Sentinel
sudo nano /etc/redis/sentinel.conf

Add:

port 26379
sentinel monitor mymaster 192.168.1.10 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
logfile /var/log/redis/sentinel.log
dir /var/lib/redis
announce-ip 192.168.1.21
announce-port 26379

Configure third Sentinel instance (192.168.1.22):

# SSH to sentinel3
ssh [email protected]

# Configure Sentinel
sudo nano /etc/redis/sentinel.conf

Add:

port 26379
sentinel monitor mymaster 192.168.1.10 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
logfile /var/log/redis/sentinel.log
dir /var/lib/redis
announce-ip 192.168.1.22
announce-port 26379

Start all Sentinel instances and verify cluster:

# On each Sentinel host
sudo systemctl start redis-sentinel
sudo systemctl status redis-sentinel

# From any Sentinel, check cluster status
redis-cli -p 26379 SENTINEL masters

# Get current master info
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster

# Check Sentinel nodes
redis-cli -p 26379 SENTINEL sentinels mymaster

# Check replicas
redis-cli -p 26379 SENTINEL replicas mymaster

Monitoring and Health Checks

Monitor Sentinel and Redis health:

# Connect to Sentinel
redis-cli -p 26379

# View all monitored masters
SENTINEL masters

# Get detailed master information
SENTINEL master mymaster

# Check current leader
SENTINEL is-master-down-by-addr 192.168.1.10 6379 0

# Get master status
SENTINEL get-master-addr-by-name mymaster

# Check replica status
SENTINEL replicas mymaster

# View Sentinel info
INFO sentinel

# Exit
EXIT

Monitor from multiple Sentinels:

#!/bin/bash
# Check Sentinel status across cluster

SENTINELS=("192.168.1.20" "192.168.1.21" "192.168.1.22")

for sentinel in "${SENTINELS[@]}"; do
  echo "=== Sentinel at $sentinel ==="
  redis-cli -h $sentinel -p 26379 SENTINEL get-master-addr-by-name mymaster
  redis-cli -h $sentinel -p 26379 SENTINEL sentinel-state mymaster | head -5
done

Create monitoring script:

#!/bin/bash
# /usr/local/bin/redis-sentinel-monitor.sh

MASTER_HOST="192.168.1.10"
MASTER_PORT="6379"
SENTINELS=("192.168.1.20" "192.168.1.21" "192.168.1.22")

echo "=== Redis Sentinel Status ==="
echo "Timestamp: $(date)"

# Check master
echo ""
echo "Master Status:"
redis-cli -h $MASTER_HOST -p $MASTER_PORT ping > /dev/null 2>&1 && echo "Master: ONLINE" || echo "Master: OFFLINE"

# Check all Sentinels
echo ""
echo "Sentinel Status:"
for sentinel in "${SENTINELS[@]}"; do
  result=$(redis-cli -h $sentinel -p 26379 PING 2>/dev/null)
  [ "$result" = "PONG" ] && echo "Sentinel $sentinel: OK" || echo "Sentinel $sentinel: FAILED"
done

# Get master address from Sentinel
echo ""
echo "Current Master:"
redis-cli -h ${SENTINELS[0]} -p 26379 SENTINEL get-master-addr-by-name mymaster

# Check replica count
echo ""
echo "Replica Status:"
redis-cli -h $MASTER_HOST -p $MASTER_PORT INFO replication | grep -E "role|connected_slaves"

Automatic Failover

Understand and monitor automatic failover:

# Simulate master failure
redis-cli -h 192.168.1.10 -p 6379 shutdown

# Monitor Sentinel detection
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
# Result will show new master after failover

# Watch logs during failover
tail -f /var/log/redis/sentinel.log | grep -E "failover|switch"

# After failover, check new master
redis-cli -h 192.168.1.11 -p 6379 INFO replication

# Rejoin original master (after recovery)
# Start Redis on original master
redis-server /etc/redis/redis.conf

# Monitor Sentinel reconfiguring it as replica
tail -f /var/log/redis/sentinel.log

Monitor failover events:

#!/bin/bash
# Monitor and log failover events

redis-cli -p 26379 SUBSCRIBE +master-failover-began \
                              +master-failover-end \
                              +convert-to-replica \
                              +role-change |
while read line; do
  echo "$(date): $line" >> /var/log/redis/failover-events.log
done

Client Configuration

Configure Redis client libraries for Sentinel support:

# Python example with redis-py
from redis.sentinel import Sentinel

# Create Sentinel connection
sentinel = Sentinel(
    sentinels=[
        ('192.168.1.20', 26379),
        ('192.168.1.21', 26379),
        ('192.168.1.22', 26379)
    ],
    socket_timeout=0.1
)

# Get master connection
master = sentinel.master_for('mymaster', socket_timeout=0.1)
master.set('key', 'value')

# Get slave connection for reads
slave = sentinel.slave_for('mymaster', socket_timeout=0.1)
value = slave.get('key')

Node.js example:

// Using node-redis with Sentinel
const redis = require('redis');

const sentinel = redis.createClient({
  sentinels: [
    { host: '192.168.1.20', port: 26379 },
    { host: '192.168.1.21', port: 26379 },
    { host: '192.168.1.22', port: 26379 }
  ],
  name: 'mymaster',
  sentinelRetryStrategy: (times) => {
    return Math.min(times * 10, 1000);
  }
});

sentinel.on('connect', () => {
  console.log('Connected to Redis via Sentinel');
});

Java example with Jedis:

// Using Jedis with Sentinel
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisSentinelPool;
import redis.clients.util.Pool;

Set<String> sentinels = new HashSet<>(Arrays.asList(
    "192.168.1.20:26379",
    "192.168.1.21:26379",
    "192.168.1.22:26379"
));

JedisSentinelPool sentinelPool = new JedisSentinelPool(
    "mymaster",
    sentinels
);

try (Jedis jedis = sentinelPool.getResource()) {
    jedis.set("key", "value");
    String value = jedis.get("key");
}

Notification Scripts

Configure scripts to execute on failover events:

# Create notification script
sudo nano /usr/local/bin/redis-failover-notification.sh

Add:

#!/bin/bash
# Redis Sentinel notification script

MASTER_NAME=$1
ROLE=$2
STATE=$3

# Log the event
echo "$(date): Sentinel event - Master: $MASTER_NAME, Role: $ROLE, State: $STATE" >> /var/log/redis/failover-notifications.log

# Send email notification
SUBJECT="Redis Failover Event: $MASTER_NAME"
BODY="Time: $(date)\nMaster: $MASTER_NAME\nRole: $ROLE\nState: $STATE"

# Uncomment to enable email
# echo -e "$BODY" | mail -s "$SUBJECT" [email protected]

# Update load balancer or DNS
case $STATE in
    "failover-end")
        echo "Failover completed for $MASTER_NAME"
        # Update load balancer to point to new master
        # curl -X POST http://load-balancer:8080/update-redis-endpoint
        ;;
esac

exit 0

Set permissions and configure:

sudo chmod +x /usr/local/bin/redis-failover-notification.sh
sudo chown redis:redis /usr/local/bin/redis-failover-notification.sh

# Update sentinel.conf to use the script
sudo nano /etc/redis/sentinel.conf

# Add line
sentinel notification-script mymaster /usr/local/bin/redis-failover-notification.sh

Operational Procedures

Perform maintenance operations safely:

# Graceful master failover (no downtime)
redis-cli -p 26379 SENTINEL failover mymaster

# Monitor failover progress
redis-cli -p 26379 SENTINEL master mymaster | grep failover

# Resync after maintenance
redis-cli -h 192.168.1.10 -p 6379 SLAVEOF 192.168.1.11 6379

# Restore original master role
redis-cli -h 192.168.1.10 -p 6379 SLAVEOF NO ONE

Reset Sentinel state:

# Clear Sentinel state files (requires restart)
sudo rm /var/lib/redis/sentinel-*.conf
sudo systemctl restart redis-sentinel

# Reconfigure Sentinel if needed
redis-cli -p 26379 SENTINEL reset \*

Troubleshooting

Debug common Sentinel issues:

# Check if Sentinel can see master
redis-cli -p 26379 SENTINEL is-master-down-by-addr 192.168.1.10 6379 0

# Get detailed master state
redis-cli -p 26379 SENTINEL master mymaster

# Check Sentinel logs
sudo tail -100 /var/log/redis/sentinel.log

# Verify network connectivity
ping 192.168.1.10
redis-cli -h 192.168.1.10 -p 6379 ping

# Check Sentinel quorum
redis-cli -p 26379 SENTINEL sentinels mymaster | wc -l

# Monitor failover attempt
strace -e trace=network redis-sentinel /etc/redis/sentinel.conf

Conclusion

Redis Sentinel provides automated failover and high availability for Redis deployments without requiring external tools. Its lightweight nature, built-in consensus mechanism, and automatic client notification make it ideal for production environments where Redis availability is critical. By properly configuring multiple Sentinel instances, monitoring health continuously, and testing failover scenarios, you ensure your Redis infrastructure maintains availability through hardware failures and planned maintenance. Sentinel's simplicity combined with its reliability makes it the recommended approach for Redis high availability in most production deployments.