Dragonfly In-Memory Data Store Installation

Dragonfly is a modern, multi-threaded in-memory data store fully compatible with Redis and Memcached APIs, designed to deliver 25x better throughput than Redis on the same hardware by leveraging a novel shared-nothing architecture. This guide covers installing Dragonfly on Linux, configuring it as a Redis/Memcached drop-in replacement, and tuning for high performance.

Prerequisites

  • Ubuntu 20.04/22.04 or CentOS 8/Rocky Linux 8+ (64-bit)
  • At least 4 GB RAM (Dragonfly is most valuable with 16+ GB)
  • Multi-core CPU (8+ cores recommended for maximum benefit)
  • Root or sudo access
  • Linux kernel 5.11+ recommended (for io_uring support)

Install Dragonfly

From binary (recommended):

# Download latest Dragonfly binary
DRAGONFLY_VERSION=$(curl -s https://api.github.com/repos/dragonflydb/dragonfly/releases/latest | grep tag_name | cut -d'"' -f4)
wget "https://github.com/dragonflydb/dragonfly/releases/download/${DRAGONFLY_VERSION}/dragonfly-amd64.tar.gz"

tar -xzf dragonfly-amd64.tar.gz
sudo install dragonfly /usr/local/bin/dragonfly

# Verify installation
dragonfly --version

From Docker:

docker run --rm -it \
  --ulimit memlock=-1 \
  -p 6379:6379 \
  -p 11211:11211 \
  -v dragonfly_data:/data \
  --name dragonfly \
  docker.dragonflydb.io/dragonflydb/dragonfly:latest

Install as a systemd service:

# Create a dedicated user
sudo useradd -r -s /bin/false -d /var/lib/dragonfly dragonfly
sudo mkdir -p /var/lib/dragonfly /etc/dragonfly
sudo chown dragonfly:dragonfly /var/lib/dragonfly

# Create the service file
cat > /etc/systemd/system/dragonfly.service << 'EOF'
[Unit]
Description=Dragonfly In-Memory Data Store
After=network.target
Documentation=https://dragonflydb.io/docs

[Service]
Type=simple
User=dragonfly
Group=dragonfly
WorkingDirectory=/var/lib/dragonfly
ExecStart=/usr/local/bin/dragonfly \
  --logtostderr \
  --requirepass= \
  --maxmemory=4gb \
  --dir=/var/lib/dragonfly \
  --dbfilename=dragonfly-dump

# Required for memory locking
LimitMEMLOCK=infinity
AmbientCapabilities=CAP_SYS_RESOURCE

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now dragonfly
sudo systemctl status dragonfly

# Test connection
redis-cli ping
# PONG

Basic Configuration

Dragonfly uses command-line flags for configuration (no separate config file required):

# Common startup flags
/usr/local/bin/dragonfly \
  --bind 0.0.0.0 \
  --port 6379 \
  --requirepass "your-strong-password" \
  --maxmemory 8gb \
  --dir /var/lib/dragonfly \
  --dbfilename dump \
  --logtostderr \
  --loglevel INFO

# Update systemd service with these flags
sudo nano /etc/systemd/system/dragonfly.service
# Edit ExecStart line

sudo systemctl daemon-reload
sudo systemctl restart dragonfly

Configure firewall:

# Allow Redis port (localhost only for security)
sudo ufw allow from 10.0.0.0/8 to any port 6379

# Or bind to specific interface
# --bind 192.168.1.5

Redis Compatibility and Client Usage

Dragonfly is a drop-in Redis replacement - no client changes needed:

# Use redis-cli directly with Dragonfly
redis-cli -h localhost -p 6379 -a your-password

# Basic data types all work
redis-cli set mykey "hello dragonfly"
redis-cli get mykey

redis-cli lpush mylist a b c
redis-cli lrange mylist 0 -1

redis-cli hset myhash field1 value1 field2 value2
redis-cli hgetall myhash

redis-cli zadd leaderboard 100 "player1" 200 "player2"
redis-cli zrevrange leaderboard 0 -1 withscores

Python client (no changes from Redis):

# pip install redis
import redis

# Connect exactly as Redis
r = redis.Redis(
    host='localhost',
    port=6379,
    password='your-password',
    decode_responses=True
)

# All Redis commands work
r.set('name', 'Dragonfly')
print(r.get('name'))

# Pipelining for high throughput
pipe = r.pipeline()
for i in range(1000):
    pipe.set(f'key:{i}', f'value:{i}')
pipe.execute()

print(f"Keys stored: {r.dbsize()}")

Node.js client:

// npm install ioredis
const Redis = require('ioredis');
const client = new Redis({
  host: 'localhost',
  port: 6379,
  password: 'your-password',
});

async function main() {
  await client.set('greeting', 'Hello from Dragonfly');
  const value = await client.get('greeting');
  console.log(value);
  client.disconnect();
}
main();

Memcached Compatibility

Dragonfly also speaks the Memcached protocol on port 11211:

# Start Dragonfly with Memcached protocol enabled
/usr/local/bin/dragonfly \
  --port 6379 \
  --memcache_port 11211 \
  --maxmemory 4gb

# Test with Memcached client
# Install memcached client tools
sudo apt install -y libmemcached-tools

# Set a value via Memcached protocol
echo -e "set testkey 0 300 5\r\nhello\r" | nc localhost 11211
# STORED

# Get the value
echo -e "get testkey\r" | nc localhost 11211
# VALUE testkey 0 5
# hello
# END

Python Memcached client:

# pip install pymemcache
from pymemcache.client.base import Client

client = Client(('localhost', 11211))
client.set('key', 'value')
print(client.get('key'))

Multi-Threaded Architecture and Tuning

Dragonfly uses a shared-nothing threading model where each thread owns a shard:

# Check number of threads Dragonfly is using
# Dragonfly automatically uses all available CPU cores

# To limit threads (e.g., to avoid starving other services)
/usr/local/bin/dragonfly --num_shards 8   # Use 8 shards instead of default (auto)

# Check thread/shard info
redis-cli info server | grep -i thread
redis-cli info server | grep -i shard

# Enable io_uring for better I/O performance (Linux 5.11+)
/usr/local/bin/dragonfly --force_epoll=false  # Allow io_uring (default behavior)

# Check which I/O model is active
redis-cli info server | grep -i uring

Kernel tuning for high performance:

# Disable transparent huge pages (reduces latency variance)
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

# Increase network backlog
echo "net.core.somaxconn = 65536" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.tcp_max_syn_backlog = 65536" | sudo tee -a /etc/sysctl.conf

# Allow memory locking
echo "dragonfly soft memlock unlimited" | sudo tee -a /etc/security/limits.conf
echo "dragonfly hard memlock unlimited" | sudo tee -a /etc/security/limits.conf

sudo sysctl -p

Memory Management

# Set memory limit
/usr/local/bin/dragonfly --maxmemory 8gb

# Set eviction policy (same as Redis)
redis-cli config set maxmemory-policy allkeys-lru

# Available policies:
# noeviction (default) - return errors when memory full
# allkeys-lru          - evict least recently used keys
# volatile-lru         - evict LRU among keys with TTL
# allkeys-random       - evict random keys
# volatile-random      - evict random keys with TTL
# volatile-ttl         - evict keys with shortest TTL

# Check memory usage
redis-cli info memory | grep -E "used_memory_human|maxmemory_human|mem_fragmentation"

# Check eviction stats
redis-cli info stats | grep evicted

Persistence and Snapshots

# Configure RDB snapshots
/usr/local/bin/dragonfly \
  --dir /var/lib/dragonfly \
  --dbfilename dump \
  --save_schedule "*:30"  # Save every hour at :30 minutes

# Trigger manual snapshot
redis-cli bgsave

# Check snapshot status
redis-cli lastsave

# Verify snapshot file
ls -lh /var/lib/dragonfly/dump.rdb

# For append-only file (AOF) - Dragonfly uses a journal
/usr/local/bin/dragonfly \
  --aof_file dragonfly.aof \
  --aof_rewrite_min_size 64mb

Benchmarking Dragonfly

# Install redis-benchmark
sudo apt install -y redis-tools

# Benchmark Dragonfly with 100 concurrent connections, 1M operations
redis-benchmark \
  -h localhost \
  -p 6379 \
  -a your-password \
  -c 100 \
  -n 1000000 \
  --threads 4 \
  -q

# Compare with Redis (if available on different port)
redis-benchmark -h localhost -p 6380 -c 100 -n 1000000 --threads 4 -q

# Benchmark specific commands
redis-benchmark -h localhost -p 6379 -c 50 -n 500000 -t set,get,lpush,lpop -q

# Test with large payloads
redis-benchmark -h localhost -p 6379 -c 50 -n 100000 -d 1024 -q

Troubleshooting

Dragonfly fails to start with "failed to lock memory":

# Check memlock limit
ulimit -l

# Set unlimited in service file:
# LimitMEMLOCK=infinity

# Or run without memory locking
/usr/local/bin/dragonfly --nolock_memory

High memory usage:

# Check if maxmemory is set
redis-cli config get maxmemory

# Set from CLI (takes effect immediately)
redis-cli config set maxmemory 4gb

# Set eviction policy to prevent out-of-memory
redis-cli config set maxmemory-policy allkeys-lru

Client connection refused:

# Check if Dragonfly is listening
ss -tlnp | grep 6379

# Check binding address
journalctl -u dragonfly -n 30 | grep -i bind

# Verify firewall rules
sudo ufw status

Performance lower than expected:

# Check number of shards (should be close to CPU count)
redis-cli info server | grep -i shard

# Increase shards to match CPU count
# --num_shards $(nproc)

# Verify no CPU throttling
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# Should be "performance" for best results
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Conclusion

Dragonfly's shared-nothing multi-threaded architecture delivers exceptional throughput for memory-intensive workloads, making it ideal for caching, session storage, and real-time analytics on multi-core servers. It is a true drop-in replacement for both Redis and Memcached clients, requiring no application code changes. Start with the binary distribution, set maxmemory to leave headroom for the OS, and benchmark against your real workload to confirm the performance gains.