Apache ZooKeeper is a coordination service that keeps a small, strongly consistent tree of data replicated across several servers. Distributed systems such as HBase, Solr, Hadoop and older Kafka releases use it for leader election, configuration and locks. In this tutorial you will deploy a three-node ZooKeeper ensemble on Ubuntu 24.04, run it as a systemd service, confirm that the nodes elect a leader, and store and protect data with the ZooKeeper CLI.

Prerequisites

To follow this tutorial, you will need:

  • Three servers running Ubuntu 24.04 LTS, for example CubePath VPS instances, connected through a private network. Each needs at least 2 GB of RAM and fast local storage (SSD or NVMe), because ZooKeeper writes every change to disk before acknowledging it.
  • A non-root user with sudo privileges on every server.
  • UFW enabled with SSH allowed.

An ensemble needs a majority of its nodes (a quorum) to work, so always use an odd number: three nodes survive the loss of one, five survive the loss of two. The examples use these addresses:

NodePrivate IPServer ID
zk110.0.0.211
zk210.0.0.222
zk310.0.0.233

Run Steps 1 to 5 on all three nodes unless a step says otherwise.

Step 1 - Installing Java

ZooKeeper 3.9 runs on Java 11 or 17. Install the headless OpenJDK 17 runtime:

sudo apt update
sudo apt install openjdk-17-jre-headless

Verify the version:

java -version

openjdk version "17.0.16" 2025-07-15
OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)

Step 2 - Downloading ZooKeeper

Create a dedicated system user that will own the data and run the service:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin zookeeper

Check the ZooKeeper releases page for the current stable 3.9.x version and set it in a variable. The Apache archive keeps every release at a permanent URL:

ZK_VERSION=3.9.3
cd /tmp
curl -fLO "https://archive.apache.org/dist/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz"
curl -fLO "https://archive.apache.org/dist/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz.sha512"

Download the -bin archive: the one without -bin contains only the source code. Compare the checksum with the published value; both hashes must match:

sha512sum "apache-zookeeper-${ZK_VERSION}-bin.tar.gz"
cat "apache-zookeeper-${ZK_VERSION}-bin.tar.gz.sha512"

Extract it into /opt and create a version-independent symlink, which makes future upgrades a matter of switching the link:

sudo tar -xzf "apache-zookeeper-${ZK_VERSION}-bin.tar.gz" -C /opt
sudo ln -sfn "/opt/apache-zookeeper-${ZK_VERSION}-bin" /opt/zookeeper

Create the data and log directories and give them to the zookeeper user:

sudo mkdir -p /var/lib/zookeeper /var/log/zookeeper
sudo chown zookeeper:zookeeper /var/lib/zookeeper /var/log/zookeeper

Step 3 - Setting the server ID

Each node identifies itself with a number stored in a file called myid inside the data directory. This value is the only thing that differs between nodes. On zk1 run:

echo 1 | sudo -u zookeeper tee /var/lib/zookeeper/myid

On zk2 write 2, and on zk3 write 3. The ID must match the server.N line for that node's address in the next step.

Step 4 - Configuring the ensemble

Create the configuration file. It is identical on all three nodes:

sudo nano /opt/zookeeper/conf/zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5

dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=60

autopurge.snapRetainCount=5
autopurge.purgeInterval=24

admin.enableServer=true
admin.serverAddress=127.0.0.1
admin.serverPort=8080

4lw.commands.whitelist=ruok,srvr,stat,mntr,conf

server.1=10.0.0.21:2888:3888
server.2=10.0.0.22:2888:3888
server.3=10.0.0.23:2888:3888

What the settings mean:

  • tickTime is ZooKeeper's basic time unit in milliseconds. initLimit and syncLimit are measured in ticks: a follower has 20 seconds to sync with the leader at startup and 10 seconds to respond afterwards.
  • dataDir holds snapshots, transaction logs and myid.
  • autopurge.* keeps the five most recent snapshots and deletes older ones every 24 hours. Without it, the data directory grows until the disk fills.
  • The admin server is a small HTTP interface for status commands. Binding it to 127.0.0.1 keeps it off the network.
  • 4lw.commands.whitelist enables the four-letter commands used for health checks below. Everything not listed is refused.
  • server.N=host:2888:3888 lists every member. Followers connect to the leader on port 2888, and nodes use port 3888 for leader election.

Configuration files use = without spaces and do not support comments at the end of a line, so keep any comments on their own lines.

Open the ZooKeeper ports to the private network only. Port 2181 is for clients; 2888 and 3888 are for the other ensemble members:

sudo ufw allow from 10.0.0.0/24 to any port 2181 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 2888 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 3888 proto tcp

Step 5 - Creating the systemd service

Run ZooKeeper in the foreground under systemd so that systemd tracks the Java process directly and its log output goes to the journal:

sudo nano /etc/systemd/system/zookeeper.service

[Unit]
Description=Apache ZooKeeper
Documentation=https://zookeeper.apache.org
After=network-online.target
Wants=network-online.target

[Service]
User=zookeeper
Group=zookeeper
Environment=ZOO_LOG_DIR=/var/log/zookeeper
Environment=ZK_SERVER_HEAP=1024
ExecStart=/opt/zookeeper/bin/zkServer.sh start-foreground
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

ZK_SERVER_HEAP sets the Java heap in megabytes. Give ZooKeeper about half of the server's RAM, and never so much that the system starts swapping: a swapping ZooKeeper node falls behind and drops out of the quorum.

Load the unit and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now zookeeper

Repeat Steps 1 to 5 on the other nodes. The first node logs connection errors until a second node is up; that is expected, because it cannot form a quorum alone.

Step 6 - Verifying the ensemble

Once all three services are running, check the role of each node:

sudo -u zookeeper /opt/zookeeper/bin/zkServer.sh status

ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

Exactly one node must report Mode: leader and the other two Mode: follower. A node that shows an error here has not joined the quorum.

The four-letter commands give a quick health check over the client port. ruok answers imok when the server is running:

echo ruok | nc 10.0.0.21 2181; echo

imok

srvr shows the role and basic statistics of any node, which is useful to check the whole ensemble from one place:

for ip in 10.0.0.21 10.0.0.22 10.0.0.23; do
  printf '%s: ' "$ip"
  echo srvr | nc "$ip" 2181 | grep Mode
done

10.0.0.21: Mode: follower
10.0.0.22: Mode: leader
10.0.0.23: Mode: follower

To test failover, stop the leader with sudo systemctl stop zookeeper, run the loop again and confirm that one of the remaining nodes became leader. Start the stopped node again afterwards; it rejoins as a follower.

Step 7 - Working with znodes

ZooKeeper stores data in a tree of nodes called znodes, addressed by paths like a file system. Each znode holds a small payload (up to 1 MB, usually much less). Connect with the CLI, passing all three servers so the client fails over automatically:

/opt/zookeeper/bin/zkCli.sh -server 10.0.0.21:2181,10.0.0.22:2181,10.0.0.23:2181

Create a parent znode and a child with configuration data:

create /myapp ""
create /myapp/config "db_host=10.0.0.30"
get /myapp/config

db_host=10.0.0.30

Update the value and list the children:

set /myapp/config "db_host=10.0.0.31"
ls /myapp

[config]

Two special znode types are the building blocks of most coordination recipes:

  • Ephemeral znodes (create -e) are deleted automatically when the client session that created them ends. Services use them to announce that they are alive.
  • Sequential znodes (create -s) get a monotonically increasing suffix appended to their name. Queues and locks rely on that ordering.
create /myapp/workers ""
create -e /myapp/workers/worker-1 "10.0.0.40"
create -s /myapp/queue- "job-data"

Created /myapp/workers/worker-1
Created /myapp/queue-0000000003

If you quit the CLI with quit and reconnect, ls /myapp/workers returns an empty list: the ephemeral znode disappeared with the session. Because every node in the ensemble stores the same tree, you can also reconnect to a different server and see the same data.

Step 8 - Protecting znodes with ACLs

By default a new znode is readable and writable by any client that can reach port 2181. The digest scheme restricts a znode to a user and password. In zkCli.sh, authenticate the current session, then create a znode whose ACL grants all permissions to the authenticated user only:

addauth digest appuser:your_strong_password
create /secure "sensitive-data" auth::cdrwa
getAcl /secure

'digest,'appuser:Vh5bZ3tKZfC4q0zU+3QmYxJzxTg=
: cdrwa

The auth scheme stores the digest of the credentials used in addauth, so you never handle the hash yourself. The permission letters are c (create children), d (delete children), r (read), w (write) and a (admin, change the ACL).

Test it from a new session without authentication:

/opt/zookeeper/bin/zkCli.sh -server 10.0.0.21:2181 get /secure

The command fails with an Insufficient permission error. After addauth digest appuser:your_strong_password in an interactive session, get /secure returns the data again.

To make a znode readable by everyone but writable only by your application user, combine several ACL entries separated by commas:

setAcl /myapp/config world:anyone:r,auth::cdrwa

Troubleshooting

zkServer.sh status reports Error contacting service. It is probably not running: the service failed or the node has no quorum. Read sudo journalctl -u zookeeper -n 100 for the actual error.

The journal repeats Cannot open channel to 2 at election address /10.0.0.22:3888: that peer is down or port 3888 is blocked. Check the service on the peer and sudo ufw status on both nodes.

The node refuses to start with myid file is missing: /var/lib/zookeeper/myid does not exist or is not readable by the zookeeper user. Recreate it as in Step 3.

echo ruok | nc ... returns nothing and the log says is not executed because it is not in the whitelist: add the command to 4lw.commands.whitelist and restart the service.

Clients see frequent session expirations: the node is swapping or its disk is slow. Check free -h, lower ZK_SERVER_HEAP if swap is in use, and keep other disk-heavy workloads off ZooKeeper servers.

Conclusion

You now have a three-node ZooKeeper ensemble running as systemd services, with a working leader election, automatic snapshot cleanup, a locked-down admin interface and digest ACLs on sensitive znodes. As next steps, scrape the mntr metrics (or enable the built-in Prometheus metrics provider) to alert on latency and outstanding requests, enable TLS for client connections on a secureClientPort, and point your applications at the full connection string 10.0.0.21:2181,10.0.0.22:2181,10.0.0.23:2181.