Celery is a Python task queue: your application puts jobs on a message broker, and one or more worker processes pick them up and run them in the background. In this tutorial you will install Celery on Ubuntu 24.04 with Redis as the broker and result backend, write tasks with automatic retries, route slow tasks to a dedicated queue, schedule periodic jobs with Celery Beat, and run the worker and scheduler as systemd services. A separate step shows how to use RabbitMQ as the broker instead.
Prerequisites
To follow this guide you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 1 GB of RAM.
- A non-root user with
sudoprivileges. - Basic knowledge of Python.
Step 1 - Installing Redis
Redis will hold the queue of pending tasks (broker) and the return values of finished tasks (result backend). Install it from the Ubuntu repositories:
sudo apt update
sudo apt install redis-server
The package enables and starts the service, listening only on 127.0.0.1:6379. Confirm it answers:
redis-cli ping
PONG
If you want to use RabbitMQ as the broker instead, still install Redis for the result backend, and follow Step 7 after completing Step 3.
Step 2 - Creating the project and installing Celery
Create a dedicated system user and a project directory. The worker will run as this user, never as root:
sudo useradd --system --home-dir /opt/celeryapp --shell /usr/sbin/nologin celery
sudo mkdir -p /opt/celeryapp
sudo chown "$USER":"$USER" /opt/celeryapp
Install the Python virtual environment module and create a virtual environment for the project:
sudo apt install python3-venv
python3 -m venv /opt/celeryapp/venv
Install Celery with the Redis extras, which pulls in the redis client library:
/opt/celeryapp/venv/bin/pip install "celery[redis]"
Check the installed version:
/opt/celeryapp/venv/bin/celery --version
5.6.3 (recovery)
Step 3 - Writing the Celery app and tasks
A Celery project needs an application object that knows where the broker is, plus functions decorated as tasks. For a small project both can live in one module. Create it:
nano /opt/celeryapp/tasks.py
import time
import urllib.request
from celery import Celery
from celery.schedules import crontab
app = Celery(
"tasks",
broker="redis://localhost:6379/0",
backend="redis://localhost:6379/1",
)
app.conf.update(
task_serializer="json",
accept_content=["json"],
result_serializer="json",
timezone="UTC",
enable_utc=True,
result_expires=3600, # drop stored results after 1 hour
task_acks_late=True, # acknowledge only after the task finishes
worker_prefetch_multiplier=1, # fetch one task at a time per process
broker_connection_retry_on_startup=True,
)
@app.task
def add(x, y):
return x + y
@app.task
def slow_report(seconds):
time.sleep(seconds)
return f"report finished after {seconds}s"
@app.task(
autoretry_for=(OSError,),
retry_backoff=True,
retry_kwargs={"max_retries": 5},
)
def fetch_status(url):
with urllib.request.urlopen(url, timeout=10) as response:
return response.status
A few details matter here:
- The broker uses Redis database
0and results use database1, so you can inspect or flush them separately. task_acks_late=Truemeans a task is only removed from the queue once it completes, so a worker crash does not lose it. Tasks should therefore be safe to run twice.fetch_statusretries automatically on network errors (urllibraises subclasses ofOSError), with exponential backoff and at most five retries.
Step 4 - Running a worker and calling tasks
Start a worker in the foreground to test the setup. -A tasks tells Celery to load the app object from tasks.py:
cd /opt/celeryapp
venv/bin/celery -A tasks worker --loglevel=INFO --concurrency=2
The banner lists the broker, the result backend and the registered tasks:
[config]
.> app: tasks:0x7f...
.> transport: redis://localhost:6379/0
.> results: redis://localhost:6379/1
.> concurrency: 2 (prefork)
...
[tasks]
. tasks.add
. tasks.fetch_status
. tasks.slow_report
[... INFO/MainProcess] celery@your_hostname ready.
Leave it running and open a second SSH session. Send a task and wait for its result:
cd /opt/celeryapp
venv/bin/python -c 'from tasks import add; r = add.delay(4, 6); print(r.id, r.get(timeout=10))'
6a1c3f0e-6d0b-4a0e-9d1b-2f3c7c9d8e11 10
delay() puts the task on the queue and returns immediately with an AsyncResult; get() blocks until the worker stores the result. In a web application you would normally store the task ID and check the result later instead of blocking.
Check that the worker is reachable through the broker:
venv/bin/celery -A tasks inspect ping
-> celery@your_hostname: OK
pong
1 node online.
Stop the foreground worker with Ctrl+C before continuing.
Step 5 - Routing slow tasks to their own queue
By default every task goes to a queue named celery. If long reports share that queue with quick tasks, a burst of reports can delay everything else. Route slow_report to a separate queue by adding this block at the end of tasks.py:
nano /opt/celeryapp/tasks.py
app.conf.task_routes = {
"tasks.slow_report": {"queue": "reports"},
}
A worker only consumes the queues you give it with -Q. You can run one worker for each queue, or, on a small server, one worker that listens to both:
cd /opt/celeryapp
venv/bin/celery -A tasks worker --loglevel=INFO -Q celery,reports --concurrency=2
In the second session, send a report and confirm which queue the worker lists as active:
venv/bin/python -c 'from tasks import slow_report; print(slow_report.delay(3).get(timeout=10))'
venv/bin/celery -A tasks inspect active_queues
report finished after 3s
-> celery@your_hostname: OK
* {'name': 'celery', ...}
* {'name': 'reports', ...}
Stop the worker with Ctrl+C.
Step 6 - Scheduling periodic tasks with Celery Beat
Celery Beat is a separate process that sends tasks to the queue on a schedule; workers then execute them. Add a schedule at the end of tasks.py:
nano /opt/celeryapp/tasks.py
app.conf.beat_schedule = {
"add-every-30-seconds": {
"task": "tasks.add",
"schedule": 30.0,
"args": (2, 3),
},
"nightly-report": {
"task": "tasks.slow_report",
"schedule": crontab(hour=3, minute=0),
"args": (5,),
},
}
crontab() uses the timezone set in the app configuration (UTC here). Run exactly one Beat process per project; two Beat processes would send every scheduled task twice. You will start Beat as a service in the next step.
Step 7 - Using RabbitMQ as the broker (optional)
RabbitMQ is a dedicated message broker with durable queues, per-queue monitoring and clustering. It is a good choice when tasks must survive broker restarts or when many services share the broker. Install it from the Ubuntu repositories:
sudo apt install rabbitmq-server
Create a user and a virtual host for Celery, and give the user full permissions on that vhost only. Replace your_strong_password with a strong password:
sudo rabbitmqctl add_user celery 'your_strong_password'
sudo rabbitmqctl add_vhost celery_vhost
sudo rabbitmqctl set_permissions -p celery_vhost celery ".*" ".*" ".*"
Change the broker line in tasks.py. Results can stay in Redis:
app = Celery(
"tasks",
broker="amqp://celery:your_strong_password@localhost:5672/celery_vhost",
backend="redis://localhost:6379/1",
)
Celery already depends on the AMQP client library, so nothing else needs installing. Start the worker again as in Step 4; the banner now shows transport: amqp://celery:**@localhost:5672/celery_vhost. The reports queue from Step 5 appears in sudo rabbitmqctl list_queues -p celery_vhost once the worker has declared it.
In production, keep the broker credentials out of the source file, for example by reading them from an environment variable with os.environ["CELERY_BROKER_URL"].
Step 8 - Running the worker and Beat with systemd
Give the celery user ownership of the project, then create a unit for the worker:
sudo chown -R celery:celery /opt/celeryapp
sudo nano /etc/systemd/system/celery-worker.service
[Unit]
Description=Celery worker
After=network-online.target redis-server.service
Wants=network-online.target
[Service]
Type=simple
User=celery
Group=celery
WorkingDirectory=/opt/celeryapp
ExecStart=/opt/celeryapp/venv/bin/celery -A tasks worker --loglevel=INFO -Q celery,reports --concurrency=2
Restart=on-failure
RestartSec=5
KillSignal=SIGTERM
TimeoutStopSec=300
[Install]
WantedBy=multi-user.target
SIGTERM makes the worker stop taking new tasks and finish the running ones (warm shutdown); TimeoutStopSec gives long tasks up to five minutes to complete.
Create the unit for Beat. StateDirectory makes systemd create /var/lib/celery, where Beat stores the time each entry last ran:
sudo nano /etc/systemd/system/celery-beat.service
[Unit]
Description=Celery beat scheduler
After=network-online.target redis-server.service
Wants=network-online.target
[Service]
Type=simple
User=celery
Group=celery
WorkingDirectory=/opt/celeryapp
StateDirectory=celery
ExecStart=/opt/celeryapp/venv/bin/celery -A tasks beat --loglevel=INFO --schedule=/var/lib/celery/celerybeat-schedule
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
If you use RabbitMQ, replace redis-server.service with rabbitmq-server.service in both After= lines, and keep Redis if it holds your results.
Start both services:
sudo systemctl daemon-reload
sudo systemctl enable --now celery-worker celery-beat
Wait a minute and check that Beat is sending tasks.add every 30 seconds and the worker is running it:
sudo journalctl -u celery-worker --since "2 minutes ago" | grep tasks.add
... Task tasks.add[3b0e...] received
... Task tasks.add[3b0e...] succeeded in 0.0012s: 5
Whenever you change tasks.py, restart both services with sudo systemctl restart celery-worker celery-beat.
Step 9 - Monitoring with Flower (optional)
Flower is a web dashboard that shows workers, queues, and the state and runtime of each task. Install it into the same virtual environment:
sudo -u celery /opt/celeryapp/venv/bin/pip install flower
Start it bound to localhost only, with a username and password (replace your_strong_password):
cd /opt/celeryapp
sudo -u celery venv/bin/celery -A tasks flower --address=127.0.0.1 --port=5555 --basic-auth=admin:your_strong_password
From your local machine, open an SSH tunnel and browse to http://localhost:5555:
ssh -L 5555:127.0.0.1:5555 your_user@your_server_ip
The Workers tab should list celery@your_hostname as online, and the Tasks tab the scheduled tasks.add runs. To keep Flower running permanently, create a third unit like the worker's with this command in ExecStart, or put it behind a reverse proxy with HTTPS.
Troubleshooting
Received unregistered task of type 'tasks.x'. The worker was started before the task existed or from a different module. Restart the worker and check the[tasks]list in its banner.result.get()raisesTimeoutError. No worker consumes the task's queue. Check the routing and the-Qoption, and confirm the worker is online withcelery -A tasks inspect ping.Error 111 connecting to localhost:6379. Connection refused.Redis is not running:sudo systemctl status redis-server. For RabbitMQ,ACCESS_REFUSEDmeans wrong credentials or missing vhost permissions; list them withsudo rabbitmqctl list_permissions -p celery_vhost.- Scheduled tasks run twice. More than one Beat process is running, often a manual one plus the service. Check with
pgrep -af "celery.*beat". - Clearing a backlog.
celery -A tasks purge -Q celery,reportsdeletes all waiting tasks in those queues after asking for confirmation. It cannot be undone.
Conclusion
You now have a Celery setup on Ubuntu 24.04 with Redis (or RabbitMQ) as the broker, tasks with automatic retries, a dedicated queue for slow jobs, periodic tasks through Celery Beat, and both processes supervised by systemd. From here you can split tasks.py into a package as the project grows, scale out by running more workers on other servers against the same broker, or integrate the Celery app into your Django or Flask project.