Grafana Loki is a log aggregation system that indexes only a small set of labels per log stream instead of the full text, which makes it cheap to run and simple to operate. For years Promtail was the standard agent for shipping logs to Loki; Grafana deprecated it in 2025 in favor of Grafana Alloy, and Promtail no longer receives updates. In this tutorial you will install Loki on Ubuntu 24.04, collect systemd journal and file logs with Alloy, convert an existing Promtail configuration to Alloy, and explore your logs with LogQL in Grafana.
Prerequisites
To follow this guide you need:
- A server running Ubuntu 24.04 LTS with at least 2 GB of RAM, for example a CubePath VPS.
- A non-root user with
sudoprivileges. - UFW enabled, with SSH allowed.
In this guide Loki, Alloy and Grafana all run on the same server. To collect logs from more servers, install only Alloy on each of them and point it at the Loki server, as described in Step 3.
Step 1 - Adding the Grafana package repository
Loki, Alloy, Promtail and Grafana are all published in Grafana's APT repository. Download its signing key into the keyrings directory:
sudo apt update
sudo apt install -y gpg wget jq
sudo install -m 0755 -d /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
Add the repository and refresh the package index:
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
Confirm that the packages are available:
apt-cache policy loki alloy grafana | grep -E '^[a-z]|Candidate'
loki:
Candidate: 3.5.5
alloy:
Candidate: 1.10.2
grafana:
Candidate: 12.2.0
Your version numbers will be newer or different.
Step 2 - Installing and configuring Loki
Install Loki:
sudo apt install -y loki
The package creates a loki system user and a service that reads /etc/loki/config.yml. The default file stores data under /tmp, which is cleared on reboot, so replace it with a configuration that keeps data in /var/lib/loki. Create the directory first:
sudo mkdir -p /var/lib/loki
sudo chown loki:loki /var/lib/loki
Open the configuration file:
sudo nano /etc/loki/config.yml
Replace its content with:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
instance_addr: 127.0.0.1
path_prefix: /var/lib/loki
storage:
filesystem:
chunks_directory: /var/lib/loki/chunks
rules_directory: /var/lib/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 744h
compactor:
working_directory: /var/lib/loki/compactor
retention_enabled: true
delete_request_store: filesystem
This runs Loki as a single process (monolithic mode) with the local filesystem as storage:
auth_enabled: falsedisables multi-tenancy, so clients do not need to send a tenant ID.schema_configuses the TSDB index with schemav13, the current recommended format. Thefromdate must be in the past; never change an existing entry, only add new ones.retention_period: 744hkeeps logs for 31 days, and the compactor deletes older data becauseretention_enabledis set.
Restart Loki and enable it at boot:
sudo systemctl restart loki
sudo systemctl enable loki
Loki takes about 15 seconds to become ready. Check its readiness endpoint:
curl -s http://localhost:3100/ready
ready
If it answers Ingester not ready, wait a few seconds and try again. If the service failed, read sudo journalctl -u loki -n 50; YAML indentation mistakes are the most common cause.
WarningWith
auth_enabled: false, anyone who can reach port 3100 can read and write logs. Keep the port closed in UFW, and open it only to the IP addresses of servers that send logs, for examplesudo ufw allow from agent_ip to any port 3100 proto tcp.
Step 3 - Collecting logs with Grafana Alloy
Alloy is Grafana's collector for logs, metrics and traces. Its configuration is a set of connected components: some find or read logs, others process them, and a writer sends them to Loki. Install it:
sudo apt install -y alloy
Alloy runs as the alloy user. Add it to the adm group to read files in /var/log, and to systemd-journal to read the journal:
sudo usermod -aG adm,systemd-journal alloy
Open the Alloy configuration file:
sudo nano /etc/alloy/config.alloy
Replace its content with:
// Send everything to the local Loki
loki.write "local" {
endpoint {
url = "http://localhost:3100/loki/api/v1/push"
}
}
// Systemd journal: one stream per unit
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}
loki.source.journal "journal" {
forward_to = [loki.write.local.receiver]
relabel_rules = loki.relabel.journal.rules
labels = {job = "systemd-journal", host = constants.hostname}
}
// Plain log files
local.file_match "files" {
path_targets = [
{"__path__" = "/var/log/syslog", "job" = "syslog", "host" = constants.hostname},
{"__path__" = "/var/log/auth.log", "job" = "auth", "host" = constants.hostname},
]
}
loki.source.file "files" {
targets = local.file_match.files.targets
forward_to = [loki.write.local.receiver]
}
How this works:
loki.source.journalreads the systemd journal. Journal fields are exposed as hidden labels starting with__journal_, andloki.relabelcopies the systemd unit name into aunitlabel.local.file_matchlists the files to tail and their labels, andloki.source.filereads them. Alloy remembers the read position of each file across restarts.- Every source forwards to
loki.write.local.receiver, which pushes to Loki's API.
Keep labels to a few low-cardinality values such as job, host and unit. Values that change per line, such as user IDs or request paths, belong in the log line and are parsed at query time (Step 5); turning them into labels creates huge numbers of streams and slows Loki down.
Check the syntax, then restart Alloy:
sudo alloy fmt /etc/alloy/config.alloy > /dev/null && echo "syntax OK"
sudo systemctl restart alloy
sudo systemctl status alloy --no-pager
Confirm that Loki is receiving streams by asking for the label names it knows:
curl -s http://localhost:3100/loki/api/v1/labels | jq
{
"status": "success",
"data": [
"filename",
"host",
"job",
"service_name",
"unit"
]
}
Alloy also has a debugging UI on 127.0.0.1:12345 that shows each component and its health. To view it from your workstation, open an SSH tunnel with ssh -L 12345:localhost:12345 your_user@your_server_ip and browse to http://localhost:12345.
To ship logs from other servers, install Alloy on each one with Steps 1 and 3, and change the url in loki.write to http://your_loki_server_ip:3100/loki/api/v1/push.
Step 4 - Migrating an existing Promtail configuration
If you already run Promtail, Alloy can convert its configuration automatically. On the server running Promtail, install Alloy (Steps 1 and 3) and run:
sudo alloy convert --source-format=promtail --output=/etc/alloy/config.alloy /etc/promtail/config.yml
The command translates scrape_configs, relabel_configs, pipeline_stages and clients into the equivalent Alloy components and prints a warning for anything it cannot convert. Review the result with sudo nano /etc/alloy/config.alloy, then switch agents so the files are not shipped twice:
sudo systemctl disable --now promtail
sudo systemctl restart alloy
Alloy keeps its own read positions and does not reuse Promtail's positions.yaml, so it may send some recent lines again once. After confirming that logs are arriving, remove Promtail with sudo apt remove promtail.
Step 5 - Querying logs with LogQL
LogQL is Loki's query language. A query always starts with a stream selector in braces that picks streams by label, followed by optional filters and parsers. You can run queries from the shell against the API. Show the five most recent lines from the SSH service:
curl -G -s http://localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={job="systemd-journal", unit="ssh.service"}' \
--data-urlencode 'limit=5' | jq -r '.data.result[].values[][1]'
Accepted publickey for your_user from 203.0.113.10 port 51514 ssh2: ED25519 SHA256:...
pam_unix(sshd:session): session opened for user your_user(uid=1000) by your_user(uid=0)
...
Grafana, set up in the next step, is the usual place to write queries. Useful LogQL patterns:
| Query | What it does |
|---|---|
{job="auth"} |= "Failed password" | Lines containing a string (!= excludes, |~ uses a regex) |
{job="auth"} |= "Failed password" | pattern "<_> from <ip> port <_>" | Extracts ip from each line at query time |
{job="systemd-journal"} | json | level="error" | Parses JSON log lines and filters on a field |
sum by (unit) (count_over_time({job="systemd-journal"}[5m])) | Log lines per unit over 5-minute windows |
topk(5, sum by (ip) (count_over_time({job="auth"} |= "Failed password" | pattern "<_> from <ip> port <_>" [1h]))) | Top 5 IPs with failed SSH logins in the last hour |
The last two are metric queries: they turn log lines into numbers you can graph or alert on.
Step 6 - Visualizing logs in Grafana
Install Grafana and start it:
sudo apt install -y grafana
sudo systemctl enable --now grafana-server
Instead of adding the Loki data source by hand, provision it from a file so it survives reinstalls. Create the file:
sudo nano /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
isDefault: true
Restart Grafana to load it, and allow access to port 3000 from your IP only, replacing your_ip:
sudo systemctl restart grafana-server
sudo ufw allow from your_ip to any port 3000 proto tcp
Open http://your_server_ip:3000, log in as admin with the password admin, and set a new password when prompted. Then:
- Open Explore, which already has the Loki data source selected.
- Switch the query editor to Code and enter
{job="auth"}, then click Run query. You see the log lines and a histogram of their volume. - Try the metric query
sum by (unit) (count_over_time({job="systemd-journal"}[5m]))and switch the visualization to a time series.
To keep a view, click Add to dashboard from Explore. A dashboard with a Logs panel for {job="auth"} next to a time series of failed logins gives a quick security overview of the server.
Troubleshooting
Loki fails to start with mkdir ... permission denied: the loki user cannot write to the paths in the configuration. Check that /var/lib/loki exists and is owned by loki:loki.
Alloy runs but no logs arrive: open the Alloy UI or run sudo journalctl -u alloy -n 50. permission denied on /var/log files or the journal means the group change in Step 3 has not taken effect; restart Alloy after usermod.
Grafana shows No data: widen the time range, check that the labels in your selector exist with curl -s localhost:3100/loki/api/v1/labels, and confirm the data source test succeeds under Connections > Data sources > Loki.
Loki returns too many outstanding requests or queries are slow: the stream selector is too broad or there are too many streams. Narrow the selector with more labels and check that you did not turn a high-cardinality value into a label.
Conclusion
Loki now stores logs from the systemd journal and your log files, Alloy ships them in place of the deprecated Promtail, and Grafana lets you search them and turn them into graphs with LogQL. As next steps, install Alloy on your other servers, add Grafana alert rules on metric queries such as failed login counts, and move Loki's storage to S3-compatible object storage when your log volume outgrows a single disk.
