--- title: "Grafana, Loki and Alloy Logging Stack on Ubuntu 24.04" description: "Build a logging stack on Ubuntu 24.04: Loki for storage, Alloy to collect and parse Nginx and journald logs, and Grafana for dashboards and alerts on log data." category: "Observability Stack" language: en source: https://cubepath.com/docs/observability-stack/grafana-loki-promtail-complete-setup updated: 2026-09-25 --- # Grafana, Loki and Alloy Logging Stack on Ubuntu 24.04 A centralized logging stack lets you search every server's logs from one place, graph error rates, and get alerted when something goes wrong. In this tutorial you will build one on a single Ubuntu 24.04 server with three components from Grafana Labs: **Loki** stores the logs, **Grafana Alloy** collects and parses them, and **Grafana** is the interface for searching, dashboards and alerts. You will parse Nginx access logs, build a dashboard with HTTP status codes, and create an alert that fires when 5xx errors spike. Earlier versions of this stack used Promtail as the collector. Promtail reached end of life in 2026 and Grafana Alloy is its replacement, so this guide uses Alloy and shows how to convert an existing Promtail configuration. ## Prerequisites To follow this tutorial you need: - A server running Ubuntu 24.04 LTS with at least 2 GB of RAM (4 GB is more comfortable), for example a CubePath VPS. - A non-root user with `sudo` privileges and UFW enabled with SSH allowed. - A domain name with an A record pointing to the server's public IP. This guide uses `your_domain` for it; Grafana will be served at `https://your_domain`. ## Step 1 - Adding the Grafana APT repository Loki, Alloy and Grafana are all published in Grafana's signed APT repository. Add its key and the repository: ```bash sudo apt update sudo apt install -y gpg wget curl jq sudo mkdir -p /etc/apt/keyrings wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list sudo apt update ``` Install the three packages, plus Nginx, which will act as the reverse proxy for Grafana and as the example log source: ```bash sudo apt install -y loki alloy grafana nginx ``` ## Step 2 - Configuring Loki The packaged Loki configuration keeps data in `/tmp`, which does not survive a reboot. Create a persistent data directory: ```bash sudo mkdir -p /var/lib/loki sudo chown loki:loki /var/lib/loki ``` Open the configuration file: ```bash sudo nano /etc/loki/config.yml ``` Replace its content with a single-node configuration that listens only on localhost, uses the TSDB index with schema v13, and deletes logs after 30 days: ```yaml auth_enabled: false server: http_listen_address: 127.0.0.1 http_listen_port: 3100 grpc_listen_address: 127.0.0.1 grpc_listen_port: 9096 common: instance_addr: 127.0.0.1 path_prefix: /var/lib/loki storage: filesystem: chunks_directory: /var/lib/loki/chunks rules_directory: /var/lib/loki/rules replication_factor: 1 ring: kvstore: store: inmemory schema_config: configs: - from: 2024-04-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h limits_config: retention_period: 720h compactor: working_directory: /var/lib/loki/compactor retention_enabled: true retention_delete_delay: 2h delete_request_store: filesystem analytics: reporting_enabled: false ``` Because Alloy and Grafana run on the same server, Loki does not need to be reachable from outside. For a detailed explanation of each block, see [How to Install Grafana Loki on Ubuntu 24.04](/docs/observability-stack/loki-installation-for-log-aggregation). Enable and restart Loki, then wait until it reports ready (about 15 seconds): ```bash sudo systemctl enable loki sudo systemctl restart loki curl -s http://127.0.0.1:3100/ready ``` ```output ready ``` ## Step 3 - Collecting logs with Grafana Alloy Alloy is configured with pipelines of components: sources read logs, processors transform them, and writers send them to Loki. You will build two pipelines: one for the systemd journal and one for Nginx access logs. Alloy runs as the `alloy` user. Give it read access to the journal and to `/var/log/nginx`, whose files belong to the `adm` group: ```bash sudo usermod -aG systemd-journal,adm alloy ``` Open the Alloy configuration: ```bash sudo nano /etc/alloy/config.alloy ``` Replace its content with the following: ```alloy // Where every pipeline sends its logs. loki.write "local" { endpoint { url = "http://127.0.0.1:3100/loki/api/v1/push" } } // Pipeline 1: systemd journal. loki.relabel "journal" { forward_to = [] rule { source_labels = ["__journal__systemd_unit"] target_label = "unit" } } loki.source.journal "system" { max_age = "12h" relabel_rules = loki.relabel.journal.rules labels = { job = "systemd-journal", host = constants.hostname } forward_to = [loki.write.local.receiver] } // Pipeline 2: Nginx access log. local.file_match "nginx" { path_targets = [{ "__path__" = "/var/log/nginx/access.log", "job" = "nginx", "host" = constants.hostname, }] } loki.source.file "nginx" { targets = local.file_match.nginx.targets forward_to = [loki.process.nginx.receiver] } loki.process "nginx" { // Drop load balancer health checks before they are stored. stage.drop { expression = `"GET /healthz ` drop_counter_reason = "healthcheck" } // Extract fields from the default "combined" log format. stage.regex { expression = `^(?P\S+) \S+ \S+ \[[^\]]+\] "(?P\S+) (?P\S+) [^"]*" (?P\d{3}) ` } // Attach the status code as structured metadata, not as an index label. stage.structured_metadata { values = { status = "" } } forward_to = [loki.write.local.receiver] } ``` How the Nginx pipeline works: - `local.file_match` defines which files to tail and the labels every line from them gets (`job` and `host`). `loki.source.file` reads them and remembers its position, so restarts do not duplicate lines. - `stage.drop` discards any line matching the regular expression. Here that is health check requests, which add volume but no information. - `stage.regex` parses each line with named capture groups into a temporary map of extracted fields. - `stage.structured_metadata` attaches the extracted `status` to each line as metadata. It can be filtered on at query time but does not create new streams. > Important: Labels create streams in Loki, and each stream has a cost. Keep labels to a handful of low-cardinality values (`job`, `host`, `unit`). Never make paths, client IPs or user IDs into labels; filter on them at query time instead. Check the syntax of the file. `alloy fmt` exits with an error pointing at the line if something is wrong: ```bash alloy fmt /etc/alloy/config.alloy > /dev/null && echo "syntax OK" ``` Enable and restart Alloy, then read its log: ```bash sudo systemctl enable alloy sudo systemctl restart alloy sudo journalctl -u alloy -n 30 --no-pager ``` There should be no `level=error` lines. Generate a few Nginx requests and confirm Loki has both jobs: ```bash for i in 1 2 3; do curl -s -o /dev/null http://localhost/; curl -s -o /dev/null http://localhost/missing; done curl -s http://127.0.0.1:3100/loki/api/v1/label/job/values | jq -c .data ``` ```output ["nginx","systemd-journal"] ``` ### Migrating an existing Promtail configuration If you already run Promtail, Alloy can translate its YAML configuration into the Alloy format: ```bash alloy convert --source-format=promtail --output=/tmp/promtail-converted.alloy /etc/promtail/config.yml ``` Review the output, merge it into `/etc/alloy/config.alloy`, and once Alloy is shipping logs, stop and disable Promtail with `sudo systemctl disable --now promtail` so lines are not sent twice. ## Step 4 - Configuring Grafana behind Nginx with HTTPS Grafana listens on port 3000 on all interfaces by default. Bind it to localhost and tell it its public URL. Open its configuration file: ```bash sudo nano /etc/grafana/grafana.ini ``` In the `[server]` section, set these three keys (remove the leading `;` that comments them out): ```ini [server] http_addr = 127.0.0.1 domain = your_domain root_url = https://your_domain/ ``` Add Loki as a data source through provisioning, so it exists as soon as Grafana starts and cannot be accidentally changed in the UI: ```bash sudo nano /etc/grafana/provisioning/datasources/loki.yaml ``` ```yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy url: http://127.0.0.1:3100 isDefault: true editable: false ``` Enable and start Grafana: ```bash sudo systemctl enable --now grafana-server curl -s http://127.0.0.1:3000/api/health | jq -r .database ``` ```output ok ``` Now create an Nginx server block that proxies to Grafana. Grafana Live uses WebSockets on `/api/live/`, which needs the `Upgrade` headers: ```bash sudo nano /etc/nginx/sites-available/grafana ``` ```nginx map $http_upgrade $connection_upgrade { default upgrade; '' close; } server { listen 80; listen [::]:80; server_name your_domain; location / { proxy_set_header Host $host; proxy_pass http://127.0.0.1:3000; } location /api/live/ { proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header Host $host; proxy_pass http://127.0.0.1:3000; } } ``` Enable the site, test the configuration and reload Nginx: ```bash sudo ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/ sudo nginx -t sudo systemctl reload nginx ``` Open HTTP and HTTPS in the firewall and request a Let's Encrypt certificate. Certbot edits the server block to add TLS and a redirect from HTTP: ```bash sudo ufw allow 'Nginx Full' sudo apt install -y certbot python3-certbot-nginx sudo certbot --nginx -d your_domain ``` Visit `https://your_domain`. Log in with the user `admin` and password `admin`; Grafana forces you to set a new password immediately. Choose a strong one. ## Step 5 - Exploring logs in Grafana Open **Explore** from the left menu. The Loki data source is already selected. Switch the query editor to **Code** mode and run: ```logql {job="nginx"} ``` You will see the requests you made in Step 3, including the Grafana traffic you are generating now. A few useful queries to try: Only server and client errors, using the structured metadata extracted by Alloy: ```logql {job="nginx"} | status >= 400 ``` Parse the line at query time with `pattern` to filter on fields you did not extract in Alloy, here the request path: ```logql {job="nginx"} | pattern ` - <_> [<_>] " <_>" <_>` | path = "/missing" ``` Failed SSH logins from the journal: ```logql {job="systemd-journal", unit="ssh.service"} |= "Failed password" ``` ## Step 6 - Building a dashboard Create a dashboard with two panels: request rate by status code, and the raw error lines. 1. Go to **Dashboards > New > New dashboard** and click **Add visualization**. Select the **Loki** data source. 2. In Code mode, enter the following query, then set the legend to `{{status}}`: ```logql sum by (status) (count_over_time({job="nginx"} [$__auto])) ``` 3. Keep the **Time series** visualization, title the panel `Nginx requests by status`, and go back to the dashboard. 4. Add a second visualization, choose the **Logs** visualization, and use this query: ```logql {job="nginx"} | status >= 500 ``` 5. Title it `5xx responses` and save the dashboard as `Nginx`. `count_over_time` turns log lines into a number per time window, and `sum by (status)` produces one series per status code. `$__auto` lets Grafana pick the window according to the zoom level. ## Step 7 - Alerting on log patterns Grafana can evaluate LogQL metric queries on a schedule and notify you when they cross a threshold. First create a contact point under **Alerting > Contact points**, for example email (which requires the `[smtp]` section of `grafana.ini` to be configured) or a Slack webhook. Use **Test** to confirm it delivers. Then create the rule under **Alerting > Alert rules > New alert rule**: 1. Name it `Nginx 5xx spike` and select the **Loki** data source. 2. Enter this query, which counts 5xx responses in the last 5 minutes: ```logql sum(count_over_time({job="nginx"} | status >= 500 [5m])) ``` 3. Set the threshold condition to **Is above 10**. 4. Create a folder and an evaluation group that evaluates every `1m`, and set a pending period of `5m` so a brief blip does not page anyone. 5. Select your contact point and save the rule. To test it, temporarily lower the threshold to 0 and request a URL that your application answers with a 5xx error. The rule moves to **Pending**, then **Firing**, and you receive a notification. Restore the threshold afterwards. ## Troubleshooting **Nginx logs do not appear in Loki.** Check `sudo journalctl -u alloy | grep -i nginx`. A `permission denied` error means the `adm` group change has not been applied; confirm with `id alloy` and restart Alloy. **Grafana shows "Data source connected, but no labels found".** Loki is up but has not received anything. Check that Alloy is running and that `curl -s http://127.0.0.1:3100/loki/api/v1/labels` lists labels. **`502 Bad Gateway` at `https://your_domain`.** Grafana is not running or not listening on `127.0.0.1:3000`. Check `sudo systemctl status grafana-server` and `sudo ss -tlnp | grep 3000`. **The `| status >= 400` filter returns nothing.** Structured metadata only exists on lines ingested after the pipeline was added. Older lines can be filtered with the `pattern` query shown in Step 5. Also confirm that your Nginx `log_format` is the default `combined`, or adjust the regular expression in `stage.regex`. **Loki logs `too many outstanding requests` in Grafana.** A dashboard is querying a very wide time range. Narrow the range or add more selective labels to the stream selector. ## Conclusion You now have a complete logging stack on Ubuntu 24.04: Alloy collects the journal and Nginx logs and parses the access log, Loki stores everything for 30 days, and Grafana, served over HTTPS, gives you searching, a dashboard and an alert on 5xx errors. To extend it, install Alloy on your other servers and point their `loki.write` at this Loki instance through a TLS reverse proxy with authentication, add pipelines for your application logs (the `stage.json` and `stage.logfmt` stages parse structured formats), and add Prometheus as a second data source to correlate metrics and logs in the same dashboards.