A centralized logging stack lets you search every server's logs from one place, graph error rates, and get alerted when something goes wrong. In this tutorial you will build one on a single Ubuntu 24.04 server with three components from Grafana Labs: Loki stores the logs, Grafana Alloy collects and parses them, and Grafana is the interface for searching, dashboards and alerts. You will parse Nginx access logs, build a dashboard with HTTP status codes, and create an alert that fires when 5xx errors spike.

Earlier versions of this stack used Promtail as the collector. Promtail reached end of life in 2026 and Grafana Alloy is its replacement, so this guide uses Alloy and shows how to convert an existing Promtail configuration.

Prerequisites

To follow this tutorial you need:

  • A server running Ubuntu 24.04 LTS with at least 2 GB of RAM (4 GB is more comfortable), for example a CubePath VPS.
  • A non-root user with sudo privileges and UFW enabled with SSH allowed.
  • A domain name with an A record pointing to the server's public IP. This guide uses your_domain for it; Grafana will be served at https://your_domain.

Step 1 - Adding the Grafana APT repository

Loki, Alloy and Grafana are all published in Grafana's signed APT repository. Add its key and the repository:

sudo apt update
sudo apt install -y gpg wget curl jq
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update

Install the three packages, plus Nginx, which will act as the reverse proxy for Grafana and as the example log source:

sudo apt install -y loki alloy grafana nginx

Step 2 - Configuring Loki

The packaged Loki configuration keeps data in /tmp, which does not survive a reboot. Create a persistent data directory:

sudo mkdir -p /var/lib/loki
sudo chown loki:loki /var/lib/loki

Open the configuration file:

sudo nano /etc/loki/config.yml

Replace its content with a single-node configuration that listens only on localhost, uses the TSDB index with schema v13, and deletes logs after 30 days:

auth_enabled: false

server:
  http_listen_address: 127.0.0.1
  http_listen_port: 3100
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 720h

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

analytics:
  reporting_enabled: false

Because Alloy and Grafana run on the same server, Loki does not need to be reachable from outside. For a detailed explanation of each block, see How to Install Grafana Loki on Ubuntu 24.04.

Enable and restart Loki, then wait until it reports ready (about 15 seconds):

sudo systemctl enable loki
sudo systemctl restart loki
curl -s http://127.0.0.1:3100/ready

ready

Step 3 - Collecting logs with Grafana Alloy

Alloy is configured with pipelines of components: sources read logs, processors transform them, and writers send them to Loki. You will build two pipelines: one for the systemd journal and one for Nginx access logs.

Alloy runs as the alloy user. Give it read access to the journal and to /var/log/nginx, whose files belong to the adm group:

sudo usermod -aG systemd-journal,adm alloy

Open the Alloy configuration:

sudo nano /etc/alloy/config.alloy

Replace its content with the following:

// Where every pipeline sends its logs.
loki.write "local" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
  }
}

// Pipeline 1: systemd journal.
loki.relabel "journal" {
  forward_to = []

  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

loki.source.journal "system" {
  max_age       = "12h"
  relabel_rules = loki.relabel.journal.rules
  labels        = { job = "systemd-journal", host = constants.hostname }
  forward_to    = [loki.write.local.receiver]
}

// Pipeline 2: Nginx access log.
local.file_match "nginx" {
  path_targets = [{
    "__path__" = "/var/log/nginx/access.log",
    "job"      = "nginx",
    "host"     = constants.hostname,
  }]
}

loki.source.file "nginx" {
  targets    = local.file_match.nginx.targets
  forward_to = [loki.process.nginx.receiver]
}

loki.process "nginx" {
  // Drop load balancer health checks before they are stored.
  stage.drop {
    expression          = `"GET /healthz `
    drop_counter_reason = "healthcheck"
  }

  // Extract fields from the default "combined" log format.
  stage.regex {
    expression = `^(?P<remote_addr>\S+) \S+ \S+ \[[^\]]+\] "(?P<method>\S+) (?P<path>\S+) [^"]*" (?P<status>\d{3}) `
  }

  // Attach the status code as structured metadata, not as an index label.
  stage.structured_metadata {
    values = { status = "" }
  }

  forward_to = [loki.write.local.receiver]
}

How the Nginx pipeline works:

  • local.file_match defines which files to tail and the labels every line from them gets (job and host). loki.source.file reads them and remembers its position, so restarts do not duplicate lines.
  • stage.drop discards any line matching the regular expression. Here that is health check requests, which add volume but no information.
  • stage.regex parses each line with named capture groups into a temporary map of extracted fields.
  • stage.structured_metadata attaches the extracted status to each line as metadata. It can be filtered on at query time but does not create new streams.

Check the syntax of the file. alloy fmt exits with an error pointing at the line if something is wrong:

alloy fmt /etc/alloy/config.alloy > /dev/null && echo "syntax OK"

Enable and restart Alloy, then read its log:

sudo systemctl enable alloy
sudo systemctl restart alloy
sudo journalctl -u alloy -n 30 --no-pager

There should be no level=error lines. Generate a few Nginx requests and confirm Loki has both jobs:

for i in 1 2 3; do curl -s -o /dev/null http://localhost/; curl -s -o /dev/null http://localhost/missing; done
curl -s http://127.0.0.1:3100/loki/api/v1/label/job/values | jq -c .data

["nginx","systemd-journal"]

Migrating an existing Promtail configuration

If you already run Promtail, Alloy can translate its YAML configuration into the Alloy format:

alloy convert --source-format=promtail --output=/tmp/promtail-converted.alloy /etc/promtail/config.yml

Review the output, merge it into /etc/alloy/config.alloy, and once Alloy is shipping logs, stop and disable Promtail with sudo systemctl disable --now promtail so lines are not sent twice.

Step 4 - Configuring Grafana behind Nginx with HTTPS

Grafana listens on port 3000 on all interfaces by default. Bind it to localhost and tell it its public URL. Open its configuration file:

sudo nano /etc/grafana/grafana.ini

In the [server] section, set these three keys (remove the leading ; that comments them out):

[server]
http_addr = 127.0.0.1
domain = your_domain
root_url = https://your_domain/

Add Loki as a data source through provisioning, so it exists as soon as Grafana starts and cannot be accidentally changed in the UI:

sudo nano /etc/grafana/provisioning/datasources/loki.yaml

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://127.0.0.1:3100
    isDefault: true
    editable: false

Enable and start Grafana:

sudo systemctl enable --now grafana-server
curl -s http://127.0.0.1:3000/api/health | jq -r .database

ok

Now create an Nginx server block that proxies to Grafana. Grafana Live uses WebSockets on /api/live/, which needs the Upgrade headers:

sudo nano /etc/nginx/sites-available/grafana

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name your_domain;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://127.0.0.1:3000;
    }

    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_pass http://127.0.0.1:3000;
    }
}

Enable the site, test the configuration and reload Nginx:

sudo ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Open HTTP and HTTPS in the firewall and request a Let's Encrypt certificate. Certbot edits the server block to add TLS and a redirect from HTTP:

sudo ufw allow 'Nginx Full'
sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your_domain

Visit https://your_domain. Log in with the user admin and password admin; Grafana forces you to set a new password immediately. Choose a strong one.

Step 5 - Exploring logs in Grafana

Open Explore from the left menu. The Loki data source is already selected. Switch the query editor to Code mode and run:

{job="nginx"}

You will see the requests you made in Step 3, including the Grafana traffic you are generating now. A few useful queries to try:

Only server and client errors, using the structured metadata extracted by Alloy:

{job="nginx"} | status >= 400

Parse the line at query time with pattern to filter on fields you did not extract in Alloy, here the request path:

{job="nginx"} | pattern `<ip> - <_> [<_>] "<method> <path> <_>" <code> <_>` | path = "/missing"

Failed SSH logins from the journal:

{job="systemd-journal", unit="ssh.service"} |= "Failed password"

Step 6 - Building a dashboard

Create a dashboard with two panels: request rate by status code, and the raw error lines.

  1. Go to Dashboards > New > New dashboard and click Add visualization. Select the Loki data source.
  2. In Code mode, enter the following query, then set the legend to {{status}}:
sum by (status) (count_over_time({job="nginx"} [$__auto]))
  1. Keep the Time series visualization, title the panel Nginx requests by status, and go back to the dashboard.
  2. Add a second visualization, choose the Logs visualization, and use this query:
{job="nginx"} | status >= 500
  1. Title it 5xx responses and save the dashboard as Nginx.

count_over_time turns log lines into a number per time window, and sum by (status) produces one series per status code. $__auto lets Grafana pick the window according to the zoom level.

Step 7 - Alerting on log patterns

Grafana can evaluate LogQL metric queries on a schedule and notify you when they cross a threshold. First create a contact point under Alerting > Contact points, for example email (which requires the [smtp] section of grafana.ini to be configured) or a Slack webhook. Use Test to confirm it delivers.

Then create the rule under Alerting > Alert rules > New alert rule:

  1. Name it Nginx 5xx spike and select the Loki data source.
  2. Enter this query, which counts 5xx responses in the last 5 minutes:
sum(count_over_time({job="nginx"} | status >= 500 [5m]))
  1. Set the threshold condition to Is above 10.
  2. Create a folder and an evaluation group that evaluates every 1m, and set a pending period of 5m so a brief blip does not page anyone.
  3. Select your contact point and save the rule.

To test it, temporarily lower the threshold to 0 and request a URL that your application answers with a 5xx error. The rule moves to Pending, then Firing, and you receive a notification. Restore the threshold afterwards.

Troubleshooting

Nginx logs do not appear in Loki. Check sudo journalctl -u alloy | grep -i nginx. A permission denied error means the adm group change has not been applied; confirm with id alloy and restart Alloy.

Grafana shows "Data source connected, but no labels found". Loki is up but has not received anything. Check that Alloy is running and that curl -s http://127.0.0.1:3100/loki/api/v1/labels lists labels.

502 Bad Gateway at https://your_domain. Grafana is not running or not listening on 127.0.0.1:3000. Check sudo systemctl status grafana-server and sudo ss -tlnp | grep 3000.

The | status >= 400 filter returns nothing. Structured metadata only exists on lines ingested after the pipeline was added. Older lines can be filtered with the pattern query shown in Step 5. Also confirm that your Nginx log_format is the default combined, or adjust the regular expression in stage.regex.

Loki logs too many outstanding requests in Grafana. A dashboard is querying a very wide time range. Narrow the range or add more selective labels to the stream selector.

Conclusion

You now have a complete logging stack on Ubuntu 24.04: Alloy collects the journal and Nginx logs and parses the access log, Loki stores everything for 30 days, and Grafana, served over HTTPS, gives you searching, a dashboard and an alert on 5xx errors.

To extend it, install Alloy on your other servers and point their loki.write at this Loki instance through a TLS reverse proxy with authentication, add pipelines for your application logs (the stage.json and stage.logfmt stages parse structured formats), and add Prometheus as a second data source to correlate metrics and logs in the same dashboards.