Grafana Alloy is Grafana's distribution of the OpenTelemetry Collector and the successor to Grafana Agent. It receives OTLP data from your applications, can also scrape Prometheus metrics and read logs itself, and forwards everything to the backends you choose. In this tutorial you will install Alloy on Ubuntu 24.04 from Grafana's APT repository, build a pipeline that accepts OTLP traces, metrics and logs and routes them to Tempo, Prometheus and Loki, add host metrics and journal logs, and verify the pipeline with a test span.
Prerequisites
To follow this tutorial you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS. Alloy itself needs little: 1 vCPU and 1 GB of RAM handle a moderate load.
- A non-root user with
sudoprivileges and UFW enabled with SSH allowed. - Running backends to send data to: Grafana Tempo (OTLP gRPC on port 4317), Prometheus with the remote write receiver enabled or Grafana Mimir, and Grafana Loki. This tutorial calls them
your_tempo_host,your_prometheus_hostandyour_loki_host. - Grafana connected to those three data sources, to look at the results.
Step 1 - Installing Alloy from the Grafana repository
Grafana publishes Alloy in the same APT repository as Grafana and Loki. Install the tool needed to convert the signing key, then store the key in /etc/apt/keyrings:
sudo apt update
sudo apt install gpg
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
Add the repository, signed by that key only:
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
Install Alloy:
sudo apt update
sudo apt install alloy
Check the installed version:
alloy --version
alloy, version v1.x.x (branch: HEAD, revision: ...)
The package creates an alloy system user, a systemd unit, the configuration file /etc/alloy/config.alloy and an environment file /etc/default/alloy that the unit reads. The default configuration only collects Alloy's own logs and does not listen for OTLP yet.
Step 2 - Understanding the configuration syntax
Alloy's configuration is made of components. Each component has a type, such as otelcol.receiver.otlp, a label you choose, and arguments. Components are connected by referencing each other's exports:
otelcol.processor.batch "default" {
output {
traces = [otelcol.exporter.otlp.tempo.input]
}
}
Here the batch processor sends traces to the input of the exporter labelled tempo. Alloy reads the whole file, builds a graph from these references, and evaluates it; the order of blocks in the file does not matter. The otelcol.* components wrap the OpenTelemetry Collector, while prometheus.* and loki.* components handle Prometheus metrics and Loki log streams natively.
Step 3 - Building the OTLP pipeline
The pipeline you will build looks like this:
otelcol.receiver.otlpaccepts OTLP over gRPC (4317) and HTTP (4318).otelcol.processor.memory_limiterrefuses data when Alloy is close to its memory limit, instead of crashing.otelcol.processor.batchgroups data into batches to reduce the number of outgoing requests.- Traces go to Tempo over OTLP; metrics are converted to Prometheus format and remote-written; logs are converted to Loki streams and pushed to Loki.
Back up the default configuration and open the file:
sudo cp /etc/alloy/config.alloy /etc/alloy/config.alloy.orig
sudo nano /etc/alloy/config.alloy
Replace its content with the following, adjusting the three backend addresses:
logging {
level = "info"
format = "logfmt"
}
// Receive OTLP from applications
otelcol.receiver.otlp "default" {
grpc {
endpoint = "0.0.0.0:4317"
}
http {
endpoint = "0.0.0.0:4318"
}
output {
metrics = [otelcol.processor.memory_limiter.default.input]
logs = [otelcol.processor.memory_limiter.default.input]
traces = [otelcol.processor.memory_limiter.default.input]
}
}
otelcol.processor.memory_limiter "default" {
check_interval = "1s"
limit_percentage = 75
spike_limit_percentage = 20
output {
metrics = [otelcol.processor.batch.default.input]
logs = [otelcol.processor.batch.default.input]
traces = [otelcol.processor.batch.default.input]
}
}
otelcol.processor.batch "default" {
output {
metrics = [otelcol.exporter.prometheus.default.input]
logs = [otelcol.exporter.loki.default.input]
traces = [otelcol.exporter.otlp.tempo.input]
}
}
// Traces to Tempo
otelcol.exporter.otlp "tempo" {
client {
endpoint = "your_tempo_host:4317"
tls {
insecure = true
}
}
}
// Metrics to Prometheus or Mimir
otelcol.exporter.prometheus "default" {
forward_to = [prometheus.remote_write.default.receiver]
}
prometheus.remote_write "default" {
endpoint {
url = "http://your_prometheus_host:9090/api/v1/write"
}
}
// Logs to Loki
otelcol.exporter.loki "default" {
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://your_loki_host:3100/loki/api/v1/push"
}
}
A few details worth knowing:
tls { insecure = true }sends plain gRPC to Tempo. Use it only on a private network; for a TLS endpoint, remove thetlsblock./api/v1/writeis Prometheus' remote write path, available when Prometheus runs with--web.enable-remote-write-receiver. For Mimir, usehttp://your_mimir_host:9009/api/v1/pushinstead.prometheus.remote_writekeeps a write-ahead log under/var/lib/alloy/data, so short backend outages do not lose metrics.
Before applying it, check the syntax. alloy fmt parses the file and prints it formatted, or prints the line and column of the first error:
alloy fmt /etc/alloy/config.alloy > /dev/null && echo "syntax ok"
syntax ok
Step 4 - Adding host metrics and journal logs
Alloy can collect data from the machine it runs on without a separate node exporter. prometheus.exporter.unix embeds the node exporter, and loki.source.journal reads the systemd journal. Append these blocks to /etc/alloy/config.alloy:
// Host metrics (embedded node exporter)
prometheus.exporter.unix "host" { }
prometheus.scrape "host" {
targets = prometheus.exporter.unix.host.targets
forward_to = [prometheus.remote_write.default.receiver]
scrape_interval = "30s"
}
// systemd journal, with the unit name as a label
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}
loki.source.journal "system" {
forward_to = [loki.write.default.receiver]
relabel_rules = loki.relabel.journal.rules
labels = {
job = "systemd-journal",
host = constants.hostname,
}
}
loki.relabel is used here only for its rules export, which is why its forward_to is empty. The alloy user must be allowed to read the journal:
sudo usermod -aG systemd-journal alloy
Check the syntax again with the same alloy fmt command.
Step 5 - Starting Alloy and opening the OTLP ports
Enable and start the service. If it was already running, restart it so the new group membership takes effect:
sudo systemctl enable alloy
sudo systemctl restart alloy
sudo systemctl status alloy
Loaded: loaded (/usr/lib/systemd/system/alloy.service; enabled; preset: enabled)
Active: active (running) since ...
If the service fails, the reason is in the journal, including the component and line that caused it:
sudo journalctl -u alloy -n 50 --no-pager
Confirm Alloy is listening on the OTLP ports:
sudo ss -tlnp | grep alloy
The output should include *:4317, *:4318 and 127.0.0.1:12345, the last one being Alloy's own HTTP server for the UI and metrics.
Only your application servers should be able to send telemetry. Allow the OTLP ports from their network only, replacing 10.0.0.0/24 with your private subnet:
sudo ufw allow from 10.0.0.0/24 to any port 4317 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 4318 proto tcp
Later configuration changes do not need a full restart. sudo systemctl reload alloy makes Alloy re-read the file, and it keeps the previous working configuration if the new one is invalid.
Step 6 - Sending a test span
You can test the trace path without instrumenting an application by posting a span in OTLP JSON format to the HTTP receiver. The start and end times are nanosecond Unix timestamps, taken from the current clock:
now=$(date +%s%N)
curl -s -X POST http://127.0.0.1:4318/v1/traces \
-H 'Content-Type: application/json' \
-d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"alloy-test"}}]},"scopeSpans":[{"spans":[{"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174","name":"test-span","kind":1,"startTimeUnixNano":"'"$now"'","endTimeUnixNano":"'"$((now + 5000000))"'"}]}]}]}'
echo
{"partialSuccess":{}}
An empty partialSuccess means the receiver accepted everything. Alloy exposes the collector's internal counters on its metrics endpoint, so you can confirm the span went through the receiver:
curl -s http://127.0.0.1:12345/metrics | grep '^otelcol_receiver_accepted_spans'
The counter shows a value of at least 1 for the otlp receiver. In Grafana, open Explore, select the Tempo data source and search for the service alloy-test; the test-span trace appears after the batch is flushed, a few seconds later. In the Loki data source, the query {job="systemd-journal", unit="alloy.service"} shows Alloy's own log lines, which proves the journal path works too.
For real applications, point the OpenTelemetry SDKs at this server with OTEL_EXPORTER_OTLP_ENDPOINT=http://your_server_ip:4317 (gRPC) or http://your_server_ip:4318 (HTTP/protobuf).
Step 7 - Inspecting pipelines in the Alloy UI
Alloy's UI shows every component, its health and its arguments, and draws the graph of how data flows. It listens on localhost only, so open an SSH tunnel from your workstation:
ssh -L 12345:127.0.0.1:12345 your_user@your_server_ip
Browse to http://localhost:12345. Each component is marked healthy or unhealthy; an exporter that cannot reach its backend shows the error here. The Graph page shows the pipeline from receiver to exporters, which is the quickest way to spot a component that is not wired to anything.
Optional: sending to Grafana Cloud instead
If you use Grafana Cloud rather than self-hosted backends, one OTLP exporter covers traces, metrics and logs. Put the credentials from your stack's OpenTelemetry configuration tile in the environment file:
sudo nano /etc/default/alloy
GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-your_region.grafana.net/otlp
GRAFANA_CLOUD_INSTANCE_ID=your_instance_id
GRAFANA_CLOUD_API_TOKEN=your_api_token
Then replace the three exporters in config.alloy with an authenticated otlphttp exporter, and point the batch processor's metrics, logs and traces outputs at otelcol.exporter.otlphttp.grafana_cloud.input:
otelcol.auth.basic "grafana_cloud" {
username = sys.env("GRAFANA_CLOUD_INSTANCE_ID")
password = sys.env("GRAFANA_CLOUD_API_TOKEN")
}
otelcol.exporter.otlphttp "grafana_cloud" {
client {
endpoint = sys.env("GRAFANA_CLOUD_OTLP_ENDPOINT")
auth = otelcol.auth.basic.grafana_cloud.handler
}
}
sys.env reads variables from the service environment, which keeps the token out of the configuration file. Restart Alloy after editing /etc/default/alloy, since environment files are only read at start.
Troubleshooting
Alloy fails to start after a change. Run alloy fmt /etc/alloy/config.alloy to catch syntax errors, and read journalctl -u alloy for evaluation errors such as a reference to a component label that does not exist.
Data is accepted but never reaches a backend. Open the UI and check the exporter components. For metrics, journalctl -u alloy | grep -i remote_write shows rejected writes; a 404 usually means Prometheus is running without the remote write receiver.
Journal logs are missing. Confirm the group membership with id alloy, and restart Alloy if you added the group after it started.
Conclusion
You installed Grafana Alloy, built an OTLP pipeline with memory protection and batching, routed traces to Tempo, metrics to Prometheus and logs to Loki, and added host metrics and journal logs from the collector itself. Because applications only talk OTLP to Alloy, you can change backends later without touching them. Next, instrument your services with OpenTelemetry SDKs, add prometheus.scrape blocks for existing exporters, and use otelcol.processor.tail_sampling to keep only slow or failed traces when volume grows.