Grafana Loki and Promtail Complete Setup
Building a complete logging stack with Grafana, Loki, and Promtail provides comprehensive log aggregation and visualization. This guide covers deploying the entire stack, configuring complex pipeline stages, integrating with Grafana, and setting up log-based alerting for production environments.
Table of Contents
- Overview
- Architecture
- System Preparation
- Loki Server Setup
- Promtail Configuration
- Advanced Pipeline Stages
- Grafana Integration
- Creating Log Dashboards
- Alerting on Logs
- Performance Optimization
- Troubleshooting
- Conclusion
Overview
A complete logging stack captures, processes, stores, and visualizes logs from all infrastructure components. Loki's label-based architecture reduces storage costs while Promtail's flexible configuration handles diverse log sources. Grafana provides unified visualization across metrics and logs.
Architecture
Stack Components
┌─────────────────────────────────────────────┐
│ Application Logs / Syslog / Files │
└───────────────────┬─────────────────────────┘
│
┌─────────▼─────────┐
│ Promtail │
│ - Collection │
│ - Parsing │
│ - Labeling │
└─────────┬─────────┘
│
┌─────────▼─────────┐
│ Loki Server │
│ - Ingestion │
│ - Indexing │
│ - Storage │
└─────────┬─────────┘
│
┌────────────┼────────────┐
│ │ │
BoltDB S3/GCS Cassandra
│
┌─────────▼─────────┐
│ Grafana │
│ - Visualization │
│ - Alerting │
└───────────────────┘
System Preparation
Prerequisites
# System updates
sudo apt-get update && sudo apt-get upgrade -y
# Install dependencies
sudo apt-get install -y \
curl wget unzip \
git gcc make \
openssl ca-certificates
# Create logging user
sudo useradd --no-create-home --shell /bin/false logging
Directory Structure
# Create directories
sudo mkdir -p /opt/logging/{loki,promtail,grafana}
sudo mkdir -p /var/lib/loki/{chunks,index,cache}
sudo mkdir -p /var/log/loki
sudo mkdir -p /etc/loki /etc/promtail
# Set permissions
sudo chown -R logging:logging /opt/logging
sudo chown -R logging:logging /var/lib/loki
sudo chown -R logging:logging /var/log/loki
Loki Server Setup
Installation
cd /tmp
wget https://github.com/grafana/loki/releases/download/v2.9.0/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki
sudo chmod +x /usr/local/bin/loki
Production Configuration
sudo tee /etc/loki/loki-config.yml > /dev/null << 'EOF'
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: info
log_format: json
timeout_shutdown_request: 10s
ingester:
chunk_idle_period: 3m
chunk_retain_period: 1m
max_chunk_age: 2h
chunk_encoding: snappy
chunk_size_target: 1048576
chunk_size_bytes: 1572864
max_streams_utilization_factor: 2.0
max_stream_entries_limit: 10000
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
num_tokens: 128
heartbeat_timeout: 5m
term_timeout: 10m
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_line_size: 2097152
ingestion_rate_mb: 100
ingestion_burst_size_mb: 200
max_entries_limit_per_second: 10000
max_global_streams_matched_per_user: 10000
retention_period: 720h
cardinality_limit: 100000
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: loki_index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /var/lib/loki/index
shared_store: filesystem
cache_location: /var/lib/loki/cache
shared_store_key_prefix: index/
filesystem:
directory: /var/lib/loki/chunks
chunk_store_config:
max_look_back_period: 0s
chunk_cache_config:
cache:
enable_fifocache: true
default_validity: 1h
memcache:
batch_size: 1024
parallelism: 100
table_manager:
retention_deletes_enabled: true
retention_period: 720h
poll_interval: 10m
creation_grace_period: 10m
query_range:
align_queries_with_step: true
cache_results: true
results_cache:
cache:
enable_fifocache: true
default_validity: 1h
loki:
auth_enabled: false
tracing:
enabled: false
metrics:
enabled: false
EOF
sudo chown logging:logging /etc/loki/loki-config.yml
Systemd Service
sudo tee /etc/systemd/system/loki.service > /dev/null << 'EOF'
[Unit]
Description=Grafana Loki
Documentation=https://grafana.com/loki
After=network.target
[Service]
User=logging
Group=logging
Type=simple
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki-config.yml
Restart=on-failure
RestartSec=5
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=loki
# Resource limits
LimitNOFILE=65536
LimitNPROC=65536
# Security
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki
Promtail Configuration
Base Configuration
sudo tee /etc/promtail/promtail-config.yml > /dev/null << 'EOF'
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /var/lib/loki/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: syslog
host: __HOSTNAME__
__path__: /var/log/{syslog,messages}
- job_name: kernel
static_configs:
- targets:
- localhost
labels:
job: kernel
__path__: /var/log/kern.log
- job_name: auth
static_configs:
- targets:
- localhost
labels:
job: auth
__path__: /var/log/auth.log
- job_name: docker
static_configs:
- targets:
- localhost
labels:
job: docker
__path__: /var/lib/docker/containers/*/*-json.log
pipeline_stages:
- json:
expressions:
output: log
stream: stream
attrs_status: attrs.status
- output:
source: output
- job_name: nginx
static_configs:
- targets:
- localhost
labels:
job: nginx
__path__: /var/log/nginx/*.log
pipeline_stages:
- multiline:
line_start_pattern: '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
- regex:
expression: '^(?P<remote>[\w\.]+) (?P<host>[\w\.]+) (?P<user>[\w\-\.]+) \[(?P<timestamp>[\w:/]+\s[+\-]\d{4})\] "(?P<method>\w+) (?P<path>[^\s]+) (?P<protocol>[\w/\.]+)" (?P<status>\d+|-) (?P<bytes>\d+|-)\s?"?(?P<referer>[^\s]*)"?\s?"?(?P<agent>[^"]*)"?'
- timestamp:
source: timestamp
format: '02/Jan/2006:15:04:05 -0700'
- labels:
status:
method:
path:
- metrics:
nginx_http_requests_total:
type: Counter
description: "Total HTTP requests"
prefix: "nginx_"
max_idle_duration: 30s
match_all: true
action: add
nginx_http_request_duration_seconds:
type: Histogram
description: "Request duration"
source: response_time
prefix: "nginx_"
buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
max_idle_duration: 30s
- job_name: application
static_configs:
- targets:
- localhost
labels:
job: app
env: production
__path__: /var/log/app/*.log
pipeline_stages:
- json:
expressions:
timestamp: timestamp
level: level
msg: message
trace_id: trace_id
user_id: user_id
request_path: request.path
status_code: request.status_code
response_time: response_time_ms
- timestamp:
source: timestamp
format: 2006-01-02T15:04:05Z07:00
- labels:
level:
trace_id:
user_id:
- drop:
expression: '.*health_check.*'
- metrics:
app_errors_total:
type: Counter
description: "Total errors"
match_all: true
action: add
value: '1'
condition:
selector: '{level="error"}'
app_request_duration_seconds:
type: Histogram
description: "Request duration"
source: response_time
buckets: [10, 50, 100, 250, 500, 1000, 2500, 5000]
EOF
sudo chown logging:logging /etc/promtail/promtail-config.yml
Promtail Service
# Download Promtail
cd /tmp
wget https://github.com/grafana/loki/releases/download/v2.9.0/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail
# Create systemd service
sudo tee /etc/systemd/system/promtail.service > /dev/null << 'EOF'
[Unit]
Description=Grafana Promtail
Documentation=https://grafana.com/loki
After=network.target loki.service
[Service]
User=logging
Group=logging
Type=simple
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail-config.yml
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=promtail
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable promtail
sudo systemctl start promtail
Advanced Pipeline Stages
Multiline Log Parsing
scrape_configs:
- job_name: java-app
static_configs:
- targets:
- localhost
labels:
job: java
__path__: /var/log/java-app/*.log
pipeline_stages:
- multiline:
line_start_pattern: '^\d{4}-\d{2}-\d{2}'
- regex:
expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<level>\w+) \[(?P<thread>[^\]]+)\] (?P<logger>[^\s]+)\s-\s(?P<message>.*)$'
- timestamp:
source: timestamp
format: '2006-01-02 15:04:05'
- labels:
level:
thread:
logger:
Complex JSON Parsing
scrape_configs:
- job_name: structured-logs
static_configs:
- targets:
- localhost
labels:
job: structured
__path__: /var/log/app/*.json
pipeline_stages:
- json:
expressions:
timestamp: '@timestamp'
level: log.level
message: message
service: service.name
trace_id: trace.id
user_email: user.email
duration_ms: duration_ms
status_code: http.status_code
- timestamp:
source: timestamp
format: '2006-01-02T15:04:05.000Z07:00'
- labels:
level:
service:
trace_id:
status_code:
- metrics:
app_request_total:
type: Counter
description: "Total requests"
match_all: true
action: add
app_error_total:
type: Counter
description: "Total errors"
match_all: true
action: add
condition:
selector: '{level="error"}'
app_duration_seconds:
type: Histogram
description: "Request duration"
source: duration_ms
buckets: [10, 50, 100, 500, 1000, 5000, 10000]
Conditional Processing
scrape_configs:
- job_name: conditional-processing
static_configs:
- targets:
- localhost
labels:
job: conditional
__path__: /var/log/app/*.log
pipeline_stages:
- regex:
expression: '(?P<method>\w+) (?P<path>\S+) HTTP'
- drop:
expression: '(health_check|status_check|metrics)'
- json:
expressions:
duration: duration
only_errors: true
- match:
selector: '{method="GET"}'
stages:
- regex:
expression: 'duration=(?P<duration>\d+)'
- metrics:
get_requests_total:
type: Counter
match_all: true
action: add
- match:
selector: '{method="POST"}'
stages:
- metrics:
post_requests_total:
type: Counter
match_all: true
action: add
Grafana Integration
Add Loki Data Source
curl -X POST http://admin:admin@localhost:3000/api/datasources \
-H "Content-Type: application/json" \
-d '{
"name": "Loki",
"type": "loki",
"url": "http://localhost:3100",
"access": "proxy",
"isDefault": false,
"jsonData": {
"maxLines": 1000
}
}'
Creating Log Dashboards
Dashboard JSON with Log Panels
{
"dashboard": {
"title": "Application Logs Dashboard",
"panels": [
{
"title": "Log Volume",
"targets": [
{
"expr": "sum(rate({job=\"app\"}[5m])) by (level)",
"legendFormat": "{{level}}"
}
],
"type": "timeseries"
},
{
"title": "Error Logs",
"targets": [
{
"expr": "{job=\"app\", level=\"error\"}",
"format": "logs"
}
],
"type": "logs"
},
{
"title": "Request Duration",
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate({job=\"app\"} | json [5m])) by (le))"
}
],
"type": "timeseries"
}
]
}
}
Alerting on Logs
Create Log-Based Alerts
# Alert on error rate
curl -X POST http://admin:admin@localhost:3000/api/ruler/grafana/rules/logs \
-H "Content-Type: application/json" \
-d '{
"uid": "error-rate-alert",
"title": "High Error Rate",
"condition": "A",
"data": [
{
"refId": "A",
"queryType": "logs",
"model": {
"expr": "sum(rate({job=\"app\"} |= \"error\" [5m]))"
}
}
],
"noDataState": "NoData",
"execErrState": "Alerting",
"for": "5m",
"annotations": {
"summary": "High error rate detected"
}
}'
Performance Optimization
Tuning Loki
# Increase chunk size for high-volume logging
# Edit loki-config.yml
chunk_size_target: 2097152 # 2MB instead of 1MB
chunk_size_bytes: 3145728 # 3MB max
# Increase ingestion rate
ingestion_rate_mb: 200
ingestion_burst_size_mb: 400
# Adjust query cache
query_range:
cache_results: true
results_cache:
cache:
default_validity: 2h
Promtail Optimization
# Increase batching
clients:
- url: http://localhost:3100/loki/api/v1/push
batchwait: 2s
batchsize: 2097152 # 2MB batches
backoff_config:
minbackoff: 100ms
maxbackoff: 10s
maxretries: 5
Troubleshooting
Health Checks
# Check Loki readiness
curl -f http://localhost:3100/ready
# Check Loki metrics
curl http://localhost:3100/metrics | grep loki_ingester
# Check Promtail metrics
curl http://localhost:9080/metrics | grep promtail
Verify Data Flow
# Query recent logs
curl 'http://localhost:3100/loki/api/v1/query_range?query={job="app"}&start=1000&end=2000&limit=100' | jq .
# Check Promtail position tracking
tail -20 /var/lib/loki/positions.yaml
Debug Issues
# Enable debug logging
# Edit loki-config.yml
server:
log_level: debug
# Restart services
sudo systemctl restart loki promtail
# Monitor logs
sudo journalctl -u loki -f
sudo journalctl -u promtail -f
Conclusion
A complete Loki stack provides cost-effective log aggregation with powerful querying and visualization. By following this guide, you've built a production-ready logging infrastructure. Focus on designing efficient label hierarchies, leveraging pipeline stages for intelligent parsing, and setting appropriate retention policies. This foundation scales to handle large-scale logging requirements while maintaining fast query performance and keeping operational costs low.