VictoriaMetrics Installation and Configuration

VictoriaMetrics is a fast, cost-efficient, and scalable time series database that is fully compatible with Prometheus remote write, PromQL queries, and Grafana. This guide covers installing VictoriaMetrics in both single-node and cluster modes on Linux, ingesting metrics, querying with MetricsQL, configuring Grafana integration, setting retention, and migrating from Prometheus.

Prerequisites

Ubuntu 20.04+ or CentOS 8+ / Rocky Linux 8+
1 GB RAM minimum (single node); 4+ GB for cluster
SSD storage recommended for production
Prometheus (optional, for remote write)
Grafana for visualization

Installing VictoriaMetrics (Single Node)

# Download the latest VictoriaMetrics single-node binary
VM_VERSION="v1.101.0"
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/victoria-metrics-linux-amd64-${VM_VERSION}.tar.gz

tar xzf victoria-metrics-linux-amd64-${VM_VERSION}.tar.gz
sudo mv victoria-metrics-prod /usr/local/bin/victoria-metrics
victoria-metrics -version

# Create data directory and user
sudo useradd -r -s /sbin/nologin victoriametrics
sudo mkdir -p /var/lib/victoriametrics
sudo chown victoriametrics:victoriametrics /var/lib/victoriametrics

# Quick test run
victoria-metrics \
    -storageDataPath=/var/lib/victoriametrics \
    -retentionPeriod=12 \
    -httpListenAddr=:8428

# Test endpoints (in another terminal)
curl http://localhost:8428/metrics           # Self-metrics
curl http://localhost:8428/-/healthy         # Health check

Running as a Systemd Service

# Create systemd service
sudo cat > /etc/systemd/system/victoriametrics.service << 'EOF'
[Unit]
Description=VictoriaMetrics
After=network.target

[Service]
User=victoriametrics
Group=victoriametrics
Type=simple
ExecStart=/usr/local/bin/victoria-metrics \
    -storageDataPath=/var/lib/victoriametrics \
    -retentionPeriod=12 \
    -httpListenAddr=127.0.0.1:8428 \
    -maxConcurrentInserts=32 \
    -search.maxConcurrentRequests=16 \
    -search.maxQueryDuration=60s \
    -selfScrapeInterval=10s \
    -loggerLevel=INFO
Restart=on-failure
RestartSec=5
LimitNOFILE=1000000

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now victoriametrics
sudo systemctl status victoriametrics

Key startup flags:

-storageDataPath - where data is stored
-retentionPeriod - months to keep data (e.g., 12 = 1 year, 3 = 3 months)
-httpListenAddr - listen address (use 0.0.0.0:8428 for external access)
-memory.allowedPercent - limit RAM usage (e.g., 60 for 60% of system RAM)

Configuring Prometheus Remote Write

Send Prometheus metrics to VictoriaMetrics instead of (or in addition to) local storage:

# prometheus.yml - add remote_write section
global:
  scrape_interval: 15s
  evaluation_interval: 15s

remote_write:
  - url: http://victoriametrics-host:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 50000
      max_shards: 30
    # Optional: filter which metrics to send
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop   # Drop Go runtime metrics to save space

VictoriaMetrics can also scrape targets directly (replaces Prometheus entirely):

# /etc/victoriametrics/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100', 'web-01:9100', 'web-02:9100']

  - job_name: 'nginx'
    static_configs:
      - targets: ['localhost:9113']

Start VictoriaMetrics with a scrape config:

victoria-metrics \
    -storageDataPath=/var/lib/victoriametrics \
    -retentionPeriod=12 \
    -promscrape.config=/etc/victoriametrics/prometheus.yml \
    -httpListenAddr=:8428

Querying with MetricsQL

VictoriaMetrics supports PromQL and extends it with MetricsQL:

# Query via HTTP API (compatible with Prometheus API)
# Instant query
curl 'http://localhost:8428/api/v1/query?query=up&time=2024-01-15T12:00:00Z'

# Range query
curl 'http://localhost:8428/api/v1/query_range' \
  --data-urlencode 'query=rate(http_requests_total[5m])' \
  --data-urlencode 'start=2024-01-15T00:00:00Z' \
  --data-urlencode 'end=2024-01-15T12:00:00Z' \
  --data-urlencode 'step=60s'

MetricsQL extensions beyond PromQL:

# rollup_candlestick - OHLC for financial-style graphs
rollup_candlestick(price[1h])

# keep_last_value - fill gaps in sparse metrics
keep_last_value(temperature{sensor="outdoor"}[10m])

# Default value when metric is absent
default(memory_usage, 0)

# Median aggregation
median(http_request_duration_seconds[5m])

# topk_avg - top K by average value
topk_avg(5, rate(http_requests_total[5m]))

# Aggregate across multiple metrics with a single query
sum(
  rate({__name__=~"http_requests_total|grpc_requests_total"}[5m])
) by (service)

Grafana Integration

Add VictoriaMetrics as a Prometheus data source in Grafana:

Configuration → Data Sources → Add data source
Select Prometheus
Set URL: http://victoriametrics-host:8428
Under Custom HTTP Headers (optional), add X-Scope-OrgID: tenant1 for multi-tenancy
Click Save & Test

Import useful dashboards from Grafana.com:

Dashboard 10229 - VictoriaMetrics self-monitoring
Dashboard 1860 - Node Exporter Full
Dashboard 3662 - Prometheus 2.0 Overview (works with VictoriaMetrics)

Configure Grafana alerting with VictoriaMetrics as the data source the same way as with Prometheus.

Cluster Deployment

VictoriaMetrics cluster consists of three components:

Component	Role	Default Port
`vminsert`	Ingestion endpoint	8480
`vmselect`	Query endpoint	8481
`vmstorage`	Storage node	8482

# Download cluster components
VM_VERSION="v1.101.0"
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/vmutils-linux-amd64-${VM_VERSION}.tar.gz
tar xzf vmutils-linux-amd64-${VM_VERSION}.tar.gz

# On storage nodes (run on each storage node)
sudo mv vmstorage-prod /usr/local/bin/vmstorage
vmstorage \
    -storageDataPath=/var/lib/vmstorage \
    -retentionPeriod=12 \
    -httpListenAddr=:8482 \
    -vminsertAddr=:8400 \
    -vmselectAddr=:8401

# On insert nodes (load balancer for writes)
sudo mv vminsert-prod /usr/local/bin/vminsert
vminsert \
    -httpListenAddr=:8480 \
    -storageNode=storage-1:8400,storage-2:8400,storage-3:8400 \
    -replicationFactor=2

# On select nodes (load balancer for reads)
sudo mv vmselect-prod /usr/local/bin/vmselect
vmselect \
    -httpListenAddr=:8481 \
    -storageNode=storage-1:8401,storage-2:8401,storage-3:8401 \
    -replicationFactor=2

# Configure Prometheus to write to vminsert
# remote_write.url = http://vminsert-host:8480/insert/0/prometheus

# Configure Grafana to query vmselect
# URL = http://vmselect-host:8481/select/0/prometheus

The /0/ in URLs is the tenant ID (0 = default). Use different tenant IDs for multi-tenancy.

Retention and Storage Management

# Check storage stats
curl http://localhost:8428/api/v1/status/tsdb?topN=20 | python3 -m json.tool
# Shows top time series by size

# Check active time series count
curl 'http://localhost:8428/api/v1/query?query=vm_new_timeseries_created_total'

# Delete time series (useful for removing noisy metrics)
curl -X POST http://localhost:8428/api/v1/admin/tsdb/delete_series \
  --data-urlencode 'match[]={job="noisy-exporter"}'

# Snapshot for backup
curl http://localhost:8428/snapshot/create
# Returns snapshot name; backup /var/lib/victoriametrics/snapshots/

# Restore from snapshot
victoria-metrics \
    -storageDataPath=/var/lib/victoriametrics \
    -snapshotRestore=/var/lib/victoriametrics/snapshots/20240115120000-XXXX

Storage usage tuning:

# Limit cardinality (protect against metric explosion)
victoria-metrics \
    -storage.maxHourlySeries=1000000 \    # Max unique series per hour
    -storage.maxDailySeries=5000000 \     # Max unique series per day
    -httpListenAddr=:8428

Troubleshooting

High memory usage:

# Check current memory stats
curl http://localhost:8428/metrics | grep vm_memory

# Limit memory usage
victoria-metrics -memory.allowedPercent=60  # Use max 60% of RAM

Slow queries:

# Check active queries
curl http://localhost:8428/api/v1/status/active_queries | python3 -m json.tool

# Enable query tracing
curl 'http://localhost:8428/api/v1/query?query=rate(http_requests_total[5m])&trace=1'
# Returns trace in response showing execution time per step

Prometheus remote write failing:

# Check remote write stats in Prometheus
curl http://prometheus:9090/api/v1/query?query=prometheus_remote_storage_failed_samples_total

# Check VictoriaMetrics ingestion errors
curl http://localhost:8428/metrics | grep vm_rows_invalid

Data not showing in Grafana:

# Test the Prometheus-compatible API directly
curl 'http://localhost:8428/api/v1/label/__name__/values' | python3 -m json.tool
# Should return list of metric names

Conclusion

VictoriaMetrics is a compelling replacement or complement for Prometheus that provides better compression (typically 5-10x), lower RAM usage, and faster queries for large time series workloads. The single-node binary is drop-in compatible with Prometheus remote write and the PromQL API, making migration as simple as adding a remote_write block to your existing Prometheus config. For long-term storage needs, the cluster mode scales horizontally while maintaining the same query interface.

VictoriaMetrics Installation and Configuration

On this page

On this page

VictoriaMetrics Installation and Configuration

Prerequisites

Installing VictoriaMetrics (Single Node)

Running as a Systemd Service

Configuring Prometheus Remote Write

Querying with MetricsQL

Grafana Integration

Cluster Deployment

Retention and Storage Management

Troubleshooting

Conclusion

Latest Video

Get $20 Free Credit