Kubernetes Monitoring with kube-prometheus-stack

The kube-prometheus-stack combines Prometheus, Grafana, AlertManager, and node-exporter into a comprehensive monitoring solution for Kubernetes clusters. This guide covers Helm installation, ServiceMonitor configuration, PrometheusRules for alerting, Grafana dashboards, and implementing production-grade observability for your VPS and baremetal Kubernetes infrastructure.

Table of Contents

• Kube-prometheus-stack Overview
• Installation
• Configuration
• ServiceMonitor
• PrometheusRules and Alerting
• Grafana Dashboards
• External Prometheus
• Practical Examples
• Conclusion

Kube-prometheus-stack Overview

Components

• Prometheus: Metrics collection and storage
• Grafana: Visualization and dashboarding
• AlertManager: Alert routing and management
• Prometheus Operator: Custom resource management
• node-exporter: Host-level metrics
• kube-state-metrics: Kubernetes object metrics

Architecture

Kubernetes Cluster
├── Prometheus Operator
├── Prometheus (time-series database)
├── AlertManager (alert routing)
├── Grafana (visualization)
├── node-exporter (host metrics)
└── kube-state-metrics (object metrics)

Installation

Prerequisites

• Kubernetes v1.16+
• Helm 3+
• 10GB+ free disk space (varies by retention)
• 4GB+ available memory

Helm Repository Setup

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Basic Installation

helm install kube-prometheus \
  prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --create-namespace

Production Values File

Create a values file for production:

# kube-prometheus-values.yaml
prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: "50GB"
    
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 2
        memory: 4Gi
    
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 100Gi
    
    externalLabels:
      cluster: "production"
      environment: "prod"
    
    # High availability
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - prometheus
            topologyKey: kubernetes.io/hostname

alertmanager:
  enabled: true
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
  
  config:
    global:
      resolve_timeout: 5m
    
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'default'
      routes:
      - match:
          severity: critical
        receiver: 'critical'
      - match:
          severity: warning
        receiver: 'warning'
    
    receivers:
    - name: 'default'
      slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts'
    
    - name: 'critical'
      slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#critical-alerts'
      pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_KEY'

grafana:
  enabled: true
  adminPassword: 'SetStrongPassword123'
  
  persistence:
    enabled: true
    type: pvc
    storageClassName: fast-ssd
    size: 10Gi
  
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
      - name: Prometheus
        type: prometheus
        url: http://kube-prometheus-prometheus:9090
        isDefault: true

prometheus-node-exporter:
  enabled: true
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 100m
      memory: 128Mi

kube-state-metrics:
  enabled: true
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 100m
      memory: 256Mi

Install with values file:

helm install kube-prometheus \
  prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --create-namespace \
  -f kube-prometheus-values.yaml

Verify Installation

kubectl get pods -n monitoring
kubectl get services -n monitoring
kubectl get pvc -n monitoring

Configuration

Accessing Prometheus

Port-forward to Prometheus:

kubectl port-forward -n monitoring svc/kube-prometheus-prometheus 9090:9090

Access at http://localhost:9090

Accessing Grafana

Port-forward to Grafana:

kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80

Access at http://localhost:3000

Default credentials: admin / password from values

Accessing AlertManager

kubectl port-forward -n monitoring svc/kube-prometheus-alertmanager 9093:9093

Access at http://localhost:9093

ServiceMonitor

What is ServiceMonitor?

ServiceMonitor is a Kubernetes custom resource that tells Prometheus where to scrape metrics. It's namespace-scoped and targets services with specific labels.

Creating ServiceMonitor

Example for Kubernetes API server:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  endpoints:
  - interval: 30s
    port: https
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      serverName: kubernetes
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  selector:
    matchLabels:
      component: kube-apiserver

ServiceMonitor for Custom Application

Monitor a custom application exposing metrics on port 8080:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: production
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
    scrapeTimeout: 10s

ServiceMonitor with Label Relabeling

Modify metric labels during scraping:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: postgres-exporter
  namespace: databases
spec:
  selector:
    matchLabels:
      app: postgres-exporter
  endpoints:
  - port: metrics
    interval: 30s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod
    - sourceLabels: [__meta_kubernetes_namespace]
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_service_name]
      targetLabel: service

PodMonitor Alternative

For monitoring pods directly (not recommended, prefer Service/ServiceMonitor):

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-app-pods
  namespace: production
spec:
  selector:
    matchLabels:
      app: my-app
  podMetricsEndpoints:
  - port: metrics
    interval: 30s

PrometheusRules and Alerting

PrometheusRule Structure

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-alerts
  namespace: monitoring
spec:
  groups:
  - name: kubernetes.rules
    interval: 30s
    rules:
    - alert: AlertName
      expr: <promql_query>
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Alert summary"
        description: "Detailed description"

Node Health Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: node-health
  namespace: monitoring
spec:
  groups:
  - name: node.rules
    interval: 30s
    rules:
    - alert: NodeNotReady
      expr: kube_node_status_condition{condition="Ready",status="true"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node {{ $labels.node }} is not ready"
        description: "Node {{ $labels.node }} has been unready for more than 5 minutes"
    
    - alert: HighNodeMemoryUsage
      expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage on node {{ $labels.instance }}"
        description: "Memory usage is {{ $value | humanizePercentage }} on {{ $labels.instance }}"
    
    - alert: HighNodeDiskUsage
      expr: (1 - (node_filesystem_avail_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes)) > 0.85
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "High disk usage on {{ $labels.instance }}"
        description: "Disk usage is {{ $value | humanizePercentage }} on {{ $labels.instance }}"

Pod and Container Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-alerts
  namespace: monitoring
spec:
  groups:
  - name: pod.rules
    interval: 30s
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0.1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
        description: "Pod has restarted {{ $value }} times in the last 15 minutes"
    
    - alert: PodNotHealthy
      expr: |
        min_over_time(sum by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed"})[15m:1m]) > 0
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is not healthy"
        description: "Pod has been in a non-running state for more than 15 minutes"
    
    - alert: ContainerHighCpuUsage
      expr: |
        (sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace) / sum(container_spec_cpu_quota{container!=""}/container_spec_cpu_period{container!=""}) by (pod, namespace)) > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Container {{ $labels.pod }} has high CPU usage"
        description: "CPU usage is {{ $value | humanizePercentage }}"

Grafana Dashboards

Accessing Dashboards

Pre-installed dashboards:

• Kubernetes Cluster Monitoring
• Node Exporter
• Prometheus Stats
• Kubernetes API Server

Access via: http://localhost:3000 → Dashboards

Creating Custom Dashboard

Access Grafana and create a new dashboard:

1. Click + icon → Dashboard
2. Add panels with PromQL queries
3. Configure visualization
4. Save with meaningful name

Example Panel Query

Query for Pod CPU Usage:

sum(rate(container_cpu_usage_seconds_total{pod=~"my-app.*"}[5m])) by (pod)

Query for Pod Memory Usage:

sum(container_memory_usage_bytes{pod=~"my-app.*"}) by (pod) / (1024^3)

Dashboard JSON Export

Export dashboard configuration:

# Get dashboard JSON
curl -s http://admin:password@localhost:3000/api/dashboards/uid/dashboard-uid | jq .dashboard > dashboard.json

# Import dashboard
curl -X POST http://admin:password@localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -d @dashboard.json

External Prometheus

Remote Storage Configuration

Write metrics to external storage (e.g., Prometheus Cloud):

prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "https://cortex.example.com/api/prom/push"
      writeRelabelConfigs:
      - sourceLabels: [__name__]
        regex: 'prometheus_tsdb_.*'
        action: drop

Federation

Aggregate multiple Prometheus instances:

# Central Prometheus scraping federated endpoints
- job_name: 'federate'
  static_configs:
  - targets:
    - 'prometheus-cluster1:9090'
    - 'prometheus-cluster2:9090'
  metrics_path: '/federate'
  params:
    match[]:
    - '{job="prometheus"}'
    - '{__name__=~"job:.*"}'

Practical Examples

Example: Complete Monitoring Stack

# Create monitoring namespace
kubectl create namespace monitoring

# Add Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install with custom values
helm install kube-prometheus \
  prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f kube-prometheus-values.yaml

# Wait for deployment
kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=kube-prometheus -n monitoring --timeout=300s

# Verify installation
kubectl get all -n monitoring

Example: Monitor Custom Application

---
# Service
apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: production
  labels:
    app: my-app
spec:
  ports:
  - name: http
    port: 8080
  - name: metrics
    port: 8081
  selector:
    app: my-app
---
# ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  namespaceSelector:
    matchNames:
    - production
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
---
# PrometheusRule
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  namespace: monitoring
spec:
  groups:
  - name: my-app.rules
    interval: 30s
    rules:
    - alert: AppHighErrorRate
      expr: |
        (sum(rate(app_requests_total{status=~"5.."}[5m])) by (instance) / 
         sum(rate(app_requests_total[5m])) by (instance)) > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate on {{ $labels.instance }}"
        description: "Error rate is {{ $value | humanizePercentage }}"

Conclusion

The kube-prometheus-stack provides comprehensive monitoring and alerting for Kubernetes clusters on VPS and baremetal infrastructure. By properly configuring ServiceMonitors for application metrics, implementing PrometheusRules for intelligent alerting, and utilizing Grafana dashboards for visualization, you create a production-grade observability platform. Start with the default dashboards and alerts, then customize based on your specific application requirements and operational needs. Regular review and tuning of alert thresholds ensure you stay informed without alert fatigue.