--- title: "Kubernetes Logging with Fluentd, Elasticsearch and Kibana" description: "Build an EFK logging stack on Kubernetes: deploy Elasticsearch and Kibana with the ECK operator, ship pod logs with a Fluentd DaemonSet and set log retention." category: "Kubernetes Advanced" language: en source: https://cubepath.com/docs/kubernetes-advanced/kubernetes-logging-with-fluentd-and-elasticsearch updated: 2026-09-25 --- # Kubernetes Logging with Fluentd, Elasticsearch and Kibana Kubernetes keeps container logs only on the node where each pod runs, and they disappear when the pod is deleted. An EFK stack collects them in one searchable place: Fluentd runs on every node, reads the container log files, enriches each line with pod and namespace metadata, and sends it to Elasticsearch, where you search it with Kibana. In this tutorial you will deploy Elasticsearch and Kibana with Elastic's ECK operator, ship logs with a Fluentd DaemonSet, check that they arrive, and add a retention policy so the cluster does not fill its disks. ## Prerequisites To follow this tutorial, you will need: - A Kubernetes cluster with a default StorageClass, for example K3s on a CubePath VPS running Ubuntu 24.04. The nodes need at least 4 GB of RAM free for Elasticsearch and Kibana, and about 20 GB of disk for log data. - `kubectl` configured with admin access, and Helm installed (`sudo snap install helm --classic` on Ubuntu). - A non-root user with `sudo` privileges on each node, to adjust a kernel setting in Step 1. - The cluster uses containerd or CRI-O as runtime (the default for K3s, MicroK8s, RKE2 and kubeadm today), so container logs are written in the CRI format. This guide builds a single-node Elasticsearch suitable for small clusters and testing. The Conclusion explains what to change for production. ## Step 1 - Preparing the nodes for Elasticsearch Elasticsearch memory-maps its index files and needs a higher `vm.max_map_count` than the Ubuntu default. Set it on every node that may run Elasticsearch, and persist it across reboots: ```bash echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf sudo sysctl --system ``` Verify the value: ```bash sysctl vm.max_map_count ``` ```output vm.max_map_count = 262144 ``` Create a namespace for the logging stack: ```bash kubectl create namespace logging ``` ## Step 2 - Installing the ECK operator Elastic Cloud on Kubernetes (ECK) is the operator Elastic supports for running Elasticsearch and Kibana on Kubernetes. It handles TLS certificates, the `elastic` superuser password, upgrades and rolling restarts. Install it from Elastic's Helm repository: ```bash helm repo add elastic https://helm.elastic.co helm repo update helm install elastic-operator elastic/eck-operator \ --namespace elastic-system --create-namespace ``` Check that the operator is running: ```bash kubectl -n elastic-system get pods ``` ```output NAME READY STATUS RESTARTS AGE elastic-operator-0 1/1 Running 0 40s ``` ## Step 3 - Deploying Elasticsearch With the operator in place, you describe the cluster you want as an `Elasticsearch` resource. This tutorial pins the 8.19 release line, which is the version the Fluentd Elasticsearch 8 image in Step 5 is built for. Check the ECK documentation for the latest 8.19 patch release and use it in both manifests. ```bash nano elasticsearch.yaml ``` ```yaml apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: logging namespace: logging spec: version: 8.19.0 nodeSets: - name: default count: 1 podTemplate: spec: containers: - name: elasticsearch resources: requests: memory: 2Gi cpu: 500m limits: memory: 2Gi volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi ``` - `count: 1` creates a single Elasticsearch node. - Elasticsearch sizes its heap automatically to about half of the container memory limit. - The volume claim must be named `elasticsearch-data`; ECK mounts it as the data directory. It uses your default StorageClass. Apply it and watch the resource until `HEALTH` is `green` and `PHASE` is `Ready`. This takes one to three minutes: ```bash kubectl apply -f elasticsearch.yaml kubectl -n logging get elasticsearch -w ``` ```output NAME HEALTH NODES VERSION PHASE AGE logging green 1 8.19.0 Ready 2m ``` Press `Ctrl+C` to stop watching. ECK has created an HTTPS Service called `logging-es-http`, a Secret with the password of the `elastic` user, and a Secret with the CA certificate. Store the password in a shell variable for the next steps: ```bash ES_PASSWORD=$(kubectl -n logging get secret logging-es-elastic-user -o go-template='{{.data.elastic | base64decode}}') ``` ## Step 4 - Deploying Kibana Kibana connects to Elasticsearch through the `elasticsearchRef` field; ECK configures the credentials and TLS for you. ```bash nano kibana.yaml ``` ```yaml apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: logging namespace: logging spec: version: 8.19.0 count: 1 elasticsearchRef: name: logging ``` ```bash kubectl apply -f kibana.yaml kubectl -n logging get kibana -w ``` ```output NAME HEALTH NODES VERSION AGE logging green 1 8.19.0 90s ``` Press `Ctrl+C` once health is `green`. ## Step 5 - Deploying Fluentd as a DaemonSet Fluentd must run on every node to read the log files the container runtime writes under `/var/log/containers`. A DaemonSet does exactly that. The official `fluent/fluentd-kubernetes-daemonset` image already contains the plugins needed here: the CRI log parser, the Kubernetes metadata filter and the Elasticsearch output. First, give Fluentd read access to pod and namespace metadata: ```bash nano fluentd-rbac.yaml ``` ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: logging --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluentd rules: - apiGroups: [""] resources: ["pods", "namespaces"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: fluentd roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluentd subjects: - kind: ServiceAccount name: fluentd namespace: logging ``` ```bash kubectl apply -f fluentd-rbac.yaml ``` Next, write the Fluentd configuration as a ConfigMap. It tails the container logs, parses the CRI format, adds Kubernetes metadata, and writes to daily indices named `k8s-YYYY.MM.DD` over verified TLS: ```bash nano fluentd-config.yaml ``` ```yaml apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: logging data: fluent.conf: | @type tail @id in_tail_container_logs path /var/log/containers/*.log exclude_path ["/var/log/containers/fluentd-*.log"] pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true @type cri @type kubernetes_metadata @type elasticsearch host logging-es-http.logging.svc port 9200 scheme https ssl_verify true ca_file /etc/es-certs/ca.crt user elastic password "#{ENV['ES_PASSWORD']}" logstash_format true logstash_prefix k8s @type file path /var/log/fluentd-buffers/kubernetes.buffer flush_mode interval flush_interval 5s chunk_limit_size 8MB retry_max_interval 30s overflow_action block ``` - `exclude_path` stops Fluentd from ingesting its own logs, which would create a feedback loop. - `pos_file` records how far each file has been read, so a restarted pod does not resend everything. - The file buffer on the host keeps unsent logs if Elasticsearch is temporarily unavailable. Apply the ConfigMap: ```bash kubectl apply -f fluentd-config.yaml ``` Now create the DaemonSet. It mounts the node's `/var/log`, the ConfigMap and the Elasticsearch CA certificate, and reads the password from the Secret ECK created. Check Docker Hub for the newest `-debian-elasticsearch8-` tag of the image and use it: ```bash nano fluentd-daemonset.yaml ``` ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: logging labels: app: fluentd spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.18-debian-elasticsearch8-1 securityContext: runAsUser: 0 env: - name: ES_PASSWORD valueFrom: secretKeyRef: name: logging-es-elastic-user key: elastic resources: requests: cpu: 100m memory: 200Mi limits: memory: 512Mi volumeMounts: - name: varlog mountPath: /var/log - name: config mountPath: /fluentd/etc - name: es-certs mountPath: /etc/es-certs readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: config configMap: name: fluentd-config - name: es-certs secret: secretName: logging-es-http-certs-public ``` Container log files in `/var/log/containers` are symlinks into `/var/log/pods`, so mounting `/var/log` covers both. The toleration lets Fluentd also run on control-plane nodes that are tainted. Apply it and check that one pod runs per node: ```bash kubectl apply -f fluentd-daemonset.yaml kubectl -n logging get daemonset fluentd ``` ```output NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd 1 1 1 1 1 45s ``` Look at the Fluentd logs for connection errors: ```bash kubectl -n logging logs daemonset/fluentd --tail=20 ``` A healthy start ends with a line such as `fluentd worker is now running worker=0`, with no repeated `Could not communicate to Elasticsearch` warnings. > Note: The `elastic` superuser keeps this tutorial short. In production, create a dedicated Elasticsearch user with a role that can only write to `k8s-*` indices and give Fluentd those credentials instead. ## Step 6 - Adding a retention policy Without retention, daily indices accumulate until the disk is full and Elasticsearch stops accepting writes. An index lifecycle management (ILM) policy deletes indices after a set age, and an index template applies it to every new `k8s-*` index. Open a port forward to Elasticsearch in a second terminal and leave it running: ```bash kubectl -n logging port-forward service/logging-es-http 9200 ``` Back in the first terminal, create a policy that deletes indices seven days after creation. The `-k` flag skips certificate name checks because you are connecting through `localhost`; the traffic never leaves the server: ```bash curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_ilm/policy/k8s-logs" \ -H 'Content-Type: application/json' \ -d '{"policy":{"phases":{"hot":{"actions":{}},"delete":{"min_age":"7d","actions":{"delete":{}}}}}}' ``` ```output {"acknowledged":true} ``` Create an index template that attaches the policy to new `k8s-*` indices. `number_of_replicas: 0` keeps a single-node cluster `green`, because there is no second node to hold replicas: ```bash curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_index_template/k8s-logs" \ -H 'Content-Type: application/json' \ -d '{"index_patterns":["k8s-*"],"template":{"settings":{"index.lifecycle.name":"k8s-logs","number_of_replicas":0}}}' ``` ```output {"acknowledged":true} ``` The template applies to indices created from now on. Apply the same settings to the index Fluentd has already created today: ```bash curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/k8s-*/_settings" \ -H 'Content-Type: application/json' \ -d '{"index.lifecycle.name":"k8s-logs","number_of_replicas":0}' ``` Adjust `7d` to your retention needs and available disk. ## Step 7 - Verifying that logs arrive Start a short-lived pod that prints a few lines: ```bash kubectl run logger --image=busybox:stable --restart=Never -- \ sh -c 'for i in 1 2 3 4 5; do echo "hello from logger $i"; sleep 1; done' ``` After about ten seconds, list the log indices: ```bash curl -sk -u "elastic:$ES_PASSWORD" "https://localhost:9200/_cat/indices/k8s-*?v" ``` ```output health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open k8s-2026.09.25 a1B2c3D4e5F6g7H8i9J0kL 1 0 15230 0 6.1mb 6.1mb ``` Search for the test pod's lines. The metadata filter stored the pod name in `kubernetes.pod_name`: ```bash curl -sk -u "elastic:$ES_PASSWORD" \ "https://localhost:9200/k8s-*/_search?q=kubernetes.pod_name:logger&size=1&pretty" ``` ```output { ... "hits" : { "total" : { "value" : 5, "relation" : "eq" }, "hits" : [ { "_index" : "k8s-2026.09.25", "_source" : { "stream" : "stdout", "logtag" : "F", "message" : "hello from logger 1", "kubernetes" : { "namespace_name" : "default", "pod_name" : "logger", "container_name" : "logger", ... ``` Delete the test pod and stop the Elasticsearch port forward with `Ctrl+C`: ```bash kubectl delete pod logger ``` ## Step 8 - Exploring logs in Kibana Kibana is only reachable inside the cluster. Forward its Service on the server: ```bash kubectl -n logging port-forward service/logging-kb-http 5601 ``` From your workstation, open an SSH tunnel to the server, replacing `your_user` and `your_server_ip`: ```bash ssh -N -L 5601:127.0.0.1:5601 your_user@your_server_ip ``` Open `https://localhost:5601`, accept the self-signed certificate warning and log in as `elastic` with the password stored in `ES_PASSWORD` (print it on the server with `echo "$ES_PASSWORD"`). To see the logs: 1. Open the main menu and go to **Discover**. 2. Click **Create data view**, set **Index pattern** to `k8s-*` and **Timestamp field** to `@timestamp`, then save it. 3. Filter with KQL, for example `kubernetes.namespace_name : "default"` or `message : "error"`. Add columns such as `kubernetes.pod_name` and `message` from the field list to get a readable log view. ## Troubleshooting **Elasticsearch pod is `Pending`.** No node has enough free memory or the PersistentVolumeClaim cannot bind. Run `kubectl -n logging describe pod logging-es-default-0` and check the events at the bottom. **Elasticsearch restarts with `max virtual memory areas vm.max_map_count [65530] is too low`.** Step 1 was not applied on the node running the pod. Set the sysctl there. **Fluentd logs show `certificate verify failed`.** The `host` in the ConfigMap must match a name in the certificate. Use `logging-es-http.logging.svc`, not an IP address. **No `k8s-*` index appears.** Check that files exist under `/var/log/containers` on the node and that Fluentd is not logging `pattern not matched` errors. If your runtime is Docker Engine with the json-file driver instead of containerd, replace the `cri` parser with `@type json`. **Index health is `yellow`.** The index has replicas that cannot be allocated on a single node. Apply the settings request from Step 6 to set `number_of_replicas` to `0`. ## Conclusion You deployed Elasticsearch and Kibana with the ECK operator, shipped every container's logs with a Fluentd DaemonSet, verified them end to end and limited retention to seven days. For production, raise `count` to three Elasticsearch nodes (and set replicas back to 1), give Fluentd a dedicated write-only user, and consider adding the `fluent-plugin-concat` filter to join multi-line stack traces into single events.