---
title: "Kubernetes Logging with Fluentd, Elasticsearch and Kibana"
description: "Build an EFK logging stack on Kubernetes: deploy Elasticsearch and Kibana with the ECK operator, ship pod logs with a Fluentd DaemonSet and set log retention."
category: "Kubernetes Advanced"
language: en
source: https://cubepath.com/docs/kubernetes-advanced/kubernetes-logging-with-fluentd-and-elasticsearch
updated: 2026-09-25
---
# Kubernetes Logging with Fluentd, Elasticsearch and Kibana
Kubernetes keeps container logs only on the node where each pod runs, and they disappear when the pod is deleted. An EFK stack collects them in one searchable place: Fluentd runs on every node, reads the container log files, enriches each line with pod and namespace metadata, and sends it to Elasticsearch, where you search it with Kibana. In this tutorial you will deploy Elasticsearch and Kibana with Elastic's ECK operator, ship logs with a Fluentd DaemonSet, check that they arrive, and add a retention policy so the cluster does not fill its disks.
## Prerequisites
To follow this tutorial, you will need:
- A Kubernetes cluster with a default StorageClass, for example K3s on a CubePath VPS running Ubuntu 24.04. The nodes need at least 4 GB of RAM free for Elasticsearch and Kibana, and about 20 GB of disk for log data.
- `kubectl` configured with admin access, and Helm installed (`sudo snap install helm --classic` on Ubuntu).
- A non-root user with `sudo` privileges on each node, to adjust a kernel setting in Step 1.
- The cluster uses containerd or CRI-O as runtime (the default for K3s, MicroK8s, RKE2 and kubeadm today), so container logs are written in the CRI format.
This guide builds a single-node Elasticsearch suitable for small clusters and testing. The Conclusion explains what to change for production.
## Step 1 - Preparing the nodes for Elasticsearch
Elasticsearch memory-maps its index files and needs a higher `vm.max_map_count` than the Ubuntu default. Set it on every node that may run Elasticsearch, and persist it across reboots:
```bash
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf
sudo sysctl --system
```
Verify the value:
```bash
sysctl vm.max_map_count
```
```output
vm.max_map_count = 262144
```
Create a namespace for the logging stack:
```bash
kubectl create namespace logging
```
## Step 2 - Installing the ECK operator
Elastic Cloud on Kubernetes (ECK) is the operator Elastic supports for running Elasticsearch and Kibana on Kubernetes. It handles TLS certificates, the `elastic` superuser password, upgrades and rolling restarts. Install it from Elastic's Helm repository:
```bash
helm repo add elastic https://helm.elastic.co
helm repo update
helm install elastic-operator elastic/eck-operator \
--namespace elastic-system --create-namespace
```
Check that the operator is running:
```bash
kubectl -n elastic-system get pods
```
```output
NAME READY STATUS RESTARTS AGE
elastic-operator-0 1/1 Running 0 40s
```
## Step 3 - Deploying Elasticsearch
With the operator in place, you describe the cluster you want as an `Elasticsearch` resource. This tutorial pins the 8.19 release line, which is the version the Fluentd Elasticsearch 8 image in Step 5 is built for. Check the ECK documentation for the latest 8.19 patch release and use it in both manifests.
```bash
nano elasticsearch.yaml
```
```yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: logging
namespace: logging
spec:
version: 8.19.0
nodeSets:
- name: default
count: 1
podTemplate:
spec:
containers:
- name: elasticsearch
resources:
requests:
memory: 2Gi
cpu: 500m
limits:
memory: 2Gi
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
```
- `count: 1` creates a single Elasticsearch node.
- Elasticsearch sizes its heap automatically to about half of the container memory limit.
- The volume claim must be named `elasticsearch-data`; ECK mounts it as the data directory. It uses your default StorageClass.
Apply it and watch the resource until `HEALTH` is `green` and `PHASE` is `Ready`. This takes one to three minutes:
```bash
kubectl apply -f elasticsearch.yaml
kubectl -n logging get elasticsearch -w
```
```output
NAME HEALTH NODES VERSION PHASE AGE
logging green 1 8.19.0 Ready 2m
```
Press `Ctrl+C` to stop watching. ECK has created an HTTPS Service called `logging-es-http`, a Secret with the password of the `elastic` user, and a Secret with the CA certificate. Store the password in a shell variable for the next steps:
```bash
ES_PASSWORD=$(kubectl -n logging get secret logging-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
```
## Step 4 - Deploying Kibana
Kibana connects to Elasticsearch through the `elasticsearchRef` field; ECK configures the credentials and TLS for you.
```bash
nano kibana.yaml
```
```yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: logging
namespace: logging
spec:
version: 8.19.0
count: 1
elasticsearchRef:
name: logging
```
```bash
kubectl apply -f kibana.yaml
kubectl -n logging get kibana -w
```
```output
NAME HEALTH NODES VERSION AGE
logging green 1 8.19.0 90s
```
Press `Ctrl+C` once health is `green`.
## Step 5 - Deploying Fluentd as a DaemonSet
Fluentd must run on every node to read the log files the container runtime writes under `/var/log/containers`. A DaemonSet does exactly that. The official `fluent/fluentd-kubernetes-daemonset` image already contains the plugins needed here: the CRI log parser, the Kubernetes metadata filter and the Elasticsearch output.
First, give Fluentd read access to pod and namespace metadata:
```bash
nano fluentd-rbac.yaml
```
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
rules:
- apiGroups: [""]
resources: ["pods", "namespaces"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluentd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluentd
subjects:
- kind: ServiceAccount
name: fluentd
namespace: logging
```
```bash
kubectl apply -f fluentd-rbac.yaml
```
Next, write the Fluentd configuration as a ConfigMap. It tails the container logs, parses the CRI format, adds Kubernetes metadata, and writes to daily indices named `k8s-YYYY.MM.DD` over verified TLS:
```bash
nano fluentd-config.yaml
```
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: logging
data:
fluent.conf: |
@type tail
@id in_tail_container_logs
path /var/log/containers/*.log
exclude_path ["/var/log/containers/fluentd-*.log"]
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
@type cri
@type kubernetes_metadata
@type elasticsearch
host logging-es-http.logging.svc
port 9200
scheme https
ssl_verify true
ca_file /etc/es-certs/ca.crt
user elastic
password "#{ENV['ES_PASSWORD']}"
logstash_format true
logstash_prefix k8s
@type file
path /var/log/fluentd-buffers/kubernetes.buffer
flush_mode interval
flush_interval 5s
chunk_limit_size 8MB
retry_max_interval 30s
overflow_action block
```
- `exclude_path` stops Fluentd from ingesting its own logs, which would create a feedback loop.
- `pos_file` records how far each file has been read, so a restarted pod does not resend everything.
- The file buffer on the host keeps unsent logs if Elasticsearch is temporarily unavailable.
Apply the ConfigMap:
```bash
kubectl apply -f fluentd-config.yaml
```
Now create the DaemonSet. It mounts the node's `/var/log`, the ConfigMap and the Elasticsearch CA certificate, and reads the password from the Secret ECK created. Check Docker Hub for the newest `-debian-elasticsearch8-` tag of the image and use it:
```bash
nano fluentd-daemonset.yaml
```
```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: logging
labels:
app: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.18-debian-elasticsearch8-1
securityContext:
runAsUser: 0
env:
- name: ES_PASSWORD
valueFrom:
secretKeyRef:
name: logging-es-elastic-user
key: elastic
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
memory: 512Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: config
mountPath: /fluentd/etc
- name: es-certs
mountPath: /etc/es-certs
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
- name: config
configMap:
name: fluentd-config
- name: es-certs
secret:
secretName: logging-es-http-certs-public
```
Container log files in `/var/log/containers` are symlinks into `/var/log/pods`, so mounting `/var/log` covers both. The toleration lets Fluentd also run on control-plane nodes that are tainted.
Apply it and check that one pod runs per node:
```bash
kubectl apply -f fluentd-daemonset.yaml
kubectl -n logging get daemonset fluentd
```
```output
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd 1 1 1 1 1 45s
```
Look at the Fluentd logs for connection errors:
```bash
kubectl -n logging logs daemonset/fluentd --tail=20
```
A healthy start ends with a line such as `fluentd worker is now running worker=0`, with no repeated `Could not communicate to Elasticsearch` warnings.
> Note: The `elastic` superuser keeps this tutorial short. In production, create a dedicated Elasticsearch user with a role that can only write to `k8s-*` indices and give Fluentd those credentials instead.
## Step 6 - Adding a retention policy
Without retention, daily indices accumulate until the disk is full and Elasticsearch stops accepting writes. An index lifecycle management (ILM) policy deletes indices after a set age, and an index template applies it to every new `k8s-*` index.
Open a port forward to Elasticsearch in a second terminal and leave it running:
```bash
kubectl -n logging port-forward service/logging-es-http 9200
```
Back in the first terminal, create a policy that deletes indices seven days after creation. The `-k` flag skips certificate name checks because you are connecting through `localhost`; the traffic never leaves the server:
```bash
curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_ilm/policy/k8s-logs" \
-H 'Content-Type: application/json' \
-d '{"policy":{"phases":{"hot":{"actions":{}},"delete":{"min_age":"7d","actions":{"delete":{}}}}}}'
```
```output
{"acknowledged":true}
```
Create an index template that attaches the policy to new `k8s-*` indices. `number_of_replicas: 0` keeps a single-node cluster `green`, because there is no second node to hold replicas:
```bash
curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_index_template/k8s-logs" \
-H 'Content-Type: application/json' \
-d '{"index_patterns":["k8s-*"],"template":{"settings":{"index.lifecycle.name":"k8s-logs","number_of_replicas":0}}}'
```
```output
{"acknowledged":true}
```
The template applies to indices created from now on. Apply the same settings to the index Fluentd has already created today:
```bash
curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/k8s-*/_settings" \
-H 'Content-Type: application/json' \
-d '{"index.lifecycle.name":"k8s-logs","number_of_replicas":0}'
```
Adjust `7d` to your retention needs and available disk.
## Step 7 - Verifying that logs arrive
Start a short-lived pod that prints a few lines:
```bash
kubectl run logger --image=busybox:stable --restart=Never -- \
sh -c 'for i in 1 2 3 4 5; do echo "hello from logger $i"; sleep 1; done'
```
After about ten seconds, list the log indices:
```bash
curl -sk -u "elastic:$ES_PASSWORD" "https://localhost:9200/_cat/indices/k8s-*?v"
```
```output
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open k8s-2026.09.25 a1B2c3D4e5F6g7H8i9J0kL 1 0 15230 0 6.1mb 6.1mb
```
Search for the test pod's lines. The metadata filter stored the pod name in `kubernetes.pod_name`:
```bash
curl -sk -u "elastic:$ES_PASSWORD" \
"https://localhost:9200/k8s-*/_search?q=kubernetes.pod_name:logger&size=1&pretty"
```
```output
{
...
"hits" : {
"total" : { "value" : 5, "relation" : "eq" },
"hits" : [
{
"_index" : "k8s-2026.09.25",
"_source" : {
"stream" : "stdout",
"logtag" : "F",
"message" : "hello from logger 1",
"kubernetes" : {
"namespace_name" : "default",
"pod_name" : "logger",
"container_name" : "logger",
...
```
Delete the test pod and stop the Elasticsearch port forward with `Ctrl+C`:
```bash
kubectl delete pod logger
```
## Step 8 - Exploring logs in Kibana
Kibana is only reachable inside the cluster. Forward its Service on the server:
```bash
kubectl -n logging port-forward service/logging-kb-http 5601
```
From your workstation, open an SSH tunnel to the server, replacing `your_user` and `your_server_ip`:
```bash
ssh -N -L 5601:127.0.0.1:5601 your_user@your_server_ip
```
Open `https://localhost:5601`, accept the self-signed certificate warning and log in as `elastic` with the password stored in `ES_PASSWORD` (print it on the server with `echo "$ES_PASSWORD"`).
To see the logs:
1. Open the main menu and go to **Discover**.
2. Click **Create data view**, set **Index pattern** to `k8s-*` and **Timestamp field** to `@timestamp`, then save it.
3. Filter with KQL, for example `kubernetes.namespace_name : "default"` or `message : "error"`.
Add columns such as `kubernetes.pod_name` and `message` from the field list to get a readable log view.
## Troubleshooting
**Elasticsearch pod is `Pending`.** No node has enough free memory or the PersistentVolumeClaim cannot bind. Run `kubectl -n logging describe pod logging-es-default-0` and check the events at the bottom.
**Elasticsearch restarts with `max virtual memory areas vm.max_map_count [65530] is too low`.** Step 1 was not applied on the node running the pod. Set the sysctl there.
**Fluentd logs show `certificate verify failed`.** The `host` in the ConfigMap must match a name in the certificate. Use `logging-es-http.logging.svc`, not an IP address.
**No `k8s-*` index appears.** Check that files exist under `/var/log/containers` on the node and that Fluentd is not logging `pattern not matched` errors. If your runtime is Docker Engine with the json-file driver instead of containerd, replace the `cri` parser with `@type json`.
**Index health is `yellow`.** The index has replicas that cannot be allocated on a single node. Apply the settings request from Step 6 to set `number_of_replicas` to `0`.
## Conclusion
You deployed Elasticsearch and Kibana with the ECK operator, shipped every container's logs with a Fluentd DaemonSet, verified them end to end and limited retention to seven days. For production, raise `count` to three Elasticsearch nodes (and set replicas back to 1), give Fluentd a dedicated write-only user, and consider adding the `fluent-plugin-concat` filter to join multi-line stack traces into single events.