Kubernetes keeps container logs only on the node where each pod runs, and they disappear when the pod is deleted. An EFK stack collects them in one searchable place: Fluentd runs on every node, reads the container log files, enriches each line with pod and namespace metadata, and sends it to Elasticsearch, where you search it with Kibana. In this tutorial you will deploy Elasticsearch and Kibana with Elastic's ECK operator, ship logs with a Fluentd DaemonSet, check that they arrive, and add a retention policy so the cluster does not fill its disks.

Prerequisites

To follow this tutorial, you will need:

  • A Kubernetes cluster with a default StorageClass, for example K3s on a CubePath VPS running Ubuntu 24.04. The nodes need at least 4 GB of RAM free for Elasticsearch and Kibana, and about 20 GB of disk for log data.
  • kubectl configured with admin access, and Helm installed (sudo snap install helm --classic on Ubuntu).
  • A non-root user with sudo privileges on each node, to adjust a kernel setting in Step 1.
  • The cluster uses containerd or CRI-O as runtime (the default for K3s, MicroK8s, RKE2 and kubeadm today), so container logs are written in the CRI format.

This guide builds a single-node Elasticsearch suitable for small clusters and testing. The Conclusion explains what to change for production.

Step 1 - Preparing the nodes for Elasticsearch

Elasticsearch memory-maps its index files and needs a higher vm.max_map_count than the Ubuntu default. Set it on every node that may run Elasticsearch, and persist it across reboots:

echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf
sudo sysctl --system

Verify the value:

sysctl vm.max_map_count

vm.max_map_count = 262144

Create a namespace for the logging stack:

kubectl create namespace logging

Step 2 - Installing the ECK operator

Elastic Cloud on Kubernetes (ECK) is the operator Elastic supports for running Elasticsearch and Kibana on Kubernetes. It handles TLS certificates, the elastic superuser password, upgrades and rolling restarts. Install it from Elastic's Helm repository:

helm repo add elastic https://helm.elastic.co
helm repo update
helm install elastic-operator elastic/eck-operator \
  --namespace elastic-system --create-namespace

Check that the operator is running:

kubectl -n elastic-system get pods

NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          40s

Step 3 - Deploying Elasticsearch

With the operator in place, you describe the cluster you want as an Elasticsearch resource. This tutorial pins the 8.19 release line, which is the version the Fluentd Elasticsearch 8 image in Step 5 is built for. Check the ECK documentation for the latest 8.19 patch release and use it in both manifests.

nano elasticsearch.yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logging
  namespace: logging
spec:
  version: 8.19.0
  nodeSets:
    - name: default
      count: 1
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 2Gi
                  cpu: 500m
                limits:
                  memory: 2Gi
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  • count: 1 creates a single Elasticsearch node.
  • Elasticsearch sizes its heap automatically to about half of the container memory limit.
  • The volume claim must be named elasticsearch-data; ECK mounts it as the data directory. It uses your default StorageClass.

Apply it and watch the resource until HEALTH is green and PHASE is Ready. This takes one to three minutes:

kubectl apply -f elasticsearch.yaml
kubectl -n logging get elasticsearch -w

NAME      HEALTH   NODES   VERSION   PHASE   AGE
logging   green    1       8.19.0    Ready   2m

Press Ctrl+C to stop watching. ECK has created an HTTPS Service called logging-es-http, a Secret with the password of the elastic user, and a Secret with the CA certificate. Store the password in a shell variable for the next steps:

ES_PASSWORD=$(kubectl -n logging get secret logging-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')

Step 4 - Deploying Kibana

Kibana connects to Elasticsearch through the elasticsearchRef field; ECK configures the credentials and TLS for you.

nano kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: logging
  namespace: logging
spec:
  version: 8.19.0
  count: 1
  elasticsearchRef:
    name: logging

kubectl apply -f kibana.yaml
kubectl -n logging get kibana -w

NAME      HEALTH   NODES   VERSION   AGE
logging   green    1       8.19.0    90s

Press Ctrl+C once health is green.

Step 5 - Deploying Fluentd as a DaemonSet

Fluentd must run on every node to read the log files the container runtime writes under /var/log/containers. A DaemonSet does exactly that. The official fluent/fluentd-kubernetes-daemonset image already contains the plugins needed here: the CRI log parser, the Kubernetes metadata filter and the Elasticsearch output.

First, give Fluentd read access to pod and namespace metadata:

nano fluentd-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: logging

kubectl apply -f fluentd-rbac.yaml

Next, write the Fluentd configuration as a ConfigMap. It tails the container logs, parses the CRI format, adds Kubernetes metadata, and writes to daily indices named k8s-YYYY.MM.DD over verified TLS:

nano fluentd-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/fluentd-*.log"]
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type cri
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <match kubernetes.**>
      @type elasticsearch
      host logging-es-http.logging.svc
      port 9200
      scheme https
      ssl_verify true
      ca_file /etc/es-certs/ca.crt
      user elastic
      password "#{ENV['ES_PASSWORD']}"
      logstash_format true
      logstash_prefix k8s
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.buffer
        flush_mode interval
        flush_interval 5s
        chunk_limit_size 8MB
        retry_max_interval 30s
        overflow_action block
      </buffer>
    </match>
  • exclude_path stops Fluentd from ingesting its own logs, which would create a feedback loop.
  • pos_file records how far each file has been read, so a restarted pod does not resend everything.
  • The file buffer on the host keeps unsent logs if Elasticsearch is temporarily unavailable.

Apply the ConfigMap:

kubectl apply -f fluentd-config.yaml

Now create the DaemonSet. It mounts the node's /var/log, the ConfigMap and the Elasticsearch CA certificate, and reads the password from the Secret ECK created. Check Docker Hub for the newest -debian-elasticsearch8- tag of the image and use it:

nano fluentd-daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccountName: fluentd
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1.18-debian-elasticsearch8-1
          securityContext:
            runAsUser: 0
          env:
            - name: ES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: logging-es-elastic-user
                  key: elastic
          resources:
            requests:
              cpu: 100m
              memory: 200Mi
            limits:
              memory: 512Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: config
              mountPath: /fluentd/etc
            - name: es-certs
              mountPath: /etc/es-certs
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: fluentd-config
        - name: es-certs
          secret:
            secretName: logging-es-http-certs-public

Container log files in /var/log/containers are symlinks into /var/log/pods, so mounting /var/log covers both. The toleration lets Fluentd also run on control-plane nodes that are tainted.

Apply it and check that one pod runs per node:

kubectl apply -f fluentd-daemonset.yaml
kubectl -n logging get daemonset fluentd

NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd   1         1         1       1            1           <none>          45s

Look at the Fluentd logs for connection errors:

kubectl -n logging logs daemonset/fluentd --tail=20

A healthy start ends with a line such as fluentd worker is now running worker=0, with no repeated Could not communicate to Elasticsearch warnings.

Step 6 - Adding a retention policy

Without retention, daily indices accumulate until the disk is full and Elasticsearch stops accepting writes. An index lifecycle management (ILM) policy deletes indices after a set age, and an index template applies it to every new k8s-* index.

Open a port forward to Elasticsearch in a second terminal and leave it running:

kubectl -n logging port-forward service/logging-es-http 9200

Back in the first terminal, create a policy that deletes indices seven days after creation. The -k flag skips certificate name checks because you are connecting through localhost; the traffic never leaves the server:

curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_ilm/policy/k8s-logs" \
  -H 'Content-Type: application/json' \
  -d '{"policy":{"phases":{"hot":{"actions":{}},"delete":{"min_age":"7d","actions":{"delete":{}}}}}}'

{"acknowledged":true}

Create an index template that attaches the policy to new k8s-* indices. number_of_replicas: 0 keeps a single-node cluster green, because there is no second node to hold replicas:

curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_index_template/k8s-logs" \
  -H 'Content-Type: application/json' \
  -d '{"index_patterns":["k8s-*"],"template":{"settings":{"index.lifecycle.name":"k8s-logs","number_of_replicas":0}}}'

{"acknowledged":true}

The template applies to indices created from now on. Apply the same settings to the index Fluentd has already created today:

curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/k8s-*/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"index.lifecycle.name":"k8s-logs","number_of_replicas":0}'

Adjust 7d to your retention needs and available disk.

Step 7 - Verifying that logs arrive

Start a short-lived pod that prints a few lines:

kubectl run logger --image=busybox:stable --restart=Never -- \
  sh -c 'for i in 1 2 3 4 5; do echo "hello from logger $i"; sleep 1; done'

After about ten seconds, list the log indices:

curl -sk -u "elastic:$ES_PASSWORD" "https://localhost:9200/_cat/indices/k8s-*?v"

health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   k8s-2026.09.25  a1B2c3D4e5F6g7H8i9J0kL   1   0      15230            0      6.1mb          6.1mb

Search for the test pod's lines. The metadata filter stored the pod name in kubernetes.pod_name:

curl -sk -u "elastic:$ES_PASSWORD" \
  "https://localhost:9200/k8s-*/_search?q=kubernetes.pod_name:logger&size=1&pretty"

{
  ...
  "hits" : {
    "total" : { "value" : 5, "relation" : "eq" },
    "hits" : [
      {
        "_index" : "k8s-2026.09.25",
        "_source" : {
          "stream" : "stdout",
          "logtag" : "F",
          "message" : "hello from logger 1",
          "kubernetes" : {
            "namespace_name" : "default",
            "pod_name" : "logger",
            "container_name" : "logger",
            ...

Delete the test pod and stop the Elasticsearch port forward with Ctrl+C:

kubectl delete pod logger

Step 8 - Exploring logs in Kibana

Kibana is only reachable inside the cluster. Forward its Service on the server:

kubectl -n logging port-forward service/logging-kb-http 5601

From your workstation, open an SSH tunnel to the server, replacing your_user and your_server_ip:

ssh -N -L 5601:127.0.0.1:5601 your_user@your_server_ip

Open https://localhost:5601, accept the self-signed certificate warning and log in as elastic with the password stored in ES_PASSWORD (print it on the server with echo "$ES_PASSWORD").

To see the logs:

  1. Open the main menu and go to Discover.
  2. Click Create data view, set Index pattern to k8s-* and Timestamp field to @timestamp, then save it.
  3. Filter with KQL, for example kubernetes.namespace_name : "default" or message : "error".

Add columns such as kubernetes.pod_name and message from the field list to get a readable log view.

Troubleshooting

Elasticsearch pod is Pending. No node has enough free memory or the PersistentVolumeClaim cannot bind. Run kubectl -n logging describe pod logging-es-default-0 and check the events at the bottom.

Elasticsearch restarts with max virtual memory areas vm.max_map_count [65530] is too low. Step 1 was not applied on the node running the pod. Set the sysctl there.

Fluentd logs show certificate verify failed. The host in the ConfigMap must match a name in the certificate. Use logging-es-http.logging.svc, not an IP address.

No k8s-* index appears. Check that files exist under /var/log/containers on the node and that Fluentd is not logging pattern not matched errors. If your runtime is Docker Engine with the json-file driver instead of containerd, replace the cri parser with @type json.

Index health is yellow. The index has replicas that cannot be allocated on a single node. Apply the settings request from Step 6 to set number_of_replicas to 0.

Conclusion

You deployed Elasticsearch and Kibana with the ECK operator, shipped every container's logs with a Fluentd DaemonSet, verified them end to end and limited retention to seven days. For production, raise count to three Elasticsearch nodes (and set replicas back to 1), give Fluentd a dedicated write-only user, and consider adding the fluent-plugin-concat filter to join multi-line stack traces into single events.