The ELK Stack combines Elasticsearch (storage and search), Logstash (parsing and enrichment) and Kibana (the web interface) to collect logs from many servers in one place. A lightweight shipper, Filebeat, runs on each server and forwards log lines to Logstash. In this tutorial you will install Elasticsearch, Logstash and Kibana 9 on one Ubuntu 24.04 server, ship Nginx access logs from a second server with Filebeat, parse them into structured fields and explore them in Kibana, with a retention policy that deletes old data automatically.

Prerequisites

To follow this tutorial, you will need:

  • An ELK server running Ubuntu 24.04 LTS with at least 4 vCPU and 8 GB of RAM (for example a CubePath VPS), and a non-root user with sudo privileges. Elasticsearch, Logstash and Kibana all run on the JVM or Node.js and together use several gigabytes of memory.
  • A client server running Ubuntu 24.04 with Nginx installed, whose access logs you will collect.
  • UFW enabled on the ELK server, with SSH allowed.

Throughout the guide, replace elk_server_ip with the private or public IP address of the ELK server and client_server_ip with the address of the client.

Step 1 - Adding the Elastic APT repository

All Elastic components are installed from the same official repository. Run these commands on the ELK server, and later on the client for Filebeat.

Import the Elastic signing key:

sudo apt update
sudo apt install -y gnupg curl
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /etc/apt/keyrings/elastic.gpg

Add the repository for the 9.x release line:

echo "deb [signed-by=/etc/apt/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt update

Confirm that the packages come from the new repository:

apt-cache policy elasticsearch

elasticsearch:
  Installed: (none)
  Candidate: 9.1.4
  Version table:
     9.1.4 500
        500 https://artifacts.elastic.co/packages/9.x/apt stable/main amd64 Packages

Step 2 - Installing Elasticsearch

Install the package:

sudo apt install -y elasticsearch

On the first installation, Elasticsearch enables security automatically: it generates TLS certificates, turns on authentication and prints a password for the built-in elastic superuser. The output looks like this:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : your_generated_password
...

Save that password in a password manager. If you lose it, reset it with sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic.

The default configuration listens only on localhost, which is what you want: only Logstash and Kibana on the same server talk to Elasticsearch. The JVM heap is sized automatically from the available memory.

Start Elasticsearch and enable it at boot:

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch

Verify that it answers over HTTPS with the generated CA certificate. Enter the elastic password when prompted:

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

{
  "name" : "elk",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "9.1.4",
    ...
  },
  "tagline" : "You Know, for Search"
}

To avoid typing the password in every command of this guide, store it in a shell variable for the current session:

read -s -p "elastic password: " ES_PASS; echo

Step 3 - Preparing retention and a Logstash user

Before any data arrives, create an index lifecycle management (ILM) policy that deletes log indices after 30 days, and an index template that applies it to every nginx-* index. The template also sets zero replicas, because a single-node cluster has nowhere to place a replica and would otherwise report a yellow status.

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \
  -X PUT https://localhost:9200/_ilm/policy/logs-30d \
  -H 'Content-Type: application/json' -d '
{
  "policy": {
    "phases": {
      "hot": { "actions": {} },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}'

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \
  -X PUT https://localhost:9200/_index_template/nginx \
  -H 'Content-Type: application/json' -d '
{
  "index_patterns": ["nginx-*"],
  "template": {
    "settings": {
      "number_of_replicas": 0,
      "index.lifecycle.name": "logs-30d"
    }
  }
}'

Each command returns {"acknowledged":true}.

Logstash should not use the elastic superuser. Create a role that can only write to nginx-* indices, and a user with that role. Replace your_logstash_password with a strong password:

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \
  -X POST https://localhost:9200/_security/role/logstash_writer \
  -H 'Content-Type: application/json' -d '
{
  "cluster": ["monitor"],
  "indices": [
    { "names": ["nginx-*"], "privileges": ["create_index", "create", "write"] }
  ]
}'

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \
  -X POST https://localhost:9200/_security/user/logstash_writer \
  -H 'Content-Type: application/json' -d '
{
  "password": "your_logstash_password",
  "roles": ["logstash_writer"]
}'

The first returns {"role":{"created":true}} and the second {"created":true}.

Step 4 - Installing and connecting Kibana

Install Kibana:

sudo apt install -y kibana

Kibana connects to the secured cluster with an enrollment token. Generate one and pass it to the setup tool:

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
sudo /usr/share/kibana/bin/kibana-setup --enrollment-token your_enrollment_token

The tool ends with a message confirming that Kibana was configured successfully.

The setup writes the Elasticsearch address, credentials and CA certificate into /etc/kibana/kibana.yml. Start Kibana:

sudo systemctl enable --now kibana
sudo systemctl status kibana

Kibana listens on localhost:5601, so it is not exposed to the internet. To open it from your workstation, create an SSH tunnel:

ssh -L 5601:localhost:5601 your_user@elk_server_ip

Then browse to http://localhost:5601 on your workstation and log in as elastic. Kibana can take a minute to become available after starting.

Step 5 - Configuring Logstash

Install Logstash:

sudo apt install -y logstash

Logstash needs the Elasticsearch CA certificate to verify the HTTPS connection. Copy it to a directory Logstash can read:

sudo mkdir -p /etc/logstash/certs
sudo cp /etc/elasticsearch/certs/http_ca.crt /etc/logstash/certs/
sudo chown -R root:logstash /etc/logstash/certs
sudo chmod 0640 /etc/logstash/certs/http_ca.crt

Create a pipeline that receives events from Filebeat, parses Nginx access log lines and writes them to a daily index:

sudo nano /etc/logstash/conf.d/nginx.conf

input {
  beats {
    port => 5044
  }
}

filter {
  if "nginx-access" in [tags] {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      remove_field => [ "timestamp" ]
    }
  }
}

output {
  if "nginx-access" in [tags] {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      user => "logstash_writer"
      password => "${LOGSTASH_ES_PASSWORD}"
      ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
      index => "nginx-%{+YYYY.MM.dd}"
      data_stream => false
      manage_template => false
    }
  }
}

What the pipeline does:

  • beats listens on port 5044 for Filebeat connections.
  • grok with COMBINEDAPACHELOG splits the Nginx combined log format into fields such as source.address, http.request.method, url.original, http.response.status_code and user_agent.original.
  • date replaces the event time with the time from the log line, so events appear at the moment they happened rather than when they were received.
  • manage_template => false keeps Logstash from installing its own template over the one you created in step 3.

The password is read from an environment variable instead of being written in the pipeline. The Logstash systemd unit loads variables from /etc/default/logstash, so create that file:

sudo nano /etc/default/logstash

LOGSTASH_ES_PASSWORD=your_logstash_password

Use the password you chose for logstash_writer in step 3, and make the file readable only by root (systemd reads it before starting the service):

sudo chmod 0600 /etc/default/logstash

Test the pipeline syntax before starting the service. The test only checks the configuration, so a dummy value for the variable is enough:

sudo -u logstash env LOGSTASH_ES_PASSWORD=dummy /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

...
Configuration OK

Start Logstash and confirm that it is listening on port 5044:

sudo systemctl enable --now logstash
sudo ss -ltnp | grep 5044

LISTEN 0      1024               *:5044            *:*    users:(("java",pid=4211,fd=110))

Logstash takes 30 to 60 seconds to start. If the port does not appear, check sudo journalctl -u logstash -n 50 and /var/log/logstash/logstash-plain.log.

Allow Filebeat connections only from the client server:

sudo ufw allow from client_server_ip to any port 5044 proto tcp

Step 6 - Shipping logs with Filebeat

Run the rest of the commands on the client server. Add the Elastic repository exactly as in step 1, then install Filebeat:

sudo apt install -y filebeat

The default /etc/filebeat/filebeat.yml ships to Elasticsearch and contains many commented examples. Keep a copy and replace it with a minimal configuration:

sudo mv /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.orig
sudo nano /etc/filebeat/filebeat.yml

filebeat.inputs:
  - type: filestream
    id: nginx-access
    paths:
      - /var/log/nginx/access.log
    tags: ["nginx-access"]

output.logstash:
  hosts: ["elk_server_ip:5044"]

The filestream input tracks its position in each file, so it resumes after restarts and log rotation without sending lines twice. The nginx-access tag is what the Logstash pipeline matches on.

Set the permissions Filebeat requires on its configuration file and test it:

sudo chown root:root /etc/filebeat/filebeat.yml
sudo chmod 0600 /etc/filebeat/filebeat.yml
sudo filebeat test config
sudo filebeat test output

Config OK
logstash: elk_server_ip:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: elk_server_ip
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

The TLS warning is expected: the connection is not encrypted yet (see the conclusion). Start Filebeat:

sudo systemctl enable --now filebeat

Generate a few requests on the client so there are fresh log lines:

for i in $(seq 1 20); do curl -s -o /dev/null http://localhost/; done
curl -s -o /dev/null http://localhost/does-not-exist

Step 7 - Verifying the data in Elasticsearch and Kibana

Back on the ELK server, list the indices that Logstash created:

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" "https://localhost:9200/_cat/indices/nginx-*?v"

health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   nginx-2026.09.25  4mV0cQ1lQ3eQx2h6yJb9Aw   1   0         21            0     58.1kb         58.1kb

The index is green because the template set zero replicas. Confirm that ILM manages it:

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" "https://localhost:9200/nginx-*/_ilm/explain?filter_path=indices.*.policy"

The response shows "policy":"logs-30d" for each index.

Now create a data view in Kibana so you can search the logs:

  1. Open Stack Management > Data Views (use the search bar at the top if you do not find it in the menu) and click Create data view.
  2. Set Name to Nginx access and Index pattern to nginx-*.
  3. Select @timestamp as the Timestamp field and save.

Go to Discover, choose the Nginx access data view and set the time range to the last 15 minutes. You will see your requests as structured documents. Try these KQL queries in the search bar:

http.response.status_code >= 400

http.request.method : "GET" and url.original : "/does-not-exist"

Click a field such as source.address in the sidebar to see its top values, or add columns to the table to build a readable request log.

Troubleshooting

  • _grokparsefailure tag on events: the log line does not match the combined format, usually because Nginx uses a custom log_format. Either switch Nginx back to the default combined format or write a grok pattern for your format and test it in Kibana under Dev Tools > Grok Debugger.
  • Filebeat talk to server fails: the firewall on the ELK server blocks port 5044 or Logstash is not running. Check sudo ufw status and sudo ss -ltnp | grep 5044 on the ELK server.
  • Logstash logs 401 Unauthorized: the password in /etc/default/logstash does not match the logstash_writer user. Fix it and run sudo systemctl restart logstash.
  • Elasticsearch stops accepting writes: when the disk passes 95% usage (the flood stage watermark), Elasticsearch makes indices read-only. Free space or shorten the ILM retention; the block is removed automatically once usage drops.
  • Events appear with the wrong time: the date filter did not match, so @timestamp is the arrival time. Check that the Nginx timestamp looks like 25/Sep/2026:10:15:32 +0000.

Conclusion

You now have a working ELK Stack: Filebeat ships Nginx logs from a client server, Logstash parses them into structured fields, Elasticsearch stores them with a 30-day retention policy and Kibana lets you search and analyze them. Adding another server only requires installing Filebeat with the same configuration and allowing its IP on port 5044.

As next steps, enable TLS between Filebeat and Logstash (the ssl_enabled and certificate options of the Beats input) before shipping logs over the public internet, add inputs for other logs such as /var/log/nginx/error.log with their own tags and filters, and put Kibana behind an Nginx reverse proxy with HTTPS if you need access without an SSH tunnel.