--- title: "How to Set Up Centralized Logging with ELK on Ubuntu 24.04" description: "Install Elasticsearch, Logstash and Kibana 9 on Ubuntu 24.04, ship Nginx logs from other servers with Filebeat, parse them with grok and search them in Kibana." category: "Observability Stack" language: en source: https://cubepath.com/docs/observability-stack/centralized-logging-with-elk-stack-complete-guide updated: 2026-09-25 --- # How to Set Up Centralized Logging with ELK on Ubuntu 24.04 The ELK Stack combines Elasticsearch (storage and search), Logstash (parsing and enrichment) and Kibana (the web interface) to collect logs from many servers in one place. A lightweight shipper, Filebeat, runs on each server and forwards log lines to Logstash. In this tutorial you will install Elasticsearch, Logstash and Kibana 9 on one Ubuntu 24.04 server, ship Nginx access logs from a second server with Filebeat, parse them into structured fields and explore them in Kibana, with a retention policy that deletes old data automatically. ## Prerequisites To follow this tutorial, you will need: - An **ELK server** running Ubuntu 24.04 LTS with at least 4 vCPU and 8 GB of RAM (for example a CubePath VPS), and a non-root user with `sudo` privileges. Elasticsearch, Logstash and Kibana all run on the JVM or Node.js and together use several gigabytes of memory. - A **client server** running Ubuntu 24.04 with Nginx installed, whose access logs you will collect. - UFW enabled on the ELK server, with SSH allowed. Throughout the guide, replace `elk_server_ip` with the private or public IP address of the ELK server and `client_server_ip` with the address of the client. ## Step 1 - Adding the Elastic APT repository All Elastic components are installed from the same official repository. Run these commands on the ELK server, and later on the client for Filebeat. Import the Elastic signing key: ```bash sudo apt update sudo apt install -y gnupg curl curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /etc/apt/keyrings/elastic.gpg ``` Add the repository for the 9.x release line: ```bash echo "deb [signed-by=/etc/apt/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list sudo apt update ``` Confirm that the packages come from the new repository: ```bash apt-cache policy elasticsearch ``` ```output elasticsearch: Installed: (none) Candidate: 9.1.4 Version table: 9.1.4 500 500 https://artifacts.elastic.co/packages/9.x/apt stable/main amd64 Packages ``` ## Step 2 - Installing Elasticsearch Install the package: ```bash sudo apt install -y elasticsearch ``` On the first installation, Elasticsearch enables security automatically: it generates TLS certificates, turns on authentication and prints a password for the built-in `elastic` superuser. The output looks like this: ```output --------------------------- Security autoconfiguration information ------------------------------ Authentication and authorization are enabled. TLS for the transport and HTTP layers is enabled and configured. The generated password for the elastic built-in superuser is : your_generated_password ... ``` Save that password in a password manager. If you lose it, reset it with `sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic`. The default configuration listens only on `localhost`, which is what you want: only Logstash and Kibana on the same server talk to Elasticsearch. The JVM heap is sized automatically from the available memory. Start Elasticsearch and enable it at boot: ```bash sudo systemctl daemon-reload sudo systemctl enable --now elasticsearch ``` Verify that it answers over HTTPS with the generated CA certificate. Enter the `elastic` password when prompted: ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200 ``` ```output { "name" : "elk", "cluster_name" : "elasticsearch", "version" : { "number" : "9.1.4", ... }, "tagline" : "You Know, for Search" } ``` To avoid typing the password in every command of this guide, store it in a shell variable for the current session: ```bash read -s -p "elastic password: " ES_PASS; echo ``` ## Step 3 - Preparing retention and a Logstash user Before any data arrives, create an index lifecycle management (ILM) policy that deletes log indices after 30 days, and an index template that applies it to every `nginx-*` index. The template also sets zero replicas, because a single-node cluster has nowhere to place a replica and would otherwise report a `yellow` status. ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \ -X PUT https://localhost:9200/_ilm/policy/logs-30d \ -H 'Content-Type: application/json' -d ' { "policy": { "phases": { "hot": { "actions": {} }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } }' ``` ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \ -X PUT https://localhost:9200/_index_template/nginx \ -H 'Content-Type: application/json' -d ' { "index_patterns": ["nginx-*"], "template": { "settings": { "number_of_replicas": 0, "index.lifecycle.name": "logs-30d" } } }' ``` Each command returns `{"acknowledged":true}`. Logstash should not use the `elastic` superuser. Create a role that can only write to `nginx-*` indices, and a user with that role. Replace `your_logstash_password` with a strong password: ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \ -X POST https://localhost:9200/_security/role/logstash_writer \ -H 'Content-Type: application/json' -d ' { "cluster": ["monitor"], "indices": [ { "names": ["nginx-*"], "privileges": ["create_index", "create", "write"] } ] }' ``` ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" \ -X POST https://localhost:9200/_security/user/logstash_writer \ -H 'Content-Type: application/json' -d ' { "password": "your_logstash_password", "roles": ["logstash_writer"] }' ``` The first returns `{"role":{"created":true}}` and the second `{"created":true}`. ## Step 4 - Installing and connecting Kibana Install Kibana: ```bash sudo apt install -y kibana ``` Kibana connects to the secured cluster with an enrollment token. Generate one and pass it to the setup tool: ```bash sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana sudo /usr/share/kibana/bin/kibana-setup --enrollment-token your_enrollment_token ``` The tool ends with a message confirming that Kibana was configured successfully. The setup writes the Elasticsearch address, credentials and CA certificate into `/etc/kibana/kibana.yml`. Start Kibana: ```bash sudo systemctl enable --now kibana sudo systemctl status kibana ``` Kibana listens on `localhost:5601`, so it is not exposed to the internet. To open it from your workstation, create an SSH tunnel: ```bash ssh -L 5601:localhost:5601 your_user@elk_server_ip ``` Then browse to `http://localhost:5601` on your workstation and log in as `elastic`. Kibana can take a minute to become available after starting. ## Step 5 - Configuring Logstash Install Logstash: ```bash sudo apt install -y logstash ``` Logstash needs the Elasticsearch CA certificate to verify the HTTPS connection. Copy it to a directory Logstash can read: ```bash sudo mkdir -p /etc/logstash/certs sudo cp /etc/elasticsearch/certs/http_ca.crt /etc/logstash/certs/ sudo chown -R root:logstash /etc/logstash/certs sudo chmod 0640 /etc/logstash/certs/http_ca.crt ``` Create a pipeline that receives events from Filebeat, parses Nginx access log lines and writes them to a daily index: ```bash sudo nano /etc/logstash/conf.d/nginx.conf ``` ```text input { beats { port => 5044 } } filter { if "nginx-access" in [tags] { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } } } output { if "nginx-access" in [tags] { elasticsearch { hosts => ["https://localhost:9200"] user => "logstash_writer" password => "${LOGSTASH_ES_PASSWORD}" ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"] index => "nginx-%{+YYYY.MM.dd}" data_stream => false manage_template => false } } } ``` What the pipeline does: - `beats` listens on port 5044 for Filebeat connections. - `grok` with `COMBINEDAPACHELOG` splits the Nginx combined log format into fields such as `source.address`, `http.request.method`, `url.original`, `http.response.status_code` and `user_agent.original`. - `date` replaces the event time with the time from the log line, so events appear at the moment they happened rather than when they were received. - `manage_template => false` keeps Logstash from installing its own template over the one you created in step 3. The password is read from an environment variable instead of being written in the pipeline. The Logstash systemd unit loads variables from `/etc/default/logstash`, so create that file: ```bash sudo nano /etc/default/logstash ``` ```ini LOGSTASH_ES_PASSWORD=your_logstash_password ``` Use the password you chose for `logstash_writer` in step 3, and make the file readable only by root (systemd reads it before starting the service): ```bash sudo chmod 0600 /etc/default/logstash ``` Test the pipeline syntax before starting the service. The test only checks the configuration, so a dummy value for the variable is enough: ```bash sudo -u logstash env LOGSTASH_ES_PASSWORD=dummy /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t ``` ```output ... Configuration OK ``` Start Logstash and confirm that it is listening on port 5044: ```bash sudo systemctl enable --now logstash sudo ss -ltnp | grep 5044 ``` ```output LISTEN 0 1024 *:5044 *:* users:(("java",pid=4211,fd=110)) ``` Logstash takes 30 to 60 seconds to start. If the port does not appear, check `sudo journalctl -u logstash -n 50` and `/var/log/logstash/logstash-plain.log`. Allow Filebeat connections only from the client server: ```bash sudo ufw allow from client_server_ip to any port 5044 proto tcp ``` ## Step 6 - Shipping logs with Filebeat Run the rest of the commands on the **client server**. Add the Elastic repository exactly as in step 1, then install Filebeat: ```bash sudo apt install -y filebeat ``` The default `/etc/filebeat/filebeat.yml` ships to Elasticsearch and contains many commented examples. Keep a copy and replace it with a minimal configuration: ```bash sudo mv /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.orig sudo nano /etc/filebeat/filebeat.yml ``` ```yaml filebeat.inputs: - type: filestream id: nginx-access paths: - /var/log/nginx/access.log tags: ["nginx-access"] output.logstash: hosts: ["elk_server_ip:5044"] ``` The `filestream` input tracks its position in each file, so it resumes after restarts and log rotation without sending lines twice. The `nginx-access` tag is what the Logstash pipeline matches on. Set the permissions Filebeat requires on its configuration file and test it: ```bash sudo chown root:root /etc/filebeat/filebeat.yml sudo chmod 0600 /etc/filebeat/filebeat.yml sudo filebeat test config sudo filebeat test output ``` ```output Config OK logstash: elk_server_ip:5044... connection... parse host... OK dns lookup... OK addresses: elk_server_ip dial up... OK TLS... WARN secure connection disabled talk to server... OK ``` The TLS warning is expected: the connection is not encrypted yet (see the conclusion). Start Filebeat: ```bash sudo systemctl enable --now filebeat ``` Generate a few requests on the client so there are fresh log lines: ```bash for i in $(seq 1 20); do curl -s -o /dev/null http://localhost/; done curl -s -o /dev/null http://localhost/does-not-exist ``` ## Step 7 - Verifying the data in Elasticsearch and Kibana Back on the ELK server, list the indices that Logstash created: ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" "https://localhost:9200/_cat/indices/nginx-*?v" ``` ```output health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open nginx-2026.09.25 4mV0cQ1lQ3eQx2h6yJb9Aw 1 0 21 0 58.1kb 58.1kb ``` The index is `green` because the template set zero replicas. Confirm that ILM manages it: ```bash sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ES_PASS" "https://localhost:9200/nginx-*/_ilm/explain?filter_path=indices.*.policy" ``` The response shows `"policy":"logs-30d"` for each index. Now create a data view in Kibana so you can search the logs: 1. Open **Stack Management > Data Views** (use the search bar at the top if you do not find it in the menu) and click **Create data view**. 2. Set **Name** to `Nginx access` and **Index pattern** to `nginx-*`. 3. Select `@timestamp` as the **Timestamp field** and save. Go to **Discover**, choose the `Nginx access` data view and set the time range to the last 15 minutes. You will see your requests as structured documents. Try these KQL queries in the search bar: ```text http.response.status_code >= 400 ``` ```text http.request.method : "GET" and url.original : "/does-not-exist" ``` Click a field such as `source.address` in the sidebar to see its top values, or add columns to the table to build a readable request log. ## Troubleshooting - **`_grokparsefailure` tag on events**: the log line does not match the combined format, usually because Nginx uses a custom `log_format`. Either switch Nginx back to the default `combined` format or write a grok pattern for your format and test it in Kibana under **Dev Tools > Grok Debugger**. - **Filebeat `talk to server` fails**: the firewall on the ELK server blocks port 5044 or Logstash is not running. Check `sudo ufw status` and `sudo ss -ltnp | grep 5044` on the ELK server. - **Logstash logs `401 Unauthorized`**: the password in `/etc/default/logstash` does not match the `logstash_writer` user. Fix it and run `sudo systemctl restart logstash`. - **Elasticsearch stops accepting writes**: when the disk passes 95% usage (the flood stage watermark), Elasticsearch makes indices read-only. Free space or shorten the ILM retention; the block is removed automatically once usage drops. - **Events appear with the wrong time**: the `date` filter did not match, so `@timestamp` is the arrival time. Check that the Nginx timestamp looks like `25/Sep/2026:10:15:32 +0000`. ## Conclusion You now have a working ELK Stack: Filebeat ships Nginx logs from a client server, Logstash parses them into structured fields, Elasticsearch stores them with a 30-day retention policy and Kibana lets you search and analyze them. Adding another server only requires installing Filebeat with the same configuration and allowing its IP on port 5044. As next steps, enable TLS between Filebeat and Logstash (the `ssl_enabled` and certificate options of the Beats input) before shipping logs over the public internet, add inputs for other logs such as `/var/log/nginx/error.log` with their own tags and filters, and put Kibana behind an Nginx reverse proxy with HTTPS if you need access without an SSH tunnel.