Filebeat is Elastic's lightweight log shipper: it tails log files, keeps track of how far it has read, and sends each line to Elasticsearch (or Logstash) with little CPU and memory. Its modules include ready-made parsers and Kibana dashboards for common software such as Nginx and the system logs. In this tutorial you will install Filebeat 9 on Ubuntu 24.04, connect it to an existing Elasticsearch cluster over TLS, enable the system and nginx modules, add a custom input for JSON application logs and switch to a least-privilege API key.

Prerequisites

To follow this tutorial you need:

  • A server running Ubuntu 24.04 LTS whose logs you want to ship, for example a CubePath VPS, with a non-root sudo user.
  • A running Elasticsearch 9.x cluster and Kibana 9.x reachable from that server on ports 9200 and 5601. We refer to them as your_es_host and your_kibana_host.
  • The password of the elastic superuser (only needed for the initial setup) and the cluster's HTTP CA certificate. On a self-managed Elasticsearch installed from packages, it is /etc/elasticsearch/certs/http_ca.crt.
  • Nginx installed on the Filebeat server if you want to follow the Nginx module part.

Step 1 - Installing Filebeat from the Elastic repository

Elastic publishes signed APT packages. Download the signing key into its own keyring so it only applies to the Elastic repository:

sudo install -m 0755 -d /etc/apt/keyrings
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /etc/apt/keyrings/elasticsearch.gpg

Add the repository and install Filebeat:

echo "deb [signed-by=/etc/apt/keyrings/elasticsearch.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt update
sudo apt install filebeat

Check the installed version:

filebeat version

filebeat version 9.1.4 (amd64), libbeat 9.1.4 [...]

Do not start the service yet: it has no valid output configured.

Step 2 - Connecting Filebeat to Elasticsearch

Copy the cluster's HTTP CA certificate to the Filebeat server, for example with scp, and place it in /etc/filebeat/certs/:

sudo mkdir -p /etc/filebeat/certs
sudo cp http_ca.crt /etc/filebeat/certs/http_ca.crt

Instead of writing the password into the configuration file, store it in the Filebeat keystore, which is readable only by root:

sudo filebeat keystore create
sudo filebeat keystore add ES_PWD

Type the elastic password when prompted. You can now reference it as ${ES_PWD}.

Open the main configuration file:

sudo nano /etc/filebeat/filebeat.yml

The packaged file contains many commented examples. Replace its content with the following minimal configuration:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

filebeat.inputs: []

setup.kibana:
  host: "http://your_kibana_host:5601"

output.elasticsearch:
  hosts: ["https://your_es_host:9200"]
  username: "elastic"
  password: "${ES_PWD}"
  ssl.certificate_authorities: ["/etc/filebeat/certs/http_ca.crt"]

processors:
  - add_host_metadata: ~

If Kibana is served over HTTPS, use https:// in setup.kibana.host. When no Kibana credentials are set, Filebeat uses the Elasticsearch output credentials for Kibana.

Validate the file and test the connection:

sudo filebeat test config
sudo filebeat test output

Config OK
elasticsearch: https://your_es_host:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.0.0.20
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 9.1.4

Never "fix" a TLS error by setting ssl.verification_mode: none. If the handshake fails, the CA file or the hostname in hosts is wrong.

Step 3 - Enabling the system and Nginx modules

Modules bundle an input definition, an Elasticsearch ingest pipeline that parses the lines, and Kibana dashboards. List them and enable the two you need:

sudo filebeat modules enable system nginx
sudo filebeat modules list | head -n 5

Enabled:
nginx
system

Disabled:

Enabling a module only renames its file in /etc/filebeat/modules.d/. The filesets inside are still disabled by default, so edit each file. Start with the system module:

sudo nano /etc/filebeat/modules.d/system.yml

- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

Then the Nginx module:

sudo nano /etc/filebeat/modules.d/nginx.yml

- module: nginx
  access:
    enabled: true
  error:
    enabled: true

The modules use the default Ubuntu paths (/var/log/syslog, /var/log/auth.log, /var/log/nginx/access.log*, /var/log/nginx/error.log*). If your logs live elsewhere, add var.paths: ["/path/to/file.log*"] under the fileset.

Step 4 - Loading index templates, pipelines and dashboards

filebeat setup creates the index template, the ILM policy, the ingest pipelines of the enabled modules and the Kibana dashboards. It needs elevated privileges, which is why it runs now with the elastic user, before you switch to a restricted API key. The --pipelines option needs the modules listed explicitly:

sudo filebeat setup -e --pipelines --modules system,nginx

The command takes a minute and ends with lines similar to:

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines

Start Filebeat and enable it at boot:

sudo systemctl enable --now filebeat
systemctl status filebeat --no-pager

Write a test line to syslog and search for it a few seconds later:

logger "filebeat test message from $(hostname)"
curl -s -u elastic --cacert /etc/filebeat/certs/http_ca.crt \
  "https://your_es_host:9200/filebeat-*/_search?q=message:%22filebeat%20test%20message%22&size=1&filter_path=hits.hits._source.message"

After typing the password you should see your message:

{"hits":{"hits":[{"_source":{"message":"filebeat test message from web-01"}}]}}

In Kibana, open Discover and select the filebeat-* data view, or open Dashboards and search for [Filebeat Nginx] or [Filebeat System].

Step 5 - Adding a custom input for JSON application logs

Applications that write one JSON object per line (see structured logging) do not need grok or dissect parsing. A filestream input with the ndjson parser decodes each line into fields. Every filestream input needs a unique id, which Filebeat uses to track read offsets.

Edit /etc/filebeat/filebeat.yml and replace filebeat.inputs: [] with:

filebeat.inputs:
  - type: filestream
    id: orders-api-json
    paths:
      - /var/log/orders-api/*.log
    parsers:
      - ndjson:
          target: "app"
          add_error_key: true
    fields:
      service: orders-api
    fields_under_root: true

target: "app" places the decoded keys under app.*, which avoids clashes with Filebeat's own ECS fields such as message or log.level. add_error_key: true adds an error.message field instead of dropping lines that are not valid JSON.

For plain-text logs with multi-line stack traces, such as Java applications, use the multiline parser. In filestream it is configured under parsers, not at the input level. This example starts a new event on every line that begins with a date and appends all other lines to the previous event:

  - type: filestream
    id: billing-java
    paths:
      - /var/log/billing/app.log
    parsers:
      - multiline:
          type: pattern
          pattern: '^\d{4}-\d{2}-\d{2}'
          negate: true
          match: after

Validate and restart:

sudo filebeat test config
sudo systemctl restart filebeat

Step 6 - Switching to a least-privilege API key

The elastic superuser should not stay in a configuration file on every server. Create an API key that can only write Filebeat data. Run this from any machine that can reach Elasticsearch, and enter the elastic password when prompted:

curl -s -u elastic --cacert /etc/filebeat/certs/http_ca.crt \
  -X POST "https://your_es_host:9200/_security/api_key" \
  -H "Content-Type: application/json" -d '
{
  "name": "filebeat-web-01",
  "role_descriptors": {
    "filebeat_writer": {
      "cluster": ["monitor", "read_ilm", "read_pipeline"],
      "index": [
        {
          "names": ["filebeat-*"],
          "privileges": ["view_index_metadata", "create_doc", "auto_configure"]
        }
      ]
    }
  }
}'

{"id":"VuaCfGcBCdbkQm-e5aOx","name":"filebeat-web-01","api_key":"ui2lp2axTNmsyakw9tvNnw","encoded":"..."}

Store id:api_key in the keystore:

sudo filebeat keystore add ES_API_KEY

Enter the value as VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw (your own id and api_key). Then edit the output in /etc/filebeat/filebeat.yml, replacing username and password:

output.elasticsearch:
  hosts: ["https://your_es_host:9200"]
  api_key: "${ES_API_KEY}"
  ssl.certificate_authorities: ["/etc/filebeat/certs/http_ca.crt"]

Remove the stored password, test and restart:

sudo filebeat keystore remove ES_PWD
sudo filebeat test output
sudo systemctl restart filebeat

When you need to run filebeat setup again (for example after enabling another module or upgrading), temporarily add superuser credentials or run it from an admin machine.

Troubleshooting

No events arrive. Read Filebeat's own log first: sudo journalctl -u filebeat -n 50 --no-pager. Errors such as 403 Forbidden point to API key privileges, x509: certificate signed by unknown authority to the CA file.

Events arrive but are not parsed. The ingest pipelines are missing. Rerun filebeat setup --pipelines --modules system,nginx with superuser credentials.

Exiting: no modules or inputs enabled. You enabled a module but left all its filesets at enabled: false, and there is no custom input. Enable at least one fileset.

A file is not picked up. Check the glob in paths with ls, and remember that each filestream input needs a unique id. Changing an id makes Filebeat treat the files as new and re-read them from the start.

Conclusion

Filebeat now ships system, Nginx and JSON application logs from Ubuntu 24.04 to Elasticsearch over verified TLS, authenticated with an API key that can only write Filebeat data. Next, adjust the filebeat ILM policy in Kibana so retention matches your disk budget, add processors such as drop_event to discard noisy health-check lines, and roll the same configuration out to your other servers with a configuration management tool.