Web server logs answer most operational questions: which pages fail, which clients generate the most traffic, which requests are slow and whether someone is scanning your site. They also grow without limit unless they are rotated. In this tutorial you will add response times to the Apache or Nginx access log on Ubuntu 24.04, analyze the logs with standard command line tools and GoAccess, and adjust log rotation with logrotate.
Prerequisites
To follow this tutorial you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with a non-root user that has
sudoprivileges. - Apache (
apache2) or Nginx installed from the Ubuntu repositories and serving some traffic, so the logs have content to analyze.
Step 1 - Finding the logs
On Ubuntu both servers write two kinds of logs: an access log with one line per request and an error log with warnings, errors and messages from modules.
| Server | Access log | Error log |
|---|---|---|
| Nginx | /var/log/nginx/access.log | /var/log/nginx/error.log |
| Apache | /var/log/apache2/access.log | /var/log/apache2/error.log |
Individual sites can use their own files, set with access_log and error_log in an Nginx server block, or CustomLog and ErrorLog in an Apache <VirtualHost>. To find every log path in use:
grep -rhE '^\s*(access_log|error_log)' /etc/nginx/ | sort -u
grep -rhE '^\s*(CustomLog|ErrorLog)' /etc/apache2/sites-enabled/ | sort -u
Keep per-site logs inside /var/log/nginx/ or /var/log/apache2/; the default rotation rules in Step 5 cover every *.log file in those directories.
Both servers use the combined format by default. A typical line looks like this:
203.0.113.25 - - [25/Sep/2026:10:14:02 +0000] "GET /pricing HTTP/1.1" 200 5123 "https://www.google.com/" "Mozilla/5.0 (X11; Linux x86_64) ..."
The fields, separated by spaces, are: client IP, identity (always -), authenticated user, timestamp, request line, status code, response size in bytes, referrer and user agent. Watch new requests arrive in real time with:
sudo tail -f /var/log/nginx/access.log
Press CTRL+C to stop.
Step 2 - Adding response times to the access log
The combined format does not record how long each request took, which is the most useful number when you investigate slowness. Define a new format that appends it.
Nginx
log_format must be defined in the http context. Ubuntu's nginx.conf includes every file in /etc/nginx/conf.d/ inside http, so create one there:
sudo nano /etc/nginx/conf.d/log-format.conf
log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time $upstream_response_time';
$request_time is the total time in seconds with millisecond precision, and $upstream_response_time is the time spent waiting for a backend such as PHP-FPM or an application server (- for static files). Use the new format in your site's server block:
access_log /var/log/nginx/your_domain.access.log timed;
Test and reload:
sudo nginx -t && sudo systemctl reload nginx
Apache
Create a configuration snippet with the new format:
sudo nano /etc/apache2/conf-available/log-format.conf
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D" timed
%D is the time taken to serve the request in microseconds. Enable the snippet and use the format in your virtual host:
sudo a2enconf log-format
CustomLog ${APACHE_LOG_DIR}/your_domain.access.log timed
sudo apache2ctl configtest && sudo systemctl reload apache2
Make a request and check that the new field appears at the end of the line:
curl -s -o /dev/null http://your_domain/
sudo tail -n 1 /var/log/nginx/your_domain.access.log
203.0.113.25 - - [25/Sep/2026:10:20:11 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/8.5.0" 0.004 0.003
The remaining steps use the default combined format, so the field numbers work with any log. With the timed format the response time is the last field ($NF in awk).
Step 3 - Analyzing logs from the command line
awk, sort and uniq answer most questions in seconds. The examples use the Nginx log; for Apache change the path to /var/log/apache2/access.log. Rotated logs are compressed, and zcat -f reads both plain and gzipped files, so replace sudo cat file with sudo zcat -f /var/log/nginx/access.log* to analyze several days at once.
Count requests per status code:
sudo awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
18234 200
1022 304
611 404
130 301
12 500
Find the ten IP addresses with the most requests:
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 10
Find the most requested URLs that return 404, which reveals broken links and vulnerability scanners:
sudo awk '$9 == 404 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 10
142 /wp-login.php
98 /.env
31 /xmlrpc.php
7 /images/old-logo.png
Requests for /.env or /wp-login.php on a site that is not WordPress are automated scans. They are harmless if those files don't exist, but they are good candidates for Fail2Ban or rate limiting.
List every request that caused a server error:
sudo awk '$9 ~ /^5/' /var/log/nginx/access.log | tail -n 20
Count requests per hour to find traffic peaks. The timestamp is field 4, for example [25/Sep/2026:10:14:02, and characters 2 to 15 contain the date and hour:
sudo awk '{print substr($4, 2, 14)}' /var/log/nginx/access.log | sort | uniq -c
1204 25/Sep/2026:08
2311 25/Sep/2026:09
4870 25/Sep/2026:10
With the timed format from Step 2, list the ten slowest requests (Nginx stores seconds, Apache microseconds):
sudo awk '{print $NF, $7}' /var/log/nginx/your_domain.access.log | sort -rn | head -n 10
For errors, the error log is often more informative than the access log. Show the most frequent messages without timestamps:
sudo tail -n 1000 /var/log/nginx/error.log | cut -d' ' -f6- | sort | uniq -c | sort -rn | head
Step 4 - Getting a visual report with GoAccess
GoAccess parses access logs and shows visitors, requested files, status codes, referrers and user agents in an interactive terminal dashboard or an HTML report. Install it from the Ubuntu repositories:
sudo apt install goaccess
Open the terminal dashboard for the combined format:
sudo goaccess /var/log/nginx/access.log --log-format=COMBINED
Use the arrow keys and TAB to move between panels and q to quit. To include rotated files, pipe them in and pass - as the file name:
sudo zcat -f /var/log/nginx/access.log* | goaccess - --log-format=COMBINED
To produce an HTML report you can open in a browser, write it to your home directory and copy it to your computer with scp:
sudo goaccess /var/log/nginx/access.log --log-format=COMBINED -o ~/report.html
WarningThe report contains IP addresses and URLs from your visitors. Don't publish it in a public web directory without protecting it with a password.
Step 5 - Configuring log rotation
Ubuntu rotates web server logs with logrotate, which runs once a day through a systemd timer. Check that the timer is active:
systemctl list-timers logrotate.timer
The rules for each server are in /etc/logrotate.d/nginx and /etc/logrotate.d/apache2. Open the one for your server:
sudo nano /etc/logrotate.d/nginx
The Nginx file looks like this (Apache's is similar and ends with a reload of apache2):
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi \
endscript
postrotate
invoke-rc.d nginx rotate >/dev/null 2>&1
endscript
}
The important directives are:
dailyandrotate 14: rotate every day and keep 14 old files, so two weeks of history.compressanddelaycompress: gzip old files, but leave the most recent one uncompressed for one more cycle.create 0640 www-data adm: create the new log with permissions that let members of theadmgroup read it.postrotate: tell the server to reopen its log files, otherwise it keeps writing to the renamed file.
To keep logs for 30 days and also rotate any file that grows beyond 500 MB before the daily run, change rotate 14 to rotate 30 and add maxsize 500M under daily:
daily
maxsize 500M
rotate 30
maxsize only takes effect when logrotate runs, so on a very busy site you may also want to run the timer more often. Keep the retention period in line with your privacy obligations: access logs contain IP addresses, which are personal data under the GDPR.
Check the configuration without changing anything. The debug mode prints what logrotate would do:
sudo logrotate --debug /etc/logrotate.d/nginx
rotating pattern: /var/log/nginx/*.log after 1 days empty log files are not rotated, (30 rotations), old logs are removed
considering log /var/log/nginx/access.log
Now: 2026-09-25 10:30
Last rotated at 2026-09-25 00:00
log does not need rotating (log has already been rotated)
To force a rotation now and verify the whole cycle, including the postrotate reload:
sudo logrotate --force /etc/logrotate.d/nginx
ls -lh /var/log/nginx/
-rw-r----- 1 www-data adm 0 Sep 25 10:31 access.log
-rw-r----- 1 www-data adm 4.1M Sep 25 10:31 access.log.1
-rw-r----- 1 www-data adm 812K Sep 24 00:00 access.log.2.gz
Make a new request and confirm it is written to the new, empty access.log, which proves the server reopened its files.
Troubleshooting
Logs stay empty after rotation and requests go to access.log.1. The postrotate script did not make the server reopen its files. Run it by hand (sudo systemctl reload nginx or sudo systemctl reload apache2) and check that the postrotate block in your logrotate file was not removed.
/var/log fills the disk. Find what is using the space with sudo du -sh /var/log/* | sort -h. If the web server logs are the problem, lower rotate, add maxsize, or disable access logging for noisy endpoints such as health checks (access_log off; inside a dedicated Nginx location).
logrotate reports error: skipping ... because parent directory has insecure permissions. A log directory is writable by a group or user other than root. Either fix the directory permissions or add a su root adm line (with the owner and group of that directory) to the rule.
Logs contain the proxy's IP instead of the client's. When the server runs behind a load balancer or CDN, configure set_real_ip_from and real_ip_header in Nginx, or mod_remoteip in Apache, so the real client address is logged.
Conclusion
Your web server now logs response times, you can answer common questions about traffic and errors with a few commands or GoAccess, and log rotation keeps a predictable amount of history on disk. As next steps, feed the scanner patterns you found into Fail2Ban, add rate limiting for abusive clients, or ship the logs to a central system such as Loki or the ELK stack when you manage more than a few servers.