Reading Server Logs: Finding the Actual Problem Faster

A checkout form failed “around four in the afternoon”. You open the application log, scroll to 16:00, and find nothing at all: clean requests, ordinary responses, no errors anywhere near that window. The temptation now is to decide the report was wrong, which is the professional version of blaming the customer. Check the server’s clock instead. Most cloud images ship set to UTC, so four in the afternoon in Patna is 10:30 in the log file, five and a half hours from where you were staring. The failure is sitting there with a database timeout and a full stack trace attached, exactly as reported. Very little about reading logs is technically difficult. What makes it slow is method: which clock, which window, which file, and whether you read forward from before the failure or backward from the last line. This is mostly an article about method.

form a hypothesis before you search

The instinct when something breaks is to run grep -i error across everything and scroll. It’s almost always the slow route, because a busy server logs errors constantly that have nothing to do with your problem: expired sessions, aborted uploads, scanners probing for admin paths you don’t have.

Write down three things first, even if only in your head. What exactly failed, from the user’s point of view. When, in the server’s timezone. Which component would have to be involved for that symptom to appear. Now you’re not searching, you’re testing a guess, and you’ll know within a minute whether the guess was wrong.

The other half is reading forward from before the failure rather than backward from the last line. Causes precede symptoms. The stack trace at 10:31 is often a consequence of something boring at 10:28, like a connection pool filling, and the boring line is the one you can fix.

where logs actually live

Two systems coexist on a modern Ubuntu or Debian box and knowing which is which saves a lot of blundering about.

The journal holds anything managed by systemd, which is every service you start with systemctl, plus kernel messages. It’s a binary store you query with journalctl, it records which unit each line came from, and it’s where you look for “did this service start, and what did it say”.

Plain files under /var/log hold whatever applications write themselves. nginx access and error logs, MySQL’s error log, PHP-FPM, mail. These are the ones you read with tail, grep and awk. Containers are a third case: their output goes to Docker’s logging driver, so docker compose logs is the right tool, and anything inside the container writing to a file is invisible from the host unless you’ve mounted a volume. That’s one of several reasons to have application containers log to stdout, as in my Docker deployment guide.

make the journal survive reboots

On some images the journal is kept in memory only, so a reboot erases it. That’s exactly the history you want after a crash, and finding it gone is a genuinely bad moment.

# is the journal persistent? this directory should exist
ls -d /var/log/journal 2>/dev/null || echo "journal is volatile"

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

# how much space is the journal using? (the permanent cap, SystemMaxUse=, is set below)
sudo journalctl --disk-usage

the journalctl flags that earn their keep

# one service, a specific window, oldest first
sudo journalctl -u nginx --since "2026-08-26 10:15" --until "2026-08-26 10:45" --no-pager

# follow live, last 50 lines for context
sudo journalctl -u myapp -f -n 50

# errors and worse, this boot only
sudo journalctl -p err -b --no-pager

# the previous boot, which is where an unexpected reboot hides
sudo journalctl -b -1 -p warning

# kernel messages: OOM kills, disk errors, network resets
sudo journalctl -k --since "1 hour ago"

# two services interleaved in time order, which is how you spot cause and effect
sudo journalctl -u nginx -u myapp --since "09:00" --until "11:00"

That last one is my favourite and it’s underused. Most real incidents involve two components, and seeing the proxy’s complaint next to the app’s exception in one ordered stream answers “which failed first” immediately. --since also accepts human phrasing like "2 hours ago" or "yesterday".

nginx logs, field by field

The default combined format gives you a line per request, and the positions are worth memorising because every one-liner below depends on them:

203.0.113.45 - - [26/Aug/2026:10:31:07 +0000] "POST /checkout HTTP/1.1" 502 166 "https://technobabu.com/cart" "Mozilla/5.0 ..."

#  $1  client address
#  $4  timestamp, in the server's timezone
#  $6  method        $7  path
#  $9  status code   $10 bytes sent
#  $11 referrer      $12 onwards, user agent

Two files, two purposes. The access log records every request and its outcome, which is what you count and aggregate. The error log records why nginx itself was unhappy, and for a proxied app that’s where the upstream failure message lives. On a 502, the access log says it happened; the error log says the upstream refused the connection or timed out reading the response header.

4xx is about them, 5xx is about you

Status	Usual meaning here	Where to look next
404	Bad link, or a scanner probing paths	Access log: is it one path or hundreds?
401 / 403	Auth failure, or file permissions	App log, then file ownership
413	Upload larger than client_max_body_size	nginx error log, then raise the limit
499	Client gave up first, nginx-specific	Upstream response time, not the client
500	Your application threw an exception	App log, at that exact second
502	Upstream refused or died	Is the app process running at all?
504	Upstream too slow to respond	Slow query log, external API calls

The mental shortcut: a 4xx is normally a statement about the request, a 5xx is a statement about your server. A flood of 404s is usually automated traffic and not an outage, while a single 502 means something behind the proxy was not answering. If your proxy configuration is new, the server block basics in my nginx setup guide will help you make sense of which upstream a given error refers to.

the timezone trap

This is the one that costs almost everybody an hour at least once, and it is worth building a habit against. Cloud images default to UTC. Your users describe times in IST. Nothing warns you and everything looks consistent, because the logs are internally correct.

# what does this box think the time is?
timedatectl

# convert a user-reported IST time (UTC+05:30) into UTC, which is what the log will say
date -u -d '2026-08-26 16:00 +0530'
# prints: Wed Aug 26 10:30:00 UTC 2026

# or set the box to local time and stop converting in your head
sudo timedatectl set-timezone Asia/Kolkata

# journalctl can render in UTC regardless of the box setting
sudo journalctl -u myapp --since "16:00" --utc

Pick one convention per machine and write it down. I’d set a single-purpose server to Asia/Kolkata, because reports from users here arrive in IST and mental arithmetic under pressure is how mistakes happen. For users in several countries, keep UTC and get comfortable with --utc. Either is defensible; not knowing which a box uses is not. Don’t change it mid-investigation either, or the log you’re reading goes discontinuous.

one-liners that answer real questions

Counting is more useful than reading. These are the handful worth knowing, and the pattern behind all of them is the same: extract a field, sort, count duplicates, sort by count.

cd /var/log/nginx

# which addresses are hitting me hardest?
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# how many of each status code today?
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# which paths are returning server errors?
awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | head

# requests per minute, to see when a spike started
awk '{print substr($4, 2, 17)}' access.log | uniq -c | tail -30

# one client's full history, in order
grep '^203.0.113.45 ' access.log | tail -40

# the last upstream complaints, which is where 502s explain themselves
tail -n 40 error.log

Two practical notes. Rotation means today’s file may not hold the window you want; check access.log.1 and use zcat or zgrep for the compressed older ones. And when you’re about to run six variations of the same search, save the window once and work on the copy instead of re-reading a large file each time:

sudo journalctl -u myapp --since "10:00" --until "11:00" --no-pager > /tmp/window.log
grep -c 'Traceback' /tmp/window.log
grep -n 'database' /tmp/window.log | head

# searching yesterday's rotated and compressed logs
zgrep ' 502 ' /var/log/nginx/access.log.2.gz | head

rotation, and logs eating your disk

Logs are a leading cause of full disks on small servers, and a full disk breaks things in confusing ways: databases refuse writes, sessions fail, the web server can’t even log the failure. logrotate handles the file-based logs on a daily timer, and the journal has its own cap.

# who is using the space?
sudo du -h --max-depth=1 /var/log | sort -h | tail
sudo journalctl --disk-usage

# trim the journal by size or age; this permanently deletes older entries, so export any incident window first
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=14d

# check a logrotate config without waiting a day
sudo logrotate -d /etc/logrotate.d/nginx

For a permanent cap, set SystemMaxUse=200M in /etc/systemd/journald.conf and restart journald. logrotate’s -d is a dry run showing what it would do, which is how you verify a config instead of discovering next week that it never matched your file.

One thing rotation cannot fix: an application that logs every request at debug level in production. That’s a volume problem, not a rotation problem, and the fix is the log level. Disk trending is also the cheapest thing to watch, which is why it’s on the short list in my monitoring guide.

common mistakes

• Grepping for “error” with no time window, then reading unrelated noise for half an hour.
• Comparing a user-reported IST time against a UTC log without converting, and concluding nothing happened.
• Restarting the service before capturing anything. The restart often clears the state that explained the failure.
• Reading only the last line of a stack trace. The cause is usually several frames up, or in a different service entirely.
• Forgetting rotated files, so the incident window has already moved into .log.1 or a gzip archive.
• Treating a burst of 404s from scanners as an attack in progress. It’s background noise on any public IP; the defences worth having are rate limits and current software.

FAQ

what are all these requests to paths i don’t have?

Automated scanners, constantly, on every public address. They probe for known admin panels, old plugin paths and forgotten backup files. It isn’t personal and it isn’t worth investigating individually. Keep software updated, don’t leave real admin panels open to the whole internet, and add rate limiting if the volume becomes a nuisance. Repeated login attempts against SSH are the same category of noise, which is what fail2ban quietly handles.

why does my log show a 200 for a page users say was broken?

Because the server successfully returned something, and that something may have been an error page, a cached stale response, or HTML with a broken script inside it. Status codes describe the HTTP exchange, not whether your feature worked. This is the same gap that makes port-based monitoring misleading, and it’s why checking for expected content beats checking for a 200. Certificate problems are another case where the server is fine and the user still sees a wall, which I cover in the Let’s Encrypt guide.

Next time something breaks, write down the symptom and the time before you touch a terminal, then convert that time into the server’s timezone. That thirty seconds of discipline saves more debugging hours than any clever command in this article.

Update history: the IST-to-UTC date command now converts correctly, and the journal comments say what each command actually does.

Sources

• journalctl(1) manual page — every flag used above, including the time expressions and vacuum options.
• nginx log module reference — the combined log format field order and how to define your own.
• logrotate(8) manual page — rotation directives and the dry-run flag for testing a config.