Knowing Your Server Is Down Before Your Users Tell You

Port 443 answers. The certificate is valid, nginx is healthy, the dashboard is a wall of green, and the site has been returning a server error to every visitor for seven hours. Nothing fires, because nothing the monitor examines has changed: a process is listening, and that was the only question being asked. The application behind it ran out of disk and has been failing every request since. This is the most common way small-site monitoring fails, and it fails quietly, which is the worst property a safety net can have. The two definitions drift apart gradually. On day one, “the port is open” and “the site works” mean the same thing. By month six the application has grown a database, a queue worker and a disk that can fill, and the check is still answering the day-one question. Closing that gap is mostly about choosing what to check.

“the port is open” is not “the app works”

There’s a ladder of things you can check, and each rung is closer to what a user experiences:

• Does it ping? Tells you the machine has power and a network. Many providers block ICMP, so a failure here may mean nothing at all.
• Is the port open? Tells you a process is listening. That process can be returning errors to every single request.
• Does HTTP return 200? Better. But a cached error page, or a login page where a dashboard should be, is still a 200.
• Does the page contain the right text? Now you’re checking the app rendered something real, not just that a web server replied.
• Does a health endpoint confirm dependencies? The app itself checks its database and anything else it needs, and reports the result.

Climb at least to the fourth rung. A keyword check costs nothing extra and catches the whole class of failure where the infrastructure is fine and the software isn’t. The fifth rung is a few lines in your app, and it’s the one I’d write first on anything with a database behind it.

# a health endpoint should answer three questions:
#   can I reach the database?
#   can I write to the disk I need?
#   are my background workers alive?
# and return a non-200 status when any answer is no.

curl -fsS --max-time 10 https://app.technobabu.com/healthz
echo "exit status: $?"

two kinds of watching, and you want both

external checks: is it reachable from the outside?

Something outside your server requests your URL every minute or so and shouts when it can’t. This is the single highest-value monitor you can set up, and the reason is uncomfortable: when the box is genuinely dead, anything running on the box is dead too, including its alerting. An external check survives the failure it’s meant to report.

agent-based metrics: why is it unhappy?

The external check tells you something is wrong. It cannot tell you that memory has been climbing steadily for nine days, or that disk crossed ninety percent last Thursday. For that you need something running on the machine recording CPU, memory, disk, load and service state over time.

The trade-off for a small box: a full metrics stack is genuinely heavy. Prometheus plus Grafana plus exporters can eat a meaningful slice of a 1 GB VPS, which is a funny way to cause the resource problem you installed it to detect. On small boxes I’d rather run a lightweight agent, or a handful of shell checks on a timer, and keep the heavy dashboards for a machine that can spare the RAM. If yours is already tight, read why cheap VPS boxes keep dying before you add anything.

Question	External check	On-box agent
Site is unreachable	Detects it	Dies with the box
DNS or domain problem	Detects it	Sees nothing wrong
Memory creeping up for days	Blind to it	Detects it
Disk about to fill	Blind to it	Detects it
One service crashed, others fine	Only if you check that URL	Detects it
Cost on a 1 GB box	None, runs elsewhere	Real, choose carefully

one hosted, one self-hosted

For external checks, a hosted service is the right answer even for a self-hosting enthusiast, because the whole point is independence from your own infrastructure. Free tiers of the well-known uptime services cover several monitors at a one or five minute interval, which is plenty for a personal project.

If you’d rather own it, Uptime Kuma is the obvious self-hosted choice. It’s a single container, the interface is clear, and it does HTTP checks with keyword matching, TLS expiry warnings, push monitors and a long list of notification channels. The rule I’d insist on: run it somewhere other than the server it watches. A Raspberry Pi at home watching your VPS, and the VPS watching your home connection, is a neat arrangement that costs nothing extra.

services:
  kuma:
    image: louislam/uptime-kuma:2
    restart: unless-stopped
    volumes:
      - kuma:/app/data
    ports:
      - "127.0.0.1:3001:3001"   # reach it through a proxy, not directly

volumes:
  kuma:

New installs should use the 2.x image; an existing 1.x install should follow the project’s v1-to-v2 upgrade notes before switching tags.

the dead man’s switch for scheduled jobs

Everything above watches things that are supposed to be running. Backups, dumps and cleanup jobs are supposed to finish, which is a different problem, and normal monitoring is completely blind to it. A job that silently stopped running months ago looks exactly like a job with nothing to report.

Invert it. The job pings a URL after it succeeds. If the ping doesn’t arrive inside the window you configured, you get alerted. What makes this pattern so good is that it detects absence, including the case where the job never started because a path was wrong or the timer was never enabled.

#!/usr/bin/env bash
set -euo pipefail

PING_URL="https://hc.example.com/ping/your-uuid-here"

# do the work; set -e means we never reach the ping if this fails
/home/anurag/bin/backup.sh

# only a genuinely successful run checks in
curl -fsS --max-time 10 --retry 3 "$PING_URL" >/dev/null

The placement of that last line is the whole design. Put it before the work and you’ve built something that reports success unconditionally, which is worse than nothing because it’s reassuring. Pair it with the scheduling patterns in cron versus systemd timers and a missed backup finally becomes loud.

getting the alert onto your phone

Email is where alerts go to be ignored. What actually gets noticed, in ascending order of insistence: a Telegram bot pushing to a private channel, ntfy for a self-hosted push you can set to bypass silent mode, and a real phone call from a paid on-call service for anything business-critical. Telegram plus a distinct sound is a good default, because that channel contains nothing but alerts, so an unread badge means something. Test it by breaking something deliberately at a convenient hour.

thresholds that don’t train you to ignore alerts

Alert fatigue kills monitoring setups, and it’s self-inflicted. You alert on CPU above eighty percent, discover the nightly backup pegs it for twenty minutes, and after a fortnight you’ve stopped reading the channel. Ignored alerts are worse than none, because you believe you’re covered.

Three rules I hold to. First, alert on symptoms users would notice, not on numbers being interesting: the site returning errors, not CPU being busy. Second, require a condition to persist before it fires; two or three consecutive failed checks removes almost all noise from momentary network blips, which on Indian residential links are frequent and meaningless. Third, every recurring alert must either be fixed or its threshold changed. An alert you’ve decided to tolerate is a bug in your monitoring.

the two checks everyone forgets

Disk space and certificate expiry. Both are common causes of small-site outages, and both give you days or weeks of warning if anything is looking.

#!/usr/bin/env bash
set -uo pipefail

# --- disk: percentage used on the root filesystem ---
USED=$(df --output=pcent / | tail -1 | tr -dc '0-9')
if [ "$USED" -ge 85 ]; then
  echo "WARN: root filesystem at ${USED}% used"
fi

# inodes fill independently of bytes, and the error is baffling
df -i / | tail -1

# --- certificate: days remaining on the live socket ---
END=$(echo | openssl s_client -connect technobabu.com:443 \
        -servername technobabu.com 2>/dev/null \
        | openssl x509 -noout -enddate | cut -d= -f2)
DAYS=$(( ( $(date -d "$END" +%s) - $(date +%s) ) / 86400 ))
[ "$DAYS" -lt 21 ] && echo "WARN: certificate expires in $DAYS days"

Check the certificate over the network rather than reading the file on disk. That way you’re testing what browsers actually receive, which catches the case where renewal worked fine and nobody reloaded the web server. I’ve explained that particular trap in more detail in the Let’s Encrypt guide, and it’s exactly the sort of failure an external monitor spots and a file-based check does not.

what to do when it fires

Have a habit rather than improvising at 3 AM. The order below is the one I’d follow, and most incidents end before the fourth step.

1. Confirm it independently. Load the site on mobile data. Monitors do produce false alarms, and a network problem at the checking end is common.
2. Check the cheap causes: df -h for disk, free -h for memory, systemctl --failed for dead units. Thirty seconds, and it’s the answer surprisingly often.
3. Read the logs from the window the alert names, not from now. The useful lines are at the moment things started failing, and the technique for finding them is in reading server logs.
4. Restore service first, understand it second, but write down what you saw before restarting anything. A restart often destroys the evidence.

common mistakes

• Running the monitoring on the machine it monitors. When the box dies, so does the thing meant to tell you.
• Checking the IP and port instead of the real hostname over HTTPS. You then miss DNS failures, certificate expiry and virtual-host misconfiguration entirely.
• Omitting curl’s -f flag in homemade scripts, so a 500 response counts as a pass.
• Alerting on every metric that moves, until the channel is noise and you’ve stopped looking at it.
• Never testing the notification path, then finding out during a real outage that the webhook token expired.
• Monitoring the web server but not the database, the queue worker or the scheduled jobs, which is where quiet failures like to live.

FAQ

what’s the minimum worth setting up for a personal site?

One external HTTP check on your real URL with a keyword match, alerting to your phone after two consecutive failures. That’s fifteen minutes of work on a free tier and it catches the large majority of outages you’d otherwise hear about from a friend. Add disk and certificate checks next, then metrics only when you have a specific question the metrics would answer.

do i need prometheus and grafana?

Not for one or two small servers. They’re excellent tools and they carry real setup cost plus meaningful memory use, which is hard to justify against a shell script on a timer and a hosted uptime check. The moment they earn their place is when you’re asking “was it slower last month than this month” and you need history to answer, or when you’re running enough machines that per-box scripts have become their own maintenance problem.

how do i avoid being woken up for nothing?

Split alerts by severity into two channels. Things that need action now, meaning the site is down or data is at risk, get a loud notification. Everything else goes to a quiet channel you read with your morning tea. Disk at eighty-five percent does not need to wake you; disk at ninety-eight does, and separating those two is the difference between monitoring you trust and monitoring you mute.

Update history: the Uptime Kuma example now uses the 2.x image, and the curl manual link points to the official curl site.

Sources

• RFC 9110, HTTP status code semantics — what a 4xx versus a 5xx response actually claims, which is what your checks interpret.
• curl(1) manual page — the exact behaviour of the fail, max-time and retry flags used above.
• df(1) manual page — output formatting and the inode reporting the disk check relies on.