The Breach You Won’t Notice Until It’s Too Late
You get compromised on a Tuesday. You find out on a Friday — when a customer emails you that their account was accessed from Lagos at 3 AM. You pull the logs. Nothing useful. Nginx access logs with status codes, some vague app errors, maybe a stack trace or two.
The attacker had four days. Your logs had nothing actionable.
This is OWASP A09:2021 — Security Logging and Monitoring Failures — and it sits at number nine on the Top 10 not because it’s rare, but because when it bites you, it turns every other vulnerability into a catastrophe. A SQL injection that gets caught in two minutes is a bad day. One that goes undetected for 96 hours is a data breach with regulatory consequences.
The good news: fixing this is almost entirely operational. No magic framework, no expensive vendor. You need to know what to log, what format to log it in, what to alert on, and what to ignore. Let’s go through it properly.
What A09 Actually Means
The OWASP definition is deliberately broad: "This category is to help detect, escalate, and respond to active breaches." Translation — if your incident response playbook starts with "first, go find the logs," you’re already losing.
Logging failures aren’t just missing logs. They include:
- Logging the right events in a useless format (plain text blobs nobody can query)
- Logging useful data but never alerting on it
- Alerting on everything and drowning in noise until engineers start clicking "acknowledge" reflexively
- Storing logs in the same system being attacked (first thing attackers do: clear logs)
- Keeping logs for 7 days when your breach detection cycle is 30+
Most teams tick the "we have logging" checkbox with an ELK stack that nobody has looked at in six months. That’s not monitoring — that’s log archiving.
What You Must Log
Not everything. That’s the trap beginners fall into. "Log everything" fills your storage, buries the signal in noise, and creates GDPR/PII headaches. Be surgical.
Authentication Events — Every Single One
{
"timestamp": "2026-05-25T14:32:11Z",
"event": "auth.login.failure",
"user_id": null,
"username_attempted": "admin",
"ip": "185.220.101.47",
"user_agent": "python-requests/2.28.0",
"request_id": "req_8f3a2b1c",
"reason": "invalid_password"
}
Log login success AND failure. Log logout. Log password reset requests. Log MFA challenges and whether they passed or failed. Log account lockouts and the triggering event.
The key insight: a single failed login is nothing. 847 failed logins across 200 different usernames from one IP in 90 seconds is a credential stuffing attack. You need the raw events to build that picture.
What NOT to log: the actual password attempted. Ever. Yes, even hashed. It’s unnecessary and creates liability.
Access Control Decisions
{
"timestamp": "2026-05-25T14:33:02Z",
"event": "authz.denied",
"user_id": "usr_9921",
"resource": "/api/admin/users",
"method": "GET",
"required_role": "admin",
"user_roles": ["viewer"],
"ip": "10.0.1.55",
"request_id": "req_9b4c1d2e"
}
Every time your authorization layer says "no," log it. One denied request is normal — a user clicking somewhere they shouldn’t. Twenty denied requests from the same authenticated user in five minutes means someone is probing your API surface.
This catches both external attackers and insider threats. A disgruntled employee who suddenly starts querying admin endpoints they’ve never touched before is a story your logs should be able to tell.
Input Validation Failures
{
"timestamp": "2026-05-25T14:34:19Z",
"event": "validation.failed",
"field": "user_id",
"value_pattern": "contains_sql_keyword",
"endpoint": "/api/products/search",
"ip": "93.184.216.34",
"request_id": "req_2a7f8e3b"
}
Don’t log the raw malicious value — log the pattern of why it failed. This avoids storing actual attack payloads in your logs while still giving you enough to detect scanning and injection attempts.
SQLi probes, XSS attempts, path traversal (../../etc/passwd), template injection — these all leave fingerprints in validation failures.
Application-Level Security Events
These are the ones teams consistently forget:
- Data export events — when a user exports 10,000 records, log it. Normal users don’t export full datasets.
- Privilege escalation — any role change, permission grant, sudo-equivalent action.
- Configuration changes — firewall rule updates, OAuth client creation, API key generation.
- Bulk operations — deleting 500 records in one API call needs a log entry with the actor.
- Token events — API key creation, rotation, revocation. Especially revocation — that’s often a sign someone knows a key leaked.
What You Don’t Need to Log
HTTP 404s at volume. Successful static asset requests. Health check pings from your load balancer. Routine cron job completion. Verbose ORM query logs in production.
Logging these wastes storage and, worse, makes the important events harder to find. Noise is the enemy of detection.
Structured Logging Is Non-Negotiable
If your logs look like this:
[2026-05-25 14:32:11] ERROR: Login failed for user admin from 185.220.101.47
You have a forensics problem. Parsing unstructured text at query time is slow, error-prone, and means you can’t build reliable alerts on field values.
Use JSON. Always. Every log line should be machine-parseable with a predictable schema.
Standard fields every log line should carry:
{
"timestamp": "2026-05-25T14:32:11.234Z",
"level": "WARN",
"event": "auth.login.failure",
"service": "user-api",
"version": "2.4.1",
"environment": "production",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"request_id": "req_8f3a2b1c",
"ip": "185.220.101.47"
}
trace_id and span_id follow OpenTelemetry conventions — they let you correlate a security event across multiple services in a distributed system. This is essential when your authentication service, API gateway, and business logic layer are separate processes.
The Logging Stack: A Practical Setup
Here’s a production-grade logging stack using Grafana Loki, which is far more resource-efficient than ELK for most self-hosted setups. ELK is powerful but hungry — Loki indexes only metadata (labels) and stores log content compressed, which makes it practical to run on a single server.
Official repo: https://github.com/grafana/loki
# docker-compose.yml
version: "3.8"
services:
loki:
image: grafana/loki:3.0.0
container_name: loki
ports:
- "3100:3100"
volumes:
- ./loki-config.yml:/etc/loki/local-config.yaml
- loki_data:/loki
command: -config.file=/etc/loki/local-config.yaml
restart: unless-stopped
promtail:
image: grafana/promtail:3.0.0
container_name: promtail
volumes:
- /var/log:/var/log:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- ./promtail-config.yml:/etc/promtail/config.yml
command: -config.file=/etc/promtail/config.yml
restart: unless-stopped
depends_on:
- loki
grafana:
image: grafana/grafana:11.0.0
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=changethis
- GF_USERS_ALLOW_SIGN_UP=false
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
restart: unless-stopped
depends_on:
- loki
volumes:
loki_data:
grafana_data:
# loki-config.yml
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
chunk_idle_period: 5m
chunk_retain_period: 30s
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
storage_config:
tsdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
filesystem:
directory: /loki/chunks
limits_config:
# Retention: keep security logs for 90 days minimum
retention_period: 2160h
# Reject logs older than 1 hour to catch misconfigured clock skew
reject_old_samples: true
reject_old_samples_max_age: 1h
compactor:
working_directory: /loki/compactor
retention_enabled: true
retention_delete_delay: 2h
# promtail-config.yml
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
# Pick up Docker container logs automatically
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: [__meta_docker_container_name]
target_label: container
- source_labels: [__meta_docker_container_label_com_docker_compose_service]
target_label: service
# System auth logs — critical for host-level security events
- job_name: system_auth
static_configs:
- targets: [localhost]
labels:
job: auth
__path__: /var/log/auth.log
> Gotcha: Promtail needs access to /var/run/docker.sock to auto-discover containers. That gives it essentially root-equivalent access. Run Promtail as a privileged container only if you understand the implication, or use a dedicated log driver instead and point Promtail at /var/lib/docker/containers.
What to Alert On
This is where most setups break down. Engineers set up alerts based on gut feeling, get paged at 2 AM for a false positive, and start ignoring alerts entirely. Alert fatigue is a real security failure — if your team is trained to dismiss pages, the real attack gets dismissed too.
The principle: alert on anomalies, not on events. A single failed login doesn’t warrant a page. The rate of failed logins does.
High-Confidence, Alert Immediately
These have near-zero false positive rates in production:
- Brute force threshold crossed: >10 failed logins for the same account in 5 minutes, or >100 failed logins from the same IP in 10 minutes
- Authentication from impossible geography: successful login from country X, then country Y 30 minutes later (impossible travel)
- First-time admin action after months of inactivity: dormant admin account suddenly making changes
- Security configuration changed: new admin user created, OAuth client added, firewall rule modified
- Bulk data access: single session accessing >1000 records where historical p99 is <50
Medium-Confidence, Alert to Slack/Email (not pager)
- Spike in authorization failures from a single authenticated user (>20 in 10 minutes)
- New IP address for an existing user (worth a notification, not a 3 AM page)
- Input validation failures above baseline (catching active scanning)
- Service account used interactively (service accounts don’t log in from browsers)
Low-Confidence, Dashboard Only
- Single failed login
- Single 403 response
- Dependency health check failures (these have their own alerting pipeline)
Here’s an example Grafana alerting rule expressed as a LogQL query for the brute force case:
# Count failed logins per IP in the last 5 minutes
sum by (ip) (
count_over_time(
{service="user-api"}
| json
| event = "auth.login.failure"
[5m]
)
) > 10
Log Storage: Keep It Off the Compromised Host
This is the most ignored requirement in A09. If your application server gets compromised and the attacker has root, your logs on the same disk are gone or tampered. Attackers know to check /var/log and your app’s log directory first.
Minimum viable setup:
- Ship logs to a separate host in near-real-time (Promtail → remote Loki, Filebeat → remote Elasticsearch,
rsyslog→ remote syslog) - The log destination should have no inbound SSH from the application server — push only, no pull
- The application server should have append-only credentials for the log destination
Production setup:
- Object storage (S3, MinIO, Backblaze B2) as the long-term archive. Loki, Elasticsearch, and most log systems support S3-compatible backends. Object storage has immutability features (S3 Object Lock) that make tampering detectable.
- 90 days minimum retention. PCI-DSS requires 12 months. SOC 2 and GDPR breach notification windows both benefit from longer retention.
> Gotcha: Loki’s reject_old_samples setting will drop logs if your app server’s clock drifts more than the configured threshold. Make sure NTP is running everywhere. Clock skew has killed more log pipelines than any config error.
The Log Format Contract
Define a schema for your security events and treat it like an API contract. When the schema changes, bump a version field. This prevents silent breaking changes where your alerts stop matching because a field got renamed.
# Example Python logging setup that enforces the schema
import logging
import json
import uuid
from datetime import datetime, timezone
class SecurityLogger:
def __init__(self, service_name: str, version: str):
self.service = service_name
self.version = version
self.logger = logging.getLogger("security")
def _base_event(self, event: str, level: str) -> dict:
return {
"timestamp": datetime.now(timezone.utc).isoformat(),
"schema_version": "1",
"level": level,
"event": event,
"service": self.service,
"version": self.version,
}
def auth_failure(self, username: str, ip: str, reason: str, request_id: str = None):
entry = self._base_event("auth.login.failure", "WARN")
entry.update({
"username_attempted": username,
"ip": ip,
"reason": reason, # "invalid_password" | "account_locked" | "mfa_failed"
"request_id": request_id or str(uuid.uuid4()),
})
# Never log the actual password
self.logger.warning(json.dumps(entry))
def authz_denied(self, user_id: str, resource: str, method: str,
required_role: str, user_roles: list, ip: str, request_id: str = None):
entry = self._base_event("authz.denied", "WARN")
entry.update({
"user_id": user_id,
"resource": resource,
"method": method,
"required_role": required_role,
"user_roles": user_roles,
"ip": ip,
"request_id": request_id or str(uuid.uuid4()),
})
self.logger.warning(json.dumps(entry))
Testing Your Detection
This is where almost everyone stops short. You set up logging, you configure alerts, and you assume it works. Then six months later you realize the alert query had a typo and never fired.
Treat your detection logic like production code. Write tests for it.
Drill: simulate a brute force attack
#!/bin/bash
# brute_force_sim.sh — run against your staging environment only
# Verify that your brute force alert fires within the configured window
TARGET="https://staging.yourdomain.com/api/auth/login"
IP_HEADER="X-Forwarded-For: 10.99.99.99" # simulated attacker IP
for i in $(seq 1 15); do
curl -s -o /dev/null -w "%{http_code}\n" \
-X POST "$TARGET" \
-H "Content-Type: application/json" \
-H "$IP_HEADER" \
-d '{"username":"admin","password":"wrongpassword'$i'"}'
sleep 1
done
echo "Sent 15 failed login attempts. Check your alerting system."
Run this quarterly. Document the expected outcome. If the alert doesn’t fire, something in your pipeline broke — Promtail config, LogQL query, label mismatch, whatever. Find it before an attacker does.
> Gotcha: Many alerting systems have a minimum evaluation interval. If your Grafana alert evaluates every 5 minutes but your threshold is "10 failures in 1 minute," you might miss a fast brute force that finishes before the next evaluation cycle. Set evaluation intervals shorter than your detection window.
PII in Logs: The GDPR Trap
You will be tempted to log user email addresses, IP addresses, and device identifiers alongside security events. IP addresses are PII under GDPR in most EU member state interpretations. Email addresses definitely are.
Practical approach: log internal IDs (user_id: "usr_9921") rather than PII. Keep a separate, access-controlled mapping from IDs to real identities. This means your log storage doesn’t need GDPR-level data protection controls, and investigators can look up the identity when needed through a proper access-controlled system.
For IPs: log them in security events — there’s a legitimate interest argument for security purposes — but document the legal basis in your data processing records and honor deletion requests by anonymizing historical log entries (replace with 0.0.0.0 or truncate the last octet).
The Monitoring Maturity Ladder
Don’t try to implement all of this at once. Here’s a realistic progression:
Week 1 — Stop flying blind: structured JSON logging for auth and authz events. Ship to a remote destination. Confirm logs arrive.
Month 1 — Basic detection: brute force alert, impossible travel check, admin configuration change notification. Test each one manually.
Quarter 1 — Behavioral baseline: dashboards showing normal request volumes, typical login patterns by hour, usual data export sizes. You can’t detect anomalies without knowing your normal.
Ongoing — Red team your detection: quarterly exercises where someone on your team (or a hired consultant) tries to do something malicious in staging and you verify your alerts fire. Treat failed detection the same way you treat a failed test — it’s a bug.
The Real Takeaway
A09 is fundamentally a discipline problem, not a technology problem. The tools are cheap and mature. The hard part is the organizational habit of treating logs as security-critical infrastructure — not a debugging afterthought.
Your logs should be able to answer: who did what, from where, at what time, on which resource, and whether it succeeded. If you can answer those five questions for every security-relevant action in your system within two minutes of an incident starting, you’ve solved A09.
If you can’t, start with authentication events today. Everything else can follow.