Pastebin is one of those deceptively simple apps. You think "a text box, a submit button, a URL" — and then six months later you’re waking up at 3am because someone dumped a 50MB base64 blob through your API, your disk is full, and your PostgreSQL table has 40 million rows of spam.
The real engineering challenge isn’t creating pastes. It’s everything around them: how you store content efficiently, how you actually delete it when it expires (not just flag it as expired), and how you stop bad actors from using your service as a free hosting platform for malware, credentials, and worse.
This article is the guide I wish existed when I built my first pastebin. We’ll go from a naive design to something you can actually run in production — with real configs, real tradeoffs, and explicit callouts for where people get burned.
What We’re Building
A pastebin with these characteristics:
- Pastes stored with configurable TTL (1 hour to forever)
- Hard-limit on paste size
- Unique, non-guessable short IDs
- Real deletion (not soft-delete theater)
- Rate limiting on creation
- Basic content abuse prevention
Official reference implementation for Privatebin (a popular self-hosted option): https://github.com/PrivateBin/PrivateBin. We’ll design something similar but break down the decisions explicitly.
Storage: The First Decision That Haunts You
The naive move is a single PostgreSQL table:
CREATE TABLE pastes (
id CHAR(8) PRIMARY KEY,
content TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ,
burn_after_read BOOLEAN NOT NULL DEFAULT false
);
This works. Until it doesn’t. Text in PostgreSQL is TOAST-ed above 2kB, which means your table bloats, VACUUM struggles to reclaim space from deleted rows, and your disk IO pattern becomes unpredictable under load. A table with millions of expired (but not yet vacuumed) rows has real performance consequences.
The better split: metadata in Postgres, content in object storage.
paste_id → metadata (Postgres)
paste_id → raw content blob (S3-compatible or local filesystem)
Your metadata table stays lean:
CREATE TABLE pastes (
id CHAR(10) PRIMARY KEY,
content_key TEXT NOT NULL, -- S3 key or filesystem path
size_bytes INT NOT NULL,
syntax VARCHAR(32),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ,
burn_after_read BOOLEAN NOT NULL DEFAULT false,
deletion_token CHAR(64) -- for user-initiated deletion
);
CREATE INDEX ON pastes (expires_at) WHERE expires_at IS NOT NULL;
Content lives in S3 (or MinIO if you’re self-hosting):
s3://your-bucket/pastes/ab/cd1234ef.txt
Using the first two characters as a prefix directory is an old S3 trick to avoid hotspotting on a single partition when you have millions of objects.
Gotcha: If you store encrypted content (like PrivateBin does — the server never sees plaintext), you can’t do content-based deduplication. Don’t try. The mental overhead isn’t worth it unless you have a very specific cost problem.
Generating IDs That Don’t Leak and Don’t Collide
Auto-increment integers are a terrible idea. They tell every user exactly how many pastes you’ve created, and sequential IDs are trivially guessable.
Use random base62 strings. 8 characters gives you 62^8 ≈ 218 trillion combinations — more than enough, even for busy services. 10 characters if you’re paranoid.
import secrets
import string
ALPHABET = string.ascii_letters + string.digits # base62
def generate_id(length: int = 10) -> str:
return ''.join(secrets.choice(ALPHABET) for _ in range(length))
On collision: just retry. At 10 characters with any reasonable volume (even millions of pastes), a collision is so unlikely that a simple retry loop is correct. Don’t overcomplicate this with distributed counters unless you’re operating at Google scale.
Gotcha: Don’t use UUID4 as your paste URL. It’s 36 characters of ugly. Your users will thank you for short IDs. UUIDs are fine internally for database PKs if you want, but expose the short ID in URLs.
Deletion Policies That Actually Work
This is where most implementations lie to themselves.
TTL Expiry
Setting expires_at and then checking it on read is soft expiry. The data is still there. Your disk is still full. You still have legal liability for the content.
You need a background job that physically deletes expired content.
# cleanup_worker.py — run this as a cron job or long-running async task
import asyncio
import asyncpg
import boto3
async def delete_expired_pastes(db_pool, s3_client, bucket: str, batch_size: int = 500):
async with db_pool.acquire() as conn:
# Grab a batch of expired paste IDs and their S3 keys
rows = await conn.fetch("""
DELETE FROM pastes
WHERE expires_at < now()
RETURNING id, content_key
LIMIT $1
""", batch_size)
if not rows:
return 0
# Delete from S3 in bulk (S3 supports up to 1000 per batch request)
objects = [{'Key': row['content_key']} for row in rows]
s3_client.delete_objects(
Bucket=bucket,
Delete={'Objects': objects}
)
return len(rows)
async def cleanup_loop():
db_pool = await asyncpg.create_pool(dsn=DATABASE_URL)
s3 = boto3.client('s3', ...)
while True:
deleted = await delete_expired_pastes(db_pool, s3, BUCKET)
if deleted == 0:
await asyncio.sleep(60) # nothing to do, back off
else:
await asyncio.sleep(1) # more work likely, keep going
Run this as a separate container — don’t couple it to your API process. If the API crashes, cleanup keeps running.
Gotcha: The DELETE ... RETURNING pattern is crucial. You atomically get the list of things to delete and remove the DB row in one operation. If you SELECT first and DELETE second, there’s a race window where a user can read an expired paste and your cleanup job deletes the S3 object while the content is in flight.
Burn After Read
This one trips people up. A "burn after read" paste should be deleted after it’s delivered to the client, not when the request comes in.
async def get_paste(paste_id: str) -> dict:
paste = await db.fetchrow("SELECT * FROM pastes WHERE id = $1", paste_id)
if not paste:
raise NotFound()
if paste['expires_at'] and paste['expires_at'] < datetime.utcnow():
# Expired — delete and pretend it never existed
await hard_delete(paste)
raise NotFound()
# Fetch content from S3 before deleting
content = await s3_get(paste['content_key'])
if paste['burn_after_read']:
# Schedule deletion AFTER we have the content ready to send
asyncio.create_task(hard_delete(paste))
return {'id': paste['id'], 'content': content, 'syntax': paste['syntax']}
If you delete the DB row and S3 object before sending the response and the connection drops — the paste is gone and the user never got it. Bad UX, hard to explain.
User-Initiated Deletion
Give users a deletion token at creation time. This is a random 64-character hex string stored hashed in the database (bcrypt or SHA-256 is fine here since it’s not a password — SHA-256 with a pepper is sufficient).
import hashlib, os
def create_deletion_token() -> tuple[str, str]:
"""Returns (raw_token_for_user, hashed_token_for_db)"""
raw = secrets.token_hex(32)
hashed = hashlib.sha256(raw.encode()).hexdigest()
return raw, hashed
Return the raw token to the user once, never store it, never show it again. Standard stuff.
Abuse Prevention: The Part Nobody Talks About Until They’re On a Blocklist
A public pastebin with no protection is a free hosting service for malware, leaked credentials, phishing pages, and CSAM. You will get all of these. The question is how fast you detect and remove them.
Rate Limiting at the Edge
The first line of defense is rate limiting paste creation. Use Redis with a sliding window:
import redis.asyncio as aioredis
redis = aioredis.from_url("redis://localhost")
async def check_rate_limit(client_ip: str) -> bool:
key = f"rl:create:{client_ip}"
pipe = redis.pipeline()
now = time.time()
window = 3600 # 1 hour
# Sliding window log
pipe.zremrangebyscore(key, 0, now - window)
pipe.zadd(key, {str(now): now})
pipe.zcard(key)
pipe.expire(key, window)
results = await pipe.execute()
count = results[2]
return count <= 20 # 20 pastes per hour per IP
Apply stricter limits for unauthenticated users. If you offer accounts, authenticated users can get higher quotas.
Gotcha: IP-based rate limiting is trivially bypassed by anyone with a residential proxy or a /48 IPv6 block. It’s not a silver bullet — it’s speed bumps for casual abuse and crawlers.
Size Limits — Enforce at Multiple Layers
Set a hard limit (I use 512KB for text, 2MB max). Enforce it:
- At the load balancer/proxy (
client_max_body_size 2min Nginx) - In your application before touching storage
- As a column constraint or S3 size check
server {
client_max_body_size 2m;
location /api/paste {
proxy_pass http://pastebin_backend;
}
}
If someone sends 500MB and your only check is in the app layer, you’re reading 500MB into memory before rejecting it. The nginx limit kills it at the socket.
Content-Based Abuse Detection
For a personal or small community pastebin, a blocklist of known bad strings (hardcoded malware hashes, obvious phishing URLs) gets you most of the way.
For anything public, you need more:
Option 1: Google Safe Browsing API
Check any URLs found in paste content. Free tier is generous.
import httpx
async def check_urls_safe_browsing(urls: list[str]) -> list[str]:
"""Returns list of flagged URLs."""
if not urls:
return []
payload = {
"client": {"clientId": "yourpastebin", "clientVersion": "1.0"},
"threatInfo": {
"threatTypes": ["MALWARE", "SOCIAL_ENGINEERING", "UNWANTED_SOFTWARE"],
"platformTypes": ["ANY_PLATFORM"],
"threatEntryTypes": ["URL"],
"threatEntries": [{"url": u} for u in urls],
}
}
async with httpx.AsyncClient() as client:
resp = await client.post(
f"https://safebrowsing.googleapis.com/v4/threatMatches:find?key={GOOGLE_API_KEY}",
json=payload
)
data = resp.json()
matches = data.get("matches", [])
return [m["threat"]["url"] for m in matches]
Option 2: ClamAV for scanning paste content
Run ClamAV as a sidecar and scan content before storing:
# docker-compose.yml excerpt
services:
clamav:
image: clamav/clamav:latest
volumes:
- clamav_data:/var/lib/clamav
healthcheck:
test: ["CMD", "clamdscan", "--ping", "3"]
interval: 30s
pastebin:
image: yourpastebin:latest
environment:
- CLAMAV_HOST=clamav
- CLAMAV_PORT=3310
depends_on:
clamav:
condition: service_healthy
import pyclamd
def scan_content(content: bytes) -> bool:
"""Returns True if content is clean."""
cd = pyclamd.ClamdNetworkSocket(host=CLAMAV_HOST, port=3310)
result = cd.scan_stream(content)
return result is None # None means clean
ClamAV’s virus definitions update automatically. This catches a huge chunk of malware pastes with minimal false positives on code content.
Reporting and Manual Review Queue
Any abuse prevention system needs a human backstop. Build a report endpoint from day one:
CREATE TABLE reports (
id SERIAL PRIMARY KEY,
paste_id CHAR(10) REFERENCES pastes(id) ON DELETE CASCADE,
reason VARCHAR(32) NOT NULL,
reporter_ip INET,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
reviewed BOOLEAN NOT NULL DEFAULT false
);
When a paste accumulates N reports (I use 3), auto-suspend it (hide from public, keep for review). Send yourself a notification — neo notify if you’re on this stack, or a simple webhook to a Telegram bot.
Gotcha: Don’t auto-delete on reports. Coordinated false-flag campaigns are real. A malicious group can report-bomb any paste into oblivion. Human review before deletion is non-negotiable for anything that isn’t clearly illegal.
The Docker Compose Setup
Here’s a production-ready compose file for the whole stack:
# docker-compose.yml
version: "3.9"
services:
pastebin:
image: yourpastebin:latest
restart: unless-stopped
environment:
DATABASE_URL: postgresql://paste:${DB_PASS}@postgres:5432/pastebin
REDIS_URL: redis://redis:6379/0
S3_ENDPOINT: http://minio:9000
S3_BUCKET: pastes
S3_ACCESS_KEY: ${MINIO_ACCESS_KEY}
S3_SECRET_KEY: ${MINIO_SECRET_KEY}
MAX_PASTE_SIZE: "524288" # 512KB
RATE_LIMIT_PER_HOUR: "20"
CLAMAV_HOST: clamav
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_started
ports:
- "8000:8000"
cleanup:
image: yourpastebin:latest
restart: unless-stopped
command: python cleanup_worker.py
environment:
DATABASE_URL: postgresql://paste:${DB_PASS}@postgres:5432/pastebin
S3_ENDPOINT: http://minio:9000
S3_BUCKET: pastes
S3_ACCESS_KEY: ${MINIO_ACCESS_KEY}
S3_SECRET_KEY: ${MINIO_SECRET_KEY}
depends_on:
- postgres
- minio
postgres:
image: postgres:16-alpine
restart: unless-stopped
volumes:
- pg_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
environment:
POSTGRES_DB: pastebin
POSTGRES_USER: paste
POSTGRES_PASSWORD: ${DB_PASS}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U paste -d pastebin"]
interval: 10s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
restart: unless-stopped
command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru
volumes:
- redis_data:/data
minio:
image: minio/minio:latest
restart: unless-stopped
command: server /data --console-address ":9001"
volumes:
- minio_data:/data
environment:
MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
clamav:
image: clamav/clamav:latest
restart: unless-stopped
volumes:
- clamav_data:/var/lib/clamav
volumes:
pg_data:
redis_data:
minio_data:
clamav_data:
A few things worth calling out: the cleanup worker is a separate service — it runs independently and won’t restart-loop your API if it hits a bug. Redis is configured with allkeys-lru eviction so rate limit counters are evicted under memory pressure rather than crashing (losing rate limit state is less bad than your API going down). ClamAV gets its own named volume so virus definitions persist across container restarts — without this, it re-downloads 300MB of definitions every time you redeploy.
PostgreSQL Housekeeping
One thing that bites self-hosters: VACUUM. When your cleanup job deletes thousands of rows an hour, PostgreSQL needs autovacuum to keep up or your table bloat will crater read performance.
Tune autovacuum for the pastes table specifically:
ALTER TABLE pastes SET (
autovacuum_vacuum_scale_factor = 0.01, -- vacuum after 1% of rows are dead
autovacuum_analyze_scale_factor = 0.005,
autovacuum_vacuum_cost_delay = 2 -- ms, faster vacuuming
);
Default scale factor is 20% — fine for tables that change slowly, terrible for one where you’re deleting millions of rows. This gets autovacuum running more aggressively on just this table without affecting the rest of your database.
What "Production-Ready" Actually Means Here
People throw that phrase around. For a pastebin, it means:
- Content is actually deleted, not just hidden. Run a monthly audit: count rows in pastes where expires_at < now(). If this number is growing, your cleanup is broken.
- You have a backup policy for the metadata DB. The S3 content is worthless without the metadata rows.
pg_dumpto a separate bucket, daily. - You’ve tested your rate limiting by actually hammering the endpoint with
aborwrkand verifying the 429s come out correctly. - ClamAV definitions are updating. Check the freshclam logs. Stale definitions are useless.
- You have an ops email or notification when the reports queue gets long. Abuse doesn’t sleep.
The difference between a weekend project and something you can run for years is mostly operational hygiene. The architecture here is solid — the rest is discipline.