Prometheus is a fantastic scraper and alerting engine. Its storage layer is not. The built-in TSDB is designed for short retention — two weeks is the sweet spot. Push it to 6 months and you’re fighting disk I/O, compaction stalls, and a --storage.tsdb.retention.time flag that feels more like a prayer than a guarantee.
The usual escape hatches — Thanos, Cortex, Mimir — come with real operational weight: object storage buckets, a fleet of sidecars, and a Helm chart that needs its own oncall rotation. If you run small-to-medium infrastructure and just want reliable metrics for 1–2 years without burning a weekend on Kubernetes operators, VictoriaMetrics is the answer nobody talks about loudly enough.
This guide walks through the full path: single-node deployment, the cluster mode with proper sharding, wiring Prometheus to use it as a remote write target, and migrating historical data without losing a single datapoint. No fluff, no "enterprise considerations" footnotes — just the config and the gotchas.
Official repo: https://github.com/VictoriaMetrics/VictoriaMetrics
Why VictoriaMetrics Specifically
The compression story alone is worth attention. VictoriaMetrics uses a custom encoding that typically gets 10–20x compression compared to Prometheus TSDB. A 30-day Prometheus dataset that occupies 50 GB shrinks to 3–5 GB. That’s not marketing copy — it’s the result of their variant of Gorilla timestamp encoding combined with aggressive delta-of-deltas compression on values.
Query performance is genuinely better too. Prometheus’s PromQL engine scans time-ordered blocks sequentially. VictoriaMetrics rewrites queries internally to its MetricsQL dialect (a strict superset of PromQL — all your existing queries work), and its storage layout is optimized for time range scans rather than just label filtering.
The single binary is a nice operational story: one process, no JVM, no Zookeeper, no Etcd. The cluster mode adds three purpose-built components but stays dependency-free.
Single-Node Setup
For most setups serving under ~1M active time series, the single-node version covers everything. Below is a production-ready Docker Compose file.
# docker-compose.yml
version: "3.9"
services:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.101.0
container_name: victoriametrics
restart: unless-stopped
ports:
- "8428:8428" # HTTP API, Prometheus-compatible query endpoint
volumes:
- vm-data:/storage
command:
- "-storageDataPath=/storage"
- "-retentionPeriod=12" # months; 12 = 1 year retention
- "-httpListenAddr=:8428"
- "-search.maxQueryDuration=60s" # guard against runaway queries
- "-search.maxUniqueTimeSeries=5000000"
# Enable downsampling for old data — keeps raw resolution
# for the last 30 days, 5m resolution beyond that
- "-downsampling.period=30d:5m,180d:1h"
volumes:
vm-data:
driver: local
driver_opts:
type: none
o: bind
device: /data/victoriametrics # use a dedicated mount point
Start it:
mkdir -p /data/victoriametrics
docker compose up -d
Verify it’s alive:
curl https://cd-linux.club:8428/health
# Returns "OK"
curl https://cd-linux.club:8428/metrics | head -30
# Exposes its own internal metrics — you'll want to scrape these
Gotcha #1: Retention period is in months, not days. Set -retentionPeriod=365d and VictoriaMetrics silently rounds it. The unit for the flag is months unless you explicitly append d. So 12 means 12 months, 365d also means 12 months. For a year, use 12. For 90 days specifically, use 3 or 90d — the d suffix works for exact day counts.
Gotcha #2: The data directory must be on a separate volume. If you leave it on the root filesystem or a shared volume that also serves other containers, you will eventually hit disk pressure and VictoriaMetrics will stop ingesting data silently while returning stale results. Dedicate a mount point from day one.
Configuring Prometheus Remote Write
Open your prometheus.yml and add the remote_write block:
# prometheus.yml (relevant section)
global:
scrape_interval: 15s
evaluation_interval: 15s
remote_write:
- url: "http://victoriametrics:8428/api/v1/write"
# Queue tuning — defaults are often too conservative for high-cardinality envs
queue_config:
capacity: 100000
max_shards: 30
min_shards: 5
max_samples_per_send: 10000
batch_send_deadline: 5s
min_backoff: 30ms
max_backoff: 5s
# Optional: write-ahead log for crash resilience
write_relabel_configs:
# Drop high-cardinality debug metrics you don't want long-term
- source_labels: [__name__]
regex: "go_gc_.*|python_gc_.*"
action: drop
Reload Prometheus:
kill -HUP $(pidof prometheus)
# or via the API:
curl -X POST https://cd-linux.club:9090/-/reload
Check that data is flowing:
# Query VictoriaMetrics directly — should return Prometheus's own metrics
curl 'https://cd-linux.club:8428/api/v1/query?query=up' | python3 -m json.tool | head -20
Gotcha #3: Prometheus keeps writing to its local TSDB in parallel. Remote write is additive — it doesn’t replace local storage. You’re now storing data in two places. Once you’re confident in VictoriaMetrics, either reduce local retention aggressively (--storage.tsdb.retention.time=7d) or point Grafana datasources at VM exclusively so local TSDB is just a fallback buffer.
Cluster Mode: When Single-Node Isn’t Enough
The cluster version splits responsibilities into three components:
vminsert— accepts writes, shards series acrossvmstoragenodes via consistent hashingvmstorage— the actual storage layer; stateful, one node per shardvmselect— handles read queries, fans out to allvmstoragenodes, merges results
This architecture means you can scale writes by adding vminsert instances (stateless) and scale storage by adding vmstorage nodes. Adding a new vmstorage node triggers automatic rebalancing.
Here’s a minimal cluster docker-compose.yml with two storage shards:
# docker-compose.cluster.yml
version: "3.9"
services:
# ---------- Storage nodes (stateful, one per shard) ----------
vmstorage-1:
image: victoriametrics/vmstorage:v1.101.0-cluster
container_name: vmstorage-1
restart: unless-stopped
ports:
- "8482:8482" # HTTP (health, metrics)
- "8400:8400" # vminsert data port
- "8401:8401" # vmselect read port
volumes:
- vmstorage-1-data:/storage
command:
- "-storageDataPath=/storage"
- "-retentionPeriod=12"
- "-httpListenAddr=:8482"
vmstorage-2:
image: victoriametrics/vmstorage:v1.101.0-cluster
container_name: vmstorage-2
restart: unless-stopped
ports:
- "8483:8482"
- "8402:8400"
- "8403:8401"
volumes:
- vmstorage-2-data:/storage
command:
- "-storageDataPath=/storage"
- "-retentionPeriod=12"
- "-httpListenAddr=:8482"
# ---------- Insert layer (stateless, scale horizontally) ----------
vminsert:
image: victoriametrics/vminsert:v1.101.0-cluster
container_name: vminsert
restart: unless-stopped
ports:
- "8480:8480"
command:
- "-storageNode=vmstorage-1:8400"
- "-storageNode=vmstorage-2:8400"
- "-httpListenAddr=:8480"
# Consistent hashing: each series always goes to the same shard
# This is the default; explicit here for clarity
- "-replicationFactor=1"
# ---------- Select layer (stateless, scale horizontally) ----------
vmselect:
image: victoriametrics/vmselect:v1.101.0-cluster
container_name: vmselect
restart: unless-stopped
ports:
- "8481:8481"
command:
- "-storageNode=vmstorage-1:8401"
- "-storageNode=vmstorage-2:8401"
- "-httpListenAddr=:8481"
- "-search.maxQueryDuration=60s"
volumes:
vmstorage-1-data:
driver: local
driver_opts:
type: none
o: bind
device: /data/vmstorage-1
vmstorage-2-data:
driver: local
driver_opts:
type: none
o: bind
device: /data/vmstorage-2
With this running, point Prometheus at vminsert instead:
remote_write:
- url: "http://vminsert:8480/insert/0/prometheus/api/v1/write"
# The "0" is the tenant ID — useful for multi-tenant setups;
# use 0 for a single-tenant deployment
And Grafana at vmselect:
http://vmselect:8481/select/0/prometheus
Gotcha #4: Sharding is consistent but not balanced. VictoriaMetrics shards by metric name + label set hash. This means a single high-cardinality metric (say, http_requests_total with thousands of label combinations) can land heavily on one shard. Monitor vmstorage disk usage and query latency per node. There’s no automatic rebalancing for existing data when adding a new shard — new series go to the new node, existing ones stay put.
Gotcha #5: Replication is not built into the open-source version. Setting -replicationFactor=2 on vminsert will attempt to write each series to two nodes, but vmselect needs -replicationFactor=2 too to deduplicate on reads. This works but doubles your storage. For HA without the overhead, run two independent single-node instances and fan out writes with Prometheus’s multiple remote_write targets.
Downsampling: Saving Space Without Losing Insight
Long-term storage is useless if every Grafana query over a 1-year range brings the server to its knees. Downsampling is the answer — keep full resolution for recent data, coarser aggregates for older data.
VictoriaMetrics handles this natively with the -downsampling.period flag (requires enterprise for the cluster version; single-node is free):
-downsampling.period=30d:5m,180d:1h
This means:
- Data older than 30 days: stored at 5-minute resolution
- Data older than 180 days: stored at 1-hour resolution
- Data within the last 30 days: full raw resolution
Downsampling runs in the background and doesn’t affect ingestion. The original raw data is not deleted immediately — it’s compacted during the normal merge cycle.
Migrating Historical Data from Prometheus
You’ve got months of data in Prometheus TSDB. You need it in VictoriaMetrics. The right tool is vmctl, VictoriaMetrics’s official migration utility.
Install vmctl:
# Grab the latest release binary
VERSION="v1.101.0"
wget "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VERSION}/vmutils-linux-amd64-${VERSION}.tar.gz"
tar -xzf vmutils-linux-amd64-${VERSION}.tar.gz
chmod +x vmctl-prod
mv vmctl-prod /usr/local/bin/vmctl
Basic migration from a live Prometheus instance:
vmctl prometheus \
--prom-snapshot-dir=/var/lib/prometheus/snapshots/$(date +%Y%m%d-%H%M%S) \
--vm-addr=https://cd-linux.club:8428 \
--prom-concurrency=4 \
--verbose
But this requires a snapshot. Take one first:
# Create a Prometheus TSDB snapshot via its admin API
# (requires --web.enable-admin-api flag on Prometheus)
curl -X POST https://cd-linux.club:9090/api/v1/admin/tsdb/snapshot
# The response gives you the snapshot name:
# {"status":"success","data":{"name":"20260523T120000Z-abc123def456"}}
# Then migrate from that snapshot directory
vmctl prometheus \
--prom-snapshot-dir=/var/lib/prometheus/data/snapshots/20260523T120000Z-abc123def456 \
--vm-addr=https://cd-linux.club:8428 \
--prom-concurrency=8 \
--vm-concurrency=8
For a remote Prometheus (when you can’t access the filesystem directly), use the HTTP API migration mode:
vmctl remote-read \
--remote-read-src-addr=http://prometheus:9090 \
--remote-read-step-interval=day \
--remote-read-filter-time-start=2025-06-01T00:00:00Z \
--remote-read-filter-time-end=2026-05-23T00:00:00Z \
--vm-addr=http://victoriametrics:8428 \
--vm-concurrency=4
Gotcha #6: Migration via HTTP is much slower than snapshot migration. The remote-read protocol is not designed for bulk transfer. Expect 50–100k samples per second versus 500k+ from direct TSDB access. For a year of data from a busy Prometheus, plan for several hours and run it in a tmux session.
Gotcha #7: Check for duplicate data after migration. If you migrated historical data and then started streaming via remote_write, you’ll have overlapping data around the migration cutoff. VictoriaMetrics handles duplicates gracefully — it deduplicates on read — but it does consume extra storage until the old data ages out or you run a manual compaction.
Production tip: Run migrations during low-traffic hours and cap concurrency. Setting --prom-concurrency=16 on a 4-core Prometheus host that’s also actively scraping will cause scrape latency and missed samples. Use 4–8 as a safe starting point.
Wiring Grafana
Add VictoriaMetrics as a Prometheus-compatible datasource in Grafana. In grafana.ini or the provisioning YAML:
# grafana/provisioning/datasources/victoriametrics.yaml
apiVersion: 1
datasources:
- name: VictoriaMetrics
type: prometheus
access: proxy
url: http://victoriametrics:8428 # or vmselect:8481 for cluster
isDefault: true
jsonData:
timeInterval: "15s"
queryTimeout: "60s"
# Use GET for queries (VictoriaMetrics handles both, GET is safer behind some proxies)
httpMethod: "GET"
MetricsQL is a strict superset of PromQL. Every dashboard you have today works without modification. The extras — median_over_time, limitk, histogram_quantiles — are available when you want them but you’re not forced to use them.
Monitoring VictoriaMetrics Itself
Don’t skip this. VictoriaMetrics exposes a rich /metrics endpoint at port 8428. Have Prometheus scrape it:
# prometheus.yml scrape config addition
- job_name: 'victoriametrics'
static_configs:
- targets: ['victoriametrics:8428']
Key metrics to alert on:
# Write queue backing up — remote_write falling behind
vm_rpc_send_duration_seconds_bucket
# Storage running low
(vm_data_size_bytes / vm_available_disk_space_bytes) > 0.85
# Query latency degradation
vm_request_duration_seconds{quantile="0.99"} > 5
The built-in Grafana dashboard from the official repo (https://grafana.com/grafana/dashboards/10229) covers all of these and is worth importing on day one.
Production Checklist
Before you call this setup production-ready:
- Separate disk mount for storage data with monitored free space
- Retention period explicitly set — never rely on defaults
- Downsampling configured if retention > 90 days
- Prometheus remote_write queue tuning reviewed
-
vmctlmigration tested and verified withvmctl verifysubcommand - Grafana datasource pointed at VM, old Prometheus datasource kept as fallback
- VictoriaMetrics own metrics scraped and basic alerts in place
- Backup strategy —
vmbackupto S3/GCS/local (also an official tool from the same team)
The path from "Prometheus is filling up my disk" to "I have a year of metrics and a Grafana dashboard to prove it" is shorter than most people expect. VictoriaMetrics doesn’t try to be a cloud-native platform — it tries to be a reliable, fast time-series database you can run on a $20 VPS. For the vast majority of monitoring needs, that’s exactly what you want.