Your cloud bill arrived. It’s 40% higher than last month. You open the AWS Cost Explorer, stare at a pile of aggregated numbers, and have absolutely no idea which team, service, or deployment caused the spike. Congratulations — you’ve hit the wall that every growing engineering org hits eventually.
Traditional FinOps is a finance problem. Cost SRE is an engineering problem. The difference is that you instrument your system to emit cost signals the same way you emit latency or error rate signals. You build dashboards that answer "how much does this endpoint cost per call?" not just "what’s the monthly EC2 bill?"
This article is a practical guide to building that system — from cloud-level tag propagation all the way down to per-request cost attribution in your application layer.
Why Your Current Cost Setup Is Lying to You
AWS, GCP, and Azure billing UIs are built for accountants. They aggregate by service (EC2, RDS, S3) and by region. They have zero idea what your microservices are or which team owns what.
The standard response is cost allocation tags. You slap team=payments on your EC2 instances and call it a day. This works until you have 12 teams sharing a Kubernetes cluster, a handful of shared databases, and a CDN that serves content for everyone. Suddenly your tags cover maybe 60% of spend and the rest is in "unallocated."
The real answer is multi-layered cost attribution:
- Cloud-level — resource tags and billing exports to get raw cost data
- Infrastructure-level — per-pod, per-namespace, per-service cost allocation in Kubernetes
- Application-level — per-request cost signals via OpenTelemetry
Most teams only do layer 1 and wonder why their cost reviews are useless.
Layer 1: Cloud Billing as a Data Source
Before you can allocate anything, you need raw cost data in a queryable form. Every major cloud has a billing export:
- AWS: Cost and Usage Reports (CUR) → S3 → Athena or a data warehouse
- GCP: Billing export → BigQuery
- Azure: Cost exports → Storage Account → Synapse or similar
For Kubernetes-heavy shops, the trick is getting your Kubernetes metadata into the billing stream. This means tagging your nodes at provisioning time with enough context (cluster, environment) and then doing the finer-grained allocation at the Kubernetes level.
AWS Node tag example via Terraform:
resource "aws_eks_node_group" "workers" {
cluster_name = aws_eks_cluster.main.name
node_role_arn = aws_iam_role.node.arn
# ...
tags = {
Environment = var.environment # prod, staging
Cluster = aws_eks_cluster.main.name
ManagedBy = "terraform"
CostCenter = var.cost_center # eng, data, platform
}
}
Node-level tags get you the cost of compute per cluster. Getting below that — to the workload level — requires something smarter.
Layer 2: OpenCost for Kubernetes Cost Allocation
OpenCost is the CNCF project that does exactly what its name says. It runs as a pod in your cluster, queries the cloud provider’s pricing API (or uses a custom price list for on-prem), and allocates node costs down to the pod level based on actual resource requests and usage.
The allocation model is straightforward: if a pod requests 25% of a node’s CPU and 10% of its memory, it gets attributed 25% of the CPU cost and 10% of the memory cost of that node. Idle capacity stays as a cluster-level cost that you can spread across teams or leave as overhead.
Deploying OpenCost
# opencost-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: opencost
labels:
app.kubernetes.io/part-of: opencost
# Install via Helm — the quick path
helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm repo update
helm install opencost opencost/opencost \
--namespace opencost \
--set opencost.exporter.cloudProviderApiKey="your-aws-key" \
--set opencost.prometheus.internal.enabled=true
For production, you almost certainly want to point OpenCost at your existing Prometheus rather than spinning up a new one:
# values-prod.yaml
opencost:
prometheus:
internal:
enabled: false
external:
enabled: true
url: "http://prometheus-operated.monitoring.svc.cluster.local:9090"
exporter:
aws:
region: us-east-1
# Use spot instance pricing — critical if you run spot nodes
spotLabel: "node.kubernetes.io/capacity-type"
spotLabelValue: "SPOT"
Once it’s running, OpenCost exposes a REST API and a UI on port 9090 (not to be confused with Prometheus). The real value is the /allocation API:
# Cost breakdown by namespace for the last 7 days
curl "http://opencost.opencost.svc:9090/allocation/compute" \
-G \
--data-urlencode "window=7d" \
--data-urlencode "aggregate=namespace" \
--data-urlencode "accumulate=true" | jq '.data[0] | to_entries[] | {namespace: .key, totalCost: .value.totalCost}'
Scraping OpenCost Metrics into Prometheus
OpenCost exposes Prometheus metrics on port 9003. Add a scrape job:
# prometheus-scrape-config.yaml — add to your existing Prometheus config
scrapeConfigs:
- job_name: opencost
scrape_interval: 1m
static_configs:
- targets: ["opencost.opencost.svc.cluster.local:9003"]
Key metrics you’ll use:
opencost_pod_total_cost— total cost per podopencost_namespace_cost_total— rolled up to namespacenode_total_hourly_cost— useful for capacity planning
Layer 3: Per-Request Cost Attribution with OpenTelemetry
This is where most guides stop, and where the interesting work actually begins.
The idea: every HTTP request, every queue message processed, every background job — it has a cost. If you know that your inference endpoint costs $0.008 per call and your team is making 50k calls a day to it from service B, you can surface that number to the service B team instead of hiding it in a shared infrastructure bill.
The mechanism is OpenTelemetry. You add cost as a span attribute on every request, then aggregate it in your metrics backend.
Estimating Per-Request Compute Cost
The math: if you know the cost per CPU-core-hour and the CPU time a request consumed, you can estimate its cost.
# cost_middleware.py
import time
import resource
from opentelemetry import trace, metrics
from opentelemetry.sdk.metrics import MeterProvider
meter = metrics.get_meter("cost.attribution")
# Cost histogram — records cost in USD per request
request_cost_histogram = meter.create_histogram(
name="request.cost.usd",
description="Estimated compute cost per request in USD",
unit="USD",
)
# Pull from your cloud provider's pricing or OpenCost API
# For AWS us-east-1, c6i.2xlarge on-demand: ~$0.34/hr = $0.000094/core-second
CPU_COST_PER_CORE_SECOND = float(os.getenv("CPU_COST_PER_CORE_SECOND", "0.000094"))
MEMORY_COST_PER_GB_SECOND = float(os.getenv("MEMORY_COST_PER_GB_SECOND", "0.000012"))
def cost_middleware(app):
def middleware(environ, start_response):
tracer = trace.get_tracer(__name__)
# Capture CPU usage before
ru_before = resource.getrusage(resource.RUSAGE_THREAD)
wall_start = time.monotonic()
span = tracer.start_span("http.request")
with trace.use_span(span, end_on_exit=True):
response = app(environ, start_response)
# Capture CPU usage after
ru_after = resource.getrusage(resource.RUSAGE_THREAD)
wall_elapsed = time.monotonic() - wall_start
cpu_user = ru_after.ru_utime - ru_before.ru_utime
cpu_sys = ru_after.ru_stime - ru_before.ru_stime
cpu_total = cpu_user + cpu_sys
# Estimate cost
cpu_cost = cpu_total * CPU_COST_PER_CORE_SECOND
# Memory cost: this is an approximation using peak RSS delta
# In practice you'd want cgroup data for accuracy
estimated_cost = cpu_cost # extend with memory if needed
# Attributes for cost breakdown
attrs = {
"service.name": environ.get("SERVICE_NAME", "unknown"),
"http.route": environ.get("PATH_INFO", "/"),
"team": environ.get("TEAM_LABEL", "unknown"),
"environment": environ.get("ENVIRONMENT", "prod"),
}
span.set_attribute("cost.estimated_usd", estimated_cost)
span.set_attribute("cost.cpu_seconds", cpu_total)
request_cost_histogram.record(estimated_cost, attrs)
return response
return middleware
Gotcha: resource.getrusage(RUSAGE_THREAD) gives you CPU time for the current thread only. In async frameworks (asyncio, Tornado), a single thread handles many requests concurrently, making per-request CPU attribution meaningless without instrumentation at the coroutine level. For async apps, consider approximating from wall time × CPU utilization of the pod, or use eBPF-based profilers.
For Go Services
// cost/middleware.go
package cost
import (
"context"
"net/http"
"runtime"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
)
var (
cpuCostPerCoreSecond = mustGetEnvFloat("CPU_COST_PER_CORE_SECOND", 0.000094)
requestCost metric.Float64Histogram
)
func init() {
meter := otel.GetMeterProvider().Meter("cost.attribution")
var err error
requestCost, err = meter.Float64Histogram(
"request.cost.usd",
metric.WithDescription("Estimated compute cost per request"),
metric.WithUnit("USD"),
)
if err != nil {
panic(err)
}
}
func Middleware(team, service string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var memBefore, memAfter runtime.MemStats
runtime.ReadMemStats(&memBefore)
start := time.Now()
next.ServeHTTP(w, r)
elapsed := time.Since(start).Seconds()
runtime.ReadMemStats(&memAfter)
// CPU cost: rough approximation via wall time
// For accurate CPU: use /proc/self/stat or cgroups
cpuCost := elapsed * cpuCostPerCoreSecond
attrs := []attribute.KeyValue{
attribute.String("team", team),
attribute.String("service", service),
attribute.String("http.route", r.URL.Path),
attribute.String("environment", getEnv("ENVIRONMENT", "prod")),
}
requestCost.Record(r.Context(), cpuCost,
metric.WithAttributes(attrs...),
)
})
}
}
Wiring It Together: Grafana Dashboards
With OpenCost feeding namespace-level costs to Prometheus, and your application emitting request.cost.usd metrics via OTLP, you can build a unified cost dashboard.
# Example Grafana dashboard panel — Cost per team, 24h
# PromQL
sum by (team) (
increase(request_cost_usd_sum[24h])
)
# Cost per endpoint — sorted by most expensive
topk(10,
sum by (service, http_route) (
rate(request_cost_usd_sum[1h]) * 3600
)
)
# Namespace cost from OpenCost
sum by (namespace) (
opencost_namespace_cost_total
)
A dashboard worth building has three rows:
- Team row: total daily spend by team, week-over-week delta
- Service row: top 10 most expensive services, cost per request for each
- Endpoint row: cost heatmap — which routes are expensive and how traffic patterns affect cost
Alerting on Cost Anomalies
Cost alerts are different from latency alerts. A 3× cost spike at 3am might be a data pipeline that legitimately ran, or it might be a runaway retry loop. You need context.
# prometheus-cost-alerts.yaml
groups:
- name: cost.anomalies
interval: 5m
rules:
# Alert if hourly cost for a team spikes 3x over the 7-day average
- alert: TeamCostSpike
expr: |
(
sum by (team) (rate(request_cost_usd_sum[1h]) * 3600)
/
sum by (team) (avg_over_time(rate(request_cost_usd_sum[1h])[7d:1h]) * 3600)
) > 3
for: 15m
labels:
severity: warning
annotations:
summary: "Team {{ $labels.team }} cost is {{ $value | humanize }}x above 7-day average"
runbook_url: "https://wiki.internal/runbooks/cost-spike"
# Alert on absolute threshold — adjust to your baseline
- alert: ServiceCostBudgetExceeded
expr: |
sum by (service) (
increase(request_cost_usd_sum[1h])
) > 50
for: 10m
labels:
severity: warning
annotations:
summary: "Service {{ $labels.service }} spent ${{ $value | humanize }} in the last hour"
Gotcha: Don’t fire on every spike. Cloud costs have natural weekly patterns (lower on weekends, higher on batch processing days). Use avg_over_time with a lookback window that matches your traffic pattern, not a flat threshold.
Gotchas and Pitfalls
Spot instance pricing ruins your averages. If you run spot nodes at 70% discount, mixing their cost signals with on-demand nodes will make your per-request cost metrics misleading. Separate the node pools and price them independently in OpenCost using the spotLabel configuration.
Shared infrastructure is the hard problem. Your RDS instance, your Redis cluster, your message queue — these serve multiple teams. OpenCost won’t automatically break those down. You have two options: charge them as platform overhead (add to each team’s bill proportionally based on their compute spend), or instrument the clients to emit their own usage metrics and do chargeback math in Grafana.
CPU time ≠ wall time in containers. If your container is CPU-throttled (you’ve set CPU limits below request), the pod uses more wall time but the same CPU time. Your cost estimates will diverge from reality. Consider using container_cpu_usage_seconds_total from cAdvisor in Prometheus instead of application-level instrumentation — it’s more accurate and you don’t need to touch app code.
Don’t attribute costs in hot paths without sampling. The middleware approach above adds measurement overhead. For high-QPS services (>10k req/s), use OpenTelemetry sampling — record cost on 10% of requests and multiply by 10 in your aggregations. The histogram will still give you accurate distributions.
Cost attribution lag. Cloud billing data is typically 24-48 hours delayed. Your OpenCost node-level costs update every few minutes, but if you’re reconciling against actual cloud bills, build in that lag. Don’t build alerting on raw billing data — use the real-time instrumentation for that.
Production-Ready Cost Attribution Pipeline
Here’s the full architecture in a single Docker Compose file for local development and testing:
# docker-compose.yml — local Cost SRE stack
version: "3.9"
services:
prometheus:
image: prom/prometheus:v2.52.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=30d"
ports:
- "9090:9090"
otel-collector:
image: otel/opentelemetry-collector-contrib:0.101.0
volumes:
- ./otel-collector.yaml:/etc/otel-collector.yaml:ro
command: ["--config=/etc/otel-collector.yaml"]
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "8888:8888" # Collector metrics
grafana:
image: grafana/grafana:11.0.0
environment:
GF_SECURITY_ADMIN_PASSWORD: admin
GF_FEATURE_TOGGLES_ENABLE: publicDashboards
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
- ./grafana/datasources:/etc/grafana/provisioning/datasources:ro
ports:
- "3000:3000"
depends_on:
- prometheus
volumes:
prometheus_data:
grafana_data:
# otel-collector.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
timeout: 10s
# Add team label if missing (fallback for services that don't set it)
attributes:
actions:
- key: team
value: "unknown"
action: insert
exporters:
prometheusremotewrite:
endpoint: "http://prometheus:9090/api/v1/write"
namespace: "otel"
# Keep traces for cost debugging
otlp/jaeger:
endpoint: "jaeger:4317"
tls:
insecure: true
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch, attributes]
exporters: [prometheusremotewrite]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/jaeger]
Where to Go From Here
Once you have this pipeline in place, the interesting questions become possible to answer:
- "The data team’s new ML pipeline — what does it cost per inference call, and how does that compare to using a managed API?"
- "Our checkout service costs 3× more per request than our catalog service. Is that justified by its complexity?"
- "If we optimize this endpoint’s CPU usage by 20%, what’s the monthly savings?"
These are engineering questions, not finance questions. That’s the shift Cost SRE is trying to make — from monthly bill reviews to real-time cost signals that engineers can act on the same way they act on latency and error rates.
The toolchain — OpenCost for infrastructure, OpenTelemetry for application, Prometheus and Grafana for visibility — is all open source and runs fine on a single medium-sized node for small clusters. There’s no excuse for flying blind on costs when the same observability investment that gives you dashboards for latency can give you dashboards for spend.