Cloud transcription APIs are a trap. You start with a prototype, Deepgram or AssemblyAI feels magical, and then your invoice arrives. At scale — podcast platforms, call center analytics, meeting recorders — you’re paying per minute of audio forever. Meanwhile, a single RTX 3090 sitting in your rack can process audio faster than real-time and handle dozens of concurrent streams.
Whisper.cpp is the key. It’s Georgi Gerganov’s C++ reimplementation of OpenAI’s Whisper model using ggml tensors. It runs on CPU, Metal (Apple Silicon), CUDA, OpenCL, and Vulkan. The large-v3 model runs at 8-10x real-time on an RTX 3080. The tiny model hits 100x real-time easily. For a microservice, that throughput translates directly to cost savings and SLA headroom.
This article covers the full stack: building whisper.cpp with CUDA support, designing a Python FastAPI wrapper that handles batching and streaming, Docker Compose wiring, and the production gotchas that will bite you if you skip them.
Why whisper.cpp Instead of the Python Whisper Wheel
The official openai-whisper Python package runs on PyTorch. PyTorch is great for research; it’s overkill for serving a fixed model at inference time. You get hundreds of megabytes of CUDA libraries, a Python GIL that complicates threading, and cold-start times measured in seconds.
Whisper.cpp compiles to a single binary. The CUDA build links against cuBLAS directly. No PyTorch, no CUDA toolkit installed in the container — you only need the runtime libraries. Memory usage is roughly half that of the PyTorch equivalent at the same model size. And critically, the C server example in the repo (examples/server) gives you an HTTP endpoint out of the box, though we’ll build something more production-oriented on top of it.
Architecture Overview
The design splits responsibility cleanly:
[Client] --> [FastAPI Gateway] --> [Job Queue (asyncio)] --> [whisper.cpp process pool]
|
└--> [WebSocket handler] --> [streaming chunks to client]
The gateway owns the API surface: REST for batch jobs, WebSocket for real-time streaming. It manages a bounded queue of transcription jobs and dispatches them to a pool of whisper.cpp worker processes. Workers are plain subprocesses — we’re not embedding the C library; we’re running it. This keeps the Python code simple and makes it trivial to run multiple model sizes concurrently.
Building whisper.cpp with CUDA
First, the build. We’ll do this inside a Docker image so the host doesn’t need a full CUDA toolkit — only the driver.
# builder.Dockerfile
FROM nvidia/cuda:12.3.2-devel-ubuntu22.04 AS builder
RUN apt-get update && apt-get install -y \
git cmake build-essential libcurl4-openssl-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
RUN git clone --depth 1 --branch v1.7.3 https://github.com/ggerganov/whisper.cpp.git
WORKDIR /build/whisper.cpp
RUN cmake -B build \
-DWHISPER_CUDA=ON \
-DWHISPER_CURL=ON \
-DCMAKE_BUILD_TYPE=Release \
&& cmake --build build --config Release -j$(nproc)
# ---- runtime image ----
FROM nvidia/cuda:12.3.2-runtime-ubuntu22.04
# Only the shared libraries we actually need
RUN apt-get update && apt-get install -y \
libcublas-12-3 libcurand-12-3 ffmpeg curl \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /build/whisper.cpp/build/bin/whisper-cli /usr/local/bin/whisper-cli
COPY --from=builder /build/whisper.cpp/build/bin/whisper-server /usr/local/bin/whisper-server
RUN mkdir -p /models
Gotcha: The -DWHISPER_CUDA=ON flag requires the devel image at build time (for nvcc and CUDA headers), but the runtime image only needs the shared libraries. Swapping to runtime in the final stage shaves ~3 GB off the image. Pin the CUDA version explicitly — 12.3 and 12.4 have subtle cuBLAS behavior differences that affect first-token latency.
Downloading and Quantizing Models
Whisper.cpp uses GGUF-style quantized models. The project ships a download script:
# Download large-v3 in Q5_0 quantization (~1.1 GB vs 3 GB fp16)
bash models/download-ggml-model.sh large-v3-q5_0
# Or grab specific sizes for different latency tiers
bash models/download-ggml-model.sh base.en # ~75 MB, real-time on CPU
bash models/download-ggml-model.sh medium.en # ~500 MB, best accuracy/speed tradeoff
For a multi-tier service (route short clips to base, long audio to large), you mount a single /models volume and the workers pick the right binary at startup based on the job’s requested quality tier.
The FastAPI Gateway
Here’s the core service. Install deps first:
pip install fastapi uvicorn[standard] python-multipart aiofiles aiohttp
# app/main.py
import asyncio
import os
import json
import tempfile
import subprocess
from pathlib import Path
from typing import AsyncGenerator
from fastapi import FastAPI, UploadFile, File, WebSocket, WebSocketDisconnect, HTTPException
from fastapi.responses import JSONResponse
app = FastAPI(title="Whisper Transcription Service")
MODEL_DIR = Path(os.getenv("MODEL_DIR", "/models"))
DEFAULT_MODEL = os.getenv("DEFAULT_MODEL", "ggml-large-v3-q5_0.bin")
MAX_QUEUE_SIZE = int(os.getenv("MAX_QUEUE_SIZE", "64"))
WORKER_COUNT = int(os.getenv("WORKER_COUNT", "2"))
# Semaphore limits concurrent whisper processes — more than GPU VRAM allows
# will cause OOM errors, not graceful queuing
_sem = asyncio.Semaphore(WORKER_COUNT)
_queue: asyncio.Queue = asyncio.Queue(maxsize=MAX_QUEUE_SIZE)
async def run_whisper(
audio_path: str,
model: str = DEFAULT_MODEL,
language: str = "auto",
output_format: str = "json",
) -> dict:
model_path = MODEL_DIR / model
if not model_path.exists():
raise FileNotFoundError(f"Model not found: {model_path}")
cmd = [
"whisper-cli",
"-m", str(model_path),
"-f", audio_path,
"-l", language,
"-oj", # output JSON
"--no-prints", # suppress progress to stderr
"-t", "4", # CPU threads (used even in GPU mode for preprocessing)
]
async with _sem:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate()
if proc.returncode != 0:
raise RuntimeError(f"whisper-cli failed: {stderr.decode()}")
return json.loads(stdout.decode())
@app.post("/transcribe")
async def transcribe(
file: UploadFile = File(...),
model: str = DEFAULT_MODEL,
language: str = "auto",
):
"""Batch transcription — returns when complete."""
if _queue.full():
raise HTTPException(status_code=503, detail="Queue full, try again later")
suffix = Path(file.filename).suffix or ".audio"
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
tmp.write(await file.read())
tmp_path = tmp.name
try:
result = await run_whisper(tmp_path, model=model, language=language)
return JSONResponse(content=result)
finally:
os.unlink(tmp_path)
Gotcha: asyncio.create_subprocess_exec is non-blocking at the Python level, but the whisper process itself is blocking CPU/GPU work. The semaphore is critical — without it, 20 concurrent requests will each spawn a whisper process, each trying to allocate the full model in VRAM, and your GPU will OOM-kill them all.
Real-Time Streaming via WebSocket
The built-in whisper.cpp server supports streaming via SSE, but for a proper microservice you want WebSocket — clients can push audio chunks and receive partial transcripts as they arrive. The trick is using whisper’s VAD (voice activity detection) mode with short segments.
# app/streaming.py
import asyncio
import os
import json
import tempfile
from pathlib import Path
from fastapi import WebSocket, WebSocketDisconnect
MODEL_DIR = Path(os.getenv("MODEL_DIR", "/models"))
DEFAULT_MODEL = os.getenv("DEFAULT_MODEL", "ggml-large-v3-q5_0.bin")
# Segment length in seconds — shorter = lower latency, more overhead
SEGMENT_DURATION = float(os.getenv("SEGMENT_DURATION", "5.0"))
async def stream_audio_chunks(websocket: WebSocket):
"""
Protocol:
Client sends: binary audio frames (raw PCM s16le, 16kHz mono)
Server sends: JSON {"text": "...", "is_final": bool, "segment_id": int}
Client sends: text "EOS" to signal end of stream
"""
await websocket.accept()
buffer = bytearray()
segment_id = 0
# 16kHz mono s16le = 32000 bytes/sec
bytes_per_segment = int(SEGMENT_DURATION * 16000 * 2)
try:
while True:
data = await websocket.receive()
if data.get("type") == "websocket.disconnect":
break
if "bytes" in data:
buffer.extend(data["bytes"])
# Process when we have a full segment
while len(buffer) >= bytes_per_segment:
chunk = bytes(buffer[:bytes_per_segment])
buffer = buffer[bytes_per_segment:]
transcript = await transcribe_pcm_chunk(chunk, segment_id)
await websocket.send_json({
"text": transcript,
"is_final": False,
"segment_id": segment_id,
})
segment_id += 1
elif "text" in data and data["text"] == "EOS":
# Flush remaining buffer
if buffer:
transcript = await transcribe_pcm_chunk(bytes(buffer), segment_id)
await websocket.send_json({
"text": transcript,
"is_final": True,
"segment_id": segment_id,
})
await websocket.close()
break
except WebSocketDisconnect:
pass
async def transcribe_pcm_chunk(pcm_data: bytes, segment_id: int) -> str:
"""Write PCM to temp WAV, run whisper, return text."""
import wave
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
tmp_path = f.name
# Write minimal WAV header
with wave.open(tmp_path, "wb") as wav:
wav.setnchannels(1)
wav.setsampwidth(2) # 16-bit
wav.setframerate(16000)
wav.writeframes(pcm_data)
model_path = MODEL_DIR / DEFAULT_MODEL
cmd = [
"whisper-cli",
"-m", str(model_path),
"-f", tmp_path,
"-oj",
"--no-prints",
"-t", "4",
"--vad", # voice activity detection — skip silent segments
]
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await proc.communicate()
result = json.loads(stdout.decode())
# Extract plain text from JSON segments
return " ".join(s["text"].strip() for s in result.get("transcription", []))
finally:
os.unlink(tmp_path)
Register the WebSocket route in main.py:
# In app/main.py
from app.streaming import stream_audio_chunks
@app.websocket("/ws/transcribe")
async def websocket_transcribe(websocket: WebSocket):
await stream_audio_chunks(websocket)
Request Batching for Throughput
The naive approach — one process per request — leaves throughput on the table. Whisper.cpp supports batching via --batch-size (processes multiple audio files in parallel on GPU). For high-throughput scenarios, collect N requests and submit them together:
# app/batcher.py
import asyncio
from dataclasses import dataclass, field
from typing import List
BATCH_SIZE = int(os.getenv("BATCH_SIZE", "4"))
BATCH_TIMEOUT = float(os.getenv("BATCH_TIMEOUT", "0.5")) # seconds
@dataclass
class TranscriptionJob:
audio_path: str
model: str
language: str
future: asyncio.Future = field(default_factory=asyncio.get_event_loop().create_future)
class BatchProcessor:
def __init__(self):
self._pending: List[TranscriptionJob] = []
self._lock = asyncio.Lock()
self._flush_task: asyncio.Task | None = None
async def submit(self, job: TranscriptionJob) -> dict:
async with self._lock:
self._pending.append(job)
if len(self._pending) >= BATCH_SIZE:
# Batch is full — flush immediately
await self._flush()
elif self._flush_task is None or self._flush_task.done():
# Schedule a timeout flush so small batches don't wait forever
self._flush_task = asyncio.create_task(self._timeout_flush())
return await job.future
async def _timeout_flush(self):
await asyncio.sleep(BATCH_TIMEOUT)
async with self._lock:
if self._pending:
await self._flush()
async def _flush(self):
jobs, self._pending = self._pending, []
if not jobs:
return
asyncio.create_task(self._process_batch(jobs))
async def _process_batch(self, jobs: List[TranscriptionJob]):
# whisper-cli doesn't support true multi-file batching in the CLI,
# so we run them concurrently under the semaphore
results = await asyncio.gather(
*[run_whisper(j.audio_path, j.model, j.language) for j in jobs],
return_exceptions=True,
)
for job, result in zip(jobs, results):
if isinstance(result, Exception):
job.future.set_exception(result)
else:
job.future.set_result(result)
batch_processor = BatchProcessor()
Gotcha: The BATCH_TIMEOUT is the latency floor for small jobs. If your p99 latency matters, keep it at 200-500ms. For throughput-heavy offline workloads, push it to 2-5 seconds. Don’t set it to zero — you’ll defeat the purpose and hammer the GPU with single-item "batches."
Docker Compose Wiring
# docker-compose.yml
version: "3.9"
services:
whisper-api:
build:
context: .
dockerfile: Dockerfile
image: whisper-service:latest
restart: unless-stopped
ports:
- "8000:8000"
volumes:
- ./models:/models:ro
- /tmp/whisper-audio:/tmp/whisper-audio # shared temp space
environment:
MODEL_DIR: /models
DEFAULT_MODEL: ggml-large-v3-q5_0.bin
WORKER_COUNT: "2" # match to GPU VRAM / model size
BATCH_SIZE: "4"
BATCH_TIMEOUT: "0.3"
MAX_QUEUE_SIZE: "128"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-sf", "https://cd-linux.club:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s # model load takes a moment
# Optional: Prometheus metrics scraping
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
ports:
- "9090:9090"
The Dockerfile for the service itself:
# Dockerfile
FROM nvidia/cuda:12.3.2-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y \
libcublas-12-3 ffmpeg curl python3 python3-pip \
&& rm -rf /var/lib/apt/lists/*
COPY --from=ghcr.io/yourorg/whisper-cpp-builder:latest \
/usr/local/bin/whisper-cli /usr/local/bin/whisper-cli
WORKDIR /app
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt
COPY app/ ./app/
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]
Gotcha: Use --workers 1 with uvicorn when you’re managing GPU resources via asyncio primitives. Multiple uvicorn workers = multiple Python processes = each one has its own semaphore = no global concurrency control. For true multi-worker setups, move the semaphore to Redis (aioredis) or use a proper task queue like Celery with GPU-aware routing.
Health Check and Observability
# In app/main.py
import time
from collections import deque
_request_times: deque = deque(maxlen=1000)
_service_start = time.time()
@app.get("/health")
async def health():
return {
"status": "ok",
"queue_size": _queue.qsize(),
"uptime_seconds": int(time.time() - _service_start),
}
@app.get("/metrics")
async def metrics():
"""Bare-bones metrics — wire Prometheus exporter for production."""
if not _request_times:
return {"p50_ms": 0, "p95_ms": 0, "p99_ms": 0, "total_requests": 0}
sorted_times = sorted(_request_times)
n = len(sorted_times)
return {
"p50_ms": sorted_times[int(n * 0.5)],
"p95_ms": sorted_times[int(n * 0.95)],
"p99_ms": sorted_times[int(n * 0.99)],
"total_requests": n,
}
Production Gotchas
Model loading time. The large-v3 model takes 3-8 seconds to load into VRAM. Don’t load it per-request. The approach here — running whisper-cli as a subprocess — reloads the model every call. For real production workloads, either use the built-in whisper-server binary (keeps the model resident) or use the C API via ctypes/cffi bindings to keep it in memory. The subprocess approach is fine for low-traffic internal APIs; switch for anything above ~50 req/min.
Audio format chaos. Whisper.cpp expects 16kHz mono WAV internally. It uses FFmpeg to handle other formats, but FFmpeg spawns another subprocess. For streaming, transcode on the client side and send raw PCM — saves ~50ms per chunk. For batch uploads, accept whatever format and let FFmpeg handle it server-side, but log a warning when it triggers so you can profile the overhead.
GPU memory fragmentation. Long-running services accumulate CUDA memory fragmentation. A 16 GB GPU running large-v3 (~6 GB model) seems to have headroom, but after hours of concurrent requests you’ll see mysterious OOMs. Set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512 if you’re using PyTorch anywhere in the same process. With pure whisper-cli subprocesses, memory is reclaimed between calls — a feature, not a bug.
Concurrent streams and thermal throttling. Running 4 concurrent large-v3 inferences on a 3090 will push it past 85°C and trigger thermal throttling. Monitor GPU temp with nvidia-smi dmon -s u and tune WORKER_COUNT accordingly. For sustained throughput, set a power limit: sudo nvidia-smi -pl 250 caps a 350W card at 250W with maybe 10% throughput loss and a 15°C temperature drop.
The -t flag trap. The --threads / -t flag controls CPU threads for preprocessing (FFT, mel spectrogram). Setting it too high on a shared server causes CPU contention that paradoxically slows GPU inference because data isn’t ready in time. Start at 4, benchmark, go up cautiously.
Client Example
# client_example.py — sync REST client
import requests
with open("interview.mp3", "rb") as f:
response = requests.post(
"https://cd-linux.club:8000/transcribe",
files={"file": ("interview.mp3", f, "audio/mpeg")},
data={"model": "ggml-large-v3-q5_0.bin", "language": "en"},
)
data = response.json()
for segment in data.get("transcription", []):
print(f"[{segment['timestamps']['from']} --> {segment['timestamps']['to']}]")
print(segment["text"])
# client_ws.py — WebSocket streaming client
import asyncio
import wave
import websockets
import json
async def stream_file(path: str):
uri = "ws://localhost:8000/ws/transcribe"
async with websockets.connect(uri) as ws:
with wave.open(path, "rb") as wav:
while True:
frames = wav.readframes(16000 * 5) # 5-second chunks
if not frames:
break
await ws.send(frames)
msg = await asyncio.wait_for(ws.recv(), timeout=15)
print(json.loads(msg))
await ws.send("EOS")
asyncio.run(stream_file("recording.wav"))
Scaling Beyond One GPU
The single-node setup handles most internal workloads. When you need more, the architecture scales naturally:
Put Nginx or Caddy in front of multiple whisper-service instances, each pinned to a different GPU via CUDA_VISIBLE_DEVICES. Nginx upstream with least_conn balancing distributes audio intelligently. For Kubernetes, add a GPU resource limit (nvidia.com/gpu: 1) and let the scheduler handle placement.
For multi-model routing — send short clips to base.en and long interviews to large-v3 — add a routing layer that checks Content-Length or a quality query param and forwards accordingly. Cheap routing logic, huge efficiency gain.
The streaming WebSocket path doesn’t fan out as easily since a single connection should stay pinned to one worker to avoid segment reordering. Use sticky sessions at the load balancer level for WebSocket connections.
You now have a self-hosted transcription service that handles real production traffic: GPU acceleration with proper concurrency control, a batch collector that keeps the GPU busy, and a streaming API that returns transcripts as the speaker talks. The cloud vendors charge per minute for what your own hardware can do in seconds.