FFmpeg ate my server: lessons from a browser video pipeline

ScreenAI, the hiring platform I built, records one-way video interviews in the browser. The candidate answers questions on camera; the recording is uploaded, the audio is transcribed, and the transcript is scored. Somewhere in the middle sits FFmpeg — the Swiss Army knife of video — and for a few months it was the source of almost every production incident I had.

None of those incidents were FFmpeg's fault, exactly. They came from not understanding what I was asking it to do. So this post starts with the five minutes of video theory I wish I'd had, then walks through what went wrong, in order.

A five-minute primer

Containers and codecs

A video file is two things nested inside each other.

The codec is how the pictures and sound are compressed. Browsers commonly record VP9 video with Opus audio; the "plays everywhere" combination is H.264 video with AAC audio.
The container is the file format that holds those compressed streams, plus an index: timestamps, total duration, and where each keyframe is. WebM and MP4 are containers.

Think of the container as a box and the codec as the language the letters inside are written in. You can move letters to a new box without reading them — or you can translate every letter into a new language, which takes far longer.

Remuxing vs transcoding

That's the difference between FFmpeg's two most common jobs:

Remuxing (-c copy) moves the same compressed streams into a fresh, properly built container. Nothing is decoded, so it takes seconds — it's limited by disk and network, not CPU.
Transcoding decodes every frame and compresses it again, usually in a different codec. It's slow and entirely CPU-bound.

Why encoding H.264 is so expensive

For every single frame, an H.264 encoder searches nearby frames for motion, transforms and quantises blocks of pixels, and entropy-codes the result — while looking ahead to decide where keyframes go. A 19-minute interview at 30 frames per second is about 34,000 frames. On a small server, one encode can own the CPU for twenty minutes.

Why a browser recording is "streamable but not seekable"

The browser's MediaRecorder writes the file while recording. It can't know the total duration, or build the table of keyframe positions (WebM calls them Cues), until the recording ends — and it never goes back to fill them in. The file plays fine from the start, but jumping to minute twelve means guessing.

And the very first chunk the recorder produces holds the only description of what's inside the file — the init segment. It's never repeated. Remember that one; it comes back.

What FFmpeg does in this pipeline

FFmpeg runs as a child process of the Node API, on the same server — first 2 vCPUs, later 4. It has three jobs:

# 1. Audio for transcription — runs on every interview
ffmpeg -i video.webm -vn -acodec libmp3lame -ab 32k -ar 16000 audio.mp3
 
# 2. Remux — fix the container so the video is seekable
ffmpeg -i in.webm -c copy -fflags +genpts -map 0 out.webm
 
# 3. Transcode to MP4 — a playback fallback, now only on request
ffmpeg -i in.webm -threads 2 -vf fps=30 -af aresample=async=1:first_pts=0 \
       -c:v libx264 -preset veryfast -crf 23 -c:a aac -movflags +faststart out.mp4

Job 1 is light: it throws the video away and writes low-bitrate speech audio. Job 2 is light: no decoding. Job 3 is the heavy one. Most of this story is about job 3 being somewhere it shouldn't have been.

Problem 1: the server froze while FFmpeg ran

The very first version ran FFmpeg with Node's spawnSync. That blocks Node's event loop for the entire run — and since one Node process serves every request, the whole API went unresponsive for ten to thirty seconds whenever audio was extracted.

Fix: use the asynchronous spawn and wait on its exit event. This happened before the first commit, but it set the theme: heavy work must never share a lane with everything else.

Problem 2: interviews stuck for days

To make recordings play in Safari, I started generating an MP4 for every interview — in parallel with audio extraction, before transcription could begin. That put the most expensive job in the pipeline directly on the critical path.

The first symptom was CPU saturation, so I limited FFmpeg to two threads, switched to a faster preset, and allowed only one encode at a time. That kept the box alive — and created a queue. Candidates sat in "transcribing" for hours, sometimes days. One 77 MB recording burned over 1 hour 54 minutes of CPU and produced a 274 MB MP4, while three other candidates waited behind it. There were no timeouts and no logging, so the only way to see what was happening was ps -ef | grep ffmpeg.

When I measured, the MP4 encode was about 98% of all the time spent in the encoding slot, with a median of about 24 minutes — for a file that transcription doesn't even need.

Fix: stop making MP4s by default. Every modern browser except Safari plays WebM natively, so recruiters get WebM, and transcription went from around 25 minutes to under one. MP4 became an on-demand admin action.

Problem 3: the phantom 1,000 fps

Even on demand, the MP4 encode regularly ran for its full 20-minute time limit and got killed. The reason was hiding in the timestamps.

MediaRecorder stamps frames on a one-millisecond timebase. FFmpeg read that as a stream running at 1,000 frames per second, and dutifully padded the output to about 1.14 million frames for a 19-minute clip — instead of roughly 34,000. On top of that, the audio timestamps weren't always increasing, so FFmpeg spent effort complaining about "queue backward in time" — tens of thousands of log lines that flooded the system journal and slowed the encode down through pipe I/O.

Fix: tell FFmpeg the real frame rate (-vf fps=30), resample audio to repair its timestamps (-af aresample=async=1:first_pts=0), and only log errors (-loglevel error). Encodes dropped to single-digit minutes.

Problem 4: one candidate, five scores

This was the worst one, because slowness turned into wrong results.

Jobs are claimed with a five-minute lease in the database: if a worker crashes, the lease expires and another tick picks the job up. Meanwhile, audio extraction and MP4 encoding shared a single FFmpeg slot. So:

An admin's 20-minute MP4 encode took the slot.
A new interview's audio extraction queued behind it.
Its transcription stage ran past the five-minute lease.
The worker saw an "expired" job and started it again. And again.

Overlapping runs each transcribed, scored and announced the same candidate — five notifications for one person.

Fix, in layers:

Separate light and heavy work. Audio extraction no longer waits for the encoding slot; it's limited by its own concurrency instead.
Heartbeat the lease. A running job renews its lease every third of its lifetime, so "slow" no longer looks like "dead".
Make completion idempotent. The final state change is conditional — only if the session isn't already complete — so a duplicate run finishes as a no-op and sends nothing.
Time-box the encode. MP4 conversion gets a timeout that scales with file size.

// 5 minutes, plus 2 s for every MB over 200 MB, capped at 20 minutes
export function mp4TimeoutForSize(sizeBytes: number): number {
  const overMb = Math.max(0, sizeBytes / (1024 * 1024) - 200);
  return Math.min(5 * 60_000 + Math.round(overMb * 2_000), 20 * 60_000);
}

Problem 5: zombies that survived restarts

A hung FFmpeg process never exited, so it never released the encoding slot — and when Node restarted, its FFmpeg children didn't die with it. They were re-parented to the operating system and kept pinning the CPU. One incident needed a full redeploy to clear.

Fix: every FFmpeg process gets a hard SIGKILL timer and is tracked in a set; on exit, SIGTERM or SIGINT, Node kills every child it started. Everything also runs at the lowest scheduling priority:

const proc = spawn("nice", ["-n", "19", "ffmpeg", "-hide_banner", "-loglevel", "error", ...args]);
liveFfmpeg.add(proc);
const timer = setTimeout(() => proc.kill("SIGKILL"), timeoutMs);

One subtlety: nice doesn't cap CPU. It means that the moment the API needs the CPU, the kernel gives it back. That's exactly what you want for background work on a shared box.

Problem 6: videos that stalled halfway

Remember "streamable but not seekable"? Some recordings arrived without duration, seek head or Cues, and Chrome would stall on playback once the file passed about 50 MB.

Fix: a copy-only remux after every interview. It rebuilds the container and its index without touching a single frame, and takes seconds even for a 200 MB file. It writes a new file next to the original, never over it. If a recording is so broken that it has almost no keyframes, remuxing can't invent them — that's when the on-demand MP4 transcode earns its keep.

Problem 7: files FFmpeg refused at byte zero

Then the scariest symptom: FFmpeg rejecting a recording outright with EBML header parsing failed. Twenty-five interviews were lost that way over four months — about 1 in 50 recordings. Every upload had reported success; the first sign of trouble was a failed job twenty minutes later.

This is the init segment from the primer. Lose that first chunk and a 280 MB upload isn't a damaged video — it isn't a video at all. Double-clicking Start created two recorders writing into one buffer, and chunks were sometimes flushed before the upload had started.

When I looked at the actual bytes, it was more interesting: 20 of the 25 still had a perfectly valid header, displaced by one to three stray bytes left behind by an abandoned recorder. Only five were truly headerless.

Fix: make Start idempotent, keep the first chunk for the whole recording, store it separately as a small backup object, and when reading a file, trim stray leading bytes or graft the backup header back on. The repair is designed to fail open: if it can't help, FFmpeg gets the original bytes exactly as before.

Problem 8: interviews too long to transcribe

The speech-to-text API rejects uploads over 25 MiB. At 64 kbps that's about 54 minutes of audio — and a 45-minute interview that ran long because of timer drift failed with the same 413 error on every retry.

Fix: extract audio at 32 kbps instead. That's about 109 minutes of headroom, with no meaningful accuracy cost — the model works on 16 kHz mono speech anyway.

Where it ended up

Audio extraction runs for every interview, outside the encoding slot, bounded by its own concurrency.
Remux runs after every interview, one at a time, in seconds.
MP4 is on demand only, time-boxed by file size.
Every FFmpeg run is low-priority, time-limited, logged at start and finish, and reaped if Node exits.
Jobs hold heartbeated leases, and completion is idempotent.

What I'd tell my earlier self

Know which job you're asking for. Remuxing and transcoding look like the same command with different flags. They differ in cost by two or three orders of magnitude.
Keep optional heavy work off the critical path. Transcription needed an MP3; it waited on an MP4.
A shared lock between light and heavy work turns slowness into incorrect results. Fix it with both a heartbeat and idempotent completion — either alone isn't enough.
Every external process needs a timeout and an owner. Otherwise a hang leaks a slot, and a restart leaves orphans.
Look at the actual bytes before designing the fix. Most of the "headerless" files weren't.
Logging is part of the fix. For weeks, the only observability I had was ps.