How we sandboxed a public FFmpeg API with bubblewrap

FFmpeg is not a filter. It's a small operating system that happens to render video: it opens sockets, speaks HTTP, reads local files, writes to stdout, spawns threads, and will happily fetch a URL from a playlist you didn't write. When the whole product is "give us an FFmpeg command, we'll run it", you can't defend by validating the command. The set of valid commands is unbounded by design. You have to run the command somewhere it can't hurt you.

The threat model, in one sentence

Any authenticated user can submit any argv to FFmpeg, and any URL as media input. Our job is to make sure that giving someone that power doesn't give them the host.

Concretely, an attacker with an API key should not be able to:

• reach the cloud metadata endpoint at 169.254.169.254, or any internal service on the VPC,
• read files outside their own render's scratch directory,
• write outside that directory,
• read another tenant's media,
• turn the runner into a free compute farm or a DDoS amplifier,
• escape into the host kernel via an FFmpeg demuxer bug.

That's the list. Everything below is how we knocked items off it.

Layer 1: bubblewrap

bwrap is a low-level sandbox that creates Linux namespaces without requiring root. We wrap every FFmpeg invocation in it, one process per render:

function buildBwrapArgs(scratchDir: string, cmd: string[]): string[] {
  return [
    '--unshare-all',        // user, PID, mount, UTS, IPC, cgroup, net
    '--die-with-parent',    // sandbox dies when Node dies
    '--cap-drop', 'ALL',    // no Linux capabilities

    '--clearenv',
    '--setenv', 'PATH', '/usr/local/bin:/usr/bin:/bin',
    '--setenv', 'HOME', scratchDir,

    // binaries + shared libraries
    '--ro-bind', '/usr', '/usr',
    '--ro-bind-try', '/lib', '/lib',
    '--ro-bind-try', '/lib64', '/lib64',
    '--ro-bind-try', '/bin', '/bin',

    // dynamic linker needs its cache to find libs
    '--ro-bind-try', '/etc/ld.so.cache', '/etc/ld.so.cache',
    '--ro-bind-try', '/etc/ld.so.conf', '/etc/ld.so.conf',
    '--ro-bind-try', '/etc/ld.so.conf.d', '/etc/ld.so.conf.d',

    '--proc', '/proc',
    '--dev', '/dev',

    // writable scratch — tmpfs FIRST, then the bind
    '--tmpfs', '/tmp',
    '--bind', scratchDir, scratchDir,

    ffmpegPath,
    ...cmd,
  ];
}

A few of these flags deserve their own paragraph:

--unshare-all is the wall

It creates fresh namespaces for user, PID, mount, UTS, IPC, cgroup, and network. The child process sees a private PID space where it's PID 1, a private /proc that contains only itself, and a private network namespace with nothing but a loopback interface that isn't connected to the host's. That last part is the one that matters most: it doesn't matter what protocol -i http://… asks for, the kernel returns ENETUNREACH because there is no route. Argument parsing can't escape a missing network interface.

Why --ro-bind /usr and not --ro-bind / /

The tempting shortcut is to bind the whole root filesystem read-only and only expose the scratch directory as writable. It's one line, and it dodges the "which library path did the base image move this week" problem entirely. We don't do it, because our runners sit inside a Kubernetes pod that also has:

• /var/run/secrets/kubernetes.io/serviceaccount/token — a bearer token for the pod's ServiceAccount,
• /run/secrets/* — anything mounted by the orchestrator,
• Vault-injected secrets, when they're enabled.

None of those are needed by FFmpeg. All of them are readable by anything that can open(2) a path, and FFmpeg can read paths. A file: input pointed at a secret, piped through a nullsink filter, and echoed to stderr is a full exfiltration primitive — if the sandbox lets FFmpeg see the file in the first place.

So we bind only what FFmpeg needs: /usr for the binary and its shared libraries, /lib and /lib64 for the symlink farm on Ubuntu-based images, /bin for the shell FFmpeg occasionally execs, and the four /etc/ld.so* files so the dynamic linker can find libdrm.so.2 and friends. Everything else on the host filesystem is invisible inside the sandbox.

Why the mount order matters

bwrap applies mounts in the order they appear on the command line, and the last mount at a given path wins. If you write:

'--bind', scratchDir, scratchDir,
'--tmpfs', '/tmp',

and scratchDir happens to live under /tmp — which it does by default — the fresh empty tmpfs shadows the bind. FFmpeg enters the sandbox, sees an empty /tmp, and fails on the first -i. The fix is to declare --tmpfs /tmp first and then bind the scratch directory. Obvious in hindsight; hours of debugging the first time.

Layer 2: the protocol whitelist

bwrap handles the network, but defense in depth means we also constrain FFmpeg's own protocol set. We walk the final argv and inject -protocol_whitelist file before every -i:

const tokens = typeof execCode === 'string'
  ? parseCommand(execCode)
  : execCode;

const substituted = tokens.map((arg) => {
  const s = String(arg);
  return s.startsWith('$') ? (env[s.slice(1)] ?? s) : s;
});

const cmdProcessed: string[] = [];
for (const arg of substituted) {
  if (arg === '-i') {
    cmdProcessed.push('-protocol_whitelist', 'file');
  }
  cmdProcessed.push(arg);
}

Three things we learned the hard way:

• The option is per-input, not global. -protocol_whitelist file -i A -i B only constrains A. You need one per -i. Injecting a single prefix at the top of the argv is a no-op for every input after the first.
• String substitution is the wrong tool. Doing ncmd.replace('-i $MEDIA_1', ...) breaks on -i $MEDIA_1 (two spaces), -i\t$MEDIA_1, or when the same placeholder appears twice — String.prototype.replace without a global flag only touches the first occurrence. Walk the token array instead.
• Placeholders can prefix-collide. With String.replace, substituting -i $MEDIA_1 inside a command that also contains -i $MEDIA_10 produces -protocol_whitelist file -i /path/to/media-1 followed by a stray 0. Tokenizing first and matching whole tokens avoids the whole class.

Inside bwrap --unshare-all, this tripwire is redundant. Outside it — on a self-hosted deployment that forgot to enable the sandbox — it's the difference between "reads your VPC" and "fails cleanly."

Layer 3: server-side URL fetching, done properly

Users can submit a media URL as an input. That means our Node process is going to make an outbound HTTP request, on our network, with our network position. If we let the user choose the destination without checking it, we've built an SSRF engine with extra steps.

String-matching hostnames for "looks like an IP" doesn't work. 169.254.169.254.nip.io, http://2130706433/ (decimal 127.0.0.1), http://0x7f.0.0.1/, and IPv6-mapped addresses all bypass a regex. What works is resolving the hostname ourselves, validating every returned address, and then pinning the connection to the address we validated:

export async function assertPublicUrl(input: string | URL) {
  const url = new URL(input.toString());

  if (!ALLOWED_PROTOCOLS.has(url.protocol)) throw new BlockedUrlError(..., 'protocol');
  if (url.username || url.password)         throw new BlockedUrlError(..., 'credentials');

  const hostname = url.hostname;
  const bare = hostname.replace(/^\[|\]$/g, '');

  // IP literal — URL parser has normalised decimal/octal/hex forms
  if (net.isIP(bare)) {
    if (isBlockedAddress(bare)) throw new BlockedUrlError(..., 'ip');
    return { url, addresses: [{ address: bare, family: net.isIP(bare) }] };
  }

  const records = await dns.lookup(hostname, { all: true, verbatim: true });
  if (!records.length) throw new BlockedUrlError(..., 'dns');

  // any blocked address rejects the whole set
  const blocked = records.find((r) => isBlockedAddress(r.address));
  if (blocked) throw new BlockedUrlError(..., 'ip');

  return { url, addresses: records };
}

isBlockedAddress covers the IANA special-purpose registries for both families — RFC1918, CGNAT, loopback, link-local, multicast, ULA, the whole set — with the IPv4-mapped IPv6 form (::ffff:127.0.0.1) unwrapped and re-checked as IPv4, because Node's URL parser hands it to us in hex notation (::ffff:7f00:1) rather than dotted-quad.

The part that actually stops DNS rebinding

Validation alone is not enough. Between the moment we call assertPublicUrl and the moment Node opens the socket, the attacker's DNS server can change its answer. `https.get` re-resolves the hostname internally; if the second lookup returns a different IP than the first, we've validated one address and connected to another.

The fix is to hand Node a custom lookup function that returns the addresses we already checked:

export function pinnedLookup(addresses: ResolvedPublicUrl['addresses']) {
  return (_hostname, options, callback) => {
    if (typeof options === 'function') {
      callback = options;
    }
    if (options?.all) callback(null, addresses);
    else callback(null, addresses[0].address, addresses[0].family);
  };
}

// and in downloadFile:
const { url: parsed, addresses } = await assertPublicUrl(url);
client.get(parsed, { lookup: pinnedLookup(addresses) }, onResponse);

Two consequences:

• The socket connects to the IP we validated. The hostname argument is ignored inside the sandbox — there is no second DNS lookup.
• Redirects are followed manually. Node doesn't follow them by default, which is convenient; if we ever want to enable it we'd have to re-run assertPublicUrl on every Location, because the target of a redirect is a new destination and needs its own validation.

Layer 4: paths, tenancy, and the boring stuff

The interesting sandboxing lives in the process isolation. The bugs that produce bug-bounty reports live in the plainer layer above it.

Per-render scratch directories

Every render gets its own documentDir/<renderId>/ directory. Inputs are downloaded into it, FFmpeg writes its output into it, the result is uploaded to S3, and the directory is deleted. Nothing is shared between renders. This is the property that makes cross-tenant traversal impossible to exploit even if a validation check is missed somewhere upstream: there simply isn't a path that reaches another tenant's data, because no two renders ever share a filesystem.

Validation at the boundary, containment at the point of use

User-supplied identifiers (project.id, media.folderId) are validated at the API boundary with a strict regex — either a UUID or a pipeline-generated slug like audio-pipeline-extract_audio. Rejecting on mismatch, rather than sanitising, means a malformed id never reaches the filesystem in the first place.

Behind that, every path-composition site goes through a containment helper:

export function safePath(...segments: string[]): string {
  const base = fs.realpathSync(documentDir());
  const full = path.resolve(base, ...segments);
  if (full !== base && !full.startsWith(base + path.sep)) {
    throw new Error(`path escapes documentDir: ${full}`);
  }
  return full;
}

Validation catches the requests we expect. safePath catches the ones we don't — a future refactor that adds a new code path, a regex that gets loosened for a legitimate reason, a symlink someone drops into the scratch tree. Either alone would work in the common case; together they fail closed when either is wrong.

Stripping trusted fields from untrusted inputs

Renders submitted through the API have their bucket and key fields removed before insert. Those fields are only ever supposed to be set by admin-authored pipeline templates, which create renders through a different code path. Enforcing that at the point of insert — rather than relying on the frontend not to send them — means a curl against the API can't point our S3 client at an object the caller doesn't own.

Resource limits

Every FFmpeg process runs with a wall-clock timeout and a SIGKILL fallback, and the runner container caps CPU and memory at the cgroup level. A filter graph that fans out 9000× and eats all available RAM on the node is a denial-of-service against every other tenant on that node, and no amount of namespace isolation helps if the process is allowed to run until the OOM killer arrives.

What we got wrong the first time

A few things looked correct on paper and weren't:

• --no-net is not a Deno flag. It doesn't exist. The deno run call for our YAML transpiler exits immediately with an "unexpected argument" error and every pipeline edit fails silently. The correct flag is --deny-net, or just not passing --allow-net at all, since network access is denied by default.
• --allow-read with no path grants the whole filesystem. It's the same privilege as -A for reads; the flag rename doesn't scope anything. Real scoping looks like --allow-read=/opt/ffmpeglab/sdk,/tmp/ffmpeglab.
• bwrap's --ro-bind /lib /lib fails on symlinked /lib. On Ubuntu-based images, /lib is a symlink to /usr/lib; --ro-bind (not --ro-bind-try) can error, or silently do the wrong thing. Using --ro-bind-try for anything that might not be a real directory is cheap insurance.
• Sandboxing a spawn without a --die-with-parent equivalent leaves orphans. When the Node process exits uncleanly, the child keeps running with its namespace still attached. bwrap's --die-with-parent is the exact feature for this; we just had to remember to add it.

Putting it together

The resulting stack, from the outside in:

1. API boundary — strict regex on identifiers, strip trusted fields from user-submitted data, reject rather than sanitise.
2. URL validation — resolve, validate every address against the special-purpose registries, pin the socket to the validated IP, follow redirects manually.
3. bwrap — --unshare-all, --cap-drop ALL, --clearenv, selective read-only binds, one writable scratch directory, --die-with-parent.
4. FFmpeg argv — -protocol_whitelist file before every -i, injected by a token walk rather than string substitution.
5. cgroup limits — CPU, memory, PID count, and a wall-clock kill.

Each layer is independently sufficient for a subset of the threat model, and independently insufficient for the whole of it. The network namespace stops FFmpeg from reaching the metadata service; the protocol whitelist stops a legitimate-looking command from trying; the URL validator stops our own Node process from being the fetch. Take any one away and the others still hold, which is the property that matters in production — because "we wrote it correctly on the first try" is not a security control.

Should you do this?

If you're running a single-tenant video editor on your own machine — no. Run FFmpeg directly, keep your interface small, and save the complexity.

If you're running a service where one user's render shares a host with another user's data, and both users are authenticated but neither is trusted, then yes, and the four layers above are close to the minimum. bwrap is a small dependency, the container runtime provides most of the rest, and the URL validation is roughly a hundred lines of Node that you will never have to think about again — as long as nobody edits it without rerunning the tests.