Vercel Sandbox Now Runs Terminal-Bench in Firecracker VMs

Vercel now runs Harbor evals on Vercel Sandbox, with each benchmark trial executing in its own isolated Firecracker microVM.

The changelog, published September 17 by Elisabeth Rülke and George Fahmy, describes Harbor as "the open-source harness behind Terminal-Bench, whose registry includes many other benchmarks such as SWE-bench, tau3-bench and OSWorld." That registry is the point: the same runner covers coding, agent and desktop benchmarks.

What the flag does

The mechanism is a single flag. Per the changelog, you "pass --env vercel to harbor run and each trial executes in its own isolated Firecracker microVM, so you can parallelize far beyond what your local machine is capable of."

That is the practical constraint being lifted. Benchmark suites are embarrassingly parallel and locally bounded — a laptop runs a handful of trials at a time, and a full sweep turns into an overnight job. Fanning trials into per-trial microVMs converts wall-clock time into concurrency you rent.

Isolation sits outside the VM

The security design is the more interesting detail for anyone benchmarking agents that run untrusted code. The changelog states that "a task's network policy is enforced at the sandbox firewall, outside the VM," and that "optional credential injection attaches secrets to matching outbound requests at that firewall, so they never enter the sandbox."

Enforcing the policy outside the guest means a task that compromises its own VM does not thereby control its egress rules. Injecting credentials at the firewall means a benchmark task can authenticate to a service without ever holding the token — the boundary holds even if the trial is actively hostile. For agent benchmarks, where the whole point is letting a model run real commands, that ordering matters more than it would for a unit-test runner.

Swapping models is a flag change

The changelog's setup example, reproduced as published:

uv tool install 'harbor[vercel]'
export VERCEL_TOKEN="..."
export AI_GATEWAY_API_KEY="..."

harbor run -d terminal-bench/terminal-bench-2-1 \
  --agent fx \
  --model vercel_ai_gateway/anthropic/claude-fable-5 \
  --env vercel \
  --n-concurrent 8

Vercel notes that with AI Gateway, "one AI_GATEWAY_API_KEY reaches hundreds of models from multiple providers, and benchmarking another model is the same command with a different --model." The changelog gives the concrete swap: change --model to vercel_ai_gateway/openai/gpt-5.6-luna to run the same benchmark against an OpenAI model. Harbor 0.22.0 or later is required.

Why this lands now

Single-key, single-flag model swapping is what makes a cross-provider comparison cheap enough to actually run. Most published model comparisons are vendor-run, on vendor-chosen tasks, with vendor-chosen labels — and the gap between those numbers and your workload is the thing no benchmark page can close for you.

Note the shape of the incentive, though: the harness is open source, but the concurrency and the gateway are billed products. The comparison gets cheaper to run in engineering time and starts costing money in compute.

What to watch

Two things worth checking before you budget a sweep: what --n-concurrent 8 costs at the concurrency you would actually want, and whether the tasks you care about are in Harbor's registry or need porting. Neither is answered in a one-minute changelog.