Needle 2 Packs an Agentic LLM Into a 14MB Binary

Cactus Compute's Needle 2 ships an agentic tool-calling model as a 14MB binary that runs a full session in about 28MB of RAM. The catch is buried in the API reference: fine-tune it, and the calibrated confidence score that makes it safe to trust with real-world actions returns None.

What Needle 2 actually is

Needle 2 is a 45-million-parameter model with a deliberately narrow mandate: tool calling, device control, and structured data extraction. No chat, no creative writing, no general Q&A. It is built to sit inside a phone app, a wearable, a smart-home hub, or a robot, and turn a request into a function call without a network round trip. The project is sitting near 6,800 GitHub stars, and the architecture is published as a paper (arXiv:2607.18363) credited to an eight-person team at the Y Combinator-backed company.

Two design choices do most of the work. Weights are compressed to roughly 2 bits each during pretraining rather than quantized after the fact, and every tool schema is compiled into a byte-level grammar that constrains decoding the whole way through — so malformed JSON isn't a training-quality claim, it's a hard decode-time constraint. A 256-token sliding window with tools pinned as KV sinks is what keeps memory flat at ~28MB no matter how long the conversation runs.

The confidence gate breaks exactly where you need it

Every Needle response carries a confidence score — the minimum of a calibrated post-hoc head and the raw decoding probability of the call tokens, so both signals have to agree. The intended pattern is to act automatically above a threshold and escalate below it. For a model meant to unlock doors or adjust thermostats, that gate arguably matters more than raw accuracy.

The docs state the limit plainly: "Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports confidence as None and warns once at construction." LoRA fine-tuning is the officially supported adaptation path — the entire toolchain encourages it — so the safety feature disappears at precisely the moment most teams adapt the model to their actual product. A one-line construction-time warning is easy to miss in a background service that only logs errors.

Read the benchmark comparison with skepticism

The README claims Needle "trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple's on-device Foundation Models, at 5x to 70x smaller, and 2 bits against their f16." Independent coverage citing the model's own tables reports 63.7% on Mobile Actions, 42.6% overall on BFCL v4 with 93.4% well-formed JSON, and 32.6% in-domain on Seal-Tools.

Those are not controlled comparisons. Needle runs at 2-bit precision with a 256-token context; the models it is measured against run at f16 with substantially larger context windows. The honest reading is "here's how far compression got us," not "beats models 70x its size on equal footing."

Where it's genuinely differentiated

Deployment surface, not accuracy, is the real argument. Needle is MIT-licensed for both weights and code and runs across macOS, Linux (x86-64, ARM64, ARMv7, RISC-V, MIPS32el), Windows, Android, iOS/watchOS/tvOS, and WebAssembly. Apple's Foundation Models run only inside Apple's stack and aren't user fine-tunable. Reported decode throughput is around 500 tokens/sec on a Raspberry Pi 5 and 300–700 tokens/sec on sub-$200 Android phones.

What to check before you ship it

If your product has real-world consequences and you plan to fine-tune, budget for building your own confidence signal — Cactus's stops at the base model's door. And treat the 256-token window as the binding constraint it is: it's the mechanism behind the flat memory number, and the sharpest limitation in the design.