AI Code Passes Tests but Only 18.4% of It Is Secure

A new benchmark ran coding agents against 145 real CVEs. Claude Code passed the functional tests on 74.0% of tasks and produced secure code on 18.4%.

The gap is the finding. According to the write-up of the study, 55.6% of Claude Code's output "passed every test while a working exploit still succeeded." If your review process treats a green test suite as a security result, that is the number to sit with.

What the benchmark measures

The write-up reports that on November 4, 2026, eight researchers (Yu et al.) posted A.S.E 2.0 to arXiv. It is described as a preprint, not yet peer reviewed, so treat the figures as a first read rather than a settled result.

The design is stricter than most agent benchmarks. The tasks are "145 tasks built from real CVEs in 51 open-source projects," and each one "runs in Docker with functional tests and a validated proof-of-concept exploit, and counts as secure only if the tests pass and the exploit fails." There is no partial credit for code that works and leaks.

What the benchmark is actually varying is context: "It tests how the repository context an LLM sees changes the security of the code it writes." That makes the results actionable in a way a raw leaderboard is not — the knobs it moves are knobs you control.

Passing tests is not a security signal

The headline pairing — 74.0% functional, 18.4% secure — is not specific to one agent. The study found that as a share of test-passing output, 73.2% to 77.3% was exploitable, "whether it was Claude Code, SWE-agent on Sonnet 4.5 or GPT-5.4, or plain retrieval with no agent loop."

That consistency matters more than the absolute numbers. Swapping the harness did not move the ratio. Neither did swapping in a different frontier model among the two tested. The failure is not a bad agent; it is what "done" means to an agent whose only stopping signal is a passing test.

The agents searched for the wrong things

The most useful section of the study is about retrieval behavior. The agents "read far more of the repository than any retrieval pipeline, yet saw the relevant safeguard (an existing validator or access policy) in only 27% to 36% of their runs."

The reason shows up in the queries. Per the write-up, "71% of their search queries named functional things, such as a file format; only 9% named security concepts." An agent asked to fix a path-handling bug goes looking for path handling. It does not go looking for the validator that the codebase already has.

Seeing the safeguard helped, and not seeing it hurt:

Condition	Secure rate
Saw and used the safeguard	34.5%
Never saw the safeguard	12.7%
Reference context of annotated safeguards	24.8%
Target file alone	12.6%
Generic "write secure code" reminder	13.9%

Two things fall out of that table. First, the generic reminder is close to worthless: it moved the secure rate from 12.6% to 13.9%, while naming the safeguards moved it to 24.8%. The instruction that works is specific, not virtuous.

Second, even visibility is not enough. The study reports that 41% of runs that observed a safeguard didn't use it. Putting the validator in front of the model roughly doubles your odds; it does not close the case.

More repository context made it worse

The intuitive fix — feed the agent more of the repo — went the wrong way. A truncated 128K-token repository "raised the functional pass rate from 56.9% to 60.9% and cut the secure rate from 12.6% to 12.1%." More context bought working code and slightly less safe code.

Worse, the repository teaches by example. Inserting one insecure example "raised correct-but-exploitable output by 6.3 points, even with the safeguard present." The dead authentication helper nobody deleted is not inert. It is training data for every agent run that greps past it.

The agent loop cost 15x and bought 0.5 points

The cost comparison is the line to bring to a budget conversation. Per the write-up, "Claude Code cost $1.41 per task against $0.09 for its retrieval setup without the loop, and read 204K tokens against 7.2K, for a secure rate of 18.4% against 17.9%."

Read that as a scoping result, not an argument against agents. The loop is paying for the functional work — exploration, iteration, getting the tests green. On the security dimension specifically, this benchmark found it bought almost nothing. If you are funding an agent loop expecting safer output as a side effect, that expectation is the thing to drop.

What to change on Monday

The study's own recommendations are concrete, and they are all about what the agent is told, not which model you buy:

• Name the safeguard in the prompt. The write-up's example is literal: "use validate_path() from utils/security and check can_edit() before writing." The agent will not reliably go looking.
• List your validators, auth checks and safe APIs, with paths, in your agent instruction file. This is the 24.8%-vs-12.6% condition, made permanent.
• Remove or label insecure examples in the code the agent reads.
• In review, check that the safeguard is called, and called in the right place. The write-up flags "a permission check that runs too late" as a real failure mode in the paper.

One caution on tooling: the write-up says static analysis "disagreed with the exploit oracle on 36.8% of 28,984 functionally correct generations." A clean scanner run is a third signal, not a verdict.

A separate project this week framed the adjacent problem — code that is correct but corrosive. AntiSlopScan's author opens with "A pull request can pass its tests and still make the next change harder," and ships what he describes as "an open-source, self-hosted audit appliance for code quality, architecture and systems-thinking indicators" that runs from the CLI, a web interface, or CI. Its stated approach is to make architectural intent checkable — a rule like "domain code must not depend on transport code" is something a reviewer can verify. The project is at the 1.0.0-alpha.1 Docker image stage and, by its author's own description, "leaves unsupported assessments ungraded." That is an honest boundary, and the same one the benchmark draws: tooling narrows the review, it does not replace it.

What this does not settle

The limits are stated in the write-up and worth repeating before anyone quotes 18.4% in a planning doc. The tasks come from known CVEs, "so they're security-sensitive by construction" — this is not the security rate of ordinary application code. The paper tested Sonnet 4.5 and GPT-5.4, "not newer models." And an exploit "can demonstrate a vulnerability but not exclude one," so the secure rates are upper bounds on what the oracle caught.

The thing to watch is whether a newer model moves the test-pass-to-secure ratio, or only the test-pass rate. Every setup in this run landed between 73.2% and 77.3% exploitable among passing output. If that band holds on the next generation, the constraint is the context you hand the agent, not the model you pick.