Prompt Ensembles Only Beat Self-Consistency When Errors Differ

Five different prompts voting beat five samples of one prompt by six points on an identical five-call budget — but only because the five prompts failed on different questions. That caveat is the whole finding.

The numbers come from an explorable simulation published on dev.to, and the author is explicit about what the engine is: "The engine simulates nothing about language." The 24 arithmetic questions are data, each with four options — the true value, an off-by-one, a digit swap and a rounded figure — and a "prompt variant" is a scoring profile over those options, not a real model call. Read the figures as a demonstration of voting mechanics, not as a benchmark of any shipping model.

The headline comparison

On the author's 24 questions at a budget of five calls each:

• Self-consistency — one prompt, five samples at temperature 0.3, majority vote: 15/24.
• Prompt ensemble — five different prompts, plurality vote: 21/24.
• The best single prompt alone: 17/24.

Self-consistency landed below the best single prompt it was sampling from. Pushed to k = 301, the simulation has it settle at 17/24, reproducing that prompt's own greedy pick on 24 of 24 questions. The author's conclusion: the prompt is the ceiling, and k buys stability rather than a second point of view.

The dial that decides whether an ensemble pays

The sharpest part of the write-up blends each member's profile toward the first member's, sliding the set from five genuinely different prompts to five rewordings of one. Cost stays identical throughout.

Member set	Error correlation	Best single	Ensemble	Gain	Agreement when wrong
Five different prompts	−0.144	17/24	21/24	+4	53.3%
Five rewordings of one	0.508	17/24	16/24	−1	87.5%

The gain does not merely shrink as correlation rises — it crosses zero and goes negative, at unchanged cost. At a correlation of 0.644, the simulation puts the ensemble at 16/24 with 92.5% agreement when wrong; five identical calls reach 100% agreement when wrong for five times the bill.

That last column is the operational risk. A diverse ensemble that is wrong disagrees internally and looks shaky. A correlated ensemble that is wrong agrees almost unanimously and looks certain. Any gate that routes high-agreement answers past human review will therefore fail in exactly the direction the ensemble fails.

Two implementation details that cost accuracy for free

Use an odd k. The simulation's accuracy by k runs 17 → 18 → 20 → 18 → 21 — adding a member is not a safe move. At k = 2, twelve of the 24 questions get decided by tie-break policy rather than by vote.

Write the tie-break down. Five members over four options tie often, and a loop that keeps a running maximum silently returns whichever tied option the first member voted for. Across all 120 permutations of the five members, that default returns different answers on 2 of 24 questions; listing the same five members in reverse moves the sheet from 21/24 to 20/24 with nothing else changed. Breaking ties on a property of the options instead — lowest listed position, summed confidence, Borda score — is invariant across all 120 orderings and costs nothing.

On aggregation rules, the author tested seven seeds and reports Borda was at least as good as plurality and confidence-weighting on 7 of 7, while plurality and confidence-weighting split three seeds each. The caveat attached: Borda needs every member to rank all options, which means more output tokens per call.

What to measure before you ship one

The question is not whether your prompts read differently. It is whether they fail on different questions. Before paying k times for an ensemble, measure on a held-out set: each variant's accuracy on its own, the pairwise error correlation, the agreement rate on wrong answers, and accuracy at k = 1, 3 and 5. If the ensemble is not beating your best single variant, or correlation is high, ship k = 1 and keep the money.