pgvector vs LanceDB: What the Benchmark Numbers Mean

A reproducible benchmark on 100,000 real OpenAI embeddings just answered the question every RAG team argues about in Slack: pgvector or a dedicated vector store. The numbers split cleanly by workload, not by which side shouts louder.

The benchmark, run straight

An independent test (published on the blog implicit-none.com, code on GitHub) measured pgvector 0.8.6 against LanceDB 0.36.0 on identical data — 100,000 DBpedia OpenAI embeddings at 1536 dimensions, 1,000 held-out query vectors, k=10, cosine distance, ground truth from exact full scan. Both systems ran on the same machine (Apple M5 Pro, 48GB RAM), with pgvector capped at 8 CPUs and 6GB of shared buffers to keep the comparison fair. The test swept both systems' tuning parameters into full recall-latency curves rather than picking one setting each, which is the only way an ANN comparison means anything.

Ingest and index build: no contest

Metric	pgvector	LanceDB	Ratio
Ingest (100k vectors)	31.3s (3,190 vec/s)	1.4s (71,474 vec/s)	22.4x
Index build	60.3s	5.2s	11.6x
Total disk	2.48GB	0.78GB	3.2x

pgvector's ingest goes through Postgres's COPY text format, paying the cost of textifying 1536-dim vectors through the SQL layer. LanceDB writes Arrow tables close to as-is. For a pipeline that regenerates or appends embeddings daily, that 22.4x ingest gap is wall-clock time someone waits on every day.

Single query vs. concurrent load: the numbers flip

At equal recall (~0.97-0.98), LanceDB answered single-threaded queries in 1.58ms versus pgvector's 2.86ms — roughly twice as fast, though that comparison folds in pgvector's TCP and SQL protocol overhead, which is genuinely part of the cost of running it that way. Push to 8 concurrent clients and the result reverses: pgvector scaled to 2,376 queries per second (6.8x from its single-thread number, one Postgres backend process per connection using all 8 cores), while LanceDB reached 1,338 QPS (2.2x), constrained by Python's GIL around its Rust search kernel. The engine that was 2x faster alone loses by 1.8x once eight clients hit it at once — a direct consequence of process-per-connection parallelism versus single-process Python concurrency, not a tuning mistake on either side.

Filtered search is where it gets uncomfortable for pgvector

Real RAG systems combine a WHERE clause with vector search, and this is where the two systems' personalities diverge most. LanceDB's prefilter held steady: recall 0.97-1.0 at 2.3-7.5ms across both 1% and 10% selectivity, tested and predictable. pgvector's behavior depends on which query plan Postgres's planner picks, and the benchmark caught it making a costly mistake: immediately after a bulk load, before running ANALYZE, the planner had no statistics and chose HNSW plus post-filter even at 1% selectivity — recall collapsed to 0.071. The identical query returned recall 1.0 after running ANALYZE. At 10% selectivity the plan flipped non-monotonically as ef_search increased, swinging between fast-but-partial and slow-but-exact; setting hnsw.iterative_scan = relaxed_order (new in pgvector 0.8) lifted recall from 0.76 to 0.97 at ef=80. The lesson the benchmark states directly: always run ANALYZE after bulk-loading into pgvector, and know that its filtered search trades LanceDB's predictability for tuning literacy.

Which one to run

The benchmark's own framing is about workload shape, not a universal winner: pgvector fits a team already running Postgres with high concurrent QPS, gets ACID and existing backup/permission tooling for free, and wins 1.8x at 8 threads. LanceDB fits a pipeline that regenerates embeddings often, needs Arrow/pandas-native tooling, or serves single-request latency (agent memory, for instance) without standing up a server. Filtered search compound queries favor LanceDB's consistency unless a team is willing to learn pgvector's planner behavior.

The other half of the bill: which embedding model feeds the index

A separate piece on choosing an embedding model puts a number on the cost side of that decision, which is easy to treat as an afterthought next to database benchmarks. Dimension count sets storage and index memory linearly — the piece states an 8x RAM cost between a 384-dimension model and a 3072-dimension one. Re-embedding is the real switching cost, since vectors from two different models are not comparable and a model swap means re-embedding the whole corpus: for 5 million chunks averaging 350 tokens (1.75 billion input tokens), the piece calculates $35 on OpenAI's text-embedding-3-small at its January 2024 launch rate of $0.02 per million tokens, versus $227.50 on text-embedding-3-large at $0.13 per million. At 500 million chunks the same math produces $3,500 and $22,750 — the point where, per the piece, the model choice becomes a real line item rather than a rounding error next to storage and index costs.

The piece also warns against leaning on MTEB leaderboard rank alone: the benchmark aggregates unrelated task types into one average score, and its retrieval sub-score — the only column that matters for a RAG system — can sit mid-table even for a model that tops the overall leaderboard. Its recommended selection process is a 50-query gold set pulled from real search logs, scored on recall@10 and MRR@10 against each shortlisted model with that model's own prefix convention applied, rather than trusting a public benchmark to predict behavior on a specific corpus.

What this means for a stack decision this week

Anyone benchmarking pgvector against a dedicated store should test at their real concurrency level, not just single-threaded latency — the ranking can invert at 8 clients, per the numbers above. Anyone running pgvector in production should confirm ANALYZE runs after every bulk load; skipping it is the difference between recall 1.0 and recall 0.07 on the same query. And before comparing vector databases, the benchmark on embedding model choice suggests pricing out the corpus at the target scale, since a wrong dimension or model pick can carry a re-embedding bill in the thousands once a corpus reaches hundreds of millions of chunks.