Vector Databases

The trade you are actually making

An embedding model turns a piece of text, an image or a product into a list of numbers — typically several hundred to a few thousand of them — positioned so that things people would call similar end up close together. Once meaning is coordinates, "find related documents" becomes "find the nearest points", which is a geometry problem with a long history and one inconvenient property: doing it exactly means measuring the distance to every point you have. At ten thousand items that is instant. At fifty million it is not a query, it is a batch job.

So every vector database at scale gives up exactness. It builds an index that lets a search visit a few hundred promising candidates instead of fifty million, and it accepts that a genuinely nearer item is sometimes never visited. The share of true nearest neighbours a search actually returns is called recall, and the single most important thing to understand about these systems is that recall is a dial. Turn it up and you visit more candidates for more latency and more CPU; turn it down and you get a fast answer that is quietly missing things.

One detail catches people before any of this matters: the distance metric has to be the one the embedding model was trained for. Cosine similarity compares direction and ignores magnitude; dot product rewards longer vectors as well as aligned ones; Euclidean distance measures straight-line separation. Use dot product on vectors the model expects to be normalised and popular items dominate every result for reasons that have nothing to do with relevance. Check the model card, use the metric it names, and normalise if it says to.

The index families, and what each one costs

Every product on the market implements some subset of four ideas, usually under its own names. Knowing the ideas rather than the brand names is what lets you read a benchmark honestly, because a benchmark comparing two products is almost always comparing two index configurations.

• Flat — no index at all

  Compare against everything. 100% recall, linear cost, no build step, no parameters. Correct up to a few hundred thousand vectors, and the baseline you measure every other index against.

• IVF — inverted file

  Cluster the space once, then search only the nearest few clusters. nprobe is the dial. Cheap to build, cheap in memory, and it degrades when the data shifts away from the clustering it was built on.

• HNSW — a navigable graph

  Layers of links, sparse on top for long jumps and dense below for precision. The best recall-per-millisecond in general use, and the default almost everywhere. efSearch is the dial; the graph itself costs real memory on top of the vectors.

• Quantization — PQ and friends

  Not a search structure but a compression: store an approximation of each vector, often 4–32× smaller. Layered under IVF or HNSW when memory is the binding constraint, at a further cost in recall.

The memory arithmetic is worth doing on paper before choosing anything, because it usually decides the answer. A vector of 1,536 dimensions in 32-bit floats is 6,144 bytes. One million of them is about 6 GB of raw vectors — and an HNSW graph over them adds roughly another 20–40% for the links, so call it 8 GB before payload, replicas or headroom. Ten million is a machine that costs real money every month. Two moves cut that dramatically: use an embedding model with fewer dimensions if one performs comparably on your data, and store vectors as 8-bit integers where the recall loss is acceptable, which is a straight 4× reduction.

Filtering is where the index fights you

Almost every real query has a condition attached: this workspace only, published documents only, in stock, in this language, not deleted. In a relational database that is unremarkable. Here it is the hardest thing on the page, because the index was built over the geometry of the whole collection and knows nothing about your condition. There are three ways to reconcile them and each fails differently.

The practical rule is to make high-selectivity boundaries structural rather than a filter. If every query is scoped to one tenant, one workspace or one language, give each its own collection or partition and the problem disappears — you are searching a small index rather than filtering a large one. Save filters for conditions that genuinely vary per query, and treat any filter that matches less than a few per cent of the collection as a signal that the data should have been separated in the first place.

Writes, deletes and the migration nobody plans for

Vector indexes are built for reading. A graph index can accept inserts while serving, but each insert adds links to an existing structure, and a graph that has absorbed millions of insertions is measurably worse than the same data indexed from scratch — recall drifts down without any error appearing. Deletes are worse: most engines mark the entry as removed rather than repairing the graph, so a deleted vector still occupies memory and still gets traversed, and a collection with heavy churn slowly fills with tombstones. Both are fixed by periodic rebuilds, which are an operational task somebody has to own and schedule.

• Bulk-load rather than insert one at a time when first populating: building the index once over the whole set is both faster and better connected.
• Treat a delete as a request that will be honoured in the results immediately and reclaimed in memory later, and check what your engine actually guarantees.
• Schedule rebuilds by churn rather than by calendar: a collection where 20% of entries have been replaced is due, whatever the date.
• Keep the source text and its metadata in your primary database, with the vector store as a derived index you can drop and rebuild.
• Store the embedding model name and version on every record, because one day the collection will contain two generations at once.

Do you need a separate database at all

The honest default for most products is an extension in the database you already run. Postgres with pgvector supports the same index families, gives you transactions, joins and one backup story, and lets a filter be an ordinary WHERE clause evaluated by a planner that knows about your other columns. It stops being enough at a scale most projects never reach — roughly when the vectors no longer fit comfortably in memory alongside everything else the database is doing, or when you need sharded, independently scaled search with per-collection replicas.

One last thing that saves projects: vector search is not always the best search. It is strong on paraphrase and weak on exact tokens — part numbers, error codes, names, rare acronyms — precisely the queries where users expect a literal match and are least forgiving. Combining a lexical index with a vector one and merging the two result lists reliably beats either alone, and it is often the cheapest quality improvement available. If your evaluation shows the vector index losing to plain keyword search on your own queries, believe the evaluation. How to run that evaluation belongs to the RAG page, which owns retrieval quality as a measured property.

Anti-patterns worth naming

• Shipping without ever measuring recall, so nobody can tell a tuning problem from a model problem.
• Post-filtering a selective condition and reporting the empty result as "the search is broken".
• A distance metric that does not match the embedding model, which produces plausible-looking nonsense.
• The vector store as the only copy of the text, so a rebuild means re-deriving data you no longer have.
• A dedicated cluster for two hundred thousand vectors, where a column in the existing database would have been faster to query and free to operate.
• Multi-tenancy enforced by a filter rather than by a boundary, so one forgotten clause is a data disclosure.

When to use it

Related pages

• DevelopmentRAG ArchitectureRetrieval-augmented generation (RAG) is how a model answers questions about data it was never trained on: find the relevant passages first, put them in the prompt, and require the answer to come from them. That makes RAG a search system with a language model on the end — and it means almost every wrong answer is a retrieval failure, not a generation one. This page covers both pipelines, how to measure retrieval separately, and the permission problem that turns a demo into an incident.
• DevelopmentNoSQLNoSQL is an umbrella term that names four different kinds of database by what they are not, which is why the choice is so often made badly. What they share is more useful than what they lack: data is placed on a node by a key you choose, so that key decides which queries are cheap, which are expensive and which are impossible. This page covers the four families, what a partition key really commits you to, what "eventually consistent" means for your users, and when a relational database is still the right answer.
• InfrastructureML PipelinesAn ML pipeline is the automated path from raw data to a model you can deploy. A trained model is a build artifact, not source code. It comes out of four inputs — the data, the training code, the configuration and the environment — and only one of those is in your repository, which is why cloning the repo and running the script gives you a different model. A pipeline exists to make all four recorded, repeatable and inspectable. This page covers the stages and why they are separate, why data breaks silently where code breaks loudly, the skew that makes an excellent offline model useless online, and when a notebook is genuinely enough.