Choosing a Vector Database for RAG: The Trade-offs That Matter at Every Scale
Your RAG prototype works. Chroma held 50,000 chunks without complaint, retrieval felt instant, demo went great. Then the roadmap says 10 million documents, metadata permissions, and sub-100ms retrieval, and suddenly the vector database for RAG decision is a real one, not a pip install you made on a Tuesday and forgot about.
This guide gives you the trade-off axes that actually matter, a comparison of the main candidates, and a recommendation by scale stage. Because here's the thing nobody says out loud: the right vector db for RAG at 100K vectors is usually the wrong one at 100M, and picking the 100M-vector tool on day one is its own kind of mistake.
Why RAG Changes What You Need From a Vector Database
Generic "best vector database" listicles miss something specific: RAG workloads have a particular shape, and that shape drives the whole decision.
• Read-heavy, with frequent incremental writes as new documents show up
• Needs metadata filtering constantly, tenant, source, date, permissions, not as an afterthought
• Benefits enormously from hybrid search for anything with exact terms, product codes, legal citations, error messages
• A recall problem here isn't a 500 error. It's a wrong answer delivered with total confidence
That last point is worth sitting with. In most software, a failure is loud, a stack trace, an error code, something breaks visibly. In a RAG pipeline (retrieve, augment, generate), a retrieval failure just quietly hands the LLM the wrong context, and the LLM, being an LLM, will confidently generate an answer anyway. A traditional database without ANN indexes can't do similarity search at any usable speed, that's the whole reason this category exists. But the deeper point practitioners learn the hard way: most "bad RAG answer" complaints trace back to retrieval quality, not the model. The database is usually innocent. Usually.
The Five Trade-off Axes That Actually Matter
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 moreManaged vs Self-Hosted (and Truly Open Source)
Managed options (Pinecone, Zilliz Cloud, Weaviate Cloud, Qdrant Cloud) mean zero ops and a faster start, at the cost of a recurring bill and, depending on your industry, some data residency questions worth asking upfront.
Self-hosted open source vector database options (Qdrant, Milvus, Weaviate, Chroma, pgvector) hand you full control at the cost of infra work and the joy of being on-call for your own vector search cluster.
A self-hosted Qdrant instance on a single cloud VM is often the actually-realistic choice for early-stage teams and student projects, the kind where engineering time is free (or at least unpaid) and a subscription is not. That equation flips hard the moment engineering time starts costing more than the monthly bill.
Hybrid Search Support
Hybrid search, in one snippet-sized definition: combining dense vector similarity with keyword scoring (typically BM25 or another sparse method), then fusing the two result sets, often via reciprocal rank fusion or a reranking step.
Why RAG specifically cares: embeddings are great at "meaning" and genuinely bad at exact strings. A product SKU, an error code, a legal citation, a person's name spelled an unusual way, dense retrieval alone will quietly miss these more often than you'd like.
• Weaviate and Elasticsearch/OpenSearch: strongest native hybrid support
• Qdrant and Milvus: support sparse vectors directly
• Pinecone: offers hybrid indexes as a first-class feature
• pgvector: needs Postgres full-text search bolted on separately, workable, just not automatic
Worth noting since it comes up constantly in shortlisting conversations: this is roughly the weaviate-vs-qdrant fork. Weaviate leans toward hybrid-search-as-default, Qdrant leans toward filter-heavy precision. Neither is wrong, they're just optimizing for slightly different first instincts.
Metadata Filtering
Pre-filtering vs post-filtering sounds like a pedantic distinction right up until it silently breaks your app in production.
Post-filtering runs the ANN search first, then throws away results that fail the metadata check afterward, which can quietly return fewer than k results and nobody notices until a user complains their search "feels broken." Databases with filter-aware ANN, Qdrant's payload indexes being the clean example, avoid this by folding the filter into the search itself, not applying it as an afterthought.
Concrete RAG example, the kind that shows up in basically every enterprise deployment: "answer only from documents this user's team can access." Get the filtering wrong here and you're not looking at a bug, you're looking at a permissions leak with extra steps.
Latency vs Recall, the Index Tuning You Can't Avoid
ANN search trades recall for speed, always, there's no free lunch here. HNSW is the graph-based algorithm most of these databases lean on, and its knobs (M, ef_construction, ef_search) are what you're actually tuning when someone says "just adjust the index parameters."
• Higher ef_search: better recall@k, more latency per query
• Higher M: better graph connectivity, more memory and slower index builds
Benchmark on your own embeddings, not a vendor's marketing chart. Vendor benchmarks are run on someone else's data with someone else's dimensionality, which tells you less than you'd hope. ANN-Benchmarks is the standard open reference if you want a starting point that isn't trying to sell you anything. And a quick reality check worth internalising early: how you chunk documents and which embedding model you pick both cap your ceiling before the index tuning even enters the picture. A perfectly tuned HNSW index over badly chunked documents just retrieves the wrong text very quickly.
Cost Model
Three pricing shapes, and they behave completely differently as you scale, which is the part most comparison pages gloss over:
• Usage-based serverless (Pinecone serverless, Zilliz): near-zero at prototype scale, grows with reads and storage. Cheap to start, worth watching closely as query volume climbs.
• Capacity or pod-based (dedicated clusters): you pay for reserved capacity whether you use it or not, predictable, occasionally wasteful.
• Infra-only self-hosted: a step function tied to VM sizes. Flat until you outgrow the box, then a real jump.
Check current entry-tier pricing on the actual provider pages before you commit anything, free tiers and pricing structures shift often enough that any number printed here would probably be stale by the time you read it.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreStop learning AI in fragments—master a structured AI Engineering Course with hands-on GenAI systems with IIT Roorkee CEC Certification
AI Engineering Course Advanced Certification by IIT-Roorkee CEC
A hands on AI engineering program covering Machine Learning, Generative AI, and LLMs - designed for working professionals & delivered by IIT Roorkee in collaboration with Scaler.
Enrol Now
Which Vector Database for RAG at Each Scale
Prototype and Side Projects (up to ~100K to 1M vectors)
Default: Chroma if you're starting fresh (pip install, native LangChain and LlamaIndex support), or pgvector if Postgres is already sitting in your stack. FAISS only if you specifically need raw library control for research or a custom pipeline.
At this scale, basically everything is fast enough. Optimise for iteration speed and zero ops, not for a scale problem you don't have yet. And here's the bit worth saying honestly: plenty of production apps live at this scale forever and never need to "graduate" to anything heavier. A well-chosen vector store here also happens to make a portfolio-worthy generative AI project if you're building to get hired, interviewers notice when someone picked the right tool for the actual problem instead of the flashiest one on the list.
Turn Learning into Career Growth
Production Applications (~1M to 100M vectors)
This is where the fork actually starts mattering, three reasonable paths:
• pgvector if the app is already Postgres-centric and QPS is modest, one database, one backup story, one less thing to monitor at 2am. With pgvectorscale-class extensions, the performance gap with dedicated engines has closed more than most people assume, one published benchmark measured pgvector plus pgvectorscale at
471.57 queries per second at 99% recall on 50 million vectors, against roughly 41 QPS for Qdrant on the same dataset, an order-of-magnitude gap that would have sounded implausible a couple of years ago. Worth reading the full methodology before treating that as gospel for your own workload, benchmarks are workload-specific by nature, but it's a real result from a real test, not vendor marketing copy.
• Qdrant (self-hosted or cloud) for filter-heavy, cost-sensitive workloads. This is roughly the qdrant-vs-pinecone fork in miniature: Qdrant if you want control and lower recurring cost, Pinecone if you'd rather pay to never think about infrastructure again.
• Pinecone serverless when the team genuinely wants zero ops and is fine paying for that peace of mind.
Migration triggers worth watching from the prototype stage: p95 latency creeping up, index rebuilds turning into a whole afternoon, filtered queries slowing to a crawl, or multiple app instances hitting the same embedded store and stepping on each other.
A pattern that comes up often enough to be worth naming: a team stays on Chroma comfortably until somewhere around 2 to 3 million vectors, then filtered queries start falling off a cliff, latency that was fine suddenly isn't. The migration to Qdrant that follows is usually less painful than people brace for. Re-embedding everything sounds like the scary part and turns out to be the easy part, mostly a compute bill and some patience. The actual work is re-testing every filter combination your app relies on, because that's exactly the thing that broke in the first place.
Large Scale and Multi-Tenant Platforms (100M to 1B+ vectors)
Milvus or Zilliz for billion-scale, distributed architecture built for exactly this. The milvus-vs-qdrant question at this size mostly comes down to whether you want Milvus's distributed-by-design approach or Qdrant's simpler model pushed to its outer limits, and honestly, past a certain point you stop asking which is "better" and start asking which ops team you'd rather have on call.
Elasticsearch or OpenSearch when vectors need to sit beside a search and analytics stack you already run and already trust. Pinecone pods or dedicated tiers for managed-at-scale, if the budget supports it.
What newly matters at this size, and didn't much at 1M: sharding, replication, quantization to keep memory sane, and genuine tenant isolation if you're serving multiple customers out of one cluster. Scaling retrieval infrastructure this far is, not coincidentally, one of the skills that separates a mid-level engineer from a senior one on the LLM engineering roadmap, it's rarely the flashy part of the job, but it's the part that keeps the flashy part working.
Common Mistakes When Choosing a RAG Vector Database
• Choosing by GitHub stars instead of your actual workload shape, popularity is not a benchmark
• Benchmarking against vendor demo data instead of your own embeddings and your own query patterns
• Ignoring filtering performance until it's already slow in production
• Picking a distributed system at prototype scale because it looks impressive on an architecture diagram, this has a name, resume-driven infrastructure, and it's more common than anyone admits
• Treating the vector store as sacred and unmigratable. Embeddings are recomputable. The store almost never is the permanent commitment people treat it as
• Blaming the database for what's actually a chunking or embedding-model problem, most retrieval quality issues start upstream of the database entirely
Worth a genuine watch if any of this sounds familiar: Why Your RAG Isn't Working walks through diagnosing exactly these failure modes. Most "vector database problems" turn out to be retrieval-pipeline problems wearing a database costume.
Evaluating infrastructure trade-offs like these, not just training models, is genuinely a core part of the AI engineer roadmap, the kind of judgement that tends to separate someone who can follow a tutorial from someone you'd trust to design the system.
A 60-Second Decision Checklist
1. How many vectors do you expect in 12 months? Under 1M, default to Chroma or pgvector and stop reading comparison articles.
2. Does your team actually want to run servers? If no, lean managed, no shame in paying for that.
3. Do your queries need metadata filters or tenant isolation? If yes, prioritise filter-aware ANN (Qdrant, Pinecone) over raw benchmark speed.
4. Do users search exact terms, codes, names, citations? If yes, hybrid search moves from nice-to-have to load-bearing.
5. Is Postgres already running your app? If yes, pgvector deserves a serious look before anything else.
6. What's your latency budget at your target recall? Write the actual number down, "fast" is not a spec.
7. What does the cost curve look like at 10 times your current scale, not just today? This is the question most teams skip and regret skipping.
FAQs
Which vector database is best for RAG?
There's no universal best. Chroma or pgvector for prototypes, Qdrant, Pinecone, or pgvector for production apps in the millions of vectors, Milvus or Elasticsearch-class systems past hundred-million scale. Choose by scale, hosting preference, filtering needs, and hybrid search requirements.
Is FAISS a vector database?
No. FAISS is an ANN search library, extremely fast, but with no server, no built-in persistence, and no metadata filtering. Databases like Milvus use FAISS-style indexes internally and add the database layer around them.
Can I use PostgreSQL for RAG?
Yes. The pgvector extension adds vector similarity search to Postgres, and modern extensions like pgvectorscale push it to tens of millions of vectors with strong recall. Often the pragmatic default when Postgres already runs your app.
Do I need a vector database for RAG at all?
Below roughly 100K vectors, an embedded store like Chroma or even plain in-memory search works fine. You need something dedicated once scale, concurrent users, filtering, or uptime requirements actually grow.
What is hybrid search in RAG?
Combining dense vector similarity with keyword scoring like BM25, then fusing the results. It catches the cases embeddings miss on their own, exact product codes, names, legal citations.
When should I move off Chroma or FAISS?
When query latency starts degrading, index rebuilds get painful, metadata filtering slows to a crawl, or you need multi-instance access and real uptime guarantees, typically somewhere past a few million vectors.
Managed or self-hosted vector database?
Managed trades money for time, zero ops, fast start. Self-hosted open source options cost engineering effort but buy control, data residency, and usually a lower infra bill, often the deciding factor for early-stage teams and student projects.
The axes here stay stable even as individual vendors leapfrog each other every few months, hosting model, hybrid search, filtering, the latency-recall trade-off, and cost. Pick for the scale stage you're actually at, with an honest migration path in mind, not for an imagined billion-vector future you may never reach. Most teams won't. And that's fine.
Designing full RAG systems, embeddings, retrieval, evaluation, and deployment, is part of the applied curriculum in Scaler's AI & Machine Learning program, if you'd rather build this stack with feedback than assemble it alone from documentation tabs.
