Vectorless RAG: The Retrieval Approach That Skips Embeddings Entirely (Hype or Shift?)
Six months after shipping, your RAG system works. Then you upgrade the embedding model, spend a weekend re-embedding 400,000 chunks, and still cannot explain to your product manager why the retriever missed the one clause that mattered. Into that frustration walks a claim: vectorless RAG, meaning retrieval without embeddings, without chunking, without a vector database, where the model navigates your documents the way you would, from the table of contents inward. This is vectorless RAG explained the way a practitioner needs it: what is claimed, what it genuinely fixes, what it genuinely costs, and a straight verdict on hype versus shift, delivered in H2-8, not both-sided.
What Is Vectorless RAG?
Vectorless RAG is a family of retrieval-augmented generation approaches that retrieve context without embedding-based similarity search. Instead of converting text into vectors and finding nearest neighbours, the system navigates document structure, searches text directly, or lets an LLM reason its way to the right passage.
Three clarifications, because the term does more marketing than technical work. First, "vectorless RAG" is a positioning label, not an academic category. It was popularised by vendor VectifyAI (for PageIndex, from mid-2025) and it bundles three genuinely different mechanisms (H2-3). We say that first because no ranking page does.
Second, "vectorless" does not mean the model stops using vectors internally; every transformer represents tokens as vectors. It means no embedding index, and no similarity search over it as the retrieval mechanism. Doing RAG without vector database infrastructure was always possible; the label made it a movement.
Third, the ideas predate the branding. Tree-structured retrieval was published as RAPTOR (ICLR 2024), which, tellingly, still embedded chunks to build its tree, and lexical retrieval (BM25) has been production-standard for decades. Treat "vectorless RAG" as an umbrella, not an invention.
Prerequisite: we assume you know standard RAG. If not, read how a RAG pipeline fits together with agents and MCP first, then come back.
What Vector Retrieval Actually Costs You (The Case Being Made Against It)
The anti-vector case first, at full strength: four costs any pipeline operator will recognise.
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 moreSimilarity is not relevance
Embeddings retrieve passages that resemble the query, not passages that answer it. Ask an Indian bank's policy corpus "can we recognise this revenue now?": the answering clause is a dense accounting paragraph that never uses the word "recognise" in that sense, and cosine similarity fetches what sounds like the question instead.
The re-indexing tax
Change your embedding model (stronger ones arrive every few months) and every document must be re-embedded and re-indexed: scheduled, recurring, GPU-hour-burning, featureless. Embedding-model migration is the vectorless camp's most visceral argument.
Chunking is a tuning problem that never ends
Chunk size, overlap and boundary strategy all change retrieval quality, settings differ per document type, and teams lose weeks here and never finish. The vectorless pitch: this whole problem category disappears because documents are never cut up. (Chunk boundaries and overlap, if that fire is yours.)
Retrieval you cannot explain
When vector retrieval misses, the only artifact is a similarity score: no reasoning trail for an auditor, nothing to debug beyond "try a different model." Retrieval exists only to fill a limited context window, the constraint defining this design space (see how large language models process their context window), and in regulated Indian sectors (banking, insurance, IT-services compliance work), an unexplainable choice of what fills it is a genuine blocker.
The honest concession: partial fixes exist inside the vector paradigm (chunking, reranking, hybrid search, contextual retrieval). The counter-argument says those are patches on a mismatched primitive. H2-3 and H2-8 decide it.
How Vectorless RAG Works: Three Different Mechanisms Under One Label
Everyone else conflates these into one product; they are three bets on one principle: retrieval decisions are made by reasoning over text and structure at query time, rather than by geometry over vectors at index time.
| Mechanism | How it retrieves | Best for |
|---|---|---|
| Tree/index navigation | LLM walks a table-of-contents index | Long structured documents |
| Agentic file search | LLM calls search/read tools in a loop | Codebases and file stores |
| Lexical search + LLM reasoning | BM25 candidates, model reranks and reads | Identifier-heavy corpora |
Tree and index navigation, or reading the table of contents
The document is parsed into a hierarchical index of its real structure (parts, chapters, sections) with a short summary per node. At query time the model starts at the root and walks down: deferred revenue lives under Notes to Accounts, section 3.2. No chunking, no embeddings, no vector store. It is also how a person uses a 200-page annual report: nobody scans every paragraph; they open the contents page. Every step is an explicit, inspectable decision, which is classical reasoning over structured knowledge (the idea behind a knowledge-based agent), not a 2025 invention.
Agentic file search, or give the model tools and let it look
No pre-built index at all: expose tools (list files, read file, grep, open at line N) and let the model iterate: search, read, refine, read again. This is how modern coding assistants already handle repositories nobody ever embedded: several cheap, precise lexical looks where one embedding query gets a single guess. Practitioners call this agentic retrieval, the point where a retrieval pipeline blurs into an agent, covered in how RAG and agentic AI actually differ.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreLexical search plus reasoning, the unglamorous option that keeps winning
BM25 or full-text search generates candidates; an LLM reranks and reasons over them. The repeatedly re-discovered fact: on corpora full of jargon, codes and identifiers, plain keyword search competes with and sometimes beats approximate nearest neighbour (ANN) search over dense vectors (Robertson & Zaragoza's BM25 reference). That is why hybrid search exists. For small corpora (a few hundred pages), the honest fourth option is skipping retrieval for a long context window, accepting documented "lost in the middle" degradation and a per-query bill that needs prompt caching.
PageIndex: The Implementation That Named the Category
PageIndex, the project behind most "PageIndex vectorless RAG" searches, is VectifyAI's open-source RAG engine (the PageIndex GitHub repo, MIT license, created April 2025). It builds a JSON tree index of each long document, where nodes carry a title, summary and page span, then does what it calls reasoning-based retrieval: the LLM traverses the tree, opening only relevant branches. No chunking, no embeddings.
The PageIndex RAG repo sits at 35,200+ stars (August 2026); a PageIndex MCP server exposes the index to agent clients; and 2026 updates added a file-level tree for corpus-scale search, vision retrieval over page images, and LlamaIndex/LangChain integration. Two disclosures: PageIndex is a commercial open-core product, and the category name is its marketing. Both are fine, provided nobody mistakes the brochure for an independent review.
The FinanceBench claim, examined
People literally search "PageIndex FinanceBench 98.7%", so let's do that number properly. VectifyAI reports that Mafin 2.5, its PageIndex-powered system, achieved 98.7% accuracy on FinanceBench, far above naive vector RAG. FinanceBench itself is legitimate: 10,231 questions over SEC filings, whose own authors saw a GPT-4-Turbo-plus-vector-store setup answer incorrectly or refuse 81% of a 150-case sample (arXiv:2311.11944). The 98.7% is plausible, then, and still earns four fair criticisms:
- Vendor-reported, never independently replicated as of writing.
- One benchmark, one domain. Financial filings are the most rigidly structured documents in commercial use, the most favourable corpus imaginable for tree navigation.
- Unstated baselines. Against naive top-k cosine the gap is huge; against a tuned hybrid retriever with reranking, much smaller. A headline number without stated conditions is uninterpretable, which is the same discipline as choosing and interpreting evaluation metrics.
- Accuracy reported without the corresponding latency or token cost, which is precisely the trade being made (H2-5).
The correct takeaway: real for that document class; not generalisable to your support-ticket archive.
Vectorless RAG vs Vector RAG: What Actually Changes
Everything people mean by "PageIndex vs RAG", in one honest table: vectorless retrieval loses several rows, which is what makes the rest meaningful.
| Dimension | Conventional Vector RAG | Vectorless RAG |
|---|---|---|
| Retrieval mechanism | Nearest-neighbour search over embeddings | LLM navigation of structure, lexical search, or tool-driven exploration |
| Index build | Chunk, embed, upsert; hours and GPU/API spend at scale | Parse structure, summarise nodes; often lighter, sometimes near-zero |
| Query cost | One ANN lookup, effectively free | Multiple LLM calls per query; the dominant recurring cost |
| Query latency | Milliseconds | Seconds; grows with navigation depth |
| Cost curve | Expensive to build, cheap to run | Cheap to build, expensive to run; scales with traffic, not corpus size |
| Model upgrades | Re-embed and re-index the whole corpus | No re-indexing; swap the reasoning model freely |
| Chunking decisions | Required, ongoing, corpus-specific | Eliminated |
| Infrastructure | Vector database to run, scale and pay for | None beyond document storage and an LLM endpoint |
| Explainability | A similarity score | An inspectable navigation path; auditable |
| Determinism | Deterministic for a fixed index | Non-deterministic; same query may take different paths |
| Corpus scale ceiling | Millions of chunks, comfortably | Hundreds to low thousands of structured documents |
| Unstructured data (tickets, chat logs, scraped pages) | Works well | Weak; there is no structure to navigate |
| Exact-match and identifiers | Weak without hybrid search | Strong (lexical paths) |
| Best for | Large, heterogeneous, high-QPS corpora | Long, well-structured, high-stakes documents at modest query volume |
The cost inversion is the insight the entire SERP misses: vector RAG front-loads cost into the index and makes queries nearly free; vectorless RAG eliminates index cost and charges you tokens on every query, forever. Which is cheaper is not a property of the architectures at all; it is a property of your query volume. Back-of-envelope, using Haiku-class pricing as of 13 August 2026 (5 per million output tokens, claude.com/pricing): a typical navigation query (about 4 hops, roughly 10k input plus 600 output tokens) costs ~KaTeX parse error: Expected 'EOF', got '₹' at position 17: ….013, or about ₹̲1.1. At ten que…337,000 a month, against a vector stack whose marginal query cost is effectively zero. Same architecture, opposite outcomes: volume decides.
It is the mirror image of graph-based retrieval, which pushes even more cost into indexing: one spectrum, three spending choices.
Turn Learning into Career Growth
What Vectorless RAG Genuinely Fixes and What It Genuinely Costs
The four things it really does solve
- No re-indexing on model change. Structural indexes are model-agnostic: swap the reasoning LLM without touching the corpus. A permanent operational win.
- Interpretable, auditable retrieval. The navigation path is a human-readable explanation of why a passage was chosen, provenance an auditor can inspect. In compliance-heavy Indian sectors, worth more than an accuracy point.
- No chunking tuning, no vector infrastructure. An entire tuning discipline and an entire paid system disappear; the "vectorless db" pitch is, simply, that there is no db. For a small team, real surface area removed.
- Genuine strength on long, structured, high-stakes documents (annual reports, filings, contracts, standards, manuals), where hierarchy is meaningful and a wrong answer is expensive. The honest core of the case.
The five costs the marketing does not lead with
- Per-query LLM cost, forever. Every retrieval is now inference. That is the trade.
- Latency in seconds, not milliseconds. Sequential hops compound; in a user-facing chat product, this is a product decision, not an engineering one.
- A hard scale ceiling. Reasoning through a few hundred documents is tractable; through a million-leaf tree in a handful of hops, it is not. No competitor states this.
- It requires structure to exist. Tickets, Slack exports, scraped pages, OCR'd scans have no navigable hierarchy, and much enterprise data looks exactly like that.
- Non-determinism. The same query can take different paths run-to-run, complicating testing, regression suites and debugging in ways teams underestimate.
On the SERP's pricing searches (is pageindex free, PageIndex pricing): open-source core under MIT; hosted service free to 1,000 pages, paid tiers above. Verify current terms on the official site before budgeting; startup pricing moves monthly. Judging these trade-offs is an engineering-judgement skill built over a whole stack, and the AI engineer roadmap covering GenAI, LLMs and deployment is the longer arc.
When Vectorless RAG Makes Sense: A Routing Decision, Not a Migration
Choose vectorless when most of these hold: documents are long and genuinely hierarchical (reports, contracts, manuals, standards); the corpus is hundreds to low thousands, not millions; accuracy on hard questions beats latency; retrieval must be explainable or auditable; query volume is modest; the corpus is stable but your model choice is not.
Stay with vector RAG when most of these hold: the corpus is large, heterogeneous or unstructured; queries are high-volume or latency-sensitive; per-query cost must stay predictable and low; the workload is mostly similarity over short passages; you already run a vector database that works.
The recommendation: route, don't migrate. Classify each query or document type: structured long-form documents go to navigation, everything else to the vector index. That is where serious teams land, and it beats both camps' absolutism.
One tip learned the hard way: freeze an evaluation set of 30 to 50 real questions from your own corpus (easy lookups, exact-identifier queries, hard multi-section questions) and run both paths against it, measuring answer quality, p95 latency and cost per query together. Most architecture arguments dissolve the moment someone does. And the common mistake: teams adopt vectorless RAG to escape a retrieval-quality problem that was actually a chunking or reranking problem, trading a cheap fix (a reranker, better chunk boundaries, detecting and fixing bad retrievals) for a permanent per-query bill.
Interview angle: "Would you use a vector database for this system, and why?" is now a standard GenAI system-design probe, and the cost-inversion argument plus a routing rule is a far stronger answer than naming a product.
Conclusion
One judgement, as promised: a real shift in scope, hype in framing.
Three beats.
What is hype: the name, and the claim that embeddings are obsolete. "Vectorless RAG" is vendor positioning over pre-existing ideas (hierarchical retrieval from RAPTOR, lexical search from BM25, tool use). Nothing supports "vector databases are dead"; the scale ceiling alone rules out replacement for most large deployments.
What is a real shift: the underlying correction is genuine and overdue. The industry spent three years treating "RAG" and "vector database" as synonyms, when the vector index was always an implementation detail mistaken for the architecture. Retrieval means getting the right context into the window; similarity search is one way to do that, not the definition. That correction outlives this product. Even the vector camp's frontier labs argue for improving embedding retrieval rather than abandoning it (Anthropic's contextual retrieval is the measured counterpoint), so the debate is about the primitive, not the goal.
What makes it more relevant over time: per-query cost, its main weakness, is the line item falling fastest, and prompt caching plus agentic file search's success in coding tools push the same direction. A trend argument, not a proof; both halves matter.
The line to remember: the future of retrieval is probably not vectorless, but it is very likely not vector-only either. The durable skill is knowing which primitive a given query deserves.
FAQs
1. What is vectorless RAG?
Retrieval-augmented generation that retrieves context without embedding-based similarity search. Instead of vectors and nearest-neighbour lookup, the system navigates document structure, searches text directly, or lets an LLM reason its way to the right passage.
2. Can you do RAG without embeddings or a vector database?
Yes. Tree-index navigation, lexical search like BM25, and agentic file search all retrieve context with no embedding index. Each trades cheap index-time cost for per-query LLM cost.
3. How is vectorless RAG different from normal RAG?
Normal RAG decides by geometry over pre-computed vectors; vectorless RAG decides at query time by reasoning over structure and text, producing an inspectable retrieval path instead of a similarity score.
4. What is PageIndex and how does it work?
VectifyAI's open-source project builds a hierarchical, table-of-contents-style tree index of long documents; at query time an LLM traverses the tree to find the relevant section. No chunking, no embeddings.
5. Is vectorless RAG faster than vector search?
No. Vector search returns in milliseconds; vectorless retrieval takes seconds, because each query needs several sequential LLM calls. It trades latency for accuracy and explainability on hard questions.
6. Is vectorless RAG cheaper than vector RAG?
Entirely volume-dependent. It removes embedding and vector-DB costs but adds LLM tokens to every query: cheaper at low volumes, far pricier at high ones.
7. Does vectorless RAG scale to millions of documents?
Not well. It suits hundreds to low thousands of structured documents. A million-leaf tree cannot be navigated in a few reasoning hops, and unstructured data offers no hierarchy to navigate.
8. Should I replace my vector database with vectorless RAG?
Usually no. Treat it as a routing decision: send long, structured, high-stakes documents down a navigation path; keep the vector index for large, unstructured or high-traffic workloads. Measure both on your own corpus first.
Conclusion
Vector search asks, "what looks like the question?" Vectorless retrieval asks, "where would the answer live?" Long structured documents reward the second question; large messy corpora reward the first. So route, don't migrate, and let a measurement on your own corpus, not a vendor benchmark or an internet argument, settle which path each query deserves.
Evaluating retrieval architectures on cost, latency and evidence, rather than on whichever approach is trending, is exactly the engineering judgement Scaler's Advanced AI Engineering program with IIT Roorkee is built to develop.
