Text Embeddings Explained: How Meaning Becomes Math
Type “cheap flights to Goa” into Google. Somewhere in the results you'll also see pages talking about “affordable tickets to Goa beaches,” and not one word matches between the two phrases. Google isn't guessing. It's doing math.
That trick sits on top of something called text embeddings, and once the idea clicks, a good chunk of modern AI stops looking like a black box and starts looking like, well, arithmetic with extra steps.
By the end of this you'll know what a text embedding actually is, how a sentence turns into a list of numbers, why one model spits out 384 numbers and another spits out 3072, and how to actually pick a model instead of googling “best embedding model 2026” at 1am and trusting the first Reddit thread.
Let's get into it.
What Are Text Embeddings, Really?
Here's the short version: text embeddings are numerical representations of text, lists of numbers (called vectors) that capture meaning. Texts with similar meanings end up with similar numbers. An embedding model is the thing that converts words, sentences, or entire documents into these vectors so a computer can compare meaning mathematically instead of just matching keywords.
If you want the one-line Hindi version for your notes: embedding matlab text ka numerical roop jo uska matlab (meaning) capture karta hai. Bookmark that if the English jargon starts blurring together somewhere around paragraph four.
The core intuition: meaning as a point on a map
Picture a city map. Two addresses close together on the map are usually close together in real life too. Embedding space works on the same logic, except instead of two axes (latitude, longitude), you've got hundreds, sometimes thousands, of invisible ones.
“Doctor” ends up sitting near “nurse” and “hospital.” It's nowhere close to “cricket bat,” unless the model was trained on an unusually specific medical drama.
Now the trick every ML person loves showing off with (we really need better party tricks as an industry): take the vector for “king,” subtract “man,” add “woman,” and you land almost exactly on “queen.” Meaning, quite literally, became math. That's the whole spine of this article.
Word embeddings vs sentence embeddings vs basically everything else
| Level | What gets one vector | Example model | Typical use |
|---|---|---|---|
| Word | A single word | word2vec | Older NLP, word similarity |
| Sentence | A full sentence or passage | SBERT / all-MiniLM | Semantic search, RAG |
| Document | An entire document | Doc2Vec, long-context embedders | Document clustering |
| Multimodal | Text and images, same space | CLIP | Image search from text |
Quick terminology cleanup, because this space loves overlapping words: “vector embeddings” is the umbrella term for turning any kind of data into a vector. “Text embeddings” means vectors specifically for text. “Word embeddings” is the older, word-level flavour of the same idea. Almost every LLM app you'll build today uses sentence or passage level text embeddings, not word-level ones. Word2vec had a good run, though.
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 moreHow Meaning Becomes Math: How Embedding Models Learn
The whole field rests on an idea linguists had decades before anyone trained a neural net: a word is known by the company it keeps. Words that show up in similar contexts across billions of sentences tend to mean similar things. Nobody tells the model what “bank” means. It just notices that word keeps hanging around “money,” “loan,” and “account,” and shapes a vector accordingly.
The word2vec era: one vector per word
Back in 2013, a Google team (Mikolov et al.) published word2vec. The idea: train a small model to predict a word from its neighbours, throw away the prediction task, and keep the byproduct, a set of dense vectors, usually around 300 dimensions, where similar words end up close together. GloVe arrived around the same time doing something similar using word co-occurrence counts instead.
Here's the catch. One vector per word means “bank” gets exactly one representation whether you mean a river bank or the place charging you a fee for a bounced cheque. Context gets flattened out entirely. If you want to see where this fits into the bigger NLP learning sequence, the NLP roadmap lays it out properly.
The transformer era: sentence embeddings that actually understand context
Transformers read the whole sentence at once, so “bank” gets a different vector depending on what's around it. Sentence embedding models, generally called sentence transformers, take those token-level vectors and pool them into a single vector for the whole sentence. If you want the mechanics of how a transformer processes a sentence in the first place, this piece on transformer architecture covers attention properly, so we won't repeat it here.
The training trick behind most modern embedding models is called contrastive learning, and it's simpler than the name suggests. Show the model pairs of text that mean the same thing, and pairs that don't. Nudge the matching pairs closer together in vector space. Push the mismatched pairs further apart. Do this a few million times and you end up with a model that's genuinely good at telling “this means the same thing” from “this absolutely does not.”
Stop learning AI in fragments—master a structured AI Engineering Course with hands-on GenAI systems with IIT Roorkee CEC Certification
AI Engineering Course Advanced Certification by IIT-Roorkee CEC
A hands on AI engineering program covering Machine Learning, Generative AI, and LLMs - designed for working professionals & delivered by IIT Roorkee in collaboration with Scaler.
Enrol Now
How Similarity Is Measured: Cosine, Dot Product, Euclidean
| Measure | What it compares | Range | When to use |
|---|---|---|---|
| Cosine similarity | Angle between vectors | -1 to 1 | Default choice for text |
| Dot product | Angle plus magnitude | Unbounded | When the model was trained for it |
| Euclidean distance | Straight-line gap between points | 0 to ∞ | Common in clustering |
Cosine similarity ignores how “long” a vector is and only asks: are these two pointing roughly the same direction? Direction is where meaning lives, not magnitude. Here's a detail that'll save you real debugging time: most modern embedding models output normalised vectors, which means cosine similarity and dot product give you the exact same ranking. If swapping one for the other in your code never changed your results, now you know why.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = [
"How do I reset my password?",
"What is the process to change my password?",
"What's the weather like in Delhi today?"
]
embeddings = model.encode(sentences)
similar = util.cos_sim(embeddings[0], embeddings[1])
unrelated = util.cos_sim(embeddings[0], embeddings[2])
print(f"Similar meaning: {similar.item():.3f}")
print(f"Unrelated meaning: {unrelated.item():.3f}")
Run that on all-MiniLM-L6-v2 and you'll typically land somewhere around 0.80 to 0.85 for the two password questions, and closer to 0.05 to 0.10 for the weather question. The exact numbers will shift slightly by model version, but the gap between “related” and “unrelated” should be obvious and wide. If it isn't, something in your pipeline is off, not the concept.
How to Choose an Embedding Model in 2026
Start with MTEB, the Massive Text Embedding Benchmark. It's the honest starting point, a public leaderboard scoring models across retrieval, classification, clustering, and more. One warning: don't leaderboard-chase blindly. The top overall models are often huge and slow. For RAG specifically, look at the retrieval column, not the overall average.
| Model | Dimensions | Open source? | Best for |
|---|---|---|---|
| text-embedding-3-small | 1536 | No (API) | Cheap, reliable default |
| text-embedding-3-large | 3072 | No (API) | Max quality, supports Matryoshka |
| Gemini embedding | 3072 (Matryoshka) | No (API) | Google ecosystem, multilingual |
| Cohere Embed v4 | Varies | No (API) | Enterprise search |
| bge-m3 | 1024 | Yes | Multilingual, long documents |
| e5 family | Varies | Yes | Strong retrieval, free to run |
| gte family | Varies | Yes | General purpose, free to run |
| Qwen3-Embedding | Varies | Yes | Top MTEB tier, open |
| all-MiniLM-L6-v2 | 384 | Yes | Laptops, free tiers, student projects |
Specs and pricing shift fast in this space, so double-check current numbers against the OpenAI embeddings documentation and MTEB before you commit to anything for production.
A decision framework that actually holds up, instead of just picking whatever's trending on X:
1. Privacy and budget: open-source model running locally, or a hosted API? Both are fine, they solve different problems.
2. Language: if there's any Indic-language content in the mix, go multilingual from day one.
3. Scale: leaning toward smaller dimensions if you're going to be storing millions of vectors.
4. Check the MTEB retrieval score specifically, not the overall average.
5. Benchmark 2 to 3 finalists on your own 50-query sample. This is the step almost everyone skips, and it's the one that actually matters.
Embeddings for Indian languages and Hinglish
English-only models like MiniLM tend to fall apart on Hindi, Tamil, or Hinglish queries. Multilingual models (bge-m3, multilingual-e5, Gemini embedding, Cohere's multilingual line) map “sasta phone” and “cheap phone” into roughly the same region of space, which is what makes cross-lingual search for Indian users actually work.
Real example: a support-search bot for an Indian fintech needs to handle a query like “KYC kaise complete karein.” A multilingual embedder correctly retrieves the English help doc. An English-only one just shrugs and returns noise. If you're building anything user-facing in India, test it with real code-mixed queries before you ship, not clean textbook English ones.
What Text Embeddings Power: From Search to RAG
Semantic search is the obvious one, meaning-match beats keyword-match, which is exactly what made the Goa flights example in the intro work in the first place.
RAG (retrieval-augmented generation) is arguably the biggest reason engineers are learning embeddings right now in 2026. Embed the user's question, retrieve the chunks of text with the closest vectors, and hand those to the LLM as grounding before it answers. No embeddings, no retrieval, no RAG.
Clustering and deduplication group similar support tickets together or catch duplicate listings without anyone manually tagging them.
Classification tasks, like spam or intent detection, get a lot easier once you're working on top of embeddings instead of raw text, often needing very little training data.
Recommendations (“users who read this also read...”) are, underneath the marketing language, just nearest-neighbour search in embedding space.
If you want to see embeddings actually doing the retrieval work inside a full build, this RAG tutorial walks through a complete pipeline end to end. Worth the watch once the concept clicks on paper.
How you split documents before embedding them, and where the resulting vectors actually get stored and queried, are both big enough topics to deserve their own articles rather than a rushed paragraph here. If you want to go further into structured RAG learning after this, Scaler's roundup of generative AI courses covering LLMs, RAG, and deployment is a reasonable next stop.
Generate Your First Text Embedding (5-Minute Start)
No API key, no cost, runs fine on a laptop or a free Colab notebook. Good news for anyone on a student budget.
pip install sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
text = "Text embeddings turn meaning into math."
vector = model.encode(text)
print(vector.shape) # (384,)
print(vector[:5]) # first five numbers, for a peek
A few mistakes that trip up almost everyone the first time around:
• Comparing vectors from two different models. Different models produce incompatible spaces, so re-embed everything if you switch.
• Embedding an entire document as a single vector, then wondering why retrieval feels vague and unhelpful.
• Mixing cosine similarity and dot product without checking whether the vectors are normalised first.
• Reaching for a 3072-dimension API model for a 200-document college project. It works, it's just overkill.
Once this feels comfortable, the natural next stops are vector storage and retrieval, both covered further along in the AI engineer roadmap if you want the full sequence laid out.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreFAQs
What is a text embedding?
A list of numbers (a vector) that represents the meaning of a piece of text. Texts with similar meanings get similar vectors, which lets computers compare, search, and cluster text mathematically instead of matching exact words.
What are the types of embeddings?
Word embeddings (one vector per word, like word2vec), sentence or passage embeddings (one vector per sentence, the modern default), document embeddings, and multimodal embeddings that place text and images in the same space, CLIP being the best-known example.
What does the dimension of an embedding mean?
It's the length of the vector, how many numbers describe each piece of text. More dimensions capture more nuance but cost more storage and slower search. 384 to 1536 covers most real projects comfortably.
What are OpenAI text embeddings?
OpenAI's embedding API models: text-embedding-3-small (1536 dimensions, low cost) and text-embedding-3-large (3072 dimensions, higher quality). Both support Matryoshka-style dimension shortening through a dimensions parameter. Full details in the OpenAI documentation.
Turn Learning into Career Growth
What is cosine similarity in embeddings?
A score of how closely two vectors point in the same direction, ranging from -1 to 1. Higher means more similar meaning. It's the default way to compare text embeddings.
Which embedding model should I use for RAG?
Start with a cheap, strong default like text-embedding-3-small, or an open-source option like bge or e5 if you want to run it for free. Check MTEB retrieval scores, use a multilingual model for any Indic-language content, and always test on your own queries before deciding anything's final.
Are word2vec and text embeddings the same thing?
Not quite. Word2vec is an early word-level embedding technique from 2013. Modern text embeddings come from transformer models that embed whole sentences with context, something word2vec was never built to do.
Do embeddings work for Hindi and Hinglish?
Yes, provided you use a multilingual model such as bge-m3, multilingual-e5, or Gemini embedding. English-only models generally perform poorly on Indic and code-mixed text, so check this before you ship anything user-facing.
Wrapping Up
Meaning turns into coordinates, coordinates turn into similarity math, and similarity math is what quietly powers semantic search, RAG, and recommendations under the hood. Once that chain clicks, a lot of what feels like AI magic just starts looking like well-organised arithmetic.
Storage and fast retrieval at scale are the natural next stops from here, worth exploring once you're comfortable generating and comparing embeddings on your own.
If you'd rather build these foundations in order, embeddings, RAG, and production AI systems, with mentorship from people who've actually shipped this stuff, explore Scaler's AI & Machine Learning program.
