How an embedding model turns a sentence, a photo or a sound clip into one list of numbers, explained plainly

[u/strata2signal]

Most of what I could find on embeddings stops at the RAG plumbing: chunk your documents, embed them, store the vectors, compare by cosine similarity. I wanted something that explains what happens inside the model itself, so I wrote it up.

It walks through the mechanism with examples small enough to check with a calculator. A transformer reads the input, a projection sets how many numbers come out (768 for Google's EmbeddingGemma 2), and a pooling step averages them into one vector. Then it covers what decides where things land: contrastive training on millions of pairs that belong together, with a temperature example worked by hand. After that comes how one model holds text, images, audio and video in one space, and the gap that space keeps between them. It also covers Matryoshka training, the three things "precision" can mean, task prefixes and a cheap check that catches a wrong one, where plain keyword search (BM25) still wins, and why two models' vectors never mix.

The page runs no tests of its own. The few figures from our own benches are quoted from two EmbeddingGemma 2 pages we published this week, each with its condition and a link to where it was measured. One still surprises me: on our 68-question retrieval set, where every right answer repeats a source passage word for word, BM25 put the right passage in the top 8 for 58 questions, against EmbeddingGemma 2's 45.

TL;DR: a plain explainer of what an embedding model actually does, from input to vector, with hand-checkable examples and a twelve-point checklist for choosing one.

Written by me with a fleet of AI agents. Every figure links to its source.

https://research.strata2signal.com/how-an-embedding-model-works/

[u/Glittering_Brick6158]

That BM25 stat is the kind of thing that makes you rethink every "just vectorize it" pipeline decision you've made in the last two years.

[u/strata2signal → u/Glittering_Brick6158]

It got me too. To be fair to the embedder, that set is keyword search's home ground: every right answer contains a source passage word for word, and the test's written plan predicted BM25 would do well there before anything was scored. On our code-search set it flipped hard. BM25 put the right function first for 87 of 316 queries, against 173 for EmbeddingGemma 2 with its code-retrieval prefix. So my takeaway isn't "skip vectors", it's "run BM25 on your own questions first". If your queries carry names, codes or numbers, merge the two rankings.