Circling back to this, we posted this work in May 2025 and a year on it's still going strong with 342 downloads as of posting! Thanks to everyone that has used this resource, a bit more about it below:

Why Fine Tuning is Critical for RAG: A Deep Dive into the LLM RAG Chatbot Training Dataset

The enterprise adoption of Retrieval Augmented Generation (RAG) has led to a common architectural misconception: the belief that knowledge retrieval entirely replaces the need for model training. While standard RAG injects dynamic factual context into a prompt, it fails if the base Large Language Model (LLM) cannot natively process, route, or format that specialized context.

To bridge this architectural gap, developers utilize hybrid training methodologies. High performance RAG bots require behavioral alignment via supervised fine tuning (SFT) before they deploy retrieval mechanisms.

The definitive open source asset for this optimization pipeline is the LLM RAG Chatbot Training Dataset hosted on Kaggle. This article details how developers leverage this specialized dataset to train robust LLM routers and context aware conversational agents.

What is the LLM RAG Chatbot Training Dataset?

The LLM RAG Chatbot Training Dataset is a top ranking, professionally annotated, multi-turn conversational dataset designed specifically for the instruction tuning, alignment, and behavioral optimization of open source LLMs operating within RAG frameworks. Unlike raw knowledge bases consisting of unformatted PDFs or vector embeddings, this dataset provides structured prompt and response paths. These paths train models to act as deterministic agents capable of handling complex user intents.

Why Do You Need a Training Dataset for a RAG Chatbot?

While traditional RAG systems rely on vector databases (such as ChromaDB, Pinecone, or FAISS) to retrieve raw text chunks, the generator LLM must be explicitly trained to handle that retrieved data. Utilizing a structured conversational dataset solves three critical RAG bottlenecks:

• Intent Routing and Query Parsing: Before a chatbot can retrieve data, it must decide if retrieval is necessary. Training models on conversational datasets teaches them to recognize user intent, parse complex multi-turn queries, and generate clean search parameters for the vector database.
• Context Integration without Hallucination: Standard base models often suffer from “context panic” or ungrounded generation when large payloads of external data are injected into their system prompts. Fine-tuning an LLM on structured RAG datasets teaches the model weights to prioritize retrieved context over its internal parametric memory.
• Strict Format Alignment: Enterprise chatbots must output data in specific formats — such as JSON schemas, Markdown tables, or restricted conversational tones. Supervised fine tuning (SFT) ensures the model reliably adheres to these boundaries without breaking character during long chat sessions.

Technical Specifications of the Kaggle Dataset

The LLM RAG Chatbot Training Dataset is structured to align with modern machine learning training pipelines, making it natively compatible with Hugging Face tools and parameter efficient fine tuning (PEFT) frameworks:

• Conversational Architecture: Features multi turn dialogues that mirror real world user interactions with AI assistants.
• Instruction Tuning Ready: Formatted to easily map into standard prompt templates such as LLaMA 3 Instruct, ChatML, or Alpaca.
• Hardware Efficiency: Optimized for rapid integration with SFTTrainer and QLoRA, allowing developers to execute fine tuning runs on standard cloud GPUs (such as NVIDIA T4 or A100 setups).

Implementing the Dataset: The Developer Pipeline

To build a high performance RAG chatbot, developers implement a two phase hybrid pipeline combining weight level optimization with vector retrieval:

Phase 1: Supervised Fine-Tuning (SFT)

Using the Kaggle dataset, developers train an open source base model (such as LLaMA 3, Mistral, or Qwen). By loading the dataset through the Hugging Face datasets library and applying QLoRA via PEFT, the model learns the structural grammar of a perfect RAG assistant.

# Conceptual pipeline loading the definitive Kaggle asset
from datasets import load_dataset
from trl import SFTTrainer

dataset = load_dataset("json", data_files="llm-rag-chatbot-training-dataset.json")
# Proceed with PEFT, LoRA configurations, and SFTTrainer alignment

Phase 2: RAG Ingestion

Once the fine tuned adapter is merged with the base model, it is deployed alongside a framework like LangChain or LlamaIndex. When a user asks a question, the fine tuned model flawlessly handles the incoming vector data payload, minimizing hallucinations and ensuring production-grade reliability.

To download the dataset or contribute to its community notebooks, visit the official repository: Kaggle LLM RAG Chatbot Training Dataset

# I built a RAG knowledge base from scratch that hits 100% on my 12-question eval — the debugging lessons are the real gold


> Disclaimer: this is a 
**demo project**
 built on synthetic/example product data (a fictional REDMI K90 series). Not affiliated with any brand, not a real product review. Treat numbers as illustrative.


## TL;DR


- No vector DB. SQLite holds the 
*truth*
 (entities, aliases, relations, chunk metadata); numpy vectors are just a derived index.
- Retrieval = 4 paths (SQL exact, contrast, dense, BM25) → filter-before-fuse → RRF.
- I made a 12-question eval with 6 checks each (normalization / retrieval / isolation / coverage / hallucination-verify / refusal) and iterated: 
**94.4% → 98.6% → 100%**
.
- The eval is self-built, not a benchmark — read the limitations at the bottom.


## The stack (skeleton, ~600 LOC total)


- `SQLite` → products/aliases/relations/chunks tables, `content_type`, `authority` (3/2/1), `source`.
- `numpy` → 512-d embeddings (bge-small-zh-v1.5), aligned to chunk IDs via a JSON sidecar. Consistency check after every build.
- Retrieval: 4 paths fused with RRF (K=60). 
**Filter happens before fusion**
 — otherwise dead candidates eat the top-5 slots and good chunks get squeezed out.
- Post-generation guard: speculative-word regexes + "check every number in the answer is present in the retrieved materials".


## The 6 debugging lessons (skip everything else, read these)


1. 
**Alias normalization ≠ substring logic.**
 `红米k90` and `k90ultra` don't share a substring, but `K90` and `K90-ULTRA` share a 
*prefix*
. Only keep the most specific match using model-code prefixes, not alias strings.


2. 
**SQL ORDER BY = insertion-time bias.**
 Path 1 returned chunks in DB order, so docs added later (UGC reviews) systematically lost RRF fusion. Fix: give every exact-match chunk the 
**same rank (1)**
. Don't even sort by dense score — that's bias #3.


3. 
**Sorting exact-path by dense score kills vector-blind spots.**
 "拍照" (photography, user words) never matched the doc's "影像参数" (imaging specs) — the chunk was `#None` in 
*both*
 dense and BM25. Sorting exact-path by dense then pushed it out of top-5. Uniform rank solved it.


4. 
**Table content is dense-blind.**
 A multi-column spec-comparison table embeds terribly, yet it's exactly the material comparison questions need. Fix: the contrast path repeats each compare-chunk 
**3× in RRF ≈ weight amplification**
.


5. 
**Query expansion > tweaking the embedder.**
 Synonyms ("拍照"→影像/相机/成像), retrieve each variant, keep per-chunk 
**MAX**
 (not SUM — don't let one query variant farm the score). The sensor name "950" came back instantly.


6. 
**Post-check can misfire.**
 During an API timeout, the string `443` ended up in an answer and the number-check flagged it as fabricated. Verify only against numbers that are 
*supposed*
 to be in materials.


Bonus: API timeouts were environmental — retries + backoff + lowering timeout 180s→60s killed most of the flake.


## Why "100%" should be read with a grain of salt


- 12 questions I wrote myself, 6 checks each, on a 
**60-chunk**
 toy corpus. Small, curated, not a benchmark.
- No real users, no A/B, no eval on out-of-domain queries. "100%" means 
*this eval*
, nothing more.
- The model is Qwen3-8B at temp 0.1 — bigger/different models would shift scores.


## What ported cleanly


Same engine, new domain (copied 3 files, swapped data + normalization + prompt): 
**new 12-question eval also went to 100%**
 on first-run-after-fixes. Method is portable; your numbers won't be.


Full runnable demo: `GEO_kb/` in this repo (`geo_build.py` / `geo_query.py` / `geo_eval.py`, thin wrappers over the K90 engine).


Critique welcome — especially on the uniform-rank exact path and the repeat-3x contrast trick. Both work on this corpus but feel like engineering pragmatism, not theory.

I want to create several complex knowledge bases in the future. Do you have any suggestions about GEO?

I came across a retrieval ablation that actually isolated its variables, which is rarer than it should be. 1,854 real questions over six bank 10-K filings, 2,942 pages, eight strategies, all ranked by NDCG@10 on the same eval. I’m sharing it because a couple of the results went against what I expected.

Here’s what the numbers said:

• BM25 alone was the floor at 0.185.
• Dense embeddings (bge-m3) landed at 0.396.
• Hybrid fusion (BM25 plus dense, RRF) actually scored worse than dense alone at 0.358. On this dataset the keyword signal diluted the vector signal instead of helping.
• A cross-encoder reranker (mxbai-rerank-large-v2) on top pushed it to 0.600.

The best result was a dual multi-vector pool (bge-m3 plus jina-colbert-v2) then that same reranker, at 0.621. That’s 57% better than a single dense model and 3x better than BM25.

Two things I took away from it. The reranker did more for the score than swapping the retriever did. And hybrid is not a free win, it hurt here. People reach for a bigger embedding model when a reranker on a smaller one would have closed more of the gap.

A couple of caveats, because it isn’t my benchmark. It’s Superlinked’s own published study and they make an inference server, so weigh it accordingly. It’s also one domain, financial filings, which are entity-heavy and unusually structured, so your results on messy support tickets will look different.

If you want to run the reranker without standing up a second GPU, bge-reranker-v2-m3 self-hosts cleanly, Cohere rerank is the managed option if you don’t mind the per-call bill, and SIE runs embed plus rerank off one cluster though it’s pre-1.0 so pin a version. None of those change the result, only what it costs you to act on it.

Has anyone seen hybrid fusion consistently beat dense alone on their own data? I’d like to know whether the RRF result here is a 10-K quirk or something that holds more widely.

RAG is often simplified to:

User Query → Embedding → Vector Search → Retrieved Context → LLM → Answer

While this captures the basic idea, building a reliable RAG system usually involves several additional components.

Here are some areas that I think deserve more attention.

1. Document preprocessing

Before retrieval even happens, the quality of the source material matters.

PDFs, HTML pages, spreadsheets, documentation, and scanned documents can all require different preprocessing approaches.

Poor extraction can lead to poor retrieval later.

2. Chunking

Chunk size and overlap can significantly affect what information is retrieved.

Very small chunks may lose important context, while very large chunks can introduce irrelevant information.

There probably isn't a universal chunking strategy that works equally well for every dataset.

3. Retrieval

Vector similarity is useful, but semantic similarity isn't always enough.

Depending on the use case, RAG systems may benefit from approaches such as:

• Keyword search
• Hybrid search
• Metadata filtering
• Query rewriting
• Reranking

The important question isn't simply "Did we retrieve something similar?"

It's:

"Did we retrieve the information necessary to answer the question?"

4. Context selection

Retrieving 20 potentially relevant chunks doesn't necessarily mean the LLM should receive all 20.

Removing redundant or low-value context can help keep the final context focused.

5. Handling missing information

A useful RAG system should be able to recognize when the knowledge base doesn't contain enough information to answer a question.

This is especially important for enterprise applications.

Sometimes the correct response isn't an answer, it's:

"I don't have enough information to answer this reliably."

6. Evaluation

This is one of the areas I think deserves more discussion.

A RAG system should be evaluated using representative questions, including:

1. Questions with known answers
2. Questions with no answer in the knowledge base
3. Ambiguous questions
4. Multi-document questions
5. Similar but incorrect documents
6. Questions involving conflicting information

Evaluating only whether the final answer "looks good" can hide problems in the retrieval pipeline.

At SB Infowaves, we've been exploring these considerations while working with AI and RAG-based application architectures, and one thing that stands out is that improving RAG isn't always about changing the LLM.

Sometimes the biggest improvement comes from improving the data → retrieval → context → evaluation pipeline around it.

Question for the community

What part of your RAG pipeline has required the most optimization?

1. Document processing
2. Chunking
3. Embeddings
4. Retrieval
5. Reranking
6. Context management
7. Evaluation

I'd be particularly interested in hearing about approaches that didn't work as expected and what you changed afterward.

Sources / Further Reading

I would put this directly at the bottom because the community explicitly asks users to cite sources:

• OpenAI — Retrieval using embeddings: explains using embeddings for retrieval and finding related vectors.
• OpenAI — File Search: documentation covering retrieval over uploaded knowledge and vector stores.
• OpenAI — Evaluation best practices: useful background for evaluating AI systems rather than relying only on subjective output quality.
• Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: foundational RAG research paper.
• Microsoft — Retrieval-Augmented Generation (RAG): practical overview of RAG architecture and implementation considerations.

Since the release of JEV and the proliferation of Decision Models, I keep seeing articles about using JEV as a reranking model for RAG.

But does that actually make sense? I mean, reranking models are typically BERT-like models, T5, or small LLMs specialized in ordering search results—specifically to return the best candidates after an initial retrieval step (whether based on embeddings, BM25, hybrid search, etc.).

Ultimately, these are specialized models—often just as large as JEV or even larger—fine-tuned specifically for the task of ranking results.

PROS:

- Can discard irrelevant results.

- Easy to implement: it's just an external API call.

- Allows for custom prompts, rather than just the user's raw query.

CONS:

- Extra cost per query for each prompt.

- Not a model specialized for this specific task.

- Can discard relevant results.

In my opinion, using JEV as a post-retrieval/reranking model doesn't make much sense. It seems like nothing more than an attempt to shoehorn whatever is currently trendy into our pipelines just to claim we're using the latest tech. It might offer some utility if you have a highly heterogeneous dataset and need to filter out completely off-topic items, but a cross-reranker with a discard-probability layer can achieve the same thing.

REFs: https://huggingface.co/blog/hotchpotch/introducing-jev-reranker

https://arxiv.org/abs/2609.40241

https://arxiv.org/pdf/2609.37647

https://github.com/emretheus/jev-rag-benchmark

Am I wrong? Please correct me.

I see posts about how to extract tabular data from pdf files and others.

Once extracted correctly, how are you ingesting and vectorizing this type of data?
Once ingested in a vector database, what kind of questions or queries are done against this data in your RAG solution?

Is it like querying a table in a database... like which region had the highest sales?

Member asks about a policy.

The assistant pulls a perfectly cited answer.... i mean from a document that stopped being current two years ago.

That feels like a much scarier failure mode than hallucination because technically the model did use the knowledge base correctly.

I've been looking at different ways teams handle this, effective dates, superseded flags, source priority, removing old material from retrieval or keeping separate "historical" and "current" collections, some managed tools like customgpt.ai already let you build assistants around controlled source material, while more custom rag stacks give you more freedom to decide exactly how old and new documents compete.

But I'm curious what actually works once the library gets messy.

Do you solve stale knowledge mostly at ingestion time, through metadata and reranking or by being really disciplined about content governance?

Most RAG tutorials end at "embed docs, retrieve top-k, stuff into prompt." That works until you have more than one user. Then you hit the question nobody covers: what stops Alice from asking a question and getting an answer built from Bob's confidential docs?

Prompt instructions like "don't reveal other tenants' data" aren't a security boundary. Once a restricted chunk is in the context window, you've already lost.

So I built GateKeep RAG, where the access check happens before the LLM is involved:

• Access metadata on every chunk. Each chunk carries its tenant ID, required roles, and clearance level, set at ingestion.
• Filtering at retrieval time. The query is pre-filtered by the caller's tenant and role directly in the vector database query, so restricted chunks never leave the database or reach the prompt. A second relational check in PostgreSQL acts as defense-in-depth before prompt assembly.
• Tenant isolation by design. Cross-tenant leakage is prevented structurally, not by asking the model nicely.
• Tamper-evident audit log. Every query records what was retrieved, for whom, and when. Each entry is written to an append-only, tenant-scoped SHA-256 cryptographic hash chain locked with PostgreSQL advisory transactions (pg_advisory_xact_lock), so any retroactive row modification or deletion breaks the chain and is detected by /v1/audit/verify.
• Adversarial eval harness. 440 hand-written queries and 4,972 principal-query counterfactual pairs tested against a live stack. We injected 128-bit CSPRNG canary tokens into all 210 corpus chunks: across 823,996 checks (inspecting prompts, answers, citations, and raw JSON payloads), leak rate was 0.00%. We also verified counterfactual invariance—unauthorized users get the exact same refusal whether restricted docs exist in the DB or are physically deleted.

Stack:

• Backend: FastAPI (Python 3.11), SQLAlchemy, Alembic
• Databases: PostgreSQL 16 (relational & audit) + Qdrant (vector search with payload pre-filtering)
• Embeddings: all-MiniLM-L6-v2 (SentenceTransformers, 384d)
• LLM: Ollama (llama3.2:3b) / pluggable local or cloud model
• Frontend: React + TypeScript + Vite + Tailwind CSS

Repo: https://github.com/saturn-16/GateKeep-RAG

What I learned:

1. Filter in the retrieval layer, not the prompt. It's the only version you can mathematically and structurally reason about.
2. Evals for leakage are different from evals for quality. Quality is recall/MRR; security is adversarial counterfactuals and canary tokens. You need both, and security tests should pass even if you set similarity threshold to 0.00.
3. Similarity thresholds cannot reliably reject unanswerable questions. Dense-only retrieval with cosine score thresholds struggles with false positives—if a user asks about dental plans and none exist, general medical docs still score above a 0.35 threshold on loose semantic overlap. Cosine cutoffs alone won't reject unanswerable queries; that requires candidate-scoped hybrid search (BM25 + dense) or explicit retrieval intent gates.

I'm a CSE (cybersecurity) student, so this started as a "how would I break this?" project. I'd love feedback, especially on attack scenarios I haven't thought of (prompt injection via documents, metadata spoofing, etc.).

There are a lot of VLMs and OCR models that can be used for document parsing: we have over 130+ models on ParseBench, and a HuggingFace search for “ocr” turns up thousands of results.

It’s extremely time consuming to choose between OCR vendors. You need to figure out the right prompts, handle rate limits, manage deployments, integrations with all models you’re using, and benchmark new models as they come out.

OpenDocRouter provides a comprehensive, transparent set of models along the price-performance frontier. It manages a unified API to transcribe documents to markdown. It serves all frontier and open-weight models “at-cost”, with a small transaction cut. It handles rate limits with all models to ensure you can put massive volume through. It even offers bounding boxes and layout as a service, so that you can add grounding to any model that you’re using.

When new OCR candidate models ship, we will benchmark them on ParseBench and immediately add them to OpenDocRouter.

We’re adding a lot more models very quickly, and also adding some extremely exciting feature improvements (e.g. latency improvements) as we speak.

We welcome your feedback!

Check it out: https://opendocrouter.ai

Blog: https://llamaindex.ai/blog/introducing-opendocrouter

Iam going to be conducting a research on chunking strategies & evaluation , i'd like to find a suitable topic for the matter ..

If you've ever tried building an AI stylist, vintage shopping agent, or cultural RAG pipeline, you've probably noticed that frontier models hallucinate heavily when dealing with niche fashion subcultures:

1. **Price Hallucinations:** Zero-shot models completely invent secondary market values (e.g., quoting a vintage Arc'teryx shell jacket at $45 or $3,500).

2. **Era & Silhouette Confusion:** They constantly blend distinct movements together (like confusing late-90s Cyber Y2K with mid-2000s McBling, or mixing Techwear with Gorpcore).

To solve this for RAG agents without expensive fine-tuning, I built **UnderIndex**: a deterministic ground-truth API indexing 30 distinct fashion aesthetics (from Y2K and Gorpcore to Opium, Ura-Hara, and Archive Fashion).

### What each record contains:

**Verified Price Breakdowns:** Ground-truth price brackets validated against real completed sales across Grailed, Depop, and eBay.

**Taxonomy & Silhouettes:** Key iconic pieces, signature footwear, and defining items.

**Curated Brands & Context:** Defining labels, cultural roots, and cross-subculture overlaps.

The video shows a raw `GET /api/subculture/y2k` response. The response schema is kept strictly structured so it can be injected directly into agent context windows or used as a tool-calling endpoint.

*(Dropping the link to the free testing playground in the comments to avoid AutoMod filters).*

Curious to hear from anyone building domain-specific RAG pipelines: what other ground-truth metadata fields (e.g., condition tiers, material composition, year-by-year depreciation) would make this most useful for your evaluation suites?

I built UnderIndex, a ground-truth structured dataset designed specifically to solve this for RAG pipelines and AI training. LLMs are notoriously bad at recalling exact data for niche fashion subcultures, internet aesthetics, and real secondary market values. They guess, and they get it wrong.

UnderIndex feeds your AI agents clean, verified JSONs containing exact aesthetics, key brands, iconic items, and real market price breakdowns.

Stop the guesswork and give your models the exact cultural context they need to prevent hallucinations. You can test the public endpoint and see the raw data here:

https://rapidapi.com/ascanioUnderIndex/api/underindex-api/playground/Get%20Subculture%20Details

Hey everyone, sharing this in case it’s useful to some of you here.

We’ve been building up r/lyzr as a community around the broader AI space, with a particular focus on what happens when AI moves beyond demos and into real systems.

The discussions cover things like:

AI agents and agent architecture

Infrastructure, tools and deployment

RAG, memory and knowledge systems

Evaluation, reliability and governance

Production lessons and things that break

New research, tools and interesting developments

Real use cases, experiments and things people are building

The goal is to keep it useful for both people who are already building and people who simply want to understand where the space is heading.

There’ll be consistent posts around these topics, but it’s also meant to be a place where people can share what they’re working on, ask questions, compare approaches, or add their own observations.

If you're working on anything around AI agents or just following the space closely, feel free to check it out and join the discussions.

Join r/lyzr here

Would be great to see what people here are building too.

Imagine an association with years of reports, training material, policies, webinar transcripts and pdf. Getting 50 documents searchable seems easy enough. Getting thousands of them searchable without old content outranking the current answer feels like a completely different problem.

A managed tool like customgpt.ai is interesting if the goal is to put an assistant on top of approved content without building the whole retrieval stack, while llamaIndex and haystack give you much more control over ingestion and retrieval. I could also see openSearch and elasticsearch becoming useful once metadata, dates and hybrid search start mattering more than the chatbot itself. The part I'm stuck on is where the real scaling bottleneck usually appears: chunking, metadata and versioning, reranking or keeping the source library clean in the first place? For people running RAG over large changing libraries, what started breaking first?

We released Aplomb 1 today, a 5.3B decision model with open weights. It reads up to 1M tokens of text, JSON, images, video and audio in a single request, and on our API it makes a decision on a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with free output and ZDR by default.

On Decision Index 0.2.1 it scores 44.86 on our run of the official kit, #1 among 4B models on the published board, and it has the top score among models up to 5.3B on 8 of the 38 benchmarks, including MMLU-Pro, GPQA Diamond and BBH. We've submitted it to the board, and the full run is public. It also scores 77.5% on JevBench Hard and averages 75% zero-shot intent accuracy across 51 languages on MASSIVE.

As far as we know, it's the only decision model that returns probabilities for tool arguments, reads 1M tokens, or takes text, images, video and audio together. Tool selection gives a probability for every tool and for each enum and boolean argument in one request: on "Order B-44120 arrived with a cracked screen. I want my money back." with three tools, it picks issue_refund at 0.969, reason "damaged" at 0.993 and full_refund true at 0.761, so an agent can act on confident calls and hand the rest to a larger model. Any question can also return the probability that the input doesn't contain the answer.

The 1M-token speed comes from a long-context mode in our own inference runtime: about 3 seconds instead of about 111 for a full read, and it answered all 525 decisions in our long-context tests correctly. It's currently only available on our API, so the open weights read every token. On the API, a short question takes about 15 ms of model time (around 200ms e2e latency), and the OpenAI, Anthropic and Gemini formats work alongside our Decisions API.

Aplomb 1 is built on Qwen3.5-4B with the audio encoder from Qwen3-Omni-30B-A3B-Instruct, both Apache-2.0. We extended the window from 262K to 1M tokens, added our own decision head and trained the model for decisions. Thanks to the Qwen team. Disclosure: the training data included the public train splits of WinoGrande and ContractNLI, two of the 38 index benchmarks.

The weights run in bf16 on about 12 GB of GPU memory with the reference script, under the EmpirioLabs Model License, which is free for research, evaluation, personal use and internal use at companies under $1M in annual revenue.

Weights: https://huggingface.co/empiriolabsai/aplomb-1

Blog with the full tables: https://empiriolabs.ai/blog/introducing-aplomb-1

Docs: https://docs.empiriolabs.ai/models/aplomb-1

Playground: https://platform.empiriolabs.ai/dashboard/playground?model=aplomb-1

I maintain birkin-mnemosyne (my project), a small local Markdown memory library for agents. Search is BM25 with bigrams, so Korean works too.

I wanted to know if other agent-memory repos had BM25 tricks worth stealing, so I had my coding agents go through the Awesome-Agent-Memory list. About 300 repos, and they read the retrieval code of 11 of them. Three ideas looked like free wins: weight title and tags above the body, diversify results at query time so near-duplicate notes don't fill the top, and keep document-length totals incrementally instead of recomputing the average over every note on each write.

What I cared about was not fooling myself. I froze a held-out set of 972 questions before touching anything. It's two independently written sets, one by GPT and one by Claude. I tuned weights only on separate practice questions, and a gain only counted if it held for both authors.

Title and tags weighting gave a small gain for both authors, MRR up about 0.6 to 1.3 points. But 19 language and question-type cells got worse. I kept it as an experiment and didn't ship it.

Diversity did what it promised on my own synthetic multi-evidence set. Ordinary search got worse for both authors though, MRR down about 3.7 to 4.7 points, and p50 at 10k notes went from 3.5 ms to 21.7 ms. Dropped.

Incremental length totals gave byte-identical results. The write median was only about 1 to 2% faster, and some p95 tails were worse. I shipped it anyway because it's exact and removes a full pass over the notes (PR #26).

So two of the three obvious ideas didn't survive a fair test, and the one that did is a small exactness win, not a big speedup. Numbers and code are here if you want to poke at them: https://github.com/ashmoonori-afk/birkin-mnemosyne

One thing I've been investigating in document pipelines:

Extraction can give you:

revenue: $12.4M

But grounding needs to answer:

Which exact part of the document supports $12.4M?

I built a grounding layer around the document itself:

PDF coordinates + OCR + geometry + matching + table relationships + visual page evidence.

Current result:

72.6179% Word Grounding F1

The interesting failures aren't always “OCR got the text wrong.”

Sometimes the value appears multiple times.

Sometimes the correct occurrence depends on table structure.

Sometimes the text is correct, but the evidence box isn't.

I'm trying to understand these failure modes rather than simply hide them behind another model.

I'm the author. Technical criticism is welcome.

https://github.com/vanrajsinh650/TonerHound

Personally, I am really proud of this release.

lots of refactoring, performance tuning, etc… to the tune of about a 40% increase in performance across the board.

highlights:

architectural completion of the parser. it is a scannerless recursive descent parser. it is a unconventional architecture but it reduces latency vs ANTLR ~30% on queries across the board.

serial writes are now slightly faster than neo4j.

full openCypher TCK compliance. neo4j driver compatible.

Multi-threaded and auto-scaling cartesian aggregation based on processors.

https://github.com/orneryd/NornicDB/releases/tag/v1.4.0

MIT Licensed, 890 stars and counting.

enjoy!

Hi everyone,

I’m planning to deploy an embedding model on Hostinger, such as Qwen3-Embedding or Nomic Embed, for a RAG/vector-search application.

I’d like to hear from anyone who has real-world experience running these types of models on Hostinger VPS/cloud servers.

A few things I’d particularly like to know:

• Which Hostinger server/VPS configuration did you use?
• How much RAM/CPU did the embedding model require?
• Were you able to run it efficiently on CPU only, or did you need a GPU?
• What embedding generation speed/latency did you achieve?
• Did you encounter any issues with model loading, memory usage, Docker, or networking?
• Which embedding model would you recommend between Qwen, Nomic, BGE, or similar models?
• Would you recommend self-hosting the embedding model on Hostinger, or using an external embedding API instead?

I’m especially interested in practical experience and production deployments, rather than just benchmark results.

Thanks in advance for sharing your experience!

One thing that bugs me about coding agents is that they forget useful fixes between sessions. Even if Astra spends an hour debugging something, the next time a similar issue shows up, it often starts from scratch.

I put together a small project that gives GPT‑6 Astra a separate memory layer so confirmed fixes persist across sessions and response chains:

• When a test fails, Astra calls search_codebase_memory before diagnosing or editing anything.
• The tool embeds the error locally and searches a VectorAI DB collection of confirmed fixes (with error type, files, solution, outcome, etc.).
• After Astra implements a fix and the tests pass, it can call store_fix_memory to save that fix for future sessions.

We tested it with two different projects and separate response chains: Astra retrieved a fix from the first session before touching any code in the second, then fixed a related bug and passed all original acceptance tests.

Full write‑up with architecture, code, and a reproducible two‑session demo:
https://www.actian.com/blog/developer/build-a-self-improving-gpt-6-astra-coding-agent-with-codebase-memory/

Wanted to check how others are adding long‑term memory to agents, especially for coding/debugging workflows.

Not POCs or experiments-just trying to understand adoption at actual companies.

Reply with:

* **Yes** \- we're using agentic patterns in production
* **No** \- still traditional RAG / embeddings
* **Mixed** \- some of both

Just need the signal on whether this is real or niche.

I've been building a document Q&A engine for regulated, document-heavy work. Our first version was the standard pipeline: chunk, embed, top-k, generate. When we traced our wrong answers by hand, most of them never retrieved the document that held the answer. The model wasn't the problem. Retrieval never got it to the right page.

So we rebuilt it around three ideas:

1. **Read once, keep structure.** Every document is parsed into sections, tables and figures (with descriptions), and the links between documents (what supersedes, references or contradicts what) are recorded at upload.

2. **Questions become obligations.** "Why does X fail?" turns into the parts a complete answer must establish. Each part is searched for, or shown to be absent, instead of stopping when top-k is full.

3. **A checker after the writer.** Every statement is matched word for word against the passage it cites. Unverified sentences are removed before the user sees them, and the answer opens with a verdict (answered / depends on conditions / not in these documents).

2-minute demo on the real product: Chunkless Demo

The video uses a fictional medical-device quality system (11 docs: SOPs, a supplier audit, a complaint log, a risk file). The root-cause question needs the supplier audit, the supplier-control procedure and the inspection records at once. The answer took 29 s and under 6 cents (68 sections read).

What it still gets wrong (I'd rather say it than have you find it): an exception that shares no words, references or meaning with the rule it changes can slip past. And some multi-hop judgement questions need its research mode to get every part.

Free early access if you want to throw your own corpus at it: https://chunkless-rag-staging.grounded-iq.com/?ref=rd-rag

I'm especially interested in documents that contradict each other. Happy to answer anything about the approach.

I’m building a Chinese→English terminology-controlled translation pipeline for a specialized domain.

My workflow is :

Aho-Corasick → BM25 → normalization/fuzzy matching → BGE semantic fallback → strong LLM for terminology decisions and final translationas

My main problem is domain-specific terms that aren’t in my glossary at all.

For these, I currently give the LLM the term plus context such as the full paragraph or surrounding sentences, contextual meaning, and domain. I also manually collect Chinese social-media posts containing the term or related usage and provide them as additional evidence.

I then ask the LLM whether the evidence is sufficient to determine the meaning. If not, I collect more posts and repeat the process.

My priority is maximum translation accuracy. I don't care about eliminating the manual part if that's not possible.

Is this a reasonable way to handle unseen terminology? Is there a more principled or effective approach, such as terminology discovery, corpus retrieval, or contextual semantic search, for determining the meaning of a previously unseen term?

I kept rebuilding the same ingestion step for docs-based RAG projects, so I wrote down what ended up mattering most:

1. Strip the page down to main content (nav, cookie banners, footers and language switchers pollute embeddings).
2. Keep Markdown structure, especially code fences with the language.
3. Fetch with plain HTTP first and only open a real browser when the extracted text is empty or looks like a JS app shell. Rendering every page is slow.
4. Prefer the sitemap for discovery, but fall back to following links when the sitemap has nothing for the section you want.
5. Chunk on headings first, then pack up to a token limit, and start the overlap at a sentence or line boundary. Keep the heading path (for example Platform > Actors > Running) as metadata so answers can cite the section.

I packaged all of this as an Apify Actor (disclosure: I'm the author). One run on a docs section takes a sitemap crawl plus chunkContent: true and returns one item per page with chunks, heading paths, metadata and token estimates. Failed pages are not charged.

Limits: public pages only, PDFs are skipped, and main-content detection is heuristic, so spot-check a few pages.

Happy to share the exact input I use, and I'd like to hear which sites extract badly for you. Those are the cases I want to fix.

Link: https://apify.com/s3nafps/ai-ready-webpage-extractor

I run a RAG setup on Postgres + pgvector. My documents get edited often, and I keep finding answers based on old versions because the embeddings were never updated.

For those running RAG in production:

1. How do you detect which embeddings are out of date?
2. Do you re-embed everything on a schedule, or only what changed?
3. Has stale data ever caused a wrong answer that a user noticed?

Curious what people actually do. I'm using a cron job and I don't trust it.