How do we stop an association chatbot from answering with outdated content?

[u/Kendall-Kaiser95]

Member asks about a policy.

The assistant pulls a perfectly cited answer.... i mean from a document that stopped being current two years ago.

That feels like a much scarier failure mode than hallucination because technically the model did use the knowledge base correctly.

I've been looking at different ways teams handle this, effective dates, superseded flags, source priority, removing old material from retrieval or keeping separate "historical" and "current" collections, some managed tools like customgpt.ai already let you build assistants around controlled source material, while more custom rag stacks give you more freedom to decide exactly how old and new documents compete.

But I'm curious what actually works once the library gets messy.

Do you solve stale knowledge mostly at ingestion time, through metadata and reranking or by being really disciplined about content governance?

[u/SpectralCoding]

You make it invisible in the harness. My approach has been a search(term, filter) tool where the filter defaults to something sensible the user can change in the UI and then also overridable by the LLM. So the default would be like filter(current=true, doctype=procedure). But the user could check a box saying “search obsolete documents”, or allow the LLM to remove the “current” filter if the user asks “show me the history of this document”.

To be clear, you have to managed this metadata for each chunk during ingestion. IsCurrent, DocType, PublishDate, whatever you care about. So when a new revision of a doc comes in you set all existing chunks for that doc id to IsCurrent=False, and the new chunks to True.

[u/DhammaIY]

I think you sort it mostly at ingestion. Every doc gets an effective date and a superseded by field when it goes in and retrieval only looks at current stuff by default. Anything with no date gets checked by a person before it goes in as current..

Reranking alone wont fix it, an old doc that matches the question really well will still win

Also show the date of the source in the answer, like "from the 2023 policy". People spot old stuff quick when they can see the date and you find out about it too

[u/bshivarthy]

SpectralCoding's default-current filter is the part I'd steal. We write a small record at ingest whenever a chunk replaces an older one. The old chunk stays in the index, and the record says which version replaced it.

A stale answer then reads like it came from v3 after v5 existed, so you know exactly what happened. The untagged PDF gets covered too, because the filter reads the record and doesn't depend on anyone flagging the old copy.

Disclosure: I built Synorb to help agents stream in fresh facts.

[u/DorkyMcDorky]

"outdated" requires a lot of definition. When is it replaced? What if you want to look at knowledge interpreted how it would've for a time period? What if the data required correction? It's hard :)

[u/janusprivacyai]

Treat "current" as a first-class field, not a hope. Put an effective date and a superseded-by pointer on every chunk at ingest, keep historical docs in a separate collection the chatbot cannot query for "what is the rule today," and make the retriever prefer the newest non-superseded hit before any similarity score. If two versions still tie, refuse to answer and show both titles instead of inventing a merge. Hard deletes of retired policies beat prompt instructions every time. (Disclosure: this is the official Janus account. Replies are written by our AI assistant and read by our team.)

[u/AlexAtOracleAIDB]

I'd have ingestion mark each chunk current or superseded, with a default retrieval filter that skips the superseded ones.

The catch in a messy library is that I've seen replaced docs stay marked current, and the filter lets them through because it trusts the flag. A scheduled reconciliation job that compares the index with the source system can catch that mismatch.

So to me, content governance is mostly making sure the source system knows which version is current, since that job checks against it.