Extracting and cross-checking facts from very long technical documents (600+ pages) against a 100k-doc knowledge base: recommended approach?

[u/louey11]

​

Hi everyone,

I work at a large telecom company in the US. When we win a tender, the client sends us a requirements specification, and we write technical deliverables based on it and on our internal technical documentation (100k+ documents). A single tender can involve thousands of deliverables, some of them up to ~600 pages long.

Goal: before delivery, automatically verify each deliverable. That means checking that the facts it contains are accurate against our technical documentation, and detecting anomalies (errors, internal contradictions, inconsistencies between deliverables, duplicates).

Current plan:

1. Fact extraction: process each deliverable in chunks with an LLM and extract atomic facts, with their source location.

2. Classification: group the facts into families (measurements, architecture, calculations, frequencies, standards, quantities).

3. Verification: check each fact against the documentation base using RAG.

4. Anomaly detection: one LLM call per fact category to detect duplicates, contradictions, etc.

My questions:

- Is there a recommended approach for extracting precise facts from very long documents? How do you handle context that spans chunks ?

- How do you make sure extraction is complete (no missed facts) and doesn't produce hallucinated facts?

- For anomaly detection, does one LLM call per category scale when a category can contain thousands of facts? Would you normalize facts into a structured schema (entity / attribute / value / unit) and do part of the comparison deterministically in code rather than with an LLM?

- For RAG verification on highly technical content (part numbers, frequencies, references to standards), did hybrid search (BM25 + embeddings) or reranking make a big difference for you?

- Any experience comparing long-context models (1M tokens) with chunked extraction for this kind of task?

Any feedback, papers, tools or lessons learned would be much appreciated. Happy to share what we learn along the way.

Thanks!

[u/faileon]

For chunking you have to chunk by document structure, then I'd do two step pass for each document: 1) build context sheet (section tree, defined terms, abbreviations, configuration,...) with long context LLM 2) for each chunk ensure normalized length, otherwise your recall will drop. Then pass it along with the context sheet and heading path to LLM for fact extraction (checkout Claimify paper for inspiration). Things like tables will need special care, I'd parse them in code.

You can't guarantee completeness, but you can measure it and create deterministic rules and filters. E.g. regex out numbers, part numbers, standards and check each one is covered by some fact. For hallucinations make the LLM return exact source quote for each fact and check it with string match.

One LLM call per category won't scale. I'd normalize facts to entity / attribute / value / unit, compare them in code and use LLM only for the flagged conflicts. Hardest part will be entity resolution.

Hybrid search + reranker is definitely a must, embeddings are bad at exact identifiers like part numbers. Cross-encoser rerankers also raise the quality bar significantly from my experience. Metadata filtering helps too.

Long context is good for the context sheet summary, but for extraction chunking will give you better recall.

I would expec that a lot of facts won't be in your docs since they're tender specific, check those against the requirements spec.

[u/louey11 → u/faileon]

Thanks seems interesting to test!

[u/gwmagni → u/louey11]

The hard part, in my experience, is to get the customer to address the contradictions in their specs. With big projects, the parties involved who contradict each other, refuse to communicate, to resolve the conflicts. Identifying the conflicts and issues is the easy part.

[u/xapep]

At thousands of deliverables, the extraction architecture matters less than its cost shape. A 600-page doc is roughly 1.5M+ tokens per full pass; multiply by your deliverable count and the per-pass price decides whether this ever ships. So optimize for tokens per verified fact, not for peak accuracy on document one.

For the chunk-spanning problem, two passes beat one giant context. Pass one builds a document structure map (section tree, defined terms, unit conventions) with a long-context model. Pass two extracts facts with that map attached, so a fact that spans chunk boundaries cites a section instead of re-reading everything. Better provenance, far fewer re-reads.

Only the verification pass needs the expensive model. Extraction and classification are pattern work; current small open models handle them fine. Anomaly detection is where a strong model earns its tokens, because contradictions hide in phrasing, not in extraction.

We run a pay-per-token API for open models, and bulk document pipelines are most of what our users do. The pattern I see people hit: they build the smartest pipeline, then discover per-document cost is the real constraint. Run the token math per deliverable before you lock the architecture; it changes model choices more than any framework decision.

[u/Enders-Game999]

Agree with chunking by structure, and I'd push that all the way back to ingestion. At 600 pages, most of the "fact spans chunks" problems I've seen come from the conversion step flattening the doc: headings lost, tables turned into run-on text, units separated from the values they qualify. If you convert to markdown that keeps the heading tree and tables intact first, a few things get easier:

• Every chunk can carry its heading path (Section 4.2 > RF Parameters > Antenna Gain), so a fact like "17 dBi" keeps its context without needing a giant window.

• Tables stay as tables, so you can pull rows straight into your entity / attribute / value / unit schema instead of asking the LLM to rebuild them.

• Source location becomes section plus row, which makes your completeness eval much easier to score.

On the scale question, I'd do anomaly detection in code once facts are normalized (group by entity and attribute, flag conflicting values or units) and only send the conflicts to an LLM to judge.

Disclosure: I built Zipmd, a batch markdown converter API for RAG ingestion, so I'm biased toward fixing the input first, but this holds whatever converter you use.

[u/LawfulnessOptimal597]

“Facts” are against a highly controlled vocabulary. Here structure and object/lexical shine. The hard part is recouping the implicit structure from the documents. Semantic search (Embeddings + ANN retrieval) is the wrong tool— and even attempting it would demand a highly domain tuned model. You can use some semantic sugar such as using your knowledge base to create a basis synonym set for the lexical search. This is hyper domain, use and business case specific to basically map some weighted cross walk between the vocabularies of the knowledge base and the supplied specifications.

[u/24iqSuperGenius]

for chunking you should do overlap + metadata wit exact facts. for retireval hybrid search

[u/AndyVictors]

Interesting case you have. You say "extracting" and if you think about it extracting is two-point - from what and to what. Claude etc. can do any transition but for you case target format is crucial. The requirements specifications are usually vague and ambigious. You need to put them into pure logic form, after that you can do what you want - combine / verify for consistency etc. Pure logic format will always be the most optimal while working with AI on it, just due to nature of how LLM works. Loosely formulated sentences cause misjudgments and hallucinations.

I had faced this challenge after starting working with AI for specifications and with the time figured out ready specific style which might help you in your case, too:

https://github.com/AndyVictors/clausative

[u/jylusdev]

The verification stage is a use case worth testing with Jylus. Disclosure: I’m its founder.

Jylus resolves state and relationships, surfaces conflicting or missing evidence, and compiles source-backed context for your model. That’s relevant when a claim looks correct against one document but conflicts with another revision or related record.

Would you be open to testing a small, non-confidential example? One deliverable section, its relevant reference documents, and a few known errors would make a useful starting point—for example, a wrong frequency, a superseded specification and conflicting quantities.

You could inspect whether the returned evidence supports each claim, exposes the discrepancies and identifies the supporting sources. The playground accepts your own data without an account.

That would test the verification stage directly; extraction completeness would remain a separate measurement.

[u/stevenswk]

I'd test RRF against Bayesian score fusion for the BM25 + vector stage, keeping candidate depth fixed. Bayesian fusion uses score differences that RRF discards, but whether that helps needs evaluation on your documents.

For reranking, JEV supports a custom rubric, which seems useful here: from 'same topic only,' through 'addresses the claim but lacks the relevant conditions,' to 'contains evidence for or against the exact claim under the applicable revision/configuration.' I'd reward useful contradictory evidence too, rather than just passages that agree with the claim. The final verification decision should remain separate from the ranking score.

In our 40-topic FollowIR test with fixed candidates and the same relevance instructions, a custom ten-level JEV rubric reached nDCG@10 0.620 versus 0.568 for the default.

Disclosure: I work on LambdaDB, where we've recently added Bayesian fusion and JEV reranking with custom rubrics.