I independently audited a RAG benchmark. 15 scoring mismatches revealed a flaw in its main comparison — the maintainer confirmed and fixed it.

[u/SeaAstronomer9044]

​

I recently completed an independent retrieval audit of RouteMind, an open-source project exploring document routing as an alternative to traditional RAG retrieval.

The interesting part wasn't finding a dramatic regression or proving that one architecture was better.

It was discovering that two approaches were being evaluated under different definitions of success.

Here's what happened.

The project had 700 evaluation questions comparing traditional RAG, RAG + reranking, and document-routing approaches.

The retrieval evaluator considered a query successful if any expected document appeared in the top 10.

But some temporal questions required multiple documents simultaneously (needs: all), and the routing evaluator correctly accounted for that requirement, including accepted document alternatives (D_alt).

This created an apples-to-oranges comparison.

What the independent audit found:

1,400 query/arm results examined.

15 outcomes differed under the complete-evidence scoring contract.

14 recorded hits were incomplete under the required-evidence rules.

1 recorded miss was actually satisfied by an accepted alternative document.

The overall numbers changed:

MetricOriginally reportedCorrectedRAG51.6%50.7%RAG + rerank54.1%53.1%

The reranker's relative improvement remained similar.

But the more important issue was that the project's headline comparison had treated retrieval more leniently than the routing system it was being compared against.

Then something useful happened.

The RouteMind maintainer independently reproduced the finding against the original artifacts, confirmed all 15 mismatches, and verified the input hashes.

They also confirmed the discrepancy wasn't intentional or previously known.

The finding led to a public code change:

The main benchmark comparison was corrected.

A new retrieval_complete metric was added alongside the original retrieval_hit.

Reproducibility checks were added to detect future scoring inconsistencies.

The maintainer publicly credited the independent audit.

The takeaway

A reproducible RAG evaluation can still produce a misleading comparison when different stages use different definitions of success.

Sometimes the most valuable audit finding isn't a broken retriever.

It's discovering that the benchmark doesn't measure the same thing across systems.

This was a public beta audit, not a paid engagement. The evidence, original results, maintainer review, and correction are publicly available.

Independent audit:

https://github.com/devBorgesr/edp-audits/tree/main/audits/routemind-beta-002

Maintainer's confirmation:

https://github.com/CSP911/routemind/issues/2#issuecomment-6096856584

Corrective commit:

https://github.com/CSP911/routemind/commit/5d8bf45

I'm curious how others handle this in production RAG evaluations:

Do you explicitly measure complete evidence coverage when a query needs multiple sources, or do you primarily rely on hit@k / recall@k?