Best fit
A RAG answer can sound plausible while the expected source was never retrieved, a cited or declared support ID never appeared in context, stale evidence displaced its replacement, or a temporal question followed an inadequate route.
// AI Agents & LLM Ops
Retrieval Trace Auditor
Audit real RAG evidence traces for missing expected sources, impossible citations, unsupported claims, stale evidence, unused context, and weak routing.
What problem does Retrieval Trace Auditor solve?
A RAG answer can sound plausible while the expected source was never retrieved, a cited or declared support ID never appeared in context, stale evidence displaced its replacement, or a temporal question followed an inadequate route.
Use it to
- Find citations and claim support that never appeared in retrieved context
- Separate retrieval-recall failures from later generation questions
- Review temporal or relational queries that used a vector-only route
- Create a portable pre-release gate from production-like RAG traces
What it returns
- Severity-ranked findings CSV for retrieval and trace-integrity failures
- Case summary CSV with PASS, REVIEW, or FAIL disposition
- JSON release gate and severity totals
- Markdown handoff preserving the boundary between ID checks and semantic evaluation
A representative input and result
input {"case_id":"policy-17","query_class":"temporal","route":"vector_only","retrieved_evidence":[{"id":"policy-2024","superseded_by":"policy-2026"}],"expected_evidence_ids":["policy-2026"],"cited_evidence_ids":["citation-ghost"],"claims":[{"claim_id":"claim-1","text":"The 2026 policy replaced the rule.","supporting_evidence_ids":["support-ghost"]}]}
result Release gate: FAIL
Critical: citation-ghost was cited but never retrieved.
Critical: claim-1 declares support-ghost, which was never retrieved.
High: policy-2026 was expected but absent; temporal query used vector_only.
Boundary: semantic entailment and answer correctness remain unverified.
Access and approval boundaries
- Read access to user-supplied JSONL retrieval traces.
- Write access only to the selected local audit output directory.
- No model, vector database, observability platform, browser, or network access is required.
Known limitations
- ID consistency does not prove that source text semantically entails a claim.
- Expected evidence is trustworthy only when an authoritative fixture or qualified reviewer established it.
- The input must expose stable evidence IDs and structured claim support; opaque answer-only logs cannot support the full audit.
// choose with context
Is Retrieval Trace Auditor the right skill?
It returns
Severity-ranked findings CSV for retrieval and trace-integrity failures. Case summary CSV with PASS, REVIEW, or FAIL disposition.
Do not use it as
It does not determine semantic entailment, answer correctness, source authority, privacy appropriateness, or safety.
Compatibility, access, version, and licence
- Reads or accesses
- Declared local scope: retrieval-trace-input/**, retrieval-trace-output/**, *.jsonl, *.csv.
- Compatibility
- Structured JSONL traces exported or mapped from LangSmith, LangChain, LlamaIndex, custom RAG pipelines, or agent observability systems. Stable evidence or chunk IDs and claim-to-support mappings. Python 3 for the included deterministic local auditor.
- Version
- 1.0.0
- Licence
- Review the package licence and seller terms at the linked destination.
Questions
Does this score answer correctness or faithfulness?
No. It checks the declared evidence path and routing metadata. Semantic correctness and entailment require a separate qualified or disclosed evaluator.
What is a stable evidence ID?
It is an identifier that lets the same source or chunk be recognized across a trace, claim, citation, and regression run without relying on mutable display text.
Can it replace LangSmith or Ragas?
No. It is a portable deterministic audit layer that can complement those systems, especially when you need an offline case record or explicit trace-integrity gate.
Why flag unused retrieved evidence?
Unused context may indicate retrieval noise or wasted context, but it is a Low review item rather than an automatic error.
Does vector-only routing always fail temporal questions?
No. The flag requires review because temporal, causal, relational, and multi-hop questions often need more than similarity retrieval. It is not proof of failure by itself.
Does it upload private traces?
No. The included helper runs locally and requires no browser, network, model, vector database, or observability-platform access.
Retrieval Trace Auditor keeps proof and approval boundaries visible.
The listing includes the tested package, realistic samples, declared permissions, and known limitations.
Get Retrieval Trace Auditor on Agensi