RAG Applied

"Three reasons. Cost — RAG retrieves only the few chunks relevant to a query; long context pays for the whole corpus on every call. Selectivity — long context suffers from 'lost in the middle,' where the model misses details buried mid-prompt; RAG puts only the relevant material in view. Operational decoupling — a RAG index updates independently of the agent, so adding a new regulation doesn't require touching the prompt. Long context is the right call when the task genuinely needs whole-document reasoning — say, drafting an impact analysis on a single 200-page regulation. RAG is the default when the corpus is bigger than any one task, when freshness matters, and when you need citations. In a compliance KB you almost always want RAG; you might use long context as a downstream step after retrieval narrows the relevant document."

"It's the tradeoff between retrieval precision and answer context. Smaller chunks — 200-500 tokens — give precise retrieval but lose surrounding context, so the model sees a sentence stripped of its section heading. Larger chunks — 1000-1500 — preserve context but dilute the embedding signal and waste tokens on irrelevant content. The pragmatic default I'd start with is 500-800 tokens with 10-20% overlap, then tune based on retrieval evals. Better than guessing size: respect document structure. For regulatory documents that means recursive chunking by chapter then section then paragraph — chunks become natural units, not arbitrary slices. And the parent-child trick is often the right answer: index small chunks for retrieval precision, but return the parent paragraph or section for generation context."

"Make refusal a first-class output, not a failure. Three layers. One, system prompt discipline — explicit instruction that the model must answer 'I don't have enough information' rather than guess when retrieved context doesn't cover the question. Two, grounding validation — a post-generation check that every claim in the answer maps back to a cited chunk; ungrounded claims fail the answer. Three, retrieval-quality signal — if top-K scores are below a threshold, the system surfaces 'no good match' and routes to human review rather than generating. For compliance specifically, the refusal itself is valuable signal — it tells the team where the KB has gaps. I'd capture refusals into the eval dataset and use them to prioritize content gaps."

"Source-of-truth lives outside the index. The index is a derived artifact rebuilt from canonical sources. Change-detection runs on the source — document version, hash, last-modified — and triggers re-ingestion only for what changed. You don't re-embed the whole corpus nightly; you re-embed deltas. Each chunk carries metadata for effective_date and superseded_date. Retrieval filters by date so a query about today's rule doesn't surface last year's superseded version — but the superseded version stays in the index because a regulator may still ask about historical decisions. For regulations published on a known cadence, schedule the refresh; for ad-hoc updates, webhook the publisher or poll. And version the index itself so audit logs can pin which index version answered which query."

"Separate the two — they're different failure modes. Retrieval quality uses Ragas-style metrics: context precision asks 'are the retrieved chunks relevant?', context recall asks 'do the chunks contain everything needed to answer?'. Those require a labeled set where you know the right answer and the right supporting chunks. Answer quality is the next layer: faithfulness — are the answer's claims supported by the cited chunks — plus answer relevance and citation correctness. The diagnostic move: when an answer is wrong, check retrieval first. If the right chunks weren't retrieved, the retriever is the bug — fix chunking, reranking, or query expansion. If the right chunks were retrieved but the answer is still wrong, the generator is the bug — fix the prompt, grounding instructions, or model choice. That split tells you where to invest."

"Goal: combine BM25 keyword matching with vector similarity so you catch both lexical exact matches and semantic matches. Two implementation paths. One, if the vector store supports hybrid natively — OpenSearch, Weaviate, recent versions of pgvector with full-text — enable the BM25 side and use reciprocal rank fusion or a weighted score to combine. Two, if not, run two retrievers in parallel: vector search returning top-50 with scores, BM25 search returning top-50 with scores, then merge with RRF — for each doc, score = sum of 1/(k + rank_in_each_retriever). RRF is simple and surprisingly robust because it sidesteps the score-normalization problem between BM25 and cosine similarity. Then layer a cross-encoder reranker on the merged top-50 to get final top-5-to-10. Eval the change against a held-out set before swapping it in — hybrid usually wins, but the weighting needs tuning per corpus."

"Pull the trace first — every agent run should have one. Find the exact retrieved chunks for that query and the model output. Then triage:

1. Was the cited regulation in the retrieved chunks? If no, the model fabricated the citation — that's a generator failure. Tighten the system prompt's citation discipline, add a post-generation validator that every cited ID exists in the retrieved set, and add a runtime guardrail: every cited reg ID must resolve in your citation registry or the answer is rejected.
2. Was it in the retrieved chunks but the cite is wrong? That's an ingestion failure — likely chunk metadata got mis-tagged during indexing. Check the source-of-truth, fix the metadata, re-index that document, add a data-quality check to the ingestion pipeline.
3. Either way, add this case to the eval set so the regression test catches it forever.
4. Report back to the investigator with what happened, what was fixed, and how we'll prevent it. That last step matters for trust — compliance teams need to see the AI's failure mode being treated as a real incident, not waved away."