A support team came to us convinced they needed a stronger model. The assistant was inventing policy that did not exist. The model was fine. The context was feeding it three versions of the same policy from three different years.
The retriever was doing its job in the narrow sense. For a question about refunds it returned every refund-related passage it could find, which included the current policy, a draft from two years earlier, and a regional exception that had since been retired. All three landed in the window with no dates and no ranking, and the model blended them into a confident answer that matched none of them.
We changed three things and touched the model zero times. We attached an effective date to every passage and dropped anything superseded before it reached the window. We cut retrieval from fifteen passages to the four highest-scoring current ones. And where two current passages genuinely conflicted, we surfaced the conflict in the context rather than hiding it, so the assistant asked a clarifying question instead of guessing.
Invented policy dropped to near zero on the same test set. The lesson was not subtle. The model had never been the problem. We had been asking it to reconcile a contradiction we put there ourselves.