Vector search gets you a pile of plausibly relevant chunks. That pile is not context yet. The half nobody budgets for is what you do with those chunks before they reach the model.
A retriever that returns the top twenty passages sounds thorough. In practice you have just handed the model twenty things to reconcile, several of which repeat each other and one or two of which are off topic enough to pull the answer sideways. More retrieved text is not more signal.
The steps that matter come after retrieval. Deduplicate near-identical passages. Re-rank so the strongest few sit first. Trim each passage to the part that is actually on point instead of pasting the whole chunk. Where two sources disagree, keep both but say so, because a silent contradiction in the context turns into a confident wrong answer downstream.
We have had more luck cutting a retriever from twenty passages to five good ones than from swapping the embedding model. The retriever finds candidates. Your pipeline decides which candidates deserve the desk.