If your retriever returns 5 documents but only 1 was actually related to answering the question, how do you penalize the retriever for the "noise"?

How do you calculate the ROI of a prompt change? If a new prompt is 5% more accurate but 50% more expensive in tokens, how do you decide if it’s worth it?

At what stage of the evaluation pipeline is a human absolutely necessary, and where can they be replaced by an automated "Judge LLM"?

What is a "Golden Dataset" (or Ground Truth set), and how many samples should it ideally contain before you can trust your evaluation metrics?