At what stage of the evaluation pipeline is a human absolutely necessary, and where can they be replaced by an automated "Judge LLM"?

Why is a standard unit test (asserting that output == "expected") often a bad way to test an LLM? How do you handle a model that gives three different, but correct, answers to the same prompt?

When would you evaluate a model without having a "correct" answer to compare it against? (e.g., checking for tone or politeness).

Explain the concept of using a "Stronger" model (like GPT-4o or Claude 3.5 Sonnet) to grade a "Weaker" model’s output. What are the risks of "Self-Preference Bias" in this setup?