What is the difference between testing your model on a static CSV file (Offline) vs. monitoring real user "Thumbs Up/Down" feedback (Online)?

Why is a standard unit test (asserting that output == "expected") often a bad way to test an LLM? How do you handle a model that gives three different, but correct, answers to the same prompt?

How do you programmatically check if an LLM is making things up that aren't in the provided search results?

How would you automate the process of trying to make your model "break" or "hallucinate"?