When would you choose to run a model locally on a user's device (using WebLLM or ONNX) instead of the cloud? Focus on privacy and cost.

When switching from one model to another (let's say Llama 3 to Llama 3.1), how do you perform a Blue/Green swap? How do you handle the state of ongoing "streaming" conversations during the switch?

How do you track which specific feature or user in your app is driving the most "Token Spend"?

If your inference latency is high because the model is too big for one GPU, do you scale horizontally or vertically? What if the latency is high because you have too many concurrent users?