When is it more cost-effective to use a "Pay-per-token" API (like OpenAI) versus hosting your own model on a dedicated cloud instance (like an AWS g5 instance)?

Explain how Continuous Batching (used in engines like vLLM) differs from traditional static batching. How does it improve GPU utilization?

Can you use "Spot" or "Preemptible" GPU instances for real-time inference? What happens to the user's request if the cloud provider reclaims the GPU mid-generation?

If your inference latency is high because the model is too big for one GPU, do you scale horizontally or vertically? What if the latency is high because you have too many concurrent users?