When is it more cost-effective to use a "Pay-per-token" API (like OpenAI) versus hosting your own model on a dedicated cloud instance (like an AWS g5 instance)?

If your inference latency is high because the model is too big for one GPU, do you scale horizontally or vertically? What if the latency is high because you have too many concurrent users?

Can you use "Spot" or "Preemptible" GPU instances for real-time inference? What happens to the user's request if the cloud provider reclaims the GPU mid-generation?

If you have 100 different customers, each with a custom-tuned LoRA adapter, do you need 100 different GPU clusters? How would you serve them efficiently on one cluster?