When an upstream provider returns a 429: Too Many Requests, how do you implement a "Circuit Breaker" pattern so your entire application doesn't crash?

Explain how Continuous Batching (used in engines like vLLM) differs from traditional static batching. How does it improve GPU utilization?

Can you use "Spot" or "Preemptible" GPU instances for real-time inference? What happens to the user's request if the cloud provider reclaims the GPU mid-generation?

When switching from one model to another (let's say Llama 3 to Llama 3.1), how do you perform a Blue/Green swap? How do you handle the state of ongoing "streaming" conversations during the switch?