In a serverless GPU environment, what is a "Cold Start"? How does the size of your model weights (e.g., a 70B model) impact the time it takes for a new instance to start serving traffic?

How would you implement a "Token Quota" system to prevent a single user or a bug in your code from spending $1,000 on API calls in an hour?

In a high-concurrency environment, how does PagedAttention prevent the GPU from running out of memory (OOM) when multiple users are chatting simultaneously?

When switching from one model to another (let's say Llama 3 to Llama 3.1), how do you perform a Blue/Green swap? How do you handle the state of ongoing "streaming" conversations during the switch?