When switching from one model to another (let's say Llama 3 to Llama 3.1), how do you perform a Blue/Green swap? How do you handle the state of ongoing "streaming" conversations during the switch?

Question

Accepted Answer

You run Llama-3 (Blue) and Llama-3.1 (Green) simultaneously. You route 1% of traffic to Green, check the "Eval" scores, and slowly ramp up. Ongoing streaming chats should be finished on the "Blue" server before it is decommissioned.

When switching from one model to another (let's say Llama 3 to Llama 3.1), how do you perform a Blue/Green swap? How do you handle the state of ongoing "streaming" conversations during the switch?

Practice Your Response

Similar Questions in Deployment & Cost (AI-Ops)