Envoy, Take the Wheel - Real-time Adaptive Circuit Breaking

Envoy, take the wheel: Realtime adaptive circuit breaking Protecting Microservices with Adoptive Concurrency
Agenda • Simulating traffic. • Concurrency and circuit breaking. • Adaptive concurrency control. • Tales from lyn infra
Simulations.
Full Simulation Output.
Simulating Traffic Overload.
Concurrency. There's some number of requests a system can service at any given time
Really, bursts are fine.
Envoy concurrency limits via circuit breaking. Cluster Max Connections Limit number of retries outstanding
Revisiting Traffic Overload!
Properly configuring circuit breakers. Understand service limitations
Properly configuring circuit breakers is hard. Service owners shouldn't have to profile concurrency
Configuring limits in an ideal world... No manual configuration
Envoy Filters
HTTP Connection Manager Network Filter Parses raw bytes over connection • Bytes are converted into HTTP messages • Allows Envoy to operate at a higher level of abstraction Contains an HTTP filter chain Agnostic to underlying protocols (HTTP/1, HTTP/2, RPC) HTTP filters
The gradient
Concurrency Limit Behavior.
There's more...
Adaptive Concurrency Control Filter
Adaptive Concurrency in Action
Adaptive Concurrency i ction
Steady-state with min_concurrency too low.
Configuring Adaptive Concurrency jitter
Experiences @lyR: Our default settings Our default adaptive concurrency settings
Experiences @ lyR: General observations Almost all adaptive concurrency events are due to service degradation, not spikes in RPS. Bad deploys Upstream dependency outages/delays.
Experiences @ lyR: The bad deploy Changes went out that resulted in elevated CPU utilization