Protecting the Kubernetes API from Observability Tools Overload

This conference talk shares the cautionary tale of how Grafana Labs accidentally took down their Kubernetes API in a major production cluster by deploying observability tools that made excessive API calls. Learn how the team recovered and implemented protective measures to shield the Kubernetes API while maintaining functionality. The speakers explain the appeal of using the Kubernetes API for observability solutions—particularly for enriching telemetry with pod and service metadata and creating service graphs—and detail the specific techniques they developed to prevent overwhelming the API. Their experience provides valuable lessons for anyone building observability tools that interact with Kubernetes, demonstrating how to balance rich metadata integration with system stability.