Harnessing the Power of Envoy Proxy for Building an LLM Gateway
CNCF [Cloud Native Computing Foundation] via YouTube
Overview
Learn how to leverage Envoy Proxy's unique architecture and capabilities to build a robust LLM gateway in this 38-minute conference talk. Discover why Envoy Proxy surpasses traditional proxies like NGINX through its extensibility features, including filter architecture, callout architecture (ext-proc, ext-auth), and dynamic library loading capabilities. Explore practical applications of Envoy proxy for managing LLM credentials, implementing prompt guarding/decorating, analyzing content safety, controlling usage, enabling context-aware failover, and enhancing observability. Gain insights into how Envoy's high-performance, asynchronous C++ core enables it to function effectively as an ingress, egress, and mesh gateway. Particularly valuable for developers, architects, and technology professionals seeking solutions for LLM implementation challenges and infrastructure platform optimization.
Syllabus
Harnessing the Power of Envoy Proxy for Building an LLM Gateway - Idit Levine, Solo.io
Taught by
CNCF [Cloud Native Computing Foundation]