Fully Automated Hardware SKU Selection System to Optimize Apache Pinot's Cost-to-Serve at LinkedIn

Learn how LinkedIn optimized Apache Pinot OLAP Database's cost-to-serve by nearly 50% across a production fleet of ~14K machines in this conference talk from SREcon25 Americas. Discover the automated hardware SKU selection system designed to handle diverse OLAP workloads with varying demands, resource usage patterns, and dataset sizes ranging from 500GB to 2PB. The presenters detail the core cost optimization algorithm that recommends optimal SKUs by considering multiple profiles, low-overhead mechanisms for collecting high-cardinality profiling data from production clusters, and various resource constraints including CPU, memory, disk IOPS, and throughput. Explore how this system supports multiple SKUs effectively for both cost optimization and minimal operational overhead through full automation, and learn how it has been integrated into LinkedIn's day-to-day operations.