Discover how FINRA built and optimized the Consolidated Audit Trail (CAT), a massive financial services database, in this AWS re:Invent 2023 conference presentation. Explore the architectural evolution of a system that processes over 400 billion daily events from market exchanges and broker-dealers while managing 600+ petabytes of storage. Gain insights into the dynamic, cost-efficient architecture that leverages 130,000 compute nodes daily, utilizing AWS services like Amazon EMR, Amazon Athena, and AWS KMS. Learn about the challenges faced, solutions implemented, and optimization strategies employed for both performance enhancement and cost reduction. Follow FINRA's journey from their initial infrastructure challenges through continuous improvement processes, including their approach to handling problematic nodes, scaling solutions, and architectural refinements that transformed their big data operation into an efficient, large-scale financial data processing system.
FINRA CAT: Building and Optimizing a Massive Financial Services Database System
AWS Events via YouTube
Overview
Syllabus
Introduction
What is FINRA CAT
Scott Donson
Steven Diamond
The Linker
Where were we when we started
Bad nodes
What did we do
Less bad nodes
Scaling
Solutions
Continuous optimization
Performance optimization
Cost optimization
Taught by
AWS Events
