Learn Apache Kafka for Beginners

Introduction

• Intro to Apache Kafka
• Apache Kafka in five minutes

1. Kafka Theory

• Kafka theory overview
• Topics, partitions, and offsets
• Brokers and topics
• Topic replication
• Producers and message keys
• Consumer and consumer group
• Consumer offsets and delivery semantics
• Kafka broker discovery
• ZooKeeper
• Kafka guarantees
• Theory roundup

2. Starting Kafka

• Important: Starting Kafka
• macOS: Download and set up Kafka in PATH
• macOS: Using brew
• macOS: Start ZooKeeper and Kafka
• Linux: Download and set up Kafka in PATH
• Linux: Start ZooKeeper and Kafka
• Windows: Download and set up Kafka in PATH
• Windows: Start ZooKeeper and Kafka

3. Command Line Interface (CLI) 101

• CLI introduction
• Kafka topics CLI
• Kafka console producer CLI
• Kafka console consumer CLI
• Kafka consumers in groups
• Kafka consumer groups CLI
• Resetting offsets
• Kafka Tool UI

4. Kafka Java Programming 101

• Intro to Kafka programming
• Creating a Kafka project
• Java producer
• Java producer callbacks
• Java producer with keys
• Java consumer
• Java consumer inside consumer group
• Java consumer with threads
• Java consumer seek and assign
• Client bidirectional compatibility

5. Kafka Real-World Project

• Real-world project overview

6. Kafka Twitter Producer and Advanced Configurations

• Producer and advanced configurations overview
• Twitter setup
• Producer, part 1: Writing a Twitter client
• Producer, part 2: Writing the Kafka producer
• Producer configurations introduction
• Acks and min.insync.replicas
• Retries and max.in.flight.requests.per.connection
• Idempotent producer
• Producer, part 3: Safe producer
• Producer compression
• Producer batching
• Producer, part 4: High throughput producer
• Producer default partitions and key hashing
• Refactoring the project

7. Kafka Elasticsearch Consumer and Advanced Configurations

• Consumer and advanced configuration overview
• Setting up Elasticsearch in the cloud
• Elasticsearch 101
• Consumer, part 1: Set up project
• Consumer, part 2: Write the consumer and send to Elasticsearch
• Consumer, part 3: Delivery semantics
• Delivery semantics for consumers
• Consumer, part 3: Idempotence
• Consumer poll behavior, part 1
• Consumer offset commit strategies
• Consumer, part 4: Manual commit of offsets
• Consumer, part 5: Performance improvement using batching
• Consumer offsets reset behavior
• Consumer, part 6: Replaying data
• Consumer internal threads

8. Kafka Ecosystem and Real-World Architectures

• Kafka in the real world

9. Kafka Extended APIs

• Kafka Connect introduction
• Kafka Connect Twitter: Hands-on example
• Kafka Streams introduction
• Kafka Streams: Hands-on example
• Kafka Schema Registry introduction

10. Real-World Insights and Case Studies

• ZooKeeper
• Case study: MovieFlix
• Case study: GetTaxi
• Case study: MySocialMedia
• Case study: MyBank
• Case study: Big data ingestion
• Case study: Logging and metrics aggregation

11. Kafka in the Enterprise for Admins

• Kafka cluster setup, high-level architecture overview
• Kafka monitoring and operations
• Kafka security
• ZooKeeper

12. Advanced Topic Configurations

• Changing a topic configuration
• Segment and indexes
• Log cleanup policies
• Log cleanup delete
• Log compaction theory
• Log compaction practice
• min.insync.replicas reminder
• Unclean leader election

13. Annexes

• What are annexes?

14. Starting Kafka Differently

• Annex 1: Overview
• Starting Kafka with the Confluent CLI
• Starting a multibroker Kafka cluster using binaries
• Start Kafka development environment using Docker
• Starting a multibroker Kafka cluster using Docker
• Kafka advertised host setting
• Starting Kafka on a remote machine