README.md

Kafka Labs

Welcome to Kafka labs bundle.

Tips

Copy-Paste Shortcuts on Jupyter terminal

• On Windows try: Shift + Insert
• On Windows try: Shift + Right Click

Intro Labs

1 - Setup Kafka

• 1 - Installing Kafka

2 - Kafka Utils

• 2.1 - Kafka Utilities
• 2.2 - Kafkacat
• 2.3 - Producing with keys
• 2.4 - Compacted topics - Optional

3 - Kafka API

• 3.1 - Importing Project
• 3.2 - Simple Producer and Consumer
• 3.3 - Kafka API

4. More Kafka API

• 4.1 - Kafka Producer benchmarking
• 4.2 - Compression in Kafka
• 4.3 - Kafka compression benchmarking

5 - Offset Management

• 5.1 - Manual offset
• 5.2 - Seeking Within Partition

6 - Practice Lab - Domain Count

• 6.1 - Analyzing clickstream traffic data

7 - Kafka Streams

• 7.1 - Streams - Intro
• 7.2 - Streams - Foreach

optional (intermediate)

• 7.3 - Streams - Filter
• 7.4 - Streams - Map
• 7.5 - Streams - GroupBy
• 7.6 - Streams - Window

Practice Lab - Fraud Detection

• Fraud detection

Design Exercise 1 (Group Exercise)

• Instructor will provide details

Intermediate Labs

8 - Confluent Platform

• 8.1 - Install Confluent
• 8.2 - Schema with Avro
• 8.3 - KSQL

9 - Metrics

• Optional - 9.1 - Metrics Intro - This is an introduction to using Metrics library. Provided as reference.
• 9.2 - Kafka Metrics
• BONUS - 9.3 - Kafka Metrics - traffic stats

10 - Kafka Connect

• 10.1 - Kafka Connect

11 - Spark and Kafka

• 11.1 - Spark + Kafka: batch processing
• 11.2 - Spark + Kafka: stream processing
• 11.3 - Fraud detection with Spark and Kafka - batch mode
• 11.4 - Fraud detection with Spark and Kafka - streaming

Design Exercise 2 (Group Exercise)

• Instructor will provide details

BONUS Labs

12 - Kafka in Docker

• Running full kafka stack in docker - Check out this guide on how to run multiple kafka brokers + zookeeper + kafka manager using Docker

Downloading Your Work from Lab Machine

On Jupyter terminal, execute these commands

$   cd ~/dev
$   zip -x '*.jar' -x '*.git*'  -x '*.ipynb_checkpoints*'  -r my-kafka-labs.zip    kafka-labs

Now right-click on my-kafka-labs.zip from Jupyter UI to download.