Kafka

Kafka is a distributed, partitioned, replicated log based service. It provides functionality of a messaging system, but with a very different design.

Why Queue?

• Makes each data set to be multi-subscriber

• Ensure the order is maintained in the processing

• Provide buffering

• Effectively decoupling producers and consumers of data within the pipeline

---

• Concept

• Architecture

---

• Broker

• Producer

• Consumer

---

• Kafka Connect

• Kafka Streams

---

• Client Protocol

Further Readings

Online Articles

• Apache Official Document

• Jay Kreps, Neha Narkhede, Jun Rao: Kafka: a Distributed Messaging System for Log Processing

• Jay Kreps: The Log: What every software engineer should know about real-time data’s unifying abstraction, December 16, 2013

• Martin Kleppmann and Jay Kreps: Kafka, Samza and the Unix Philosophy of Distributed Data, December 2015

---

Books

• I Heart Logs - Event Data, Stream processing, and Data integration; Jay Kreps

• Kafka: The Definitive Guide