Download - Stream Processing with Big Data: Knowledgent Big Data Palooza Meet-Up

©2014 Knowledgent Group Inc. All Rights Reserved

Stream Processing with Big Data

Learn Apache KafkaKishore VeletiBig Data Engineer

©2014 Knowledgent Group Inc. All Rights Reserved2

• Big Data Engineer at Knowledgent

• Background in enterprise application development using Hadoop stack, Java, PHP

• Worked in Healthcare, Banking, and Social Media Applications

• Passionate in sharing knowledge

About Me


Tutorial


• What is Apache Kafka?

• Apache Kafka Terminology

• Apache Kafka – about Topic & Partition

• Apache Kafka hands-on

We will discuss:


• Apache Kafka is a publish-subscribe messaging system implemented as a distributed commit log

• It is written in Java/Scala

• Built by LinkedIn to process activity stream data from their website

What is Apache Kafka?


• All the messages in Kafka are real-time

• There are many subscribers to a message

• Kafka persists messages to the disk

• Messages are retained for a specific time period

• Subscribers/clients store the state of their reads

• Easy to replay messages

What is Apache Kafka?


• Message: A datum to send

• Topic: Kafka maintains messages in categories called “topics”

• Partition: A logical division of a topic

• Producer: An API to publish messages to Kafka topic

• Broker: A server

• Cluster: Kafka cluster comprises one or more brokers

• Consumer: API to consume published messages and process further

• Replication: Kafka replicates log for each partition across servers

Apache Kafka Terminology


Message Topic Partition Producer Broker

Consumer

At a high level, producers send messages over the network to the Kafka cluster.

Kafka cluster in turn serves them up to consumers.

Apache Kafka Terminology & Big Picture



Consumer

Let’s do a hands-on exercise of Kafka with knowledge we’ve learned until now




Consumer

In Kafka for each topic a partition log is maintained.

Each partition is an ordered, immutable sequence of messages that is appended to

Each message in the partition is assigned a sequential id number called the offset

Apache Kafka: About Topic and Partition

Partition 1

Writes

Partition 2

Partition 3


Message Topic Partition Producer Broker Consumer

In Kafka, a Producer is an API to publish messages to topic



Message Topic Partition Producer Broker Consumer

In Kafka, a Consumer is an API to consume messages from topics




Consumer

Let’s do a hands-on exercise of Kafka with knowledge we’ve learned until now



• Trading Systems- Risk Identification in real-time

• Change Data Capture- Capturing the changed data into data lake environment

• Online Gaming- Identifying top scorers of a game

Apache Kafka Use Cases


Thank you!

Questions?

Download - Stream Processing with Big Data: Knowledgent Big Data Palooza Meet-Up

Top Related