kafka to hadoop ingest with parsing, dedup and other big data transformations

© 2016 DataTorrent

Chaitanya CheboluCommitter, Apache Apex

Engineer, DataTorrentSep 14, 2016

Data Ingestion - Kafka ETL

© 2016 DataTorrent

Agenda

2

• Introduction about Apache Apex (Architecture, Application, Native Hadoop Integration)

•What is Data Ingestion•Use Case : Kafka ETL•Brief about Kafka•Kafka ETL App•Kafka ETL Demo

© 2016 DataTorrent3

Apache Apex •Platform and runtime engine that enables development of

scalable and fault-tolerant distributed applications•Hadoop native (Hadoop >= 2.2)

No separate service to manage stream processingStreaming Engine built into Application Master and

Containers•Process streaming or batch big data•High throughput and low latency•Library of commonly needed business logic•Write any custom business logic in your application

© 2016 DataTorrent4

Apex Architecture

© 2016 DataTorrent5

An Apex Application is a DAG(Directed Acyclic Graph)

A DAG is composed of vertices (Operators) and edges (Streams).A Stream is a sequence of data tuples which connects operators at end-points called PortsAn Operator takes one or more input streams, performs computations & emits one or more output streams

● Each operator is USER’s business logic, or built-in operator from our open source library● Operator may have multiple instances that run in parallel

© 2016 DataTorrent6

Apex - Native Hadoop Integration

• YARN is the resource manager

• HDFS used for storing any persistent state

© 2016 DataTorrent

What is Data Ingestion?

7

•Data IngestionA process of obtaining, importing, and analyzing data for

later use or storage in a database•Big Data Ingestion

Discovering the data sources Importing the data Processing data to produce intermediate data Sending data out to durable data stores

© 2016 DataTorrent

Use Case: Kafka ETL

8

•Consuming data from Kafka

•Processing data to produce intermediate data

•Writing the processed data to HDFS

© 2016 DataTorrent

Brief about Kafka

9

● Distributed Messaging System.

● Data Partitioning Capability.

● Fast Read and Writes.

● Basic Terminology○ Topic ○ Producer○ Consumer○ Broker

© 2016 DataTorrent

Kafka ETL App

10

Kafka Parser Dedup Transform Formatter

HDFS

© 2016 DataTorrent

Kafka ETL Demo

11

Demo

© 2016 DataTorrent

Resources

12

• Apache Apex - http://apex.apache.org/• Subscribe - http://apex.apache.org/community.html• Download - https://www.datatorrent.com/download/• Twitter

ᵒ @ApacheApex; Follow - https://twitter.com/apacheapexᵒ @DataTorrent; Follow – https://twitter.com/datatorrent

• Meetups - http://www.meetup.com/topics/apache-apex• Webinars - https://www.datatorrent.com/webinars/• Videos - https://www.youtube.com/user/DataTorrent• Slides - http://www.slideshare.net/DataTorrent/presentations • Startup Accelerator Program - Full featured enterprise product

ᵒ https://www.datatorrent.com/product/startup-accelerator/

© 2016 DataTorrent

We Are Hiring

13

• jobs@datatorrent.com• Developers/Architects• QA Automation Developers• Information Developers• Build and Release• Community Leaders