ereignis-getriebene microservices mit apache kafka...mit apache kafka guido schmutz jug saxony day -...

43
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A. M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Ereignis-getriebene Microservices mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com

Upload: others

Post on 26-Apr-2020

14 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF

HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

Ereignis-getriebene Microservices mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018

@gschmutz guidoschmutz.wordpress.com

Page 2: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Guido Schmutz

Working at Trivadis for more than 21 years

Oracle ACE Director for Fusion Middleware and SOA

Consultant, Trainer Software Architect for Java, Oracle, SOA and

Big Data / Fast Data

Head of Trivadis Architecture Board

Technology Manager @ Trivadis

More than 30 years of software development experience

Contact: [email protected]

Blog: http://guidoschmutz.wordpress.com

Slideshare: http://www.slideshare.net/gschmutz

Twitter: gschmutz

Building event-driven Microservices with Kafka Ecosystem

Page 3: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

COPENHAGEN

MUNICH

LAUSANNE

BERN

ZURICH BRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

14 Trivadis branches and more than

600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:

CHF 5.0 million

Financially self-supporting and

sustainably profitable

Experience from more than 1,900

projects per year at over 800

customers

Building event-driven Microservices with Kafka Ecosystem

Page 4: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Agenda

1. Where do we come from?

2. What are Microservices?

3. Why not more Event Driven?

4. Why Kafka for Event-Driven Microservices?

5. What about streaming sources?

6. What about integrating legacy applications?

7. What about (historical) data analytics?

8. Summary

Building event-driven Microservices with Kafka Ecosystem

Page 5: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Building event-driven Microservices with Kafka Ecosystem

Where do we come from?

Page 6: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Shop Rich UI

Shop Backend Application

Traditional Approach

Search Facade

Customer DAO

Order DAO

Order Facade

Shop UI

Product DAO

UI Logic

Data Business

GUI

Customer Fat Client App

Customer BO Customer UI

Data GUI

Data Storage

Shared Database

Building event-driven Microservices with Kafka Ecosystem

sync request/response

Page 7: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Shop UI App

Business Activity Service

SOA Approach

Building event-driven Microservices with Kafka Ecosystem

Contract-first

Web Services

Technical layers

offer their own

interfaces

Reuse on each

level

Lower layer

often wraps

legacy code

Search BAS

Customer DAO

Order DAO

Order BAS

Shop UI

Product DAO

UI Logic

GUI

Business Entity Service Shop Web App

Shop UI UI Logic

GUI

Data Storage

Customer Database

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order and Product DB

SOAP

SOAP

SOAP SOAP

SOAP

SOAP

SOAP

Page 8: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Shop UI App

Business Activity Service

Virtualized SOA Approach

Search BAS Customer DAO

Order DAO

Order BAS

Shop UI UI Logic

GUI

Business Entity Service

Shop Web App

Shop UI UI Logic

GUI

Data Storage

Customer Database

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order and Product DB

Service Virtualization Layer

Service Bus

SOAP SOAP

SOAP

SOAP

SOAP

SOAP

SOAP

Building event-driven Microservices with Kafka Ecosystem

Page 9: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Building event-driven Microservices with Kafka Ecosystem

What are Microservices?

Page 10: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Microservice Approach

Building event-driven Microservices with Kafka Ecosystem

Tightly Scoped behind clear

interfaces

Responsible for managing their

own data (not necessarily the

infrastructure)

Should be highly decoupled

Independently deployable, self-

contained and autonomous

SOA done right ?!

Tightly Scoped Responsible for managing their own data Highly decoupled Independently deployable, self-contained and autonomous

{ }

Customer API Customer

Customer Logic

Order Microservice

{ }

Order API Order

Order Logic

Product Microservice

{ }

Product API Product

Product Logic

Stock Microservice

{ }

Stock API Stock

Stock Logic

Shop Web App

Shop UI UI Logic

GUI

REST

REST

REST

REST

Page 11: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Microservice Approach

with API Gateway

Customer Microservice

{ }

Customer API Customer

Customer Logic

Order Microservice

{ }

Order API Order

Order Logic

Product Microservice

{ }

Product API Product

Product Logic

Stock Microservice

{ }

Stock API Stock

Stock Logic

Shop Web App

Shop UI UI Logic

GUI

REST

REST

REST

REST

API Gateway

Building event-driven Microservices with Kafka Ecosystem

Page 12: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Synchronous World of Request-Response leads to tight,

point-to-point couplings

Building event-driven Microservices with Kafka Ecosystem

problem in lower end of chain have a ripple

effect on other service • crash of service

• overloaded service / slow response time

• change of interface

Service 2 Service 1

{ }

API Logic

{ }

API Logic

State State

Service 3

{ }

API Logic

State

Service 4

{ }

API Logic

State

Service 5

{ }

API Logic

State

Service 7

{ }

API Logic

State

Service 6

{ }

API Logic

State

Page 13: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Building event-driven Microservices with Kafka Ecosystem

Why not more Event-Driven?

Page 14: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Three mechanisms through which services can interact

Request-Driven (Imperative) Event Driven (Functional)

Service

Logic

State

Consume Event “Event Ordered”

OrderEvent

Command ”Order IPad”

boolean order(IPad)

Publish Event ”Event Validated”

OrderValidatedEvent Query

”Retrieve my Orders List<Orders> getAllOrders(for)

Building event-driven Microservices with Kafka Ecosystem

Event Broker

Service

Logic

State

Page 15: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Event-Driven (Async)

Microservice Approach

Customer Microservice

{ }

Customer API Customer

Customer Logic

Order Microservice

{ }

Order API Order

Order Logic

Product Microservice

{ }

Product API Product

Product Logic

Stock Microservice

{ }

Stock API Stock

Stock Logic

Shop Web App

Shop UI UI Logic

GUI

REST

REST

REST

REST

API Gateway

Building event-driven Microservices with Kafka Ecosystem

Event Store

sync request/response

async request/response

async, event pub/sub

Page 16: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Why Kafka for Event-Driven Microservices?

Building event-driven Microservices with Kafka Ecosystem

Page 17: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Apache Kafka – A Streaming Platform

Building event-driven Microservices with Kafka Ecosystem

High-Level Architecture

Distributed Log at the Core

Scale-Out Architecture

Logs do not (necessarily) forget

Page 18: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Hold Data for Long-Term – Data Retention

Producer 1

Broker 1

Broker 2

Broker 3

1. Never

2. Time based (TTL) log.retention.{ms | minutes | hours}

3. Size based log.retention.bytes

4. Log compaction based

(entries with same key are removed):

kafka-topics.sh --zookeeper zk:2181 \

--create --topic customers \

--replication-factor 1 \

--partitions 1 \

--config cleanup.policy=compact

Building event-driven Microservices with Kafka Ecosystem

Page 19: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Keep Topics in Compacted Form

0 1 2 3 4 5 6 7 8 9 10 11

K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2

V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11

Offset

Key

Value

3 4 6 8 9 10

K1 K3 K4 K5 K2 K6

V4 V5 V7 V9 V10 V11

Offset

Key

Value

Compaction

Building event-driven Microservices with Kafka Ecosystem

V1

V2

V3 V4

V5

V6

V7

V8 V9

V10

V11

K1

K3

K4

K5

K2

K6

Page 20: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Topic Viewed as Event Stream or State Stream (Change

Log)

Event Stream State Stream (Change Log Stream)

2015-10-02

11,Peter,Muster,3010,Berne

2016-10-04

12,Paul,Steffen,8001,Zurich

2016-12-02

21,Lisa,Meier,3043,Ittigen

2017-05-03

11,Peter,Muster,3015,Berne

2017-05-03

21,Lisa,Steffen,8001,Zurich

2017-07-03

11,Peter,Muster,3052,Zollikofen

Building event-driven Microservices with Kafka Ecosystem

2015-10-02

11,Peter,Muster,3010,Berne

2016-10-04

12,Paul,Steffen,8001,Zurich

2016-12-02

21,Lisa,Meier,3043,Ittigen

2017-05-03

11,Peter,Muster,3015,Berne

2017-05-03

21,Lisa,Steffen,8001,Zurich

2017-07-03

11,Peter,Muster,3052,Zollikofen

Page 21: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Keep Topics both in Original and Compacted Form

Building event-driven Microservices with Kafka Ecosystem

Kafka Broker A B C B A E K A E A

C K

B E A C K

Producer

Kafka Broker A B C B A E K A E A

C K

B E A C K

Producer

Consume and Produce

OR

TX

Page 22: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

What about streaming sources?

Building event-driven Microservices with Kafka Ecosystem

Page 23: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

How to work with

streaming data sources

Customer Microservice

{ }

Customer API Customer

Customer Logic

Order Microservice

{ }

Order API Order

Order Logic

Product Microservice

{ }

Product API Product

Product Logic

Stock Microservice

{ }

Stock API Stock

Stock Logic

Shop Web App

Shop UI UI Logic

GUI

REST

REST

REST

REST

Event Store

Location

Social

Click stream

Sensor Data

Mobile Apps

Weather Data

Event Stream

Building event-driven Microservices with Kafka Ecosystem

Page 24: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Hadoop Clusterd Hadoop Cluster

Stream Processing Cluster

Streaming Processing & Microservices Architecture BI Tools

Search / Explore

Online & Mobile Apps

Search

Event Stream

Results Stream Analytics

Reference / Models

Dashboard

Location

Social

Click stream

Sensor Data

Mobile Apps

Weather Data

Microservice Cluster

Microservice State

{ }

API

Event Stream

Event Stream

Event Store

Building event-driven Microservices with Kafka Ecosystem

Page 25: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Stream Processing Capability - Windowing

Introduction to Stream Processing

Computations over events done using

windows of data

Due to size and never-ending nature of it,

it’s not feasible to keep entire stream of data in memory

A window of data represents a certain

amount of data where we can perform

computations on

Windows give the power to keep a

working memory and look back at recent

data efficiently

Time

Stream of Data Window of Data

Page 26: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Sliding Window (aka Hopping Window) - uses eviction and trigger policies that are based on time: window length and sliding interval length

Fixed Window (aka Tumbling Window) - eviction policy always based on the window being full and trigger policy based on either the count of items in the window or time

Session Window – composed of sequences of temporarily related events terminated by a gap of inactivity greater than some timeout

Stream Processing Capability - Windowing

Introduction to Stream Processing

Time Time Time

Page 27: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Stream Processing Capability - Joining

Introduction to Stream Processing

Challenges of joining streams

1. Data streams need to be aligned as they

come because they have different timestamps

2. since streams are never-ending, the joins

must be limited; otherwise join will never end

3. join needs to produce results continuously as

there is no end to the data

Stream to Static (Table) Join

Stream to Stream Join (one window join)

Stream to Stream Join (two window join)

Stream-to-Static

Join

Stream-to-Stream

Join

Stream-to-Stream

Join

Time

Time

Time

Page 28: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Stream Processing Capability - State Management

Introduction to Stream Processing

Necessary if stream processing use case

is dependent on previously seen data or

external data

Windowing, Joining and Pattern Detection use State Management behind

the scenes

State Management services can be made

available for custom state handling logic

State needs to be managed as close to

the stream processor as possible

Options for State Management

How does it handle failures? If a machine

crashes and the/some state is lost?

In-Memory Replicated, Distributed

Store

Local, Embedded

Store

Operational Complexity and Features

Low high

Page 29: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Apache Kafka – scalable message processing and

more!

Building event-driven Microservices with Kafka Ecosystem

Source Connector

trucking_ driver

Kafka Broker

Sink Connector

Stream Processing

Schema Registry

Kafka Kafka

Page 30: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Kafka Connect - Overview

Source Connector

Sink Connector

Introduction to Stream Processing

Page 31: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Kafka Streams - Overview

Introduction to Stream Processing

Designed as a simple and lightweight library in Apache Kafka

no other dependencies than Kafka

Supports fault-tolerant local state

Supports Windowing (Fixed, Sliding and

Session) and Stream-Stream / Stream-Table Joins

Millisecond processing latency, no

micro-batching

At-least-once and exactly-once

processing guarantees

KTable<Integer, Customer> customers = builder.stream(”customer"); KStream<Integer, Order> orders = builder.stream(”order"); KStream<Integer, String> enriched = orders.leftJoin(customers, …); joined.to(”orderEnriched");

trucking_ driver

Kafka Broker

Java Application

Kafka Streams

Page 32: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

KSQL - Overview

Introduction to Stream Processing

STREAM and TABLE as first-class

citizens

• STREAM = data in motion

• TABLE = collected state of a stream

Stream Processing with zero coding

using SQL-like language

Built on top of Kafka Streams

Interactive (CLI) and headless (command

file)

ksql> CREATE STREAM customer_s \ WITH (kafka_topic=‘customer', \ value_format=‘AVRO'); Message ---------------- Stream created ksql> SELECT * FROM customer_s \ WHERE address->country = ‘Switzerland’; ...

trucking_ driver

Kafka Broker

KSQL Engine

Kafka Streams

KSQL CLI Commands

Page 33: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

What about integrating legacy applications?

Building event-driven Microservices with Kafka Ecosystem

Page 34: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Data Store

Integrate existing systems through CDC

Customer

Event Store

Integration

Microservice

State Logic

CDC CDC Connector

Customer Fat Client App

Customer BO Customer UI

Stream Processing

Results Stream Processor

Reference / Models

Dashboard

Capture changes directly on database

Change Data Capture (CDC) => think like a

global database trigger

Transform existing systems to event

producer

Building event-driven Microservices with Kafka Ecosystem

Page 35: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Legacy Microservice

Change Data Capture (CDC) with Kafka

Building event-driven Microservices with Kafka Ecosystem

RDBMS cdc-

source trucking_

driver

Customer Topic elasticsearch

-sink NoSQL

Page 36: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Hadoop Clusterd Hadoop Cluster

Stream Processing Cluster

Integrate existing systems through CDC BI Tools

Search / Explore

Online & Mobile Apps

Search

Event Stream

Results Stream Processor

Reference / Models

Dashboard Location

Social

Click stream

Sensor Data

Mobile Apps

Weather Data

Microservice Cluster

Microservice State

{ }

API

Event Stream

Event Stream

Event Store

Billing & Ordering

CRM / Profile

Marketing Campaigns

Building event-driven Microservices with Kafka Ecosystem

Page 37: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

What about (historical) data analytics?

Building event-driven Microservices with Kafka Ecosystem

Page 38: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Streaming & (Big) Data Analytics Architecture

Event Stream

Hadoop Clusterd Hadoop Cluster

Big Data Cluster

Parallel Processing

Storage

Storage

Ra

w

Re

fin

ed

Results

Microservice Cluster

Microservice State

{ }

API

Stream Processing Cluster

Stream Processor

State

{ }

API

Event Stream

Event Stream

SQL

Search

BI Tools

Enterprise Data Warehouse

Search / Explore

Online & Mobile Apps

Event Store

Service

Location

Social

Click stream

Sensor Data

Mobile Apps

Weather Data

Billing & Ordering

CRM / Profile

Marketing Campaigns

File Import / SQL Import

Building event-driven Microservices with Kafka Ecosystem

Page 39: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Summary

Building event-driven Microservices with Kafka Ecosystem

Page 40: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Hadoop Clusterd Hadoop Cluster

Big Data

Summary

Billing & Ordering

CRM / Profile

Marketing Campaigns

SQL

Search

BI Tools

Enterprise Data Warehouse

Search / Explore

Online & Mobile Apps

File Import / SQL Import

Event Hub

Parallel Processing

Storage

Storage

Ra

w

Re

fin

ed

Results

Microservice State

{ }

API

Stream Processor

State

{ }

API

Event Stream

Event Stream

Service

Location

Social

Click stream

Sensor Data

Mobile Apps

Weather Data

Stream Processing

Microservices

Page 41: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Summary

Building event-driven Microservices with Kafka Ecosystem

• service autonomy is key in a Microservices Architecture!

• not all communication need to be synchronous => separate into

• commands

• events

• queries

• Kafka is well suited as an event broker / event store

• brings many more interesting features beyond just “message passing”

Page 42: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

References

Building event-driven Microservices with Kafka Ecosystem

Microservices Blog Series, Ben Stopford, Confluent:

• https://www.confluent.io/blog/tag/microservices

Apache Kafka for Microservices: A Confluent Online Talk Series:

• https://www.confluent.io/landing-page/microservices-online-talk-series/

Turning the database inside-out with Apache Samza, Martin Kleppmann, Con

• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/

Immutability Changes Everything, Pat Helland, Salesforce:

• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:

• https://www.youtube.com/watch?v=B1-gS0oEtYc

Page 43: Ereignis-getriebene Microservices mit Apache Kafka...mit Apache Kafka Guido Schmutz JUG Saxony Day - 28.9.2018 @gschmutz guidoschmutz.wordpress.com Guido Schmutz Working at Trivadis

Technology on its own won't help you. You need to know how to use it properly.

Building event-driven Microservices with Kafka Ecosystem