data microservices with spring cloud stream, task, and data flow #jsug #springday

‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.

Data Microservices with Spring Cloud Stream, Task, and Data Flow

Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug

‹#›© 2016 Pivotal Software, Inc. All rights reserved.

Who am I ?• Toshiaki Maki (@making) http://blog.ik.am

•Sr. Solutions Architect @Pivotal

•Spring Framework enthusiast

bit.ly/hajiboot2

http://blog.ik.am

http://bit.ly/hajiboot2


Input

Motivation

HTTP

TCP

S3

JDBC

Rabbit

JMS

Twitter

Syslog

Output

HDFS

Cassandra

HAWQ

Greenplum

JDBC

S3

TCP

Gemfire

❓❔❓

• Real Time Analysis • Data Ingestion • ETL Process


Spring XD (eXtreme Data)


Spring XD (eXtreme Data)

🤔


Spring XD Architecture

Container Containergpfdist

Cassandra

jms

http

ZooKeeper

Message Broker

XD Admin

stream1 = http | cassandrastream2 = jms | gpfdist

On Metal/VMs




Cassandra

jms

http

ZooKeeper

Message Broker

XD Admin


On Metal/VMs

😇




Cassandra

jms

http

ZooKeeper

Message Broker

XD Admin


On Metal/VMs


Modern Platforms

Cloud Foundry YARN Kubernetes Mesos

Docker Swarm Nomad OpenShift


Cloud Native Redesign

•From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms


Data Microservices


Microservice for each data processing


Data Microservices



bound with Message Brokers


Data Microservices



bound with Message Brokers

on the modern platform


Spring Cloud Data Flow• Microservices-based Distributed Data Pipelines •Long Lived Stream Applications

•Short Lived Task Applications



•Short Lived Task ApplicationsSpring Cloud Stream




Spring Cloud Task




Spring Cloud Task

Orchestration Layer


StructureSpring Cloud Data Flow

Spring Cloud Deployer (SPI)

Spring Cloud Stream Spring Cloud Task

Spring Integration Spring Boot Spring Batch


StructureSpring Cloud Data Flow

Spring Cloud Deployer (SPI)

Spring Cloud Stream Spring Cloud Task

Spring Integration Spring Boot Spring Batch

• Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos


SCDF Deployment Platform

Spring Cloud Data Flow Server

Deployer SPI

REST API

SCDF Shell SCDF Flo


Spring Cloud Data Flow Architecture

gpfdist

cassandra

jms

http


Message Broker

Data Flow Server DB

Platform Runtime


Spring Cloud Stream


Spring Cloud Stream•Event-driven microservice framework

•Built on battle-tested components (Spring Boot / Spring Integration)

•Opinionated primitives for streaming applications

•Persistent Pub/Sub

•Consumer Groups

•Partitioning Support

•Pluggable messaging middleware bindings

source | processor | sink


JJUG CCC in 2 weeks


Spring Cloud Stream Applications

Twitter Stream

Cassandra

java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest

Source

Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest


Spring Cloud Stream Applications

Twitter Stream

Cassandra

java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest

Source

Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest

TwitterStream Cassandraingest


Message Binders• @EnableBinding•Binder Implementations

•Production-Ready

•Rabbit MQ

•Apache Kafka

•Experimental

• JMS

•Google PubSub


Programming Model (Sink)@SpringBootApplication @EnableBinding(Sink.class) public class DemoSinkApp { @StreamListener(Sink.INPUT) void receive(Message<String> message) { System.out.println("Received " + message); } public static void main(String[] args) { SpringApplication.run(DemoSinkApp.class, args); } }


Sink

Properties (Sink)

spring.cloud.stream.bindings.input.destination=demo-strm

demo-strm in

put


Programming Model (Source)@SpringBootApplication @RestController@EnableBinding(Source.class) public class DemoSourceApp { @Autowired Source source; @GetMapping void send(@RequestParam String text) { source.output() .send(MessageBuilder.withPayload(text).build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); } }


Programming Model (Source)@SpringBootApplication @RestController@EnableBinding(Source.class) public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() { return () -> MessageBuilder.withPayload("Hi").build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); }}


Properties (Source)

spring.cloud.stream.bindings.output.destination=demo-strm

demo-strm

Source

outp

ut


Binder (RabbitMQ)<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId></dependency>

demo-strm


Binder (Apache Kafka)<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId></dependency>

demo-strm


Sink

Pipeline

demo-strm in

put

Source

outp

ut

source | sink


Programming Model (Processor)@SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) void receive(String text) { return "[[" + text + "]]"; } public static void main(String[] args) { SpringApplication.run(DemoProcessorApp.class, args); } }


Properties (Processor)

spring.cloud.stream.bindings.output.destination=my-source

spring.cloud.stream.bindings.input.destination=my-sourcespring.cloud.stream.bindings.output.destination=my-proc

spring.cloud.stream.bindings.input.destination=my-proc

Source

Processor

Sink


Pipeline

my-source

Source

outp

utsource | processor | sink

Processor

outp

ut

inpu

t

my-proc

Sink

inpu

t


Reactive API Support@SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT) public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) { return stream.map(text -> "[[" + text + "]]"); } public static void main(String[] args) { SpringApplication.run(DemoProcessorRxApp.class, args); } }


Reactive API Support

@StreamListener @Output(Processor.OUTPUT) public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) { return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group));}


Spring Cloud Stream App StartersSource Processor Sink

file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit

s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream

bridge filter groovy-filter groovy-transform httpclient pommel scriptable-transform splitter tcp-client transform

aggregate-counter cassandra counter field-value-counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc

log rabbit redis-pubsub router s3 sftp task-launcher-local task-lancher-yarn tcp throughput web socket

http://cloud.spring.io/spring-cloud-stream-app-starters/



Stream Orchestration in SCDFingest = twitterstream | cassandra

dataflow:> stream create --name=ingest --definition="twitterstream | cassandra"dataflow:> stream deploy --name=ingest

|


Import apps

dataflow:> app import --uri http://bit.ly/1-0-4-GA-stream-applications-rabbit-maven

rabbit mavenkafka docker

stream-applications-


http://bit.ly/1-0-4-GA-stream-applications-rabbit-maven



Register your own apps

dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar

dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0

dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0

http://example.com/foo-source-1.0.jar

maven://com.example:foo-source:1.0


Instance Count

dataflow:> stream create --name=s1 --definition="http | work | hdfs"dataflow:> stream deploy --name=s1 --properties="app.http.count=2,app.work.count=3,app.hdfs.count=4"


Instance Count

http

work

hdfs

http

work

work

hdfs

hdfs

hdfs

Load

Bal

ance

r


Resource Management

dataflow:> stream deploy --name=s1 --properties="app.work.spring.cloud.deployer.cloudfoundry.memory=2048"


Resource Management

http

http

work

work

work

hdfs

hdfs

hdfs

hdfs


Spring Flo http://projects.spring.io/spring-flo/

http://projects.spring.io/spring-flo/


Stream Core Features•Persistent Pub-Sub •Consumer Group •Partitioning Support


Persistent Pub-Sub

HTTP Average Top Ns1.http s1.ave

Message Broker


Persistent Pub-Sub

HTTP Average

HDFS

Top Ns1.http s1.ave

Message Broker


Persistent Pub-Sub

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker


Persistent Pub-Sub

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker

{"id":1, "temperature":38}


Persistent Pub-Sub

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker

{"id":1, "temperature":38}{"id":1, "temperature":38}


Persistent Pub-Sub

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker





Persistent Pub-Sub

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker


{"id":1, "temperature":38} {"id":1, "temperature":38}


HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker

AverageAverage

HDFSHDFS


HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker


AverageAverage

HDFSHDFS


HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker



AverageAverage

HDFSHDFS






HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker



AverageAverage

HDFSHDFS





😩


HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker



AverageAverage

HDFSHDFS





😩

Consumer Group


HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker



AverageAverage

HDFSHDFS





😩spring.cloud.stream.bindings.<channelName>.group=ave

Spring Cloud Stream

group is configured by default 😁

Spring Cloud Data Flow

Consumer Group


Consumer Group

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker

AverageAverage

HDFSHDFS

group=ave

group=hdfs


Consumer Group

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker

AverageAverage

HDFSHDFS

group=ave

group=hdfs


{"id":1, "temperature":38} 🤓


Consumer Group

HTTP Average

HDFS

Top N

FaultDetection

s1.http s1.ave

Message Broker

AverageAverage

HDFSHDFS

group=ave

group=hdfs


{"id":1, "temperature":38} 🤓consumer group subscriptions

are durable 😁


HTTP

Average

s1.http

Average


HTTP

Average

s1.http

Average

{"id":1, "temperature":38}{"id":2, "temperature":41}{"id":2, "temperature":42}



HTTP

Average

s1.http

Average





HTTP

Average

s1.http

Average





😩


HTTP

Average

s1.http

Average





😩

Partitioning Support(Stateful Stream)


HTTP

Average

s1.http

Average





😩stream deploy --name=xxxx --properties="app.http.producer.partitionKeyExpression=payload.id"

spring.cloud.stream.bindings.<channelName>.producer.partitionKeyExpression=payload.id

Spring Cloud Stream

Spring Cloud Data Flow




HTTP

Average

s1.http

Average



HTTP

Average

s1.http

Average

{"id":1, "temperature":38}{"id":2, "temperature":41}{"id":2, "temperature":42}




HTTP

Average

s1.http

Average






HTTP

Average

s1.http

Average




🤓


Simple Real Time Analyticstweets = twitterstream | hdfs

analytics = :ingest.twitterstream > field-value-counter --fieldName=lang

HTTP

s1.http

HDFS COUNTER

Data Flow ServerREST API


Demo (Twitter Stream!)


Spring Cloud Task


Spring Cloud Task•Spring Boot based framework for short lived task

•Spring Boot's CommandLineRunner or Spring Batch

•Result of each process persists in Task Repository

•Well Integrated with Spring Batch (Job Repository)


Programming Model (CLR)@SpringBootApplication @EnableTaskpublic class DemoTaskApp { @Bean CommandLineRunner clr() { return args -> System.out.println("Task!"); } public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); } } spring.application.name=hello


Task Execution ListTask Name ID Start Time End Time Exit Code

hello 3 Tue Nov 15 16:49:08 JST 2016

Tue Nov 15 16:49:08 JST 2016

0

hello 2 Tue Nov 15 16:48:54 JST 2016

Tue Nov 15 16:48:55 JST 2016

0

hello 1 Tue Nov 15 16:48:23 JST 2016

Tue Nov 15 16:48:23 JST 2016

0


Programming Model (Spring Batch)@SpringBootApplication @EnableBatchProcessing@EnableTaskpublic class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory; @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()} public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); }}

https://github.com/making/cf-spring-batch-demo

spring.application.name=hello-batch

https://github.com/making/cf-spring-batch-demo


Task Execution ListTask Name ID Start Time End Time Exit Code

hello-batch

5 Tue Nov 15 17:28:49 JST 2016

Tue Nov 15 17:28:49 JST 2016

0

hello-batch

4 Tue Nov 15 17:27:56 JST 2016

Tue Nov 15 17:27:57 JST 2016

0


Job Execution ListID Task ID Start Time Step Execution Count Definition Status

2 5 Tue Nov 15 17:28:49 JST 2016

2 Created

1 4 Tue Nov 15 17:27:56 JST 2016

2 Created


Step Execution ListID Step Name Job Exec Id Start Time End Time Status

1 step1 1 Tue Nov 15 17:28:49 JST 2016

Tue Nov 15 17:28:49 JST 2016

COMPLETED

2 step2 1 Tue Nov 15 17:27:56 JST 2016

Tue Nov 15 17:27:57 JST 2016

COMPLETED


Task Orchestration in SCDF>task create hello --definition="timestamp --format=¥"yyyy¥"">task launch hello

timestamp

Data Flow Server DB Task NameStart TimeEnd TimeExit CodeExit MessageLast Updated TimeParameters

Message Broker


Task Execution List


Job Execution List


Job Execution Details


Spring Cloud Data Flow -> Taskhttp | tasklaunchrequest --uri=... | task-lancher

timestamp

Data Flow Server DB

Message Broker


Demo (Scalable Tasks)


Pivotal Cloud Foundry

PDF PDF

PDF PDF

S3

Spring Cloud Data FlowTask

TaskTask

Task

PDF

PDF

Spring Cloud Task Spring Batch

index

index

source sink


Getting Started with SCDF on Cloud Foundry


Install PCF Dev• https://docs.pivotal.io/pcf-dev

•Cloud Foundry on your laptop

• Included

•Redis / RabbitMQ / MySQL

•Spring Cloud Services

• Install with cf dev start

https://docs.pivotal.io/pcf-dev


cf create-service p-mysql 512mb df-mysqlcf create-service p-rabbitmq standard df-rebbitmqcf create-service p-redis shared-vm df-rediscf push dataflow-server -p spring-cloud-dataflow-server-cloudfoundry-1.0.1.RELEASE.jar

Deploy SCDF Server on PCF Dev


Tutorial

•📗 https://blog.ik.am/entries/396

•💎 https://github.com/making-demo-scdf/spring-cloud-dataflow-cookbook

https://blog.ik.am/entries/396

https://github.com/making-demo-scdf/spring-cloud-dataflow-cookbook


Upcoming Features•Some ‘porting’ from XD

• Batch Job DSL + Designer • Role based access

• Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks

•SCDF Tile for Pivotal Cloud Foundry


Thanks!•References

• http://www.slideshare.net/SpringCentral/data-microservices-in-the-cloud ✨

• https://cloud.spring.io/spring-cloud-dataflow/

• http://cloud.spring.io/spring-cloud-stream/

• http://cloud.spring.io/spring-cloud-stream-app-starters/

• https://github.com/spring-cloud/spring-cloud-task/

• http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry/

http://www.slideshare.net/SpringCentral/data-microservices-in-the-cloud

https://cloud.spring.io/spring-cloud-dataflow/

http://cloud.spring.io/spring-cloud-stream/


https://github.com/spring-cloud/spring-cloud-task/

http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry/

data microservices with spring cloud stream, task, and data flow #jsug #springday

Technology