data microservices in the cloud
TRANSCRIPT
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Data Microservices in the Cloud
By Mark Pollack
HTTP
JMS
Kafka
File
HDFS
Cassandra
HAWQ
JDBC
?
Real Time Analytics
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/3
Spring XD Streams
Container
Container
gpfdist
Cassandrajms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandrastream2 = jms | gpfdist
On Metal/VMs
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/4
Spring XD Limitations• How to scale up/down instances at runtime?• How to upgrade/downgrade module instances at runtime?• How to specify resources unique to each module, e.g. memory?• Container architecture lead to parent/child class loader issues• Too many libraries in root classpath
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/5
Refactoring to a Microservice ArchitectureFrom multiple modules embedded in a container
to standalone executable applications
From our own runtime to delegating to existing platforms
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/6
Data Microservices• Stand-alone, production grade applications focused on data
processing• Communicating with ‘lightweight mechanisms’ – messaging
middleware• ‘Event Driven’ - Microservices“Write programs that do one thing and do it well.”
“Write programs to work together.”“Write programs to handle text streams, because that is a universal interface.”
$ cat book.txt | tr ' ' '\ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/7
Application Types• Long lived Stream applications
• Spring Cloud Stream • Short lived Task Applications
• Spring Cloud Task
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/8
Spring Cloud Stream• Spring Boot based event-driven microservice framework• Opinionated primitives for streaming applications
• Persistent Publish/Subscribe semantics• Consumer Groups• Partitioning
• Pluggable messaging middleware bindings• Programming model focused on input/output objects
• Adaptable to different event processing APIs
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Stream Applications
9
Cassandra
java –jar cassandrasink-1.0.0.RELEASE.jar --ingestQuery=<some cql> --spring.cassandra.keyspace=tweetdata --spring.cloud.stream.bindings .input.destination=dataDest
http
java –jar twittersource-1.0.0.RELEASE.jar --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings .output.destination=dataDest
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/10
Stream orchestration in Spring Cloud Data Flow
ingest = twitterstream | cassandraStream Definition
|Spring Cloud Stream ApplicationsMap DSL names to maven/docker artifacts
dataflow:>stream create --name ingest --definition “twitterstream | cassandra” --deploy
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/11
Features: Persistent Messaging
HTTP log
s1.http DLQ
Message Broker
• Production Ready• RabbitMQ• Kafka
• Experimental • JMS• Google PubSub
• Planned • Kinesis
s1 = http | log
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/12
Features: Named Destinations
HTTP
JMS
S3
myInputDestination
s1 = http > :myInputDestination
s2 = jms > :myInputDestination
s3 = aws-s3 > :myInputDestination
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/13
Features: Consumer Groups
HTTP
s1.http
HDFS
s1 = http | hdfs
s2 = :s1.http > counter
COUNTER
group: s1 group: s2
HDFS
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/14
Simple Real Time Analyticstweets = twitterstream | hdfs
analytics = :ingest.twitterstream > field-value-counter --fieldName=lang
HTTP
s1.http
HDFS
COUNTER
Data Flow ServerREST API
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/15
Spring Cloud Stream Programming Model
@EnableBinding(Processor.class)public class TransformProcessor {
@StreamListener(“input”) @SendTo(“output”) public String transform(String s) { return s.toUpperCase(); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/16
Spring Cloud Stream Programming Model
@EnableBinding(Processor.class)public class TransformProcessor {
@StreamListener @Output(“output”) public Flux<String> transform(@Input(“input”) Flux<String> input) { return input.map(s -> s.toUpperCase()); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/17
Spring Cloud Stream Programming Model
@EnableBinding(Processor.class)public class TransformProcessor {
@StreamListener @Output(“output”) public Flux<WordCount> countWords(@Input("output") Flux<String> words) { return words.window(ofSeconds(5), ofSeconds(1)) .flatMap(window -> window.groupBy(word -> word) .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1) .map(count -> new WordCount(group.key(), count)))); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/18
Platform Runtimes
Docker Swarm Apache YARN
Apache Mesos + Marathon
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/19
Spring Cloud Data Flow Deployment Platforms
Data Flow ServerREST API
Deployer SPI
SCDF FloSCDF Shell
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/20
Spring Cloud Data Flow Streams
gpfdist
cassandrajms
http
stream1 = http | cassandrastream2 = jms | gpfdist
Message Broker
Data Flow Server DBPlatform Runtime
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/21
Deployment: Partitioning and Instance Count
http
http
work
work
work
hdfs
hdfs
hdfs
hdfs
Load Balancer
stream create s1 --definition “http | work | hdfs”
stream deploy s1 --propertiesFile ingest.properties
app.http.count=2app.work.count=3app.hdfs.count=4app.http.producer.partitionKeyExpression=payload.id
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/22
Deployment: Resource Management
http
http
work
work
work
app.work.spring.cloud.deployer.cloudfoundry.memory=2048
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/23
Spring Cloud Task• Spring Boot based framework for short lived processes
• Auto-configuration provides a task repository and pluggable data source
• Result of each process persists beyond the life of the task for future reporting
• Tasks can be any arbitrary short lived code• Well integrated with Spring Batch
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/24
Task Orchestration in Spring Cloud Data Flow>task create jdbchdfs –sql=‘select * from table’>task launch jdbchdfs
jdbc2hdfs
Data Flow Server DB Task NameStart TimeEnd TimeExit CodeExit MessageLast Updated TimeParameters
task-event
Message Broker
job-execution-eventsstep-execution-
eventsitem-read-eventsitem-process-eventsitem-write-eventsskip-events
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/25
Spring Cloud Data Flow Tasks
spark
Data Flow Server DB
http | task-launcher
sqoop
Message Broker
task-event
Platform Runtime
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/26
Spring Cloud Task Programming Model@SpringBootApplication@EnableTaskpublic class ExampleApplication {
@Bean public CommandLineRunner commandLineRunner() { return strings -> System.out.println("Executed at :" + new SimpleDateFormat().format(new Date())); }
public static void main(String[] args) { SpringApplication.run(ExampleApplication.class, args); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/27
Provided Applications
• ~60 stream and task apps• https://github.com/spring-cloud/spring-cloud-stream-app-starters• https://github.com/spring-cloud/spring-cloud-task-app-starters/
• Customize provided apps - http://start-scs.cfapps.io/ • Create new stream/task apps - http://start.spring.io/• Easy import of provided apps/tasks
• dataflow>import app --uri http://bit.ly/1-0-2-GA-stream-applications-kafka-maven
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/28
UI : Dashboard with Designer
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/29
XD to SCDF - Terminology
XD-Admin Data Flow Server(local, CF, YARN, k8s, Mesos)
XD-Container N/A
Modules Applications
Admin UI Dashboard
Message Bus Binders
Job Task
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/30
DEMO
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/31
Upcoming features• Some ‘porting’ from XD
• Batch Job DSL + Designer• Role based access
• Looking forward• Spring Cloud Sleuth• JavaDSL• In-place application version upgrades with Spinnaker• Application Groups• Polyglot• Expanded analytics with Redis and Python/R ecosystem• More provided apps/tasks
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/32
Related Talks• Building Resilient and Evolutionary Data Microservices – Tuesday
2:00pm• Cloud Native Java – Tuesday 2:00pm• Task Madness - Modern On Demand Processing – Tuesday 2:40pm• Spinnaker – Land of a 1000 Builds – Tuesday 5:00pm• Spring and Big Data – Tuesday 5:00pm• Migrating from Spring XD to Spring Cloud Data Flow – Thursday
10:10am• Orchestrate All the Things! with Spring Cloud Data Flow – Thursday
11:10am• Cloud Native Streaming and Event-Driven Microservices – Wednesday
4:20pm
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/33
Get Started…• http://cloud.spring.io/spring-cloud-dataflow/• http://cloud.spring.io/spring-cloud-stream/• http://cloud.spring.io/spring-cloud-task/• https://github.com/spring-cloud/spring-cloud-deployer
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Learn More. Stay Connected.
http://cloud.spring.io/spring-cloud-dataflow/
@springcentralspring.io/blog
@pivotalpivotal.io/blog
@pivotalcfhttp://engineering.pivotal.io
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/35
Spring XD Log Module<channel id="input"/> <logging-channel-adapter channel="input" level="$
{level}" logger-name="xd.sink.${name}" expression="$
{expression}"/>
log.xml
info.shortDescription = Logs the message payload. options_class
=o.s.xd.dirt.modules.metadata.LogSinkOptionsMetadata
log.properties
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/36
Spring Cloud Data Flow Log Application@EnableBinding(Sink.class)
@EnableConfigurationProperties(LogSinkProperties.class)public class LogSinkConfiguration {
@Autowiredprivate LogSinkProperties properties;
@Bean@ServiceActivator(inputChannel = Sink.INPUT)public LoggingHandler logSinkHandler() {
LoggingHandler loggingHandler = new LoggingHandler(this.properties.getLevel().name());
loggingHandler.setExpression(this.properties.getExpression());
loggingHandler.setLoggerName(this.properties.getName());return loggingHandler;
}}
LogSinkConfiguration.java