big data and messaging with spring
DESCRIPTION
TRANSCRIPT
![Page 1: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/1.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
with Artem Bilan
by Pivotal
Big Data & Messaging
![Page 2: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/2.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Artem Bilan
Spring Integration Team
Spring FrameworkSpring AMQP
Spring XDReactor
https://github.com/artembilanhttp://www.linkedin.com/in/cleric
https://spring.io/team/artembilan http://stackoverflow.com/users/2756547
![Page 3: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/3.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Messaging and why do I care?
![Page 4: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/4.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Messaging is very simple
Headers
Payload
public static void main(String[] args) {...}
![Page 5: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/5.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Messaging in Spring IO
![Page 6: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/6.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Spring Integration
Meet IoC!
Uses
Written
![Page 7: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/7.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
The heart of Spring Integration
– Endpoints (Filters) connected through– Channels (Pipes) exchanging– Message
![Page 8: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/8.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Spring Integration Message Flow
![Page 9: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/9.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
How does it work?
![Page 10: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/10.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
What is a Big Data application? Not just writing a few MR or Hive/Pig jobs.
The full lifecycle involves
• Ingestion
• Stream Processing
• Workflow Orchestration
• Enterprise Integration
• Export
• Horizontal scalable deployment
How do you write one of these?
• Status quo is to combine different projects – not ideal
![Page 11: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/11.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Spring XD
• Unified Platform• Developer Productivity• Modular Extensibility• Distributed Architecture• Portable Runtime • Hadoop Distribution Agnostic• Proven Foundation • XD = ‘eXtreme Data’
![Page 12: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/12.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
TapsCompute
HDFS
Wor
kflow
Export
Spring XD Runtime
Inge
st
Jobs
Export
Files Sensors Mobile Social
RDBMS
NoSQL
R, SAS
Spring XD Shell
Streams
Redis
Gemfire
Predictive modeling
Spring XD
![Page 13: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/13.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
http | filter | file
Stream Processing Model
How can we make this easier?
Non-linear stream definitions also supported
![Page 14: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/14.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Runtimes
http | filter | file
Rabbit, Redis, (Pluggable)
XD Admin
CLUSTERED NODE
FilterModule
CLUSTERED NODE
HTTPModule
CLUSTERED NODE
FileModule
In MemoryTransport
http | filter | file
SINGLENODE
AllModules
![Page 15: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/15.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
StreamsMessageStore
HTTPTailFileMail
TwitterGemfireSyslog
TCPJMS
RabbitMQMQTT
FilterTransformer
SplitterAggregator
Groovy ScriptCounters
HTTPJSON
Java Code
FileHDFSJDBCTCPMail
RabbitMQGemfireSplunkMQTT
Dynamic Router
![Page 16: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/16.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Taps “Listen” to data from another stream
• Other stream is unaffected by the tap and unaware of its presence
• EAI ‘Wiretap’
![Page 17: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/17.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Analytics
• Simple Counter• Field Value Counter
– Count occurrences of named fields
• Aggregate Counter– Pre-aggregate counts in time
buckets
• Gauge– last value
• Rich Gauge– Last value, running average,
min/max
• Abstract API• Implementations for
– Memory– Redis
• Looking at– Pattern– JPMML
![Page 18: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/18.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Jobs Jobs are a directed graph of steps
Steps
• Copy or Process data
• Files, Databases, MR, Pig, Hive, Cascading
Step executions are persisted
• Checkpointing with restart
• Rich error handling capabilities
Single node or distributed with data partitioning
Jobs can be Triggered from streams
Executing jobs generate a stream of event data
![Page 19: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/19.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
S P R I N G X D
Demo
![Page 20: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/20.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Learn More. Stay Connected.
• Spring IO: https://spring.io/platform• GitHub: https://github.com/spring-projects• Spring Integration: http://projects.spring.io/spring-integration• Spring XD: http://projects.spring.io/spring-xd• Reactor: https://github.com/reactor• EIP: http://www.eaipatterns.com• Spring Batch: http://projects.spring.io/spring-batch• Spring for Hadoop: http://projects.spring.io/spring-hadoop
![Page 21: Big data and messaging with Spring](https://reader033.vdocument.in/reader033/viewer/2022061223/54c6380f4a79594e588b45ad/html5/thumbnails/21.jpg)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
ありがとう