lessons learned - migrating from spring xd to spring cloud data flow

© 2016 zData Inc.© 2016 zData Inc.

Spring Cloud Data Flow

Lessons Learned Migrating from SpringXD to Spring Cloud Data Flow

© 2016 zData Inc.

zData provides Big Data consulting and solutions for both commercial and enterprise corporations, employing experts in all areas of the field from software engineers to data scientists. They work with top hardware and software providers for on-site and off-site consulting, managed services, trainings, and long term scalable data solutions both on-premise and in the cloud.

Introduction to zData

Partners

Certifications

© 2016 zData Inc.

Agenda

• A Common Use Case• Comparing SpringXD to Spring Cloud

Data Flow• Building a Data Pipeline• Demonstration• Conclusion

VS.

© 2016 zData Inc.

Gaining analytic insights from real-time data streams is a multi step process. Data must be captured from disparate and unreliable data sources, transformed based on business rules, analyzed using machine learning models, and streamed to web and mobile applications. Coordinating the movement and execution of these data pipelines is a challenging problem. In this session we will point out key differences between two tools that solve this coordination problem: Spring Cloud Data Flow and its predecessor SpringXD. We will highlight the lessons learned as we migrated between these two tools.

Use Case Description

© 2016 zData Inc.

Desired Pipeline

? ?

?

?

SpringXD OR Spring Cloud Data

© 2016 zData Inc.

Technologies Description

Orchestration via provided Spring XD Runtime

Spring Cloud Data FlowDevelopment for the successor to SpringXDwas started in mid 2015 with a GA release on July 12, 2016.

Spring XDOriginally released on July 30, 2014, final release was on February 24, 2016. Now officially in maintenance mode.

VS.

Runs in any modern application runtime: CloudFoundry, Kubernetes, YARN, Mesos

Must provide and maintain your own message bus Use application runtime services as message bus

Logs spread across many Admin and container servers Logs centralized and exposed via application runtime utilities such as cf-cli

Manually scale transform modules/containers to relieve bottlenecks

Easy (or even automatic) scaling provided by application runtime

© 2016 zData Inc.

Streaming Pipeline Architecture

© 2016 zData Inc.

Creating a Data Pipeline

© 2016 zData Inc.

Creating a Data Pipeline

ingeststream:http | transform | pmml | jdbc

appstream:ingeststream:pmml > websocket


Get Ready…

DEMO


We’re Hiring

• Big Data Cloud Architect

• Big Data Engineer

• Big Data Solutions Architect

Pioneer, design, architect, implement, and administer Big Data solutions

Submit your resume to [email protected]

TEXT 208.850.5582 to interview at Spring One

lessons learned - migrating from spring xd to spring cloud data flow

Technology