microservicesmicroservices 2018 7 live stream processing / fast data architecture fast data is data...
TRANSCRIPT
![Page 1: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/1.jpg)
MicroservicesBuilding Blocks
Microservices 2018 1
![Page 2: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/2.jpg)
Content
1. History of Data Processing & Services2. Beyond Microservices3. Monolithic vs. Microservice4. Important Concepts
Microservices 2018 2
![Page 3: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/3.jpg)
A Definition Data Processing / Services
Data processing is, generally, "the collection and manipulation of items of data to produce meaningful information." In this sense it can be considered as a subset of information processing, "the change (processing) of information in any manner detectable by an observer."
In the contexts of software architecture the term service refers to a software functionality or a set of software functionalities with a purpose that different clients can reuse for different purposes, together with the policies that should control its usage.
Microservices 2018 3
![Page 4: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/4.jpg)
History Data Processing
Microservices 2018 4
![Page 5: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/5.jpg)
Offline Batch Processing
In computing, batch processing refers to a computer working through a queue or batch of separate jobs (programs) without manual intervention (non-interactive).
Microservices 2018 5
![Page 6: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/6.jpg)
MapReduce
MapReduce is a programming model suitable for processing of huge data. Hadoop is capable of running MapReduce programs written in various languages.
Microservices 2018 6
![Page 7: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/7.jpg)
Lambda Architecture
The LA aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being able to serve a wide range of workloads and use cases, and in which low-latency reads and updates are required. The resulting system should be linearly scalable, and it should scale out rather than up.
Microservices 2018 7
![Page 8: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/8.jpg)
Live Stream Processing / Fast Data Architecture
Fast data is data in motion. It has different characteristics, different requirements and needs different technology to deal with it, technology that has the ability to analyze, decide, and act on – that is, offer recommendations, make decisions, or take other actions – as fast as data arrives, typically in milliseconds.
Microservices 2018 8
![Page 9: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/9.jpg)
History of Services
Microservices 2018 9
![Page 10: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/10.jpg)
Client Server
e.g. FatClient / ThinClient !
VV Semester 4
Microservices 2018 10
![Page 11: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/11.jpg)
CORBA, DCOM & RMI
CORBA – Common Object Request Broker ArchitectureDCOM – Distributed Component Object ModelJava/RMI – Java/Remote Method Invocation
Microservices 2018 11
![Page 12: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/12.jpg)
EJB & N-Tier
Enterprise Java Beans (EJB) is a development architecture for building highly scalable and robust enterprise level applications to be deployed on J2EE compliant Application Server such as JBOSS, Web Logic. EJB 3.0 is being a great shift from EJB 2.0 and makes development of EJB based applications quite easy.
Microservices 2018 12
![Page 13: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/13.jpg)
SOA & ESB
An enterprise service bus (ESB) implements a communication system between mutually interacting software applications in a service-oriented architecture (SOA). It implements a software architecture as depicted in the picture.
Microservices 2018 13
![Page 14: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/14.jpg)
HTTP-API & REST
— Resources— Nouns, not Verbs— Coarse Grained, not Fine Grained— Architectural style for use-case
scalability— Keep it Simple
— Collection Resource /users/— Instance Resource /users/1
— Behavior— GET, PUT, POST, DELETE
("CRUD"), Head— Use HTTP Response Codes— Use query params for offeet and limit!
VV Semester 4Microservices 2018 14
![Page 15: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/15.jpg)
Microservices
— A microservice is a software development technique that structures an application as a collection of loosely coupled services.
— It parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently
Microservices are NOT a Silver Bullet
Microservices 2018 15
![Page 16: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/16.jpg)
Beyond MicroservicesToday Data Processing and Services have shared requirements:
— Available— Elastically scalable— Resilient & self-healing— Loosely coupley & upgradeable
Microservices 2018 16
![Page 17: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/17.jpg)
Reactive Systems1
Available (reponse), under failure (resilience) and load (elasticity), by being message-driven.
1 The Reactive Manifesto.
Microservices 2018 17
![Page 18: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/18.jpg)
Reactive Streams2
Provides a standard for asynchronous stream processing with non-blocking back pressure.
=> Fast Data Services are based on Reactive Streams.
2 Reactive Streams & java.util.concurrent.Flow JDK9+
Microservices 2018 18
![Page 19: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/19.jpg)
Future: Convergence
— Evented Services are stream processing stages. — Events-First DDD & EventStorming— Data-driven Continuous Deployment— User of the right tools for each component— Microservices are a part of a streaming pipeline— A pipeline can now be exposed as Microservices— We can indepedently upgrade parts of the pipeline
--> And then extend to the Edges! // Not this lecture!
Microservices 2018 19
![Page 20: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/20.jpg)
Monolithic vs Microservice
Microservices 2018 20
![Page 21: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/21.jpg)
Monolithic vs. Microservice
— Most of the time just one huge project with one data source for all logic
— Scale out in monolithic systems is hard and often not possible
Microservices 2018 21
![Page 22: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/22.jpg)
Monolithic vs. Microservice
— Services are sliced into small parts. Every service should use its own data source.
— Microservices are much more complex, but you can easily scale up and out.
— You need a high automation level
Microservices 2018 22
![Page 23: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/23.jpg)
Pro Microservice
— scalable (up & out)— replaceability— inhomogeneous techstack— bounded context— loosely coupled— Independent services & teams— fast deployment and roll out— caching— highly automated
Microservices 2018 23
![Page 24: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/24.jpg)
Con Microservice
— Good operation is hard— Monitoring— Logging— Deployment— Cluster-Setup— Storage— Backup— Recovery
— tech stack— cascading failures— latency
Microservices 2018 24
![Page 25: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/25.jpg)
Important Concepts
Microservices 2018 25
![Page 26: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/26.jpg)
Domain Driven Design (DDD)
Define services corresponding to Domain-Driven Design (DDD) subdomains. DDD refers to the application’s problem space as the domain.
A domain consists of multiple subdomains. Each subdomain corresponds to a different part of the business.
--> More about this Topic as external TalkMicroservices 2018 26
![Page 27: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/27.jpg)
Event Driven
— flow of the program is determined by events
— Needed for data exchange between different services in a cluster
— A lot of programming languages use the actor pattern. akka is one implementation for scala and java
— Deep dive in lesson "Persistence” we will look at Event Sourcing and CQRS.
Microservices 2018 27
![Page 28: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/28.jpg)
The Actor Model3
Actors communicate with each other by sending asynchronous messages. Those messages are stored in other actors' mailboxes until they're processed.
3 The Actor Model - http://www.brianstorti.com/the-actor-model/
Microservices 2018 28
![Page 29: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/29.jpg)
Scale Up & Scale Out
— Scale Up: Better Resources - e.g. CPU, RAM, Storage— Scale Out: More Nodes
Microservices 2018 29
![Page 30: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/30.jpg)
Load Balancer
How does the client of a service - the API gateway or another service - discover the location of a service instance? (Deep dive In lesson "Service Discovery & API Gateways")
Microservices 2018 30
![Page 31: MicroservicesMicroservices 2018 7 Live Stream Processing / Fast Data Architecture Fast data is data in motion. It has different characteristics, different requirements and needs](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0997727e708231d4279108/html5/thumbnails/31.jpg)
Circuit Breaker
How to prevent a network or service failure from cascading to other services?
A service client should invoke a remote service via a proxy that functions in a similar fashion to an electrical circuit breaker.
Microservices 2018 31