modern data architecture at scale in the cloud : best ... · §multiple data sources §multiple...

CONFIDENCEAT EVERY TURNCONFIDENCEAT EVERY TURN

Prakriteswar Santikary, PhDVice President and Global Chief Data Officer

MODERN DATA ARCHITECTURE AT SCALE IN THE CLOUD : BEST PRACTICES OF SERVERLESS, LAMBDA AND MICROSERVICES ARCHITECTURE

ABOUT ERT

│ © Copyright ERT 20182

Founded in 1977; privately held

Supporting Pharmacos, Biotechs & CROs around the world

Operations in 12 countries

2500+ employeesMINIMIZING RISK &

UNCERTAINTY, SO

YOU CAN MOVE

AHEAD QUICKLY

WITH CONFIDENCE

Agenda

• Definition• Characteristics• Advantages• Disadvantages

Microservices Architecture


Serverless Architecture


Monolithic Architecture

| Copyright ERT 20183

01

Monolithic Architecture – a refresher

TIPS ; Fresh business idea, needs quicker turn around, small engineering team – start with monolith – build simple and lightweight application

Monolithic Architecture – Weaknesses and challenges

6

SOFTWARE COMPLEXITY HAS BEEN GROWING STEADILY


Forces behindMicroservicesArchitecture

§ Multiple data sources§ Multiple channels§ Multiple data types§ Scale requirements§ Real time insight§ Big Data (unstructured)

§ Small incremental release§ Continuous deployment§ Continuous delivery§ Quick time to market§ Competitive pressure

§ New Programming language§ New Programming framework§ NoSQL databases§ API-first design§ Cloud-first design

§ Focus on code, not on infrastructure – less IT

§ Serverless deployment§ Pay per use§ Cost reduction

World of Big Data

Cloud/Serverless computing

DevOps and Automation01 02

0304 Emerging Technology

Monolith to Microservices Transition – loosely coupled, independent services communicating via lightweight protocol

TIPS : Microservices architecture is not a silver bullet for a bad software design. Design discipline must be exercised as to when to use monolith Vs Microservices.

Monolith is hard to scale; Microservices enable y-axis scaling by breaking the application into smaller independent services. Each service is indepedently scaled based on need, enabling horizonal scaling.

Microservices Logical Architecture

Let’s take a n-tier e-commerce application to explain monolith and microservices

11

BREAK IT DOWN INTO MICROSERVICES


12

MICROSERVICES ARCHITECTURE – A PROCESS VIEW


13

MICROSERVICES ARCHITECTURE- ADVANTAGES


14

MICROSERVICES ARCHITECTURE- DISADVANTAGES


15

MICROSERVICES ARCHITECTURE DESIGN PATTERNS


MicroservicesBest

Practice

§ By Business Capability§ By business domain§ Single responsibility principle§ Independently deployable§ Small service/small team§ You build it, you own it§ Domain-driven design

§ Service per host/VM§ Multiple services per host/VM§ Serverless Deployment§ On-premise§ Hybrid (with Cloud)

§ Polyglot persistence§ Database per service§ Data consistency§ Event sourcing§ Eventual consistency§ CAP theorem

§ Log Aggregation§ Audit logging§ Distributed Tracing§ Exception Tracking§ Health-check API

Decomposition Pattern

Monitoring/Tracing Pattern

Deployment Pattern01 02

0304 Data Management Pattern

16

MICROSERVICES ARCHITECTURE – BEST PRACTICES


Microservices Architecture – a summary view

• Decentralized• Independent• Do one thing well• Polyglot• API-first Design• You build it; You own it

Benefits

• Agility• Innovation• Quality• Scalability• Availability

Challenges

• Distributed systems• Monolith->Microservices

transition not easy• Organizational issues

(DevOps)• Skillsets

Characteristics


TIPS : Microservices is not a silver bullet. You still need to follow sound software design principles.

Serverless ArchitectureFocus on your application, not on infrastructure

• No server to maintain• No server to provision• Focus on code• Ephemeral; Stateless• Event-driven• Horizontal scaling (auto)• Pay per use; no idle time

Benefits

• Faster innovation• Lower operational cost• Reduce operational

management• Speed to market• Fosters innovation

Challenges

• Skillsets• Vendor lock-in concerns• Lack of operational tools• Architectural complexity• Restriction on execution time• Restriction on memory

Characteristics


TIPS : Build scalable platform and applications using a combination of serverless and microservices in the cloud

SERVERLESS ARCHITECTURE – FEW EXAMPLES

| Confidential – Internal Use Only

SERVERLESS ARCHITECTURE USING MICROSERVICES

| Confidential – Internal Use Only

Lambda Architecture – execution of real time and batch together

• Real time data processing• Batch data processing• Merging of real time and

batch in single architecture

Benefits

• Serve both real and batch use cases simultaneously

Challenges

• Architectural complexity• skillsets

Characteristics


TIPS : One architecture supports both real time and batch

22

LAMBDA ARCHITECTURE- PHYSICAL VIEW ON AWS (SERVERLESS)


KEY TAKEAWAYS FROM TODAY’S DISCUSSION


Monolithic architecture

Simple application; quick turn around – good for simple lightweight application – validate ideas quickly. Not ideal for complex and growing applications – scalability is the biggest issue.

MicroservicesComplex application; scalability need is high; strong engineering skillsets available; distributed computing is complex; Microservices is no silver bullet – still depends on sound software design.

Serverless ArchitectureCloud-native; no HW to maintain; focus on code, not on infrastructure; works well with microservices;

24

QUESTIONS?

?…Prakriteswar Santikary, [email protected]

modern data architecture at scale in the cloud : best ... · §multiple data sources §multiple...

Documents