api world monolith to microservices · interest for the past several years, including 10+ years...

Madhu Chetuparambil & Snezana Sahter

San Jose, CA - October 11, 2019

API World Monolith to Microservices

2

Madhu

Chief Architect for FDP

In Intuit since 2016

Snezana Distinguished Architect in FDP and Identity

In Intuit since 2017

3

Snezana Sahter is a distinguished architect in Intuit focusing on the Financial Data

Platform and Identity. She has been with Intuit since 2017, expanding her e-commerce

and marketplace domain knowledge into FinTech.

Domain driven design and modeling APIs while dealing with legacy has been her area of

interest for the past several years, including 10+ years working at eBay as a principal

architect in Identity and Risk Management.

Originally from Serbia, she lives and works in San Francisco Bay Area.

Madhu Chetuparambil is the chief architect for the Intuit Financial Data Platform,

responsible for acquiring and managing consumer’s financial data from over 30,000 FI

across a variety of channels and authorization schemes. In this role, he leads the

technology, strategy and architectural decisions for the services and tools on this

platform.

Madhu has worked as a technologist since 1996 in a number of companies including

Intuit, eBay Inc, IBM and Transarc Corporation.

He received his Master's in Computer Science from Clemson University in 1996 and has

a Bachelor of Engineering, Comp Sc from the University of Mysore (1989 - 1993)

… or more formally

Context: Case for Change

Target State

Benefits and Challenges

Journey Line

Next Steps

Takeaway

Agenda

5

Sales

Accounting

Billing & Commerce

Payments

Payroll

Experience

Data Aggregation

Connectivity & Data

Transformation

Enrichment & Categorization

Support Tools

Personal Finance Mgmt

Money Movement

Ecosystem Capabilities

Depends on FDP

FDP

Time Tracking

Capital

Foundational Capabilities

...

...

...

Financial Data Platform

Identity

Customer Success

Financial Data Platform in Intuit

6

Runtime ● Runs in one runtime

● Limits: scalability, performance,

availability, resiliency

Team Productivity ● Longer release cycle (build, test, etc)

● Organizational scale

Security ● Outdated security models / standards

From “Application” to “Platform”

Signs of Monolith: Prior Application

Codebase Health ● Huge codebase / big ball of mud / too many

different features / all-in-one

● Thousands of unit tests (still test escape)

● High complexity

Integration ● No consistent APIs

● Tight coupling on modules, datastore

Quality ● Lot of functional tests (still gap in coverage)

● Band-aid solutions / workaround

7

Credential Set Account

Provider

Transaction

Profile (User/

Business)

Credential 0..n

0..n

0..n

1 1

1

1

0..n 1

1..n

Document (Tax, Statement,

etc) 0..n

Bill 0..n

1..n

Connectivity

Enablers

Financial Data

Opportunities & Considerations

• Taking another look across

the stack: data, application,

contracts

• Starting with the model to

enable flexibility

• Managing data quality and

variability across 30k

providers

• Enhancing end2end security

• Isolating the 3rd party

dependencies to increase

availability

• Compliance is still a must

The Approach to Breaking the Monolith

8


PC

I

Zo

ne

PC

I

Zo

ne

Mes

sag

ing

Bus

Providers (Financial Institutions)

DOCUMENT ACCOUNT CREDENTIAL

SET

PROFILE

BULK IMPORT/

FEEDS

API (DIRECT)

CONNECTIVITY WEB INTEGRATION RATE LIMIT

CREDENTIAL TRANSACTION PROVIDER

CATEGORIZATION ONBOARDING APP

SERVICE

ACQUISITION RESOLUTION &

RECONCILIATION

CHANNEL

MIGRATION

DATA

ACCESS

Log

gin

g

Co

nfig

urat

ion

Business Process Service

Entity Service

Connectivity Agent Service

The Target State

9

Having data in services bounded context enabled horizontal scale

Connectivity layer enabled growing number of supported providers and introducing new channels

Connectivity layer improved availability and scalability because the 3rd party impact for insulated from the rest of the platform

Asynchronous processing enabled in the “green” layer improved the performance and resiliency

Clear ownership enabled the organization to scale from 2 to 10 scrum teams

The products have more integration options with or without the experience

Team choice: technology and protocol that is the best suited for the task


PC

I

Zo

ne

PC

I

Zo

ne

DOCUMENT ACCOUNT CREDENTIAL

SET

PROFILE

BULK IMPORT API (DIRECT)

CONNECTIVITY WEB INTEGRATION RATE LIMIT

CREDENTIAL TRANSACTION PROVIDER

CATEGORIZATION ONBOARDING APP

SERVICE

ACQUISITION RESOLUTION &

RECONCILIATION CHANNEL

MIGRATION

DATA

ACCESS

The New Platform Benefits ...

10

Execution Journey Line

11

Execution Journey Line

12

Execution: Journey Line

Acknowledge Monolith

Case for Change with 3 year Target

Leaders and Stakeholders Commitment

Define Microservices

Phases / Sequence

Milestone 1..N (Dev, Test, Deploy)

Milestone 1..N (Enablement, Products)

Address TechDebt / Optimization

Planning Execution Improve

13






Phases / Sequence





1. Clearly articulate how this technology

shift will help the business

2. Agree on a rough timeline

3. Get buy-in from senior management

for the long haul

14






Phases / Sequence





● Reuse/Rewrite

● Contract definitions

● Granularity of services

● Where do you start breaking the

monolith ?

● What about new features ?

● How do you sequence this ?

15






Phases / Sequence





● Be clear about

what success is

and how you are

measuring it

16






Phases / Sequence





Learnings: ● Have quarterly delivery milestones vs big

bang approach (doomed to fail)

● Plan vs. Reality: bumpy ride

● Contain Scope: tech debt - speed vs target

● Change is inevitable: Industry trends (AWS,

CI/CD, Security standard); New features (bank

statement, additional tax forms)

● Avoid too many changes at the same time

● Stay on course beyond distractions

○ Perseverance: critical mass and beyond

● Controlled slow live transition / traffic

routing

○ A/B testing, parity, performance, data, etc

● Microservice strategy: strangler pattern vs.

rewrite vs. reuse

○ One-size fits none

○ Ex: dedupe library vs credential service

17

● Improve Operational Overhead / NoOps

○ Better CI/CD Pipeline

○ Containerization

○ Container Orchestrator

● Move intermediate solution to Target ○ tight coupling still exists between higher layers

● Streaming Architecture

○ Avoid point to point service interaction

● Service Mesh

○ Decentralize API Routing and AuthN/Z

● Service consolidation / Reduce Hops

○ Regroup: Nano-service to appropriate Microservice

○ Runtime consolidation (sidecar deployment)

● Improve intelligence

○ High quality data

○ AI/ML

● Standardize Negative (Error) scenarios

Next Steps

18

● In SOA / Platform, NFRs are more important than delivery speed but contain the scope

○ Scalability, Availability, Performance, Supportability, etc

● Execution challenge

○ Manage dependencies, scope and stakeholders

○ One-size fits none

● Success Metrics

○ More independent releases on each services

○ Organizational Scale: More scrum teams with clear ownership

○ Able to handle product volume (like TurboTax, etc) with scale, performance and

availability expectations

● External References: ○ https://martinfowler.com/articles/break-monolith-into-microservices.html ○ https://thenewstack.io/from-monolith-to-microservices/ ○ https://www.infoq.com/presentations/monolith-microservices-refactoring-analysis-tools

● Blog with the same topic: https://medium.com/@snezana_sahter/from-monolith-to-microservices-cec202a3699

Takeaway

api world monolith to microservices · interest for the past several years, including 10+ years...

Documents