decomposing applications for scalability and deployability (april 2012)
DESCRIPTION
Today, there are several trends that are forcing application architectures to evolve. Users expect a rich, interactive and dynamic user experience on a wide variety of clients including mobile devices. Applications must be highly scalable, highly available and run on cloud environments. Organizations often want to frequently roll out updates, even multiple times a day. Consequently, it’s no longer adequate to develop simple, monolithic web applications that serve up HTML to desktop browsers. In this talk we describe the limitations of a monolithic architecture. You will learn how to use the scale cube to decompose your application into a set of narrowly focused, independently deployable back-end services and an HTML 5 client. We will also discuss the role of technologies such as NodeJS and AMQP brokers. You will learn how a modern PaaS such as Cloud Foundry simplifies the development and deployment of this style of application.TRANSCRIPT
Decomposing applications for deployability and scalability
Chris Richardson
Author of POJOs in ActionFounder of the original CloudFoundry.com
1
Presentation goal
2
Decomposing applications to improve deployability and scalability
and How Cloud Foundry helps
About Chris
3
(About Chris)
4
About Chris()
5
About Chris
6
About Chris
http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
7
vmc push About-Chris
8
Developer Advocate for CloudFoundry.com
Signup at http://cloudfoundry.com
Promo code: CFOpenTour2012
9
Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps
Let’s imagine you are building an e-commerce application
10
Tomcat
Traditional web application architecture
11
Browser
WAR
MySQL Database
developtestdeployscale
ShippingService
AccountingService
InventoryService
StoreFrontUI
Simple to
Apache
But there are problems
12
Users expect a rich, dynamic and interactive experience on mobile devices and desktop
13
Java Web Application
Browser
HTTP Request
HTML/Javascript
Old style UI architecture isn’t good enough
Real-time web ≅ NodeJS
One dimensional scalability
14
§ Scales to handle transaction volumes§ Higher transaction rate required ⇒ run more instances§ Some clouds will do this automatically, others easily§ No code changes required
BUT§ Does not scale with increasing data volumes§ Caching is less effective§ Requires more memory§ Requires more I/O
Components that don’t cluster§ Most application components are easily clustered:•Most services are stateless•Server-side session state: sticky session routing/state
server
BUT there are exceptions
§ Singletons that maintain global application state:• e.g. CEP engine analyzing event stream over HTTP
§ Commercial software with a per-server license fee
§ If one application component can’t be clustered ⇒ you can’t cluster the application!
15
Obstacle to frequent deployments§ Need to redeploy everything to change one component§ Interrupts long running background (e.g. Quartz) jobs§ Increases risk of failure
Fear of change
§ Updates will happen less often§ e.g. Makes A/B testing UI really difficult 16
Large application
Slow IDESlow start times for web container
17
Scaling development
18
!=Scalable development
WAR
Shipping
Accounting
InventoryService
StoreFrontUI
§ Forces teams to synchronize development efforts§ Teams need to coordinate updates
Long-term commitment to a single technology stack§ Let’s suppose you go with the JVM•Some polyglot support•Many (but not all) JVM languages interoperate easily• e.g. Scala, Groovy modules within a Java application•But you can’t use non-JVM languages
§ Application depends on a particular:•Framework - e.g. Spring or Java EE•Container - e.g. Tomcat
§ Switching technology stack ⇒ touches entire application•Painful•Rarely done
19
20
Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps
3 dimensions to scaling
21
X axis - horizontal duplicationZ ax
is - d
ata pa
rtitio
ning
Y axis - functionaldecomposition
Scale by cloning
Scale
by sp
litting
simila
r thin
gsScale by splitting different things
X axis scaling - application level
22
Tomcat
Load Balancer
WAR
X axis scaling - database level
23
MySQL Master
MySQL Slave 1
MySQL Slave 2
MySQL Slave N
Writes Consistent reads
Inconsistent reads
Slaves are clones
Benefits and drawbacks of X axis splits§ Benefits•Easy and fast to implement•Handles increasing transaction volumes•Simple to manage
§ Drawbacks•Doesn’t scale with increasing data volumes•Doesn’t address application/development complexity•Couples entire application to a particular technology stack
24
Z axis scaling - data partitioning§ Each server runs the identical code base § Different servers responsible for different data§ Route request based on•Attribute of the request• Identity/attribute of the requestor/customer
§ For example:•Sharding - partition data by primary key•Free customer vs. paying customer
25
“Router”
Server 1
Server 2
...
Request:... Some attribute ...
Database server 2
Z axis scaling - database sharding
26
ID Name …
1 Ajanta
2 Eggshop
…
Partition column
RESTAURANT table
ID Name …
1 Ajanta
…
ID Name …
2 Eggshop
…
Database server 1
X axis scaling too!
Search ServiceSearch Service
Z axis scaling - application example
27
Database server 2Database server 1
Search Service Search Service
Query AggregatorUpdate Router
QueryContent
Cache Cache
Partition 1 Partition 2
Benefits and drawbacks of Z axis splits§ Benefits•Each server only deals with a subset of the data• Improved caching•Reduced memory usage and reduced I/O• Improved transaction scalability• Improved fault isolation - failure only makes part of the data
inaccessible§ Drawbacks•Doesn’t scale with increasing development and application
complexity• Increased management/deployment complexity
28
Y axis scaling - functional partitioning§ Splits monolithic application into a set of services§ Each service implements related set of functionality§ Partitioning schemes:•Partition functionality by noun or by verb•Single Responsibility Principle• e.g. Unix utilities - do one focussed thing well
29
ShippingService
AccountingService
billing web application
shipping web application
InventoryService
inventory web application
30
Y axis scaling - application level
Store Front
Store front web application
Apply X axis cloning and/or Z axis partitioning to each service
MySQL MySQL
Y axis scaling - databases
31
MySQL
Order table
....
MySQL
Customer table
....
Apply X axis cloning and/or Z axis partitioning to each database
Application
Real world examples
32
http://highscalability.com/amazon-architecture
Between 100-150 services are accessed to build a page.
http://techblog.netflix.com/
http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf
http://queue.acm.org/detail.cfm?id=1394128
Benefits and drawbacks of Y axis splits§ Benefits•Scales development: focussed two pizza devops teams•Deploy services independently•Scale services independently• Improves fault isolation•Eliminates long-term commitment to a single technology
stack•Enforces well defined interfaces between components
§ Drawbacks• Implementation complexity•Deployment complexity
33
Two levels of architecture§ System level•Defines the inter-service glue: interfaces and
communication mechanisms•Slow changing
§ Service level•Defines the internal architecture of each service•Far fewer constraints on technology•Each service could use a different technology stack•Rapid evolving
34
Versus
If services are small...§ Regularly rewrite using a better technology stack§ Pick the best developers rather than best <pick a
language> developers§ Adapt system to changing requirements and better
technology without a total rewrite
35
Fred George “Developer Anarchy”
Moreover: you are not the same you ...§ Cell lifetimes:• hours - some white blood cells• days - stomach lining cells• years - bone cells• lifetime - brain cells
§ 50 to 70 billion of your cells die each day§ Yet you (the system) remains intact
36
http://dreamsongs.com/Files/WhitherSoftware.pdf
http://dreamsongs.com/Files/DesignBeyondHumanAbilitiesSimp.pdf
Can we build software systems with these
characteristics?
Too much technical debt ⇒ component death?
37
Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps
Inter-service communication options§ Multiple collaborating services ⇒ need a communication
mechanism§ Many choices:•Synchronous ⇔ asynchronous•Transports: HTTP, AMQP, ...•Formats: JSON, XML, Protocol Buffers, Thrift, ...•Even via the database
§ Distributed application ⇒ error handling strategies
38
ShippingService
StoreFrontUI
wgrus-store.war
AccountingService
wgrus-billing.war
wgrus-shipping.war
InventoryService
wgrus-inventory.war
MySQL
39
REST
SOAP
Thrift
...
Synchronous communication
Lots of choices
Service discovery options§ Clients need to know the coordinates of their dependencies
§ Option #1: Caller is configured with dependencies’s URL• e.g. Environment variables, system properties
§ Option #2: Services announce their location•Relies on ‘broadcast’ protocol or Gossip protocol to
announce location• e.g. RabbitMQ message broker
§ Option #3: configuration server•Maintains all the configuration information• e.g. Zookeeper
40
Benefits and drawbacks§ Benefits•Simple to implement and use
§ Drawbacks•Poor fault isolation: caller is blocked if server is down/slow•Caller needs to know server’s coordinates (URL, ...)• Less scalable than asynchronous protocols
41
StoreFrontUI
wgrus-store.war
AccountingService
wgrus-billing.war
WidgetInventoryService
wgrus-inventory.war
InventoryService
wgrus-inventory.war
MySQL
42
RabbitMQ(Message Broker)
Asynchronous message-based communication
Benefits and drawbacks§ Benefits•Decouples caller from server•Caller unaware of server’s coordinates (URL)•Message broker buffers message when server is down/slow
§ Drawbacks•Additional complexity of message broker •RPC using messaging is more complex
43
Shared databases?
44
Order Management Customer Management
Database
OrdersCustomers
Benefits and drawbacks
45
§ Benefits•Simple•Single (consistent) source of data
§ Drawbacks•Defeats the purpose of having multiple services• Less scalable•More coupling
Better: separate databases
46
Order Management Customer Management
Database
OrdersCustomers (copy)
Database
Customers
Customer Update Event
Maintaining consistency without 2PC
47
begin transaction update order tables save intent to update customer tablescommit transaction
for each saved intent begin transaction delete intent queue message to update customer commit transaction
BASE: An Acid Alternative by Dan Pritchetthttp://queue.acm.org/detail.cfm?id=1394128
dequeue messagebegin transaction if (update has not been applied) { record update as applied update customer tables } commit transaction acknowledge message
Order management
Customer management
Writing code that calls services
48
Composable futures§ Problem:•Service A needs to call services B and C and then D•Makes sense to call B and C parallel•Yet most concurrency APIs are low-level, error-prone etc
§ Solution:•Use Akka composable futures = really nice abstraction
49
val futureB = callB()val futureC = callC()
val futureD = for { b <- futureB.mapTo[SomeType] c <- futureC.mapTo[SomeType] d <- callD(b, c) } yield d
val result = Await.result(futureD, 1 second). asInstanceOf[SomeOtherType]
http://doc.akka.io/docs/akka/2.0.1/scala/futures.htmlhttp://en.wikipedia.org/wiki/Futures_and_promises
Two calls execute in parallel
And then invokes D
Get the result of D
Spring Integration
§ Builds on Spring framework
§ High-level of abstraction for building message based applications
§ Implements EAI patterns
§ Provides plumbing for exchanging messages between application components
§ Promotes loosely coupled components
§ Integrates with external messaging infrastructure: JMS, AMQP, HTTP, Email, File transfer
50
Spring Integration concepts
§ Message channel
• Virtual pipe connecting producer and consumer
§ Message endpoints
• The filter of a pipes-and-filter architecture
• Read from and/or write to channel
§ Endpoint types:
• Transformer
• Filter
• Router
• Splitter
• Aggregator
• ServiceActivator
• Inbound channel adapter - read from external source, writes to channel
• Outbound channel adapter - read from channel write to external destination
51
Spring Integration insulates components from the underlying
communication mechanism
52
Order Service
Messaging Gateway
Channel Service Activator
Shipping service
53
Example of reconfigurability - local@Servicepublic class OrderServiceImpl {
@Autowiredprivate ShippingService shippingService;
public void placeOrder() { String orderId = generateOrderId(); … shippingService.shipOrder(orderId);}
}
@Servicepublic class ShippingServiceImpl {
public void shipOrder(String orderId) { System.out.println("shipped order: " + orderId);}
}
Order Service
Messaging Gateway
Channel Service Activator
Shipping service
AMQP
RabbitMQ
AMQP Channel
54
Example of reconfigurability - distributed
Code unchanged in new deployment
Handling failure§ Errors happen (especially in distributed systems)§ Use timeouts and retries•Never wait forever•Some errors are transient so retry
§ Use per-dependency bounded thread pool with bounded queue• Limits number of outstanding requests•Fails fast if service is slow or down
§ Use circuit breaker•High rate of errors ⇒ stop calling temporarily •Avoids calling service that has issues
§ On failure•Returned cached or default data• Invoke custom error handler
55http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
56
Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps
Modular application
Choice of presentation layer technology+
Redeploy UI frequently/independently
57
NodeJS is the fashionable technology
58
Many JavaScript client frameworks have a NodeJS counterparte.g. socket.io
NodeJS isn’t the only game in town
59
JVM-based http://vertx.io/
A modern web application
60
Browser
Service 1
Service 2
...
HTML 5Application
Socket.ioclient
Events
RESTful WS
Server Application
Socket.ioserver
Node JS
Alternatively: multiple front-end servers
61
Browser
Node JS Service 1
Service 2
...
HTML 5Application
Node JS
Node JS
Needs single-sign-on
NodeJS - using RESTful WS and AMQP
62
Node JS
Service
RabbitMQ Service
REST
AMQP AMQP
Requests
Eventssocket.io
Updating the UI is easy
63
§ Update the UI independently of rest of system§ Easily run A/B tests § Enables fast iteration of the UI
http://theleanstartup.com/principles
But coordination with backend changes required§ Let’s imagine that you are deploying an advanced search
feature:•Enhancements to search service•Enhancements to UI
§ Before•Deploy new war
§ Now:•Some coordination required•Deploy updated backend service•Deploy updated NodeJS and browser code•Enable feature using feature switch
64
65
Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps
The challenge§ In the beginning:• It would be easy to use Y axis/functional decomposition•But you don’t need it and it would slow you down
§ Later on:•You do need it•But your application logic is tangled and decomposition is
difficult
66
Lower cost way to start with Y axis decomposition§ Enforce boundaries between services, e.g. DTOs§ But run in a single JVM with a single database§ Use Spring Integration•Course-grained message-based communication•Via in-memory channels
§ Downside:• Implementing the glue layers are extra work
67
Application
Special case: Components that don’t cluster
68
Monitoring
Esper
Rest of domain classes
AC
D
B
Doesn’t cluster ⇒ No X axis scaling/cloning
Tangled dependencies prevent Y axis scaling
Don’t let this happen!
Use well-defined interfaces
Ensure loose coupling
69
Monolithic application
Traditional application architecture
70
Service A Service B Service C
Shared domain model
A C
B
X Z
Y
Difficult to untangle
Service
Domain model 2
Application
Extracting a service = untangling dependencies
71
Service A Service BService C
Domain model 1
A C
B
X
Z
Y
Trouble!
Untangling dependencies
72
§ Domain object is part of an association•Read only - replicate•Update only - notification interface•Update then read - replicate
§ Domain object is a request parameter•Read-only - replace with DTO•Update only - notification interface•Update then read - tricky
§ Note - Need acyclic dependencies• If used object isn’t part of a service = assumption is that it
belongs in calling service
class A { private B other;}
class A { void m(B other) { ... }}
Bounded context is a useful idea§ Different parts of a system ⇒ different domain models§ Different services have a different view of some domain
objects, e.g.•User Management = complex view of user•Rest of application = PK + ACL + Name
§ Different services can have a different domain model§ Services exchange messages to synchronize data
73
Domain model 2
Untangling dependencies 1
74
Domain model 1
A C
B
X
Z
Y
doSomething(Y)
getSomething()
doSomethingElse()
Service A Service B Service C
Domain model 2
Untangling dependencies 2
75
Domain model 1
A C
B
X
Z
Y
doSomething(YDTO)
doSomethingElse()
Facade A
Facade B
Service A Service B Service C
Domain model 2
Untangling dependencies 3
76
Domain model 1
A
C
B
X
Z
Y
doSomething(YDTO)
doSomethingElse()
Facade A
Facade B
Message Broker Listener
Listener
Service A Service B Service C
77
Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps
Traditional tools: monolithic applications
78
Developing service-oriented apps is harder§ Many more moving parts to manage• Infrastructure services: SQL, NoSQL, RabbitMQ•Application services
§ Who is going to setup the environments:• the developer sandbox?• ...•QA environments?
79
Cloud Foundry makes it easier§ Easy to deploy applications§ Easy to provision services§ Manifest describes the structure of a set of services§ vmc env-add for binding 'app services' together§ Micro Cloud Foundry is your personal cloud/sandbox§ Caldecott exposes services for use by integration tests
80
Services available on Cloud Foundry
81
Creating a service instance$ vmc create-service mysql --name mysql1Creating Service: OK
$ vmc services......
=========== Provisioned Services ============
+-------------+---------+| Name | Service |+-------------+---------+| mysql1 | mysql |+-------------+---------+
Multi-application manifest - part 1--- applications: inventory/target: name: inventory url: cer-inventory.chrisr.cloudfoundry.me framework: name: spring info: mem: 512M description: Java SpringSource Spring Application exec: mem: 512M instances: 1 services: si-rabbit: type: :rabbitmq si-mongo: type: :mongodb si-redis: type: :redis
83
Path to application
Required services
Multi-application manifest - part 2 store/target: name: store url: cer-store.chrisr.cloudfoundry.me framework: name: spring info: mem: 512M description: Java SpringSource Spring Application exec: mem: 512M instances: 1 services: si-mongo: type: :mongodb si-rabbit: type: :rabbitmq
84
Path to application
Required services
One command to create services and deploy application$ vmc push Would you like to deploy from the current directory? [Yn]: Pushing application 'inventory'...Creating Application: OKCreating Service [si-rabbit]: OKBinding Service [si-rabbit]: OKCreating Service [si-mongo]: OKBinding Service [si-mongo]: OKCreating Service [si-redis]: OKBinding Service [si-redis]: OKUploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (12K): OK Push Status: OKStaging Application 'inventory': OK Starting Application 'inventory': OK Pushing application 'store'...Creating Application: OKBinding Service [si-mongo]: OKBinding Service [si-rabbit]: OKUploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (5K): OK Push Status: OKStaging Application 'store': OK Starting Application 'store': ... 85
vmc push:•Reads the manifest file•Creates the required services•Deploys all the applications
Application configuration via environment variables
String value = System.getenv("PAYMENT_SVC")
@Value("#{systemEnvironment['PAYMENT_SVC']}")private String envVariable;
$ vmc env-add cf1 PAYMENT_SVC=http://...Adding Environment Variable
[PAYMENT_SVC=http://...]: OKStopping Application: OKStaging Application: OK
Micro Cloud Foundry: new developer sandbox
87
Open source Platform as a Service project
App Instances Services
10.04
A PaaS packaged as a VMware Virtual Machine
Use as a developer sandbox
• Use the services from Junit integration tests
• Deploy your application for functional testing
• Remote debugging from STS
Using Caldecott…$ vmc tunnel1: mysql-135e02: mysql1Which service to tunnel to?: 2Password: ********Stopping Application: OKRedeploying tunnel application 'caldecott'.Uploading Application: Checking for available resources: OK Packing application: OK Uploading (1K): OK Push Status: OKBinding Service [mysql1]: OKStaging Application: OK Starting Application: OK Getting tunnel connection info: OK
Service connection info: username : uMe6Apgw00AhS password : pKcD76PcZR7GZ name : d7cb8afb52f084f3d9bdc269e7d99ab50
Starting tunnel to mysql1 on port 10000.1: none2: mysqlWhich client would you like to start?: 2
…Using CaldecottLaunching 'mysql --protocol=TCP --host=localhost --port=10000 --
user=uMe6Apgw00AhS --password=pKcD76PcZR7GZ d7cb8afb52f084f3d9bdc269e7d99ab50'
Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 10944342Server version: 5.1.54-rel12.5 Percona Server with XtraDB (GPL),
Release 12.5, Revision 188
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
Running JUnit test with Caldecott
90
Configure your test code to use port + connection info
Summary§ Monolithic applications are simple to develop and deploy
BUT applying the scale cube
§ Decomposes your application into services§ Enables scaling for transactions and data volumes§ Tackles application complexity§ Enables scaling for development§ Enables frequent, independent deployments§ Make it easy to leverage other technologies
AND
§ Cloud Foundry simplifies the development and deployment of “service-oriented” applications
91