1 saas - performance management 20 th nov 2013. 2 itaas platform we studied the business challenges...
TRANSCRIPT
![Page 1: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/1.jpg)
1
SaaS - Performance Management
20th Nov 2013
![Page 2: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/2.jpg)
2
ITaaS Platform
We studied the business challenges of SMBs to redefine the Technology Consumption Model
On the software side the solution was to build a set of multi-tenant applications to leverage the SaaS model
We studied the business challenges of SMBs to redefine the Technology Consumption Model
On the software side the solution was to build a set of multi-tenant applications to leverage the SaaS model
Business Model: Pre-Integrated suite of HW, NW, SW and Services
Commercial Model: Opex based, Build-as-you-Grow and Pay-as-you-Use
Business Process:Embedded best in class practices for quick start
Technology Platform: Leveraging the Cloud for
easy distribution
Configurability: Meet individual SMB needs and improve adoption
Pre-built Compliance:Ensure security and statutory requirements
![Page 3: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/3.jpg)
3
IntroductionSaaS Performance Management
1. The SaaS platform & Solution stack
2. Performance challenges
3. Three tier Architecture
4. Performance optimizations for each tier
5. SaaS Three Tier Architecture
![Page 4: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/4.jpg)
4
ITaaS – Solution Stack
Software Solution Network Solution Hardware Solution
Desktop/Laptops Printers On Premise Servers All System Software
CRM F&A HRMS Payroll Projects
Business Analytics
Document Mgmt
Online Presence Mgmt
Enterprise Intranet Mgmt
Messaging Solution
Network Connectivity Data Center HostingNetwork Devices
Manufacturing Retail Wellness Prof. Services
L5P Biz Services Reward-Recognition Virtual Community
L5
L4
L3
L2
L1
L6
Office Mgmt Solution
Assessment
Education
P&I
Hardware Layer
Networks Layer
Common Office Applications Layer
Common Business Applications Layer
Vertical Core Applications Layer
Customer Supplier Integration Layer
Niche Vertical Application Layer
![Page 5: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/5.jpg)
5
IntroductionSaaS Performance Management
1. The SaaS platform & Solution stack
2. Performance challenges
3. Three tier Architecture
4. Performance optimizations for each tier
5. Deployment Architecture & Monitoring
![Page 6: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/6.jpg)
6
Performance Challenges
• Can users get a fast response over the low bandwidth network?
• Whether to host different solutions on different servers or on the same server?
• How do we find if anything is slow in production?
• Can the applications on the cloud scale as transactions and users increase?
• Will business operations continue in face of a disaster?
• Will use of Open source tools and servers serve the requirement?
• How would we avoid bad code entering production?
• How would we identify if any bad code has entered production?
![Page 7: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/7.jpg)
7
IntroductionSaaS Performance Management
1. The SaaS platform & Solution stack
2. Performance challenges
3. Basic three tier Architecture
4. Performance optimizations for each tier
5. SaaS Three Tier Architecture
![Page 8: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/8.jpg)
8
Three tier Architecture
Perimeter Security
Internet/MPLS
C1
C2
C1
C2
Application Tier
Web Tier
Database Tier
Applications
Reports
Master Slave
![Page 9: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/9.jpg)
9
IntroductionSaaS Performance Management
1. The SaaS platform & Solution stack
2. Performance challenges
3. Basic three tier Architecture
4. Performance optimizations for each tier
• Web
• Application
• Database
5. SaaS Three Tier Architecture
![Page 10: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/10.jpg)
10
IntroductionSummary of achieving speed
1. Lookup DNS2. Make TCP Connection
a) Reduce number of TCP reconnections3. Negotiate SSL
a) Make the negotiation quickb) Reduce number of renegotiations
4. Send Requesta) Reduce content to be sent
5. Server Sends Responsea) Send response quicklyb) Minimize response
6. Parse Response and Download additional Resourcesa) Minimize number of resources to be downloaded
7. Render Contenta) Keep content lean, clean and simpleb) Ensure all required styles for rendering are available at the earliest
8. Compile and execute JavaScript (Single Threaded Blocking)a) Keep minimal JavaScriptb) Defer JavaScript Execution
![Page 11: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/11.jpg)
11
Web tier – Minimize round-trip times
What is a Round-trip time (RTT) = Request time + Response time
Ways to minimize RTT -1. Minimize the number of requests
• Avoid bad requests (e.g. 404)• Combine external JavaScript• Combine external CSS• Optimize the order of styles and scripts• Use CSS Sprites for images
2. Parallelize them• Parallelize downloads across hostnames (CDN)
![Page 12: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/12.jpg)
12
Web tier – Set HTTP Keep Alive
What is it? In HTTP/1.1 the keep-alive extension allows persistentconnections . Basically the HTTP sessions allow multiplerequests to reuse the same underlying TCP connection.
Advantage?Speed up latency for html documents with lots of static content
Example of Web tier setting - Apache has 2 directives - KeepAlive and KeepAliveTimeout These directives keep the connection to the Web server alive for a pre-defined number of requests or for pre-defined time period (Default 100, 15 seconds)
Browsers limit the number of parallel downloads to six - eight(per hostname over HTTP/1.1). It is beneficial to increase the no. of concurrent connections to reduce page load time.Browsers limit the number of parallel downloads to six - eight(per hostname over HTTP/1.1). It is beneficial to increase the no. of concurrent connections to reduce page load time.
Make the negotiation quick
![Page 13: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/13.jpg)
13
Web tier – Optimize SSL Negotiation
1. Offload SSL Negotiation to a Hardware Device2. Enable SSL Session Cache3. Set a proper Timeout for SSL Session Cache
Session Timeout should be a based on the security concerns and normal Session Time of the userSession Timeout should be a based on the security concerns and normal Session Time of the user
Make the negotiation quick
![Page 14: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/14.jpg)
14
Web tier – Minimize HTML content
Criteria Object type to use Example
Records less than 5 Radio button/Check box Male/ Female
Records rarely increase & is easy to scroll & choose
Drop down City, State, Country
• Records increase in a narrow range • Limited filters to search
AJAX look up(AJAX call can be triggered after certain no. of characters are typed, esp. in case of Codification)
Clinic names, Site names.
•Records are in thousands and can increase •Many filters to search
Use Lookups (popup screen) Search an item by item id, item name, store name, item type, etc
![Page 15: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/15.jpg)
15
Web tier – Pagination
• Paginate depending on the context - need not be same for all screens• To reduce navigation show the latest/relevant records first• Leverage various Pagination design patterns
![Page 16: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/16.jpg)
16
Web tier – Filters
• Mandate filters for reports - To narrow the number of records fetched. • Put appropriate Constraints – Date field should always be constrained.• A constraint would typically require Functional knowledge• Set default for certain filters e.g. Status filter defaulted to OpenNote: A file of 3 MB was downloaded with various bandwidths and latencies using
WANEM. (WAN emulator). Please find its outcome and its impact on the end user experience
Test case Bandwidth Latency Time taken1 114 mbps ** NIL 3 secs2 128 kbps NIL 3 mins 40 secs3 256 kbps NIL 1 min 57 secs4 128 kbps 50 ms 4 mins 45 secs5 256 kbps 50 ms 2 mins
**(without WANEM)
Tools Tip: Create an excel sheet listing all reports names v/s its filters. This would give a holistic view of all the filters in the solution. It would not be relatively easy to implement policies related to mandating and constraining filters with some functional knowledge obviously.
Tools Tip: Create an excel sheet listing all reports names v/s its filters. This would give a holistic view of all the filters in the solution. It would not be relatively easy to implement policies related to mandating and constraining filters with some functional knowledge obviously.
![Page 17: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/17.jpg)
17
Web tier – Serve static content
• A typical web request = Static + Dynamic
• Static content - images, style sheets, JavaScript, html, pdf, video files, etc should be served by Web tier
• Advantage:
1. To avoid use of expensive Application server threads
2. Avoid delay to serve static content during Garbage collections
3. Web servers serve static content faster than Application servers
4. Sets stage for further static content consolidation and scale with use of CDN
![Page 18: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/18.jpg)
18
Web tier – Minimize round-trip times – Avoid bad requests
HTTP requests are expensive, making an HTTP request and getting a useless response (i.e. 404 Not Found) is unnecessary. Will slow down the user experience.
Firefox: Use add-ons like Firebug, Page Speed, YSlow, Fiddler, For IE: Fiddler Firefox: Use add-ons like Firebug, Page Speed, YSlow, Fiddler, For IE: Fiddler
![Page 19: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/19.jpg)
19
Web tier – Minimize round-trip times – Optimize order of styles & scripts
Correct order of external style sheets and external/inline scripts enables better parallelization of downloads and speeds up browser rendering time.
Scenario 1
<head><link rel="stylesheet" type="text/css" href="stylesheet1.css" /><script type="text/javascript" src="scriptfile1.js" /><script type="text/javascript" src="scriptfile2.js" /><link rel="stylesheet" type="text/css" href="stylesheet2.css" /><link rel="stylesheet" type="text/css" href="stylesheet3.css" /></head> Scenario 2
<head><link rel="stylesheet" type="text/css" href="stylesheet1.css" /><link rel="stylesheet" type="text/css" href="stylesheet2.css" /><link rel="stylesheet" type="text/css" href="stylesheet3.css" /><script type="text/javascript" src="scriptfile1.js" /><script type="text/javascript" src="scriptfile2.js" /></head>
Scenario 1 the second two style sheets must wait until the JS files are finished downloading. Scenario 2 merely by changing the order of the resources - 100 ms is shaved off the total download time.
Scenario 1 the second two style sheets must wait until the JS files are finished downloading. Scenario 2 merely by changing the order of the resources - 100 ms is shaved off the total download time.
![Page 20: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/20.jpg)
20
Web tier – Minimize round-trip times – Combine external Javascripts
Example of the download profile of an HTML file containing requests for 13 different .js files from the same domain
All files are downloaded serially, and take a total of 4.46 seconds to complete. Now here is the profile for the same document, with the same 13 files collapsed into 2 files. The same 729 kilobytes now take only 1.87 seconds to download.
Tip #0 Externalize JavaScriptTip #1 Deduplicate JavaScriptTip #1 JavaScript files should be combined into fewer output files (except third party JS libraries)Tip #2: Serve the file only when that component is requested by a user.
Tip #0 Externalize JavaScriptTip #1 Deduplicate JavaScriptTip #1 JavaScript files should be combined into fewer output files (except third party JS libraries)Tip #2: Serve the file only when that component is requested by a user.
![Page 21: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/21.jpg)
21
Web tier – Minimize round-trip times – Sprite Images
CSS Sprite - A single image file that contains number of different images.
Reduced 14 HTTP hits as 15 images are combined in one single sprite improving page load time. Try SpriteMe bookmarkletReduced 14 HTTP hits as 15 images are combined in one single sprite improving page load time. Try SpriteMe bookmarklet
Without Sprite With Sprite
![Page 22: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/22.jpg)
22
Web tier – Minify static content
Minification - Removes whitespace, Line Breaks, comments, Shrinks Variables & Optimizes code in js and css files. Decreases file size & improves page load time.
Tools available - Google Closure Compiler, YUI Compressor, JSMIN, Dean Edward’s Packer, DOJO Compressor. Tools available - Google Closure Compiler, YUI Compressor, JSMIN, Dean Edward’s Packer, DOJO Compressor.
1.14
0.716
0.00 MB
0.20 MB
0.40 MB
0.60 MB
0.80 MB
1.00 MB
1.20 MB
BeforeMinimization
AfterMinimization
Payroll JS
File size reduction ~62%
File
siz
e
![Page 23: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/23.jpg)
23
Web tier – Compress static content
1. GZIP generally reduces the response size by about 70%. 2. Compression may sometimes not be supported by the proxy setting at customer end3. Zipping of images, pdf or any binary file not done as most are already compressed
1. GZIP generally reduces the response size by about 70%. 2. Compression may sometimes not be supported by the proxy setting at customer end3. Zipping of images, pdf or any binary file not done as most are already compressed
![Page 24: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/24.jpg)
24
Web tier – Cache locally to take heat off back-end servers
• Set suitable static content expiry in the web tier – Example:
ExpiresActive ONExpiresByType text/javascript
"access plus 1 month“Cache-Control “public max-
age=290304000”
• Use a unique identifier to append every static request, to refresh browser cache when the content changes.
Note: Incase of server farms do not use Etags. Note: Incase of server farms do not use Etags.
290304000
![Page 25: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/25.jpg)
25
Web tier – Ajax lookups – Optimization tip
AJAX auto lookups needs to have a limit to the characters entered before which the lookup is shown. A lookup done for every character entered would lead to executing a select query. Infact it is directly proportional to the characters typed. Assuming there is no delay. For SMB we recommend to set it to at least 4 characters.
- 25 -
1 Character entered
4 Characters entered
x
Note: Enable Caching of AJAX Requests.Note: Enable Caching of AJAX Requests.
![Page 26: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/26.jpg)
26
Web tier – Bandwidth/Latency & User Experience
Impact on a transaction response time due to Bandwidth, Latency, Browser Cache &Static file compression in Web tier.
Response times vs. varying bandwidth/latency
Without browser cache With cache
Compression On Off On On Off On On On On On
Bandwidth
57 Mbps (Without WANem)
57 Mbps (Without WANem)
64 Kbps
128 Kbps
128 Kbps
128Kbps, Latency
200msec
128Kbps, Latency
400msec128 Kbps
57 Mbps (Without WANem)
64 kbps
Resp. time (secs) 0.61 0.98 48.38 22.89 56.88 44.82 76.09 4.18 0.38 9.81
Note: Time does not include – Browser rendering or processingWeb Page size: With compression – 260 KB, Without compression – 730 KBUsed TCS' WANem (WAN Emulator)
Note: Time does not include – Browser rendering or processingWeb Page size: With compression – 260 KB, Without compression – 730 KBUsed TCS' WANem (WAN Emulator)
![Page 27: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/27.jpg)
27
Web tier – Avoid Repetitive Requests
Communicate the system status to the user once a transaction is initiated.
Example: Use the loader feature which creates a grey transparent layer over the existing User
interface and does not allow the access to the submit button.
![Page 28: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/28.jpg)
28
Web tier – Time to test
Analysis of web service http://www.webtest.org for http://www.tcs.comAnalysis of web service http://www.webtest.org for http://www.tcs.com
![Page 29: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/29.jpg)
29
IntroductionSaaS Performance Management
1. The SaaS platform & Solution stack
2. Performance challenges
3. Basic three tier Architecture
4. Performance optimizations for each tier
• Web
• Application
• Database
5. SaaS Three Tier Architecture
![Page 30: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/30.jpg)
30
IntroductionTypical Activity in the Application Layer
1. Read and interpret the request parameters
2. Validate the Request Parameters
a) Read from Data Store
3. Perform Business Operations
a) Read from Data Store
4. Persist Data to Data Store
5. Form and Send the HTML Response
Optimize ProcessingMinimize fetches from Data StoreFetch only Required Data
Optimize ProcessingMinimize fetches from Data StoreFetch only Required Data
![Page 31: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/31.jpg)
31
App tier – Review default server settings
• Server Log Level (In thread dumps, several threads in blocked state are observed in several instances where too many logs at debug, info logs in the code in combination with low log level )
• Clients/Threads in Web, Application and Database tiers
• JVM settings
• Max File Descriptors
Notes: Apache – maxClients, JBoss – maxThreads, MySQL - max_connectionsGenerally - maxClients > Maxthreads < Max_connectionsEvery layer works with a limited number of connections and resources (read Threads)
Notes: Apache – maxClients, JBoss – maxThreads, MySQL - max_connectionsGenerally - maxClients > Maxthreads < Max_connectionsEvery layer works with a limited number of connections and resources (read Threads)
![Page 32: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/32.jpg)
32
Application tier – Memory allocation for JEE Application Server
But bigger the heap, longer the major GC and hence pause times.
64bit JVM allows us to set higher values of heap size
OS: Room for me?
![Page 33: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/33.jpg)
33
Application tier – Out of memory?
Major GCMinor GC
Metadata of loaded classes
Exception in thread “main” java.lang.OutOfMemoryError: PermGen space ?Exception in thread "main" java.lang.OutOfMemoryError: Java heap space ?
Heap default: java -Xms32m -Xmx128m
• Jvisualvm• Jmap• GC Viewer
Note: Specific to SUN JVM. Internals may vary depending on choice of JVMNote: Specific to SUN JVM. Internals may vary depending on choice of JVM
![Page 34: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/34.jpg)
Application tier – Exploring jmap (Memory Map) command
Heap Space -1. jmap -heap <PID>- To get Heap details. Here look at the section named - "PS Old generation" - if its above 80-90% we may then execute the next two commands in that order to get more details 2. jmap -histo <PID>-For each Java class, number of objects, memory size in bytes, and fully qualified class names are printed. VM internal class names are printed with '*' prefix. This helps to find who is occupying the heap. -If the live suboption is specified, only live objects are countedNote - jmap –histo:live triggers a FULL GC as compared to jmap -histo 3. jmap -dump:live,format=b,file=<jmap_date_server#_dump.bin > <PID>jmap -dump:format=b,file=<jmap_date_server#_dump.bin > <PID>
Note – Depending on JVM size, the heap dump can be sizeable. To open the heap dump in IBM Heap analyzer to check leak suspects the overall VM memory required is generally greater than twice the heap dump size
Permanent Generation Space - jmap –permstat <PID>Prints class loader wise statistics of permanent generation of Java heap. For each class loader, its name, liveness, address, parent class loader, and the number and size of classes it has loaded are printed. In addition, the number and size of interned Strings are printed.
34
![Page 35: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/35.jpg)
35
Application tier – Scale Out vs Scale up
Methods to add more resources for a particular application fall in two broad categories. Scale-out and Scale-up.
Simple Complex
Applications designed to scale out will almost always scale up, yet applications designed to scale up will almost never scale out. Example - Before development, we would need to think - How do we make the Singletons and Synchronized code blocks work across processes ?
Applications designed to scale out will almost always scale up, yet applications designed to scale up will almost never scale out. Example - Before development, we would need to think - How do we make the Singletons and Synchronized code blocks work across processes ?
![Page 36: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/36.jpg)
36
Application tier – Synchronous v/s Asynchronous Communication
Phone call (Synchronous) or vs email (Asynchronous).
Lets look at options to send emails related for alerts, notification, reminder, etc -
1) Send email in the same transaction – Synchronous - Each request blocks the calling thread
2) Send email by spawning a new thread - Asynchronous - Current thread is not blocked whilst the separate thread awaits a response from SMTP
3) Set a flag to send email and let the batch job do the needful - Asynchronous- A batch process wakes up after set time –> Checks a status flag – sends emails in configurable batch mode (i.e. throttle the traffic (count of emails) to be sent (with or without priority) )Note – Configuration would depend on capacity of SMTP server and/or traffic at that instance. Also, here optimization techniques like query tuning would be important.
![Page 37: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/37.jpg)
37
Application tier – Server side caching mechanisms
• In JVM– Occupies precious JVM heap space
– May trigger early frequent Garbage Collection
– Can slow down and crash system
– Not easy to replicate changes across JVMs in a cluster
• Out of JVM– Relatively slower
– Will overcome all the above mentioned problems
![Page 38: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/38.jpg)
38
Application tier – Server side caching mechanisms
1. Memcached
2. CouchBase/ Membase (A fork of memcached)
3. Redis from Google
4. Infinispan from JBoss
5. Terracotta (now from Software AG)
6. Coherence from Oracle
7. Gemfire from Gemstone
![Page 39: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/39.jpg)
39
Application tier – Use latest stable libraries
Do’s & don’ts
1. Avoid duplicates jar files
2. Avoid multiple version of same jar files
3. Remove unwanted jars
4. Use latest stable versions of library files
5. Use Server default Library NOT Application library
![Page 40: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/40.jpg)
40
Application tier – Sticky Sessions
1. Session Identifies a User
2. Needs replication in clustered environment
3. Should NOT be used for storing large data
4. Session has a time out parameter
5. Sticky sessions should be enabled
![Page 41: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/41.jpg)
41
Application tier – String concatenation
Method Purpose
Built-in concatenation (+) operator
To concatenate static stringsString s1 = "aa" + "bb";
String.concat() To concatenate two strings
StringBuffer • Is a synchronized class• To be used in a multithreaded environment (when more than one thread access the same StringBuffer instance).
StringBuilder(Added in Java 5)
• Identical in all respects to StringBuffer except that is NOT synchronized, which means that if multiple threads are accessing it at the same time, there could be trouble (Not thread safe). • To be used in places where string concatenation is to be done by a single thread
![Page 42: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/42.jpg)
42
High level differences between DOM parser and SAX Parser in Java:
1) DOM parser loads whole xml document in memory while SAX only loads small part of XML file in memory.
2) DOM parser is faster than SAX because it access whole XML document in memory.
3) SAX parser in Java is better suitable for large XML file than DOM Parser because it doesn't require much memory.
4) DOM parser works on Document Object Model while SAX is an event based xml parser.
Thumb rule – Use DOM parser over SAX parser if XML file is small enough and go with SAX parser if you don’t know size of xml files to be processed or they are large.
Application tier – XML Parsing ? Memory is limited
![Page 43: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/43.jpg)
43
FileReader (DataInputStream - deprecated)- Common and simple method to read a file but is extremely slow. The reason for the poor performance is that the DataInputStream class does no buffering. The resulting reads are done one byte at a time.For e.g. One of the benchmark says that to read a 250 KB file it took 28,198 milliseconds
BufferedReader (BufferedInputStream-deprecated) – A simple improvement involves buffering the FileInputStream by interposing a BufferedInputStream in the stream chain. This buffers the data, with the default buffer size of 2048 bytes
Result - The resulting performance increase for the medium sized file (250 kilobytes) was 91%, from 28,198 milliseconds to 2,510 -- over an order of magnitude with just a simple change i.e.
Example - try {
BufferedReader in = new BufferedReader(new FileReader(args[i]), 8192); while ((line = in.readLine()) != null) { nlines++; } } catch (Exception e) { System.out.println(" BR8192Test: exception:" + e );}finally { in.close(); }Note - The buffer size used in buffering schemes is important for performance. As a rule of thumb, bigger is better to a point. Refer - http://www.kegel.com/java/wp-javaio.html. Match the buffer size to the underlying storage sector size. Adopt similar strategy for writing to files.
Application tier – File Handling (Leverage stream-chaining techniques)
![Page 44: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/44.jpg)
44
Application tier – Blocking API - Synchronized
Note: In case of Multiple JVM scenario there is no default JVM synchronization between the Application server instances. Hence, usage of Sync blocks should be done wisely and at granular level
Blocking/Slow API
JVM
![Page 45: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/45.jpg)
45
Application tier – Static Code Review
Jill runs Static Code review
John runs Static Code review locally
SVN Server
Commit Code Commit Code
Central Build Machine
1. Compile
2. Static Code Review
3. Copy Paste instances
4. Design analysis
5. JUnit/Code coverage
Publish reports
Project Code
![Page 46: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/46.jpg)
46
Application tier – An approach to find time taken by request on each tier
• Use Non-intrusive code weaving (Aspect Oriented Programming
– Available for • Java• .Net
– Weave at run time– Weave at compile time
• Can build all-inclusive monitoring
– Not dependent on any framework
– But can leverage standards like Servlet API, JDBC API etc
![Page 47: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/47.jpg)
47
Application tier – An approach to find time taken by request on each tier
Web Layer Business Layer Data Access LayerCapture start Time, IP Address, Port, Request URL
SQL, SQL Start Time, SQL End Time & Request End Time
Capture query end time
Capture end time
1. For each request a unique thread is assigned by the Application server2. For each thread a Statistic Entity Object is created and its relationship maintained 3. Using Unique thread id retrieve the Statistics Entity object at each layer and fill the respective
attributes4. Once request is finished calculate time taken to process request, time taken to execute sql
statement for respective URL5. Convert summary information in to string and stored it in the Collection object. 6. Flush the thread and object relationship7. At periodic interval a service saves the collection object to a log file
DB
JBoss
![Page 48: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/48.jpg)
48
Application tier – Fault Isolation Architectural Structure – Swim Lanes
There are four general principles for designing and implementing swim lanes in a system’s architecture:
1) There must not be any shared hardware or software between lanes other than possibly highly available network gear such as paired loadbalancers or border routers. (Nothing is shared)
2) No synchronous calls can take place between swim lanes. If cross swim lane communication is required, i.e. grabbing search results to display on the same page as login, it must be done asynchronously. (Nothing crosses a swim lane boundary, Transactions occur along swim lanes)
3) Limit asynchronous communication. While permitted, more calls lead to a greater chance of failure propagation.
4) Use timeouts with asynchronous calls. There is no need to tie up the user’s browser or your server waiting for the eventual TCP timeout.
Note - Swim lane the money maker, Swim lane the problem areas, Swim lane natural barriers
![Page 49: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/49.jpg)
49
IntroductionSaaS Performance Management
1. The SaaS platform & Solution stack
2. Performance challenges
3. Basic three tier Architecture
4. Performance optimizations for each tier
• Web
• Application
• Database
5. SaaS Three Tier Architecture
![Page 50: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/50.jpg)
50
Database tier – Check for Open database connections
Database memory requirement primarily depends on –
1. Memory allocated per connection (Limited resource)2. Global memory 3. Memory required for background processes.
To identify connection leaks at various levels
• Static code review
• Verify pattern Connect – Query – Quit in dB general logs
• Consolidate database connection closure code in framework itself
• Use AOP to find Line of Code that invoked an open database connection or auto close it
• Kill any idle connections for more than specified time -Server policy
![Page 51: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/51.jpg)
51
Database tier – Reduce table joins
• Requirement – To fetch list of all products sold, along with their rate, quantity, sell date and customer to whom products were sold
• Typical Join
– Customer Master (100000)
– Product Master (100000)
– Bill Header (100000)
– Bill Details (500000) (Five items per bill)
![Page 52: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/52.jpg)
52
Database tier – Reduce table joins
• Instead– Join only Bill Header and Bill Details
– Fetch the Customer Name, and the Product Name from a cache of Hashmaps
• Customer Id – Customer Name
• Product Id – Product Name
![Page 53: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/53.jpg)
53
Database tier – Track slow queries
C2
C1
Optimize queries that:1. repeatedly run slow2. examine lot of records 3. fetch lot of records4. take more time to execute5. take up locks for long time
Optimize queries that:1. repeatedly run slow2. examine lot of records 3. fetch lot of records4. take more time to execute5. take up locks for long time
>1 second
Select * from emp;
Slow Query Log
Parse & Save in db
![Page 54: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/54.jpg)
54
Database tier – Avoid full table scans
• Full table scans are where the database will read the entire table without an index.
• Reasons why full table scans are performed– no WHERE clause
– no index on any field in WHERE clause
– poor selectivity on an indexed field
– too many records meet WHERE conditions
– using SELECT * FROM
– function used on the indexed column in the query Generate Explain plans to understand how database query optimizer chooses to execute a query
Generate Explain plans to understand how database query optimizer chooses to execute a query
![Page 55: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/55.jpg)
55
Database tier – Consider chunk at a time v/s row at a time
One insert - one connection
All inserts – one connection
Batch Insert – one connection
Prepared Inserts – one connectionGood for repeated inserts
Extended Inserts – one connectionGood for bulk Insertion
In case of MySQL extended insert – one connection technique is seen the fastest. Note: Need to watch out on the limit on size of the query and regular commits should be done after set frequency to avoid large rollbacks. Use Prepared Statement and Bulk Commit
In case of MySQL extended insert – one connection technique is seen the fastest. Note: Need to watch out on the limit on size of the query and regular commits should be done after set frequency to avoid large rollbacks. Use Prepared Statement and Bulk Commit
ConnectionID_1: INSERT INTO table(A,B) values(1,2);ConnectionID_1: QuitConnectionID_2: INSERT INTO table(A,B) values(3,4);ConnectionID_2: Quit
ConnectionID_1: INSERT INTO table(A,B) values(1,2);ConnectionID_1: INSERT INTO table(A,B) values(3,4);ConnectionID_1: Quit
Batch start ConnectionID_1: INSERT INTO table(A,B) values(1,2);ConnectionID_1: INSERT INTO table(A,B) values(3,4);Batch endConnectionID_1: Quit
ConnectionID_1: INSERT INTO table(A,B) values(1,2),(3,4);ConnectionID_1: Quit
better
Begin TransactionConnectionID_1: INSERT INTO table(A,B) values(?,?);Set Value (1, 2); Execute; SetValue(3,4);Commit TransactionConnectionID_1: Quit
![Page 56: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/56.jpg)
56
Database tier – Generate Explain plans
Developer – Query works fast in my environment but is slow in Production, is something different there?
DBA -1. Records in development environment less than production2. Optimizer’s perspective matters than yours
Recommendation -
Generate a Query execution plan to understand the execution plan chosen by the database optimizer.
An execution plan (Explain plan) helps to understand• Number of rows examined• Use of indexes • Missing indexes• Type of join• Information on sorting, etc
![Page 57: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/57.jpg)
57
Database tier – Stored Procedures
Set of SQL statements stored on the server and are pre-compiled. It takes incertain arguments and processes that code with those arguments atexecution time
Points to ponder before we use Stored Procedures:
1. Ease to scale database compared to application tier?2. Ease to write and maintain?3. Ease to debug4. Need for specialized skill set5. Performance gain6. Reduction in traffic between application and database tier7. Reusability & transparency8. Security
![Page 58: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/58.jpg)
58
Database tier – Use prepared statements
Prepared statements are the ability to set up a statement once, and then execute itmany times with different parameters.
Advantages:
• Save on query parsing (especially if we execute the query several times)
• Prepared statement caching inside the J2EE server connection pool avoids re-preparing of the statement
• Cache is database-wide – Improves efficiency across applications if they all use similar parameterized SQL
![Page 59: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/59.jpg)
59
Database tier – Application tier switch for Database servers
Good Practice For Multi-Tenanted Applications – Provide an Application level switch that can dynamically select the
database to connect which itself may reside on one more database instances
![Page 60: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/60.jpg)
60
System – File descriptors
Linux limits the number of file descriptors that any one process may open; the default limits are 1024 per process which generally is not sufficient for production usePlan to set this to a value which is more than the maximum number of TCP connections expected on the server.
![Page 61: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/61.jpg)
61
F5 BIG-IP® Local Traffic Manager
Internet/MPLS
IntroductionSaaS Three Tier Architecture – Consolidate & Scale
C1
C2
C1
C2
Application Tier
Web Tier
Database Tier
Applications
Reports
![Page 62: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/62.jpg)
62
SaaS Three Tier Architecture – Consolidate & Scale
Perimeter Security
Internet/MPLS
C1
C2
C1
C2
Application Tier
Web Tier
Database Tier
Applications
Reports
C3
C4
C3
C4
Cloud Capacity increases to serve more users and is resilient to failures
Cloud Capacity increases to serve more users and is resilient to failures
![Page 63: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/63.jpg)
63
ECached DataC
Internet/MPLS
SaaS Three Tier Architecture – Caching
Application Tier
Web Tier
Database Tier
Applications
Reports
Perimeter Security
Compress Static Content
C1
C2
C1
C2
Static RequestUse browser cacheDynamic Request
Transactions
Server Cache
Cache-enabled Browser
Reports
Static RequestDynamic RequestReport Request
Cached Data
Minified & compressed Static content served by Web tier and cached by browser with appropriate version
Minified & compressed Static content served by Web tier and cached by browser with appropriate version
![Page 64: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/64.jpg)
64
SaaS Architecture – Monitoring Production Servers
Performance counters on Production Servers monitored using TCSMonITPerformance counters on Production Servers monitored using TCSMonIT
![Page 65: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/65.jpg)
65
SaaS Architecture – Monitoring Remote Machines (On-premise)
Internet/MPLS
Home/ Mobile User
LAN: Clinic
Audit
Remote backup
Monitor
Install Software
Administer from any location
Remote Management Server
Distributed LAN: Corp HQ
Behind NAT: Franchisee
Experts connect remotely for immediate support
Experts connect remotely for immediate support
![Page 66: 1 SaaS - Performance Management 20 th Nov 2013. 2 ITaaS Platform We studied the business challenges of SMBs to redefine the Technology Consumption Model](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d955503460f94a7e6f1/html5/thumbnails/66.jpg)
66
SaaS Architecture & Business Continuity
SaaS users continue to operate in face of disaster
SaaS users continue to operate in face of disaster