scalability: pushing the limits · pushing the limits neha rai, tim schooley, tejas patil. 2. 3 ......
TRANSCRIPT
![Page 1: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/1.jpg)
PNSQC Presentation, October 2014
Scalability: Pushing the Limits
Neha Rai, Tim Schooley, Tejas Patil
![Page 2: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/2.jpg)
2
![Page 3: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/3.jpg)
3
So what is “Scalability”?
“Scalability is the ability of a
system to successfully handle an
increasing workload, or its ability
to be expanded without major
architectural changes, or
detriment.”
For a good read, check out “Characteristics of Scalability and Their Impact on
Performance”, André B. Bondi, AT&T Labs
![Page 4: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/4.jpg)
4
Once upon a time, there was…
![Page 5: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/5.jpg)
5
Policy
User authentication
Auditing
Reports
Key escrow
![Page 6: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/6.jpg)
6
[1] By Brian Snelson (originally posted to Flickr as Final assembly) [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
[2] By U.S. Navy photo by Lt. Arwen Chisholm [Public domain], via Wikimedia Commons
Mission Critical(Photographs for example only ; not indicative of actual customers)
[1] [2]
![Page 7: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/7.jpg)
7
DBAH AH
McAfee ePolicy Orchestrator
Drive Encryption
Agent-Server Communication Interval (ASCI)
(Agent Handler(s))
McAfee Agent
![Page 8: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/8.jpg)
8
Effects of changing the ASCI, with 100,000 clients
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Ave
rag
e n
um
be
r o
f clie
nt re
qu
ests
p
er
se
co
nd
Agent-Server Communication Interval (hours)
![Page 9: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/9.jpg)
9
So we integrated into ePO...
Policy
Users
Auditing
Reports
Key escrow
![Page 10: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/10.jpg)
10
• Does it meet our scalability expectation?
We had a number in mind, based on existing ePO scalability guidelines (goal of 100,000).
• Will it work for existing customers?
Mission critical. It has to work.
•Does it meet our quality goals?
Do we know what happens when the system reaches its limits?
Are we ready to roll it out?
![Page 11: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/11.jpg)
11
Without testing the limits,
bad things™ can happen.
[Confidence in] ability to meet demand
Inve
stm
ent ($
) in
pu
shin
g t
he lim
its
0
![Page 12: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/12.jpg)
12
Key take-away #1:
Understand the risks of
not doing Scalability Testing(this will help you determine if you need to do it)
![Page 13: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/13.jpg)
13
DBAH AH
“5_G>I’N^O!”
What to test?
• Covers many
components
• High impact failure
case
• Simple result to
interpret
• Covers high
complexity code
• Covers a very
common use case
1.5x ASCI
![Page 14: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/14.jpg)
14[1] By David B. Gleason from Chicago, IL (The Pentagon) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons
[2] By Rev Stan, Harry Potter studio tour: The cupboard under the stairs [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Flickr
OK, where are you going to get allthe clients from?
(Note: this will depend on your architecture)
You might not have
one of these!
![Page 15: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/15.jpg)
15
DBAH AH
ePolicy Orchestrator
“5_G>I’N^O!”
N nodesN nodes
![Page 16: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/16.jpg)
16
So why did we have to simulate?
(Optimization)
x 100
Not testing Steve’s true ability
to cook under heavy demand.
![Page 17: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/17.jpg)
17
So why did we have to simulate?
Meaningful data helps uncover
the limitations of the system.
(for us, it was user data)
![Page 18: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/18.jpg)
18
Example causes of limitations
Larger calculations
Cache memory
Connection pools
Contention
Disk IO
Network IO
Recommendation: keep the hardware consistent, and don’t
use virtualization unless you expect your customers to use it.
![Page 19: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/19.jpg)
19
![Page 20: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/20.jpg)
20
![Page 21: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/21.jpg)
21
Key take-away #2:
Define your test scenarios sensibly.
Aim for broad coverage
Keep acceptance criteria simple
Target complex areas
Suitable tools for gathering results
![Page 22: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/22.jpg)
22
# Nodes
# r
eq
ue
sts
/se
con
dSo how did we run the tests?
(the goal was 100k, but we needed to find the limit)
Increasing cost (setup time)
![Page 23: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/23.jpg)
23
• The first scalability tests were fireworks.
– Crashes, memory leaks, deadlocks.
– All uncovering high severity defects.
• We identified bottlenecks, then optimized.
– Expensive calculations.
– Expensive SQL transactions.
• We finally obtained a level of confidence.
– Now we’re ready to sell it.
What were our findings?(bearing in mind this was a new integration)
![Page 24: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/24.jpg)
24
The results
ePO, Agent Handler and SQL server hardware:
Dell PowerEdge R515, 2.6GHZ 6C, 8GB, 7.2K SATA
Dell PowerEdge R715, 2x 2.0GHZ 8C, 8GB, 15K SAS
ASCI: 4 hours
Nodes: 100,000
Average requests per second (to DB): ~7
All tests passed on this configuration.
Notes: no other point products were installed.
These results are advisory only.
![Page 25: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/25.jpg)
25
How might this apply elsewhere?
![Page 26: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/26.jpg)
26
Cost vs Gain
[Confidence in] ability to meet demand
Inve
stm
ent ($
) in
pu
shin
g t
he lim
its
0
Law of diminishing returns
![Page 27: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/27.jpg)
27
Key take-away #3:
Invest in Scalability appropriately
(it’s a bottomless pit, if you want it to be)
![Page 28: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/28.jpg)
28
Summary
• Understand the risks of your system not meeting its Scalability requirements.
• Define your test scenarios sensibly.
• Invest appropriately in Scalability testing.
• Have fun, and enjoy the fireworks!
![Page 29: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/29.jpg)
29
Remember to take the in-app Presentation Survey!
![Page 30: Scalability: Pushing the Limits · Pushing the Limits Neha Rai, Tim Schooley, Tejas Patil. 2. 3 ... (it’s a bottomless pit, if you want it to be) 28 Summary •Understand the risks](https://reader034.vdocument.in/reader034/viewer/2022050205/5f58cc420b6b3a703b3b5788/html5/thumbnails/30.jpg)