1
Perforce Load Testing - AgendaPerforce Load Testing - Agenda
• Speaker introduction• Presentation scope• Corporation background• Load Test initiating: motivation, goals• Load Test planning: software, hardware, licenses, clients• Load Test execution: user emulation limits, escalation• Load Test control: monitor clients and metrics• Load Test closing: charts, analysis, lessons learned• Questions?
2
Perforce Load Test – Speaker IntroductionPerforce Load Test – Speaker Introduction
• Dan Healy – [email protected]– Current SCM admin based in London, UK– Former SCM admin for BofA in San Francisco & Concord, CA– Former index, currency and hedge fund application developer at
Wells Fargo Investment Advisors and Barclays Global Investors inSan Francisco
– Former Defence Dept. developer at White Sands, NM– Academic background in physics, astronomy and computer science– Solar eclipse chaser and mountaineer
3
Perforce Load Test – Presentation ScopePerforce Load Test – Presentation Scope
• Primary focus on why and how the load test was conducted• Sterling test results should not be a surprise to anyone• Provide a list of items to consider when planning,
configuring and executing a Perforce load test• Project Management Institute (PMI) methodology
4
Perforce Load Test – Corporation BackgroundPerforce Load Test – Corporation Background
• Bank of America– First/second largest financial institution in the world– 200,000+ employees, 10,000 developers– 21,000,000 online banking customers– 443,000,000 ATM transactions per month– 3.9 billion bankofamerica.com hits per month– Pioneered consumer loans and branch banking– Historical technology innovator
• First credit card – Bankamericard now known as Visa• Magnetic ink numbering on checks
– Current technology innovations• Distributed Computing• Continuous Integration• Nanosecond f/x transactions
5
Perforce Load Test - InitiatingPerforce Load Test - Initiating
• Application development highly dependant on globally distributed teams– ClearCase insufficient– 3 separate rogue development teams began using Perforce
• Architecture Review Board (ARB) certification required– Management skeptical regarding Perforce
• Perforce load test added to certification submission– Determine any hardware or performance limitations on BofA
hardware running extraordinary usage– Attempt to exceed 10x peak ClearCase usage, 24/7
6
Perforce Load Test - PlanningPerforce Load Test - Planning
• 2000 temp user licenses from Perforce• Software
– Latest version of Perforce – 2006.1– Perforce support load test engine
• Created a client workspace template called “bulk”• Runs sync, flush, info & files, “x” iterations, “y” users, unique prefix for user IDs
– Server performance monitoring – RRDTool– Control Panel Administrative Tools – Performance– Excel spreadsheet
• Test servers– similar setup to production environment
• Primary server in UK – Linux 4 CPU i686, 2.6.9 kernel• Proxy servers in Chicago & Charlotte, Solaris 5.8, Sun Ultra-250• 3 155MB T1’s between US & UK
• Test clients– Cooperation from several colleagues to use their PCs– Invaluable use of training room PCs in Chicago & London
7
Perforce Load Test - ExecutingPerforce Load Test - Executing
• How many users can each PC emulate?– PC disk space limitations vs. ample workspaces– Settled on a 300MB depot and workspace– 5-60 users per PC
• Used Control Panel Scheduled Tasks– Typical activity: 20 users/50 iterations every 10 minutes– Annoying pop up window
• Escalated usage to 800 users• Occasional manual remove & submit
– 969 files, 8 folders, 56.2MB• Focused on average background and peak activity
8
Perforce Load Test – Monitoring & ControlPerforce Load Test – Monitoring & Control
• PC performance tool and graphics ensured clients were not limiting activity
• Every morning, made sure all clients were active using Perforce GUI
• Use RRDTool before and during major activity– Number of connections– Number of processes– % CPU– Load average– Memory usage– Manually recorded of number of users
• Determined production ClearCase and Perforce usage• Determined server peak levels
– Memory– % CPU– Load average
9
Perforce Load Test – RRDTool – Number of ProcessesPerforce Load Test – RRDTool – Number of Processes
10
Perforce Load Test – RRDTool – Active ConnectionsPerforce Load Test – RRDTool – Active Connections
11
Perforce Load Test – RRDTool – Production Active ConnectionsPerforce Load Test – RRDTool – Production Active Connections
12
Perforce Load Test – RRDTool – Production ClearCase ServerPerforce Load Test – RRDTool – Production ClearCase Server
13
Perforce Load Test – RRDTool – Memory UpgradePerforce Load Test – RRDTool – Memory Upgrade
14
Perforce Load Test – Closing – Background ActivityPerforce Load Test – Closing – Background Activity
15
Perforce Load Test – Closing – Peak ActivityPerforce Load Test – Closing – Peak Activity
16
Perforce Load Test - ClosingPerforce Load Test - Closing
• Analysis– BofA hardware remains effective up to about 500
simultaneous intensive users– Most vulnerable metric is memory
• Extrapolated peak at about 1000 users
• Lessons learned– More user participation and/or training rooms– Employ UNIX/Linux clients– Capture of PC performance graphics– Automate peak activity– Run Scheduled Tasks as alternative user