seagate technologies: an insider’s guide to best practices for establishing a performance testing...

Reneé WilsonSQA, Seagate Technology

Seagate Technologies: an insider‟s guide to best practices for establishing a Performance Testing Center of Excellence

Establishing a PC Center of Excellence

June 2010

22

Establishing a Performance Testing CoE- Agenda

Introduction

History

Transition from LR to PC

Building a Performance Testing Center of Excellence

The Performance Testing Lifecycle

CoE Quality Elements

Scripting Tips and Tricks

Survival Guide


June 2010

33

Seagate: Storage Leader

Seagate is the world’s leading provider of storage devices

Q3 FY2010*: 50.3M drives shipped; revenue of $3.05B

Provides storage devices for Enterprise, Desktop, Mobile

Computing, Consumer Electronics and Retail Markets

Builds Hard Disk Drives and Solid State Drives

31% overall market share

Broadest product offering in the industry – largest customer base

Ownership and vertical integration of critical technologies: heads

and media

52,394** employees worldwide

* For third fiscal quarter ended April 2, 2010** Includes interns, contractors, and agency temps


June 2010

44

Seagate Global Presence

Oklahoma City, OK Teparuk & Korat, Thailand

Paris, France

Longmont, CO

Springtown, N. Ireland

Wuxi, China

Shanghai, China

Tokyo, JapanBeijing, China

Delhi, India

Shrewsbury, MA

Suzhou, China

Taiwan

Amsterdam, Netherlands

Shakopee, MN

Bloomington, MN

Sunnyvale, CA

Santa Clara, CA

Fremont, CA

Scotts Valley, CA

Science Park, Singapore

Johor & Penang, Malaysia

Woodlands, Singapore

Ang Mo Kio, Singapore

HQs, Admin and/or Sales

Design

Customer Support

Drive & Component Mfg

Data Centers


June 2010

55

History- Our LoadRunner Roots

LoadRunner users for the past 10 years

3 Controllers at one site - 6 Load Generators at primary data center

Growing test volume made test setup and results data very hard to manage

Loss of data due to decentralized desktop operations

Customers needed more visibility to previous testing and trends over time

VU Licensing spread across the three controllers

IT growing beyond what LoadRunner capabilities

Initial motivation of upgrade from LoadRunner to Performance Center based on

need for distributed testing

Data Centers started to globalize

Needed better processes to schedule and allocate assets

Required more effective use of pool assets


June 2010

66

Transition from LR to PC

Did not happen over night… Transition took six to nine months

Support for early adopters

Change management for late bloomers

Discovered new infrastructure requirements

Staging

Install &

Test

Prod

Install %

Validation

Production

Testing with

PC

Test Process

redesign

Accounts for

DC and Devel

teams

SiteScope

install and

usage

Fully

weaned

from LR

1 2 3 4 5 6 7 8 9 10 11 12Months

LoadRunner Performance Center

Built new

CoE

Processes

Process

continuous

improvement

A few “hiccups” along the way – Protocols, Script activities, Scenarios

Overall improvement to our Performance Test processes – Performance

Center helped us get better at what we do


June 2010

77

Performance Testing CoE- Essential Elements

Group Charter

Connections with other CoE‟s and enterprise processes

Documented process

Leverage decentralized/global operations

Performance testing lifecycle

Quality and best practice standards

Script Best Practices

Knowledge management

Collaborating with Customers and Stakeholders

Customer education


June 2010

88

Charting A Course– Developing a Group Charter

Who do you represent? Charter may be different depending your organizational reporting structure

What services do you provide? Standard fare of testing functions vs. one-off test activities

What tasks are the Performance Testing Team chartered to perform?

What tasks are NOT offered?

Who are your Customers?

What is your service SLA?

What are your hours of service?

Is there a project prioritization process?

Is the Performance Test Team sized appropriately for success?

What are your communication mediums and who are your target

audiences?


June 2010

99

Making the Connections– Connecting with Other CoE

Tools and Processes

Application

Release Data

Identification

of latent

defects

Monitor

use cases

Readiness for

Production

Identification of

Performance-

based defects

Test use

cases

Readiness for

Test


June 2010

1010

Connecting The Dots- Drawing the Critical Communication Lines

SDLC Where does Performance Testing fit in to you development processes

Diagnostics What other tools can be leveraged to enhance the Performance test effectively

Security Testing and Management What synergies between the two groups can be leveraged

Release Management Reduce testing “surprises” by having a better radar for upcoming service needs

Deployment Readiness Processes Where does Performance testing fit in to this process

Does the Performance test team get a vote?

Post Deployment Monitoring What “intimate” application knowledge can be leveraged from the Performance test team?

Communication How closely is the Performance test reporting aligned with other reporting mediums including general Project

Management Reporting?


June 2010

1111

Documenting Your Process …and Making It Visible

High level descriptive process

Drives Performance Test Team functions

Sets Customer expectations forengagement and timeline

Key Elements of Process

Entry point for new requests

Entry point for retest

Touch points with Customers

and Developers

Quality oversight

Needs to be Accessible and Visible

We use wiki in Seagate

Pe

rfo

rma

nc

e T

es

t G

rou

pC

us

tom

er

Contact

Performance test

Team with test

schedule/scope &

Clarity info

Estimate

Test

efforts

Prioritize test

in

Performance

Test queue

Perform walkthru

of application and

use cases

Write/validate

Performance

Center

scripts

Perform Load

tests (nominal,

peak, scalability,

WAN)

Start

Publish Test

Report and

Archive

Data

Done

Perform test

review with

project team

if required

Provide

required data

sets user

accounts, etc.

Write

Performance

Test Plan

Approve

Test Plan

Provide Load

Test

Requirements

Initial Contact

and Discovery

1-Week

Test Pipeline Lead Time

2-Weeks

Test Planning

and Approval

1-Week

Scripting and

Shakedown

Testing

1-Week

Test

Performance and

Reporting

1-Week

Performance Test Timeline

Performance Testing Business Process


June 2010

1212

Process & Service Improvement Tips

Engage with Development Teams early in SDLC process

Earlier engagement of Performance Test Team yields a better chance for success

Better test requirements through dialog and collaboration yields more meaningful

tests and data

More up-front commitment yields high ROI through delivery of a higher-quality test

assets

Offer a standard set of

Performance Testing Services

Follow your process

Look for improvementopportunities

Look for points of connectivity

with other CoEs

Educate your Customers


June 2010

1313

Global collaboration platform for performance testing

Meeting place for test planning and execution

Available to Performance Test Team

Available to test customers and data consumers

Collaboration elements

Schedule and planning

Test assets

Real-time test data

Test results and trend analyses

Allows almost 24x7 test support and development with small staff

Operations activities performed at a lower cost

Higher-priced resources can perform more planning and liaison functions

Team presence resident at developer‟s venue

Has facilitated specialization of Performance Test Team members

Empowers and motivates team members

Maximizing CoE Assets- Leveraging Global Operations


June 2010

1414

Performance Testing Lifecycle-Making Your CoE Process Visible to

Customers and Team

Preplanning

• Test calendar

• Test Services

• Test Capabilities

• Example Test Plans

• Example Test Reports

• Maintenance windows

• Blackouts

Planning

• Test Plan

• Test Schedule

• Customer engagement

• Environment avail.

• Diagnostics

• Script and test

walkthroughs

Testing

• Test alerts

• User accounts

• Test collaboration

meeting

• Diagnostics

Post test

• Test Result

Announcement

• Test Reports

• Trend analyses

Performance testing is a process rather than an event

Each phase of Performance test lifecycle includes artifacts and deliverables

Needs to be visible and available to stakeholders

Performance Test Lifecycle


June 2010

1515

Critical

Artifact

Performance Testing Lifecycle- Preplanning

Mostly a Customer resource

Provides Performance Test Calendar visibility

Planning for environment availability

Includes educational material

Example artifacts and deliverables

Helps set Customer expectations

for services and lead times

We post all of this on our (wiki)

webpage


June 2010

1616

Performance Testing Lifecycle- Planning

Close engagement with customer

Earlier in the SDLC the better

Development of detailed test plan

Agreement on test schedule

Needs to be coordinated with Test Calendarin line with available resources and assets

Be cognizant of rework-loops

Test Plan walkthrough with stakeholders

Plan for required diagnostics

Script quality review with test team

Posting of final test plan (document management)

• Change history

• Goals and scope

• Topology being tested

• User account info

• Special application info

• Test use cases

• Scenarios

• Acceptance criteria

What’s in a

Performance Test Plan?

Critical

Artifact


June 2010

1717

Performance Testing Lifecycle- Testing

Schedule test meetings

Stakeholder can log into Performance Center and watch test in real-time

Email Load Test Alert to all Stakeholders

Queue and validate diagnostic

functionality

Perform full battery of required testing

Analyze each test before moving on to next

Often tests do not succeed on the first try

Perform Testing

Critical

Artifact


June 2010

1818

Performance Testing Lifecycle- Post Test

Analyze and create report

Post report to common accessible location

Send result email with executive summary of results

Send to same stakeholders that received Load test Alert

Have standard nomenclature for results (i.e. pass, fail, conditional pass, interim

results, etc)

Provide links to reports

Include Trend data if possible

Always follow-up with Customer

Document any lessons-learned

We post 3 reports Traditional Word doc

HTML Summary

Wiki news item

Critical

Artifact

Wiki news item page


June 2010

1919

Quality Touch-points- A Few Simple Activities That Can Have

a Big Impact

Performance Test Plan is your contract with Customer Formalize the plan

Tests should not be performed until this is correct/approved

Walkthroughs with Customer critical

Scripting standards and best practices Keep a list – follow it!

Script code walkthroughs

Small-scale shakedowns

Help Customers develop realistic schedules Tests rarely succeed on the first pass

Always have test review with Customer and Stakeholders

Look for ways to make CoE processes and work products visible

Look for ways to improve your processes – avoid open loops


June 2010

2020

Be the Scryptkeeper ! Write „em down – Integrate into your own Best Practices

Annotate and comment! It‟s just code…

Use Actions to manage script flow

References to external sites like hitbox or doubleclick should be removed

Randomize think times – be aware of distribution

Run script as a thread (as opposed to a process)

Make use of “Snapshot on Error”

Use “Continue on Error” like garlic… sparingly

Use “simulate parameter” tool to preview parameter allocations – especially useful when using “unique” option

Avoid “Silent” Failures – always encapsulate everything in named transaction and use web_reg_find

- Our Top-Ten Scripting Best Practices


June 2010

2121

Scripting Fun- A Few Scripting Techniques We Really Like

Sybil Make use of variable transaction names when you

need more measurement granularity – scripting

short cut – use sparingly!

Don't Re-invent the Wheel Use or create Correlation Libraries for hard-to-script applications. Some vendors provide

them! Recording Options/Correlation

The Sledge Hammer Sometimes the only way to

verify content is to see how

much stuff came back

Oracle Reform When recording Oracle NCA Protocol requires modification

to default.cfg file (in script folder) as well as “Names”

change on Oracle Server side


June 2010

2222

Leveraging the Knowledge- Building Organizational Knowledge into Your CoE

An organization continues to advance its knowledge by preserving the lessons it learns

Fourth Generation Management, Brian Joiner

Scripting with new or multiple protocols

Tips, tricks, and time-savers

Quality and reliability improvements

Architectural knowledge

Troublesome applications

Ways to leverage existing assets

Better runtime settings

Scenario management

Transaction name nomenclature

Documentation management

Always be on the lookout for ideas and concepts

Make sure your team has forum for collecting and managing

Find a way to store and institutionalize the knowledge

Use quality reviews as a medium to find new ideas and reinforce existing ones

Areas where we’ve benefited and grown our organizational knowledge


June 2010

2323

Collaboration- Making Customers and Stakeholders Active Partners

Real-time performance worth a thousand reports Encourage Stakeholders to watch the test

Keep test plans accessible and visible Document management

Quality Center

Ensure visibility of test results

Tie-in with other Stakeholder groups Quality-assurance

Operations

Monitor Team

After test failures – be part of “solution” Collaborate with teams to look for root-cause

Suggest other tests that could help drill down to root-cause

Engage in post-mortem activities

Always look at what went “right” and what went “wrong”

Feedback lessons-learned into CoE Business Processes


June 2010

2424

Customer Education- Supporting Your Customers

Give them as much access to self-help as possible

Engage with Customers on all projects

Offer training Primers

Familiarity and training sessions

Account Management Group accounts

Tailor permissions and views for test participation

Write reports (or amend them) for Customer consumption Plain English vs. IT-Speak goes a long way!

Customers LIKE trend analyses

Be accessible

Schedule regular business reviews with Customers and Stakeholders Review upcoming challenges, issues, and testing


June 2010

2525

Surviving the Game- Practical Survival Tips for Performance CoE

Establish an SLA for Performance CoE Services

Prioritize your projects

Establish and enforce standards

Engage Customers – they need to have “skin” in the game

Leverage collective group knowledge

Apply the appropriate level of diagnostics

Attempt to preserve scenarios for reuse

Reboot controllers and load generators on a regular basis

Make use of user notesin results summary

Perform analysis in Performance Center immediately after run

Provide interpretation of results in test reports


June 2010

2626

CoE Enablers- Performance Center Capabilities

We Really Like Customer Involvement

Global access

Test calendar and scheduling

Asset pooling

Automated central repository Scripts

Scenarios

Test Data

Trending Big time saver!

Customers greatly appreciate

User privilege management

Peripheral tool access

A few features we’d really like to see…

Tighter integration with Quality Center

Leverage test plans and reference results

Defect/requirement connectivity

Document repository on a project level

Hierarchical project architecture

Projects within programs


June 2010

2727

Q&A

28 ©2010 Hewlett-Packard Development Company, L.P.

To learn more on this topic, and to connect with your peers after

the conference, visit the HP Software Solutions Community:

www.hp.com/go/swcommunity

seagate technologies: an insider’s guide to best practices for establishing a performance testing...

Documents