May 29th, 2003

Curtis Anderson

Sivaprasad Padisetty

Microsoft Confidential

Overview of Presentation

Part I – Software Development and Automation• The Project Cycle• Benefits of Automation• Effective Automation

Part II – Test Distribution and Development• Test Distribution Systems• Test Development • Summary

Software Development and Automation

Curtis Anderson

Microsoft Confidential

The Project CycleTest Investment in Automation

• The investment in test automation starts early• Focus on adding testability to spec/design phase• Investment usually peaks as Development slows feature coding

Planning (M0)

Major MilestonePhase M1,2…

Release Phase

Time

TestingInvestment

TestingPhase

Microsoft Confidential

The Project Cycle Automation Payback

Planning (M0)

Major MilestonePhase M1,2…

Release Phase

Time

AutomationPayback

TestingPhase

• Return on Investment is near zero early• Largest gains at project end game and sustaining engineering

Microsoft Confidential

Benefits of AutomationWhat does Automation Bring to the Project?

•Consistent Test Execution•Stability Guarantees•Reliability Metrics•Reproduce-able Quality Assessment

Microsoft Confidential

Advantages of AutomationWhy Automate ?

• Machines instead of People• Run existing tests automatically on a new build

or version• Exact same test happens every time

• Do More Testing• Simplistic, repetitive (BORING) tests are run for

free. People spend time looking only at results• People are free to ad hoc and search for bugs

Microsoft Confidential

Advantages (continued)• Automation can perform costly, hard, or impossible to

execute tests• Perf, Stress, Scalability, Config, OM/API, Matrix

• Consistency and Reproducibility• Performance, Regression

• Reusability• How many File Open drivers are there?• Logging, etc.

• Ensure a level of stability before a human spends time• Keeps the ‘patient healthy’• Easy to hand off (external teams & sustaining engineering)

Microsoft Confidential

Risks and Problems• Over-reliance on automation results• Its easy to generate lots of results that creates too

much noise• Maintenance• Exact same test happens every time• False failures and false positives• Skill set to produce• Re-inventing the wheel

Time Management is Key!

Microsoft Confidential

Test Categorization

• Unit Test• Interoperability• Localizability• Interoperability• Performance• Reliability• Scalability• Security• Globalization

• Checkin Test• Build Verification• Acceptance• Test Pass/Comprehensive• Regression• DogFood/Deployment• System Integration• Code Coverage

What does it do vs. When does it run

Categorizing tests helps target testing and keep things focused.

Microsoft Confidential

Make Your Automation Effective

• Time Management is key• Cost of authoring• Re-use everything • Write solid automation• Document well• Base all automation on test cases • Be willing to punt if it’s not working• Run the right automation at the right time• Roll up the right metrics to your project

Microsoft Confidential

Test Distribution System OverviewWhy do we need a system?• Standard way to execute tests• Automate the execution• Standard reporting and history

What does it do?• Repository for managing automation• Test distribution• Report Results• Track & manage failures

Test Distribution and Development

Sivaprasad Padisetty

Microsoft Confidential

Test Distribution System Components

• User Interface• Middle Tier• Database• Machine Management/Controller• Client Agent• Test Harness• Machine Imaging and other common tasks• Results Viewer• Email

Microsoft Confidential

Test Distribution System Features/Requirements of a good system

• User Friendly, Modular, Flexible• Integrated Automated/Manual test support• Distributes execution – lab management• Multi stage, multi machine• Offline execution and Private network support• Test Version Control• Object Model• Queuing and Resource management• Hardware and Software tracking• Configuration management & Matrix support• Deployment/Image management• Results reporting

Microsoft Confidential

Test Development

• Abstract from code as much as possible – the less code that changes to add a test case the better

• Separate infrastructure code from test code• Re-use as much code as possible• Modeling and Data Driven tests help scale

Microsoft Confidential

Test DevelopmentGood Authoring is:

• Focused• Consistent• Accurate & Reproducible• Portable, Easily Leveraged• Impervious to Feature Changes

• Feature churn• Language, even application, user, etc.

Cost / Benefit is the ONLY measurement (Don’t forget sustain engineering)

Microsoft Confidential

Test Development Writing Automation is Software Development

• Log-Log-Log-Log-Log everything• Plan & Design • Develop coding standards• Write maintainable code• Document you code and your test cases• Peer-reviews before check in• Use a version control system (Source Depot or VSS)• Test your test.• Log-Log-Log-Log-Log everything

This code will last as long as the product code!

Microsoft Confidential

Test DevelopmentDevelop Configuration Agnostic tests

No hard coded anything!• links like files in the definition• No dependency on a specific machine name• No dependency on a specific lab• Should be package-able.• Configuration changes & dependencies should be

clearly defined.

Microsoft Confidential

Reporting and Metrics

• Good logging saves time• Individual Log files from Test Execution• Roll up summary reports for all tests• Available through system UI and email• Project roll up• Metrics help with project status

Microsoft Confidential

Summary

• Test Automation is a competitive advantage for MS• Lack of Automation hurts innovation in the product• Test Development is Software Development• Many resources and experts exist across the

company

Microsoft Confidential

Test Automation ResourcesAutomated Testing Handbook – Linda G. HayesSoftware Test Automation, Efffective Use of Test Execution Tools, Mark Fewster and Dorothy

Graham

Internal• Windows (WTT & WCTI)

• http://wttweb, http://team/sites/wtt, http://winwebtest/wcti and http://wttforum• Test Categoization doc link: http://teams/sites/wcti/stc/default.aspx• Join Partner DL wttpart, to participate in Shared Development community.

• Office • http://office and http://oasysnet• Automation Strategy Doc -

http://officenet/Teams/DnR/DnR%20Test/Specs/Office%20Automation%20Strategy.htm

• Office automation contacts – mailto:offauto• DevDiv

• http://devdiv and http://maddog/ • MadDog forums - http://maddogforum/forums/• MadDog General Docs -

http://maddog/MadDog%20General%20Docs/Forms/AllItems.htm

Questions?