testing – microsoft office david hansen twc test manager

Testing – Microsoft OfficeDavid Hansen TWC Test Manager

13 years (+2 internships) at Microsoft Started in Outlook (email, attachments, setup, automation) Now shared Office TWC team (security, performance,

reliability, feedback) Education

BS in Mechanical Engineering from BYU in 1992 MBA from University of Washington in 2006

Personal Grew up in Reno, NV. Married for 15 years to Wendy 3 children (Hayley 13, Dillon 10, Mali 8)

Contact info: [email protected]

Testing’s role in shipping commercial software The kinds of challenges testers face What makes a great tester

Not (intentionally) Recruiting for Microsoft

Typical Office/Microsoft PM/Dev/Test team ratio is 0.5 : 1 : 1

More precise definitions at http://www.microsoft.com/college

Micro-version of product lifecycle

Designed for quick spin of information between triad

Goal is ‘dogfoodable’ feature quality before checking into ‘main’

PlanningStart Feature Crew

Deliver Feature Specification

Deliver Dev Design Doc

Deliver Test Design Spec

Deliver Detailed Feature ScheduleSpec Inspection

Implementation

Deliver Feature Code

Deliver Test Automation

Check-in Test

VerificationPerform Code review

Run check-in tests

Test code on a private release build

Check code into main branchFeature Crew SignoffBuild Verification Test

*Feature Development

Office TWC Challenges

Data has significant impact on performance results

Scale and performance testing about pushing limits

How do you get all that data in there?

Automation? Disadvantages

Homogenous data sets. Lists all have similar fields Stress and functionality test cramming data

Memory leak could bust you Change in functionality/paths break test

Generating lots of data takes a long time. Time is your most valuable resource as a tester.

Attach existing database? Disadvantages

Toughest problem in developing a server product: upgrade

Product changes constantly break solution End up writing lots of build x to build y migration code

instead of new features

Tester owns the solution Drive any product changes with Dev/PM Author tools as necessary Validate process Generate performance results that are consistent,

reliable, and accurate Then you can start entering bugs…

Targeted attacks are more popular (going after assets instead of fame)

Office installed on a vast number of PCs (especially government/business)

Fuzzing (http://en.wikipedia.org/wiki/Fuzzing) has risen as popular tool for finding buffer overruns

Smart vs. Dumb fuzzing Competing against lots of motivated people with

more time. How do you scale? Distribute the problem Automate to deal with data explosion

Move the battlefront Soldiers inside the walls Sanitize (Wringer or MOICE) Source analysis (catch it in the code)

Aware and constantly learning about changing attack vectors

Build infrastructure and tools to allow scaling out Validate data coming from the tools Provide ideas on moving the battlefront Validate the features

Try to break out of the box Verify box doesn’t break workflow, functionality,

performance, etc…

Feature only works with Office in a bad state How do you reproduce various states Are you giving someone a broken fire extinguisher?

Validity of the data Can developers fix the bug? Minimal impact on user experience. They’re already

mad. Categorize crashes for decision making No Personally identifiable information

Write test plan Automation Validate data for developers and managers Scale of the service

Tools Simulate crash and doc recovery Scale test the service Service health monitoring

Process Set goals for the organization Drive investigation and action on the data Help other testers across office use the data

Fixing bugs creates bugs Beta feedback:

Verbose – Lots of data Squeaky wheel effect with customers. Not

democratic. How can you tell if you’re making it better or

worse?

WinXP SP1example:

SharePoint survey and update feature Offline store to local store migration Null in filename Office Min/Max functionality Assembly code for exploit

Passion for software and technology Curious about how things work Raw smarts and quick learners CS background and coding skill Test mentality (malicious mindset) Communication & collaboration skills Deal with ambiguity & drive for results Customer focus Detail oriented and persistence Broad and deep technical knowledge

A genuine passion for the work we do.From the day I started in Testing I loved it. I didn’t just like

it, I loved it. I love finding bugs. I love learning about how things are implemented and using that knowledge to break the software. I love coding tools that help make it easier to find bugs faster and more efficiently (wEatmem, WinWhere).

I knew only Pascal, Fortran and a little Basic when I came

to MS. I immediately set out teaching myself to program in C and write Windows apps after that. I loved learning to code and I applied what I learned to writing simple games and tools (Pegged, Dr. BlackJack, WinWhere, wEatmem, etc). I loved all of this. I had an innate genuine passion for testing and coding and applying it to making software better.

testing – microsoft office david hansen twc test manager

Documents