using managed test teams an alternative to out-sourcing or in-sourcing testing ståle amland, hulda...

UsingManaged Test Teams

An Alternative to Out-sourcing or In-sourcing

testing

Ståle Amland,Hulda Garborgsv. 2, N-4020 STAVANGER, NORWAY

Mob:+47 905 28 930, Phone: +47 51 58 05 87, Fax: +47 51 58 55 24e-mail: [email protected]

ICSTEST 2001

©Amland Consulting 2

Presentation OutlineManaged test teams (MTT):

What?Why?Experiences

Alternative risk and reward profiles

Alternative test organisations

Summary


The Managed Team Concept

Externally resourced – internally managedBest practice from various sectorsShared risk via mutual agreed objectives

The team must understand their responsibility

Optimise use of internal resources, IT & BusinessProcedural framework

GuidelinesStandardsTemplatesCoverage criteriaBut: the team should use their expertise


The CasesCase 1 – Railway Industry

Y2K programme100+ Business Critical ApplicationsUSD30 million budget, 140 staff4 x Managed Test Teams – 40 staff

Case 2 – Telecommunication IndustrySoftware house, product releaseUSD2,2 million budget (testing)12 - 15 staff, 6 months

1 manager1 operation10 – 13 planning / preparation


ExpectationsProfessional Approach

Testing expertise and abilityProfessional team management

FlexibilityTeam Rivalry

Potential conflicts

Delivery to timescaleCommercial Inertia

Concentrating on T&Cs instead of deliveries


Experiences in a Multi-Vendor Programme

BackgroundHigh profile, multi/thread, multi environmentTime driven / no flexibilityInternal resource heavily committedTesting a critical activityExternal expertise availableExternal remediation / development


Experiences continuedIssues

Integration with internal resourcesManagement (corporate and project)Test Environment / operationUsers (availability)

Integration with external resourcesProgramme management / test authority

Learning curveStandards, methods and proceduresApplicationsReporting


Experiences continuedIssues (continued)

AccountabilityTest CoverageQuality standards

Team structure Permanent vs. contractorTeam Expertise (leaders and doers)Combination of business knowledge and testing experienceNumber of teams (balance load, avoid management overhead)


Experience: Expectations vs. reality

Professional Approach - Achieved

Flexibility - Achieved

Team Rivalry - Not Met

Delivery to timescale - Achieved

Commercial Inertia - Not met


Expectations vs. reality cont.

Lessons learntEnsure correct team structure

Permanent / ContractorsBusiness Knowledge / testing experienceRetain experienced teams

Benefit from external experienceChallenged existing practise

Ensure critical mass is retainedKeep the core team together by providing new challenges continuouslyAvoid dilution of core team


Presentation OutlineManaged test teams (MTT)

Alternative risk and reward profilesFixed PriceTarget BudgetTime and Material

Alternative test organisations

Summary


Risk Profile – Fixed PriceAgreed fixed price prior to test execution

Used for System Test , Acceptance Test or equivalent “stand alone” test phasesPrice agreed as part of specification, design and/or codingTest organisation takes all risk

Obvious problemsLack of information about qualityChallenge to agree acceptance criteriaCooperation between client and test organisation

Pros. & Cons.Easy budgeting Test organisation might take short cuts…Price will have to include high risk contingency


Risk Profile – Target Budget

Incentives Time & MaterialAgree target budget

If overrun: reduced payment to test organisationIf under run: split the savings (”profit”) between client and test organisationTest organisation takes risk

Obvious ProblemsChallenge to agree acceptance criteria

Pros. & Cons.Incitements for both organisation to increase productivityTest organisation might take short cuts…


Risk Profile – Time & Material

Testing take as long as it takes…Deadlines will be agreed in project planExtensions must be documentedTest organisation takes very limited risk

Obvious problemsTo meet deadlinesAgree criteria for extensions / acceptance criteria

Pros. & Cons.Focus on quality and product risk, not deadlines onlyComplicated to control for client


Presentation OutlineManaged test teams (MTT)

Alternative risk and reward profiles

Alternative test organisationsTest FactoryOutsourcing

Summary


Test Factory – ObjectivesA permanent organisational unit, responsible for

Test methodologyTemplates / standardsTrainingTest environment (hw&sw)

Services providedTrainingTest managementTest environment management

NOT responsible forTests and test dataQualityHand-over to production


Test Factory Pros. & Cons.Benefits

Small but permanent organisational unitEfficient / focusing on re-use across businessGood control of test level in each projectEfficient management of test environmentGood integration with project management and development

DrawbacksNo additional resources, lack of flexibilityNot focused on managing contractors


Outsourcing – ObjectivesDefinition

Tasks which are carried out by a third partyTypically done offsiteOption:

Integrating tasks into current test process

Building relations to an experienced test and project management organisation

Providing experienced personnel when required


“Outsourcing” alternativesOutsourcing the total SQA function

Adding Testing Expertise

Adding Testers

Insurance (“Double checking”)

And variations…


Outsourcing Pros. & Cons.Benefits

High flexibility regarding resourcesTest expertise availability

DrawbacksIntegration with internal organisationTeam rivalry – lack of communicationPotential contention between developers and testersLack of business knowledgeMust be internally managed


SummaryManaged Test Team

A viable Approach?Absolutely!

BenefitsProfessional Expertise and flexibility

ChallengesSelecting the right teams and team structure

Variation of Outsourcing


References

Hamblen, G., Managing Outsourced Testing (On time and On Budget), www.stickymind.com, [email protected], J., Outsourcing, STQE Magazine Nov./Dec. 1999, Volume 1, Issue 6, www.stqemagazine.com

using managed test teams an alternative to out-sourcing or in-sourcing testing ståle amland, hulda...

Documents

expertise slide

system test

test execution

amland consulting3

acceptance test

amland consulting4

dilution of core team

deliveries slide