test magazine - december 2012-january 2013

Inside: Regression testing | Crowd sourcing | Application complexity

Gojko Adzic asks are we testing the right things?

The right stuff?

Visit TEST online at www.testmagazine.co.uk

Volume 4: Issue 6: December 2012

INNOVAT ION FOR SOFTWARE QuAL I Ty

TE

ST

:

IN

NO

VA

TI

ON

F

OR

S

OF

TW

AR

E

QU

AL

IT

YV

OL

UM

E

4:

IS

SU

E

6:

DE

CE

MB

ER

2

01

2

Feature | 1

December 2012 | TESTwww.testmagazine.co.uk

Leader | 1

INNOVAT ION FOR SOFTWARE QuAL I Ty

Inside: Regression testing | Crowd sourcing | Application complexity

Gojko Adzic asks are we testing the right things?

The right stuff?


Volume 4: Issue 6: December 2012

INNOVAT ION FOR SOFTWARE QUAL I TY

TE

ST

: I

NN

OV

AT

IO

N F

OR

SO

FT

WA

RE

QU

AL

IT

Y

VO

LU

ME

4

: I

SS

UE

6

: D

EC

EM

BE

R

20

12

So the smartphone is 20 years old. It wasn’t called the ‘smartphone’ at the time it appeared – the firstcommerciallyavailable

devicehadthemorespecificepithetthe‘IBMSimon’.Althoughdevicescombining telephony and computing werefirstmootedasfarbackas1973 (were there any precedents insci-fiIwonder?)itwasnotuntilearly 1994 that the Simon went on sale, and the term ‘smartphone’ did not come into use until 1997, when Ericsson described its GS 88 ‘Penelope’ concept as a ‘Smart Phone’ (accordingtoWikipedia).DespitehavingworkedforEricssonuntilacoupleofyearsbeforethisdate,Ican’tconfirmthisfactoid,butseenoreason why it shouldn’t be true, they werealwaysgreatatbeingfirstwitha concept; but less accomplished at capitalising on their prescience.

The defining feature that makes a smartphone is that it allows the running of third-party applications – the ubiquitous app as we now know it (presumably thanks to Apple). Arguably a major milestone on the route to smartphone dominance was RIM’s launch of the first Blackberry in 1999, but the launch of a Blackberry with voice, data, browser, messaging and organiser applications in 2002 marked the first true smartphone as we now know it.

It was in 2007 (that long ago!) that Apple launched its trailblazing iPhone, the product that really put mobile computing on the map and took it to the masses – amidst much skepticism at the time I seem to recall. The iPhone itself is now on its fifth iteration and there are numerous other brands and operating systems snapping at their heels. We have come a long way in a short time (a motto that could be used for computing as a whole).

Of course all that mobile computing power needs applications to run on it and we have seen an explosion of activity in this area with apps hitting the market that vary from the sublime to the ridiculous, from the pointless to the invaluable. The challenge for the tester is that any of these apps might be expected to port over to any number of platforms and operating systems and work as well on any device. Literally every day there is a system update, new device or novel interface for the tester to deal with and the pile gets bigger and bigger as anyone getting a shiny new smartphone for Christmas will surely agree.

Until next year have a very happy Christmas!

Matt Bailey, Editor

The rise of the ‘smartphone’

It was in 2007 (that long ago!)

that Apple launched its trailblazing

iPhone, the product that really

put mobile computing on the

map and took it to the masses –

amidst much skepticism at the time

I seem to recall. The iPhone itself is

now on its fi fth iteration and there

are numerous other brands and

operating systems snapping

at their heels. We have come a

long way in a short time (a motto

that could be used for computing

as a whole).

Matt Bailey, Editor

Editor Matthew [email protected] Tel: +44 (0)203 056 4599

Toadvertisecontact:Grant [email protected]: +44(0)203 056 4598

Production & DesignToni Barrington [email protected] Cook [email protected]

Editorial&AdvertisingEnquiries31 Media Ltd, Unit 8a, Nice Business ParkSylvan Grove London, SE15 1PD

Tel: +44 (0) 870 863 6930Fax: +44 (0) 870 085 8837Email: [email protected] Web: www.testmagazine.co.uk

Printed by Pensord, Tram Road, Pontllanfraith, Blackwood. NP12 2YA

© 2012 31 Media Limited. All rights reserved.

TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available.

Opinions expressed in this journal do not necessarily reflectthoseoftheeditororTESTMagazineoritspublisher, 31 Media Limited.

ISSN 2040-0160

For exclusive news, features, opinion, comment, directory, digital archive and much more visit

www.testmagazine.co.uk

Subscribe to TEST free!

Published by 31 Media Ltd

www.31media.co.uk

Telephone: +44 (0) 870 863 6930

Facsimile: +44 (0) 870 085 8837

Email: [email protected]

Website: www.31media.co.uk



Inside: Test Automation | Software security | Performance testing

Derk-Jan de Grood asks is it time to say

goodbye to a separate testing phase?

The end of the road

for the Test Phase?


Volume 4: Issue 3: June 2012INNOVAT ION FOR SOFTWARE QUAL I TY

TE

ST

: I

NN

OV

AT

IO

N F

OR

SO

FT

WA

RE

QU

AL

IT

Y

VO

LU

ME

4

: I

SS

UE

3

: J

UN

E

20

12

Test Automation | Software security | Performance testing





The end of the road The end of the road The end of the road

for the Test Phase?for the Test Phase?


UAL I TY

Inside: Agile development | Mobile apps testing | Testing techniques

Richard Eldridge explains how to tackle testing in the fi nancial sector.

Institutional applications testing


Volume 4: Issue 4: August 2012


Sponsoring TEST’s 20 Leading

Testing Providers.Editor’s Focus page 29-39.

TE

ST

: I

NN

OV

AT

IO

N F

OR

SO

FT

WA

RE

QU

AL

IT

YV

OL

UM

E

4:

IS

SU

E

4: A

UG

US

T

20

12

Agile development | Mobile apps testing | Testing techniques

Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to Richard Eldridge explains how to tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.tackle testing in the fi nancial sector.

Institutional Institutional Institutional Institutional applications testingapplications testingapplications testingapplications testingapplications testingapplications testingapplications testingapplications testingapplications testingapplications testingapplications testing


OFTWARE QUAL I TY

Sponsoring TEST’s 20 Leading

Testing Providers.Editor’s Focus page 29-39.

Inside: Test automation | Agile testing | Mobile app testing

What graduates can expect from a career in testing.

A bright future


Volume 4: Issue 5: October 2012


Sponsoring TEST’s 20 Leading Testing Providers.Pages 35-47

TE

ST

: I

NN

OV

AT

IO

N F

OR

SO

FT

WA

RE

QU

AL

IT

Y

VO

LU

ME

4

: I

SS

UE

5

: O

CT

OB

ER

2

01

2


Contents | 3

1 Leader column The rise of the smartphone.

4 News6 Are we testing the right things? Isitjustifiedtokeepinvestingintestingandmaintainingfeaturesthatarenotused

enough? Software delivery consultant Gojko Adzic has seen far too many teams that suffer from huge maintenance costs of their legacy test suites, but never ask if the features tested by those suites are still actually useful.

10 Determiningregressiontestsviadefectprobability Smart decisions have to be made about what to perform regression testing on to

reduce the risk of releasing software with potential defects. Faisal Qureshi has some formulae that could help.

12 Rising in the east Currently Russia controls three percent of the offshore software development market and

is the third leading country (after India and China) among software exporters. Russian test automation engineer EvgenyTkachenko shares his experience of how the Russian testing industry is rising to cope with this growth.

14 Productprofile–Greattoolsmakebettertests Becky Wetherill explains what’s happening with Borland’s open test management

solution Silk Central.

16 Manyhandsmakesoftwarework Dieter Speidel and Mithun Sridharan of PASS Technologies explain why crowdsourcing

can offer a rapid and cost-effective approach to software testing and provide a remedy for the apps quality crisis

22 ThepathtosuccessfulAgiledevelopment TechExcelexplainswhySpecDD(specification-drivendevelopment)isleadingtheway

down the path to successful Agile software development.

26 The Testing World – Count your blessings In this most festive of seasons, Angelina Samaroo is applying the logic of Agile to work out where all the money has gone.

28 Theexternalquestion Why don’t companies like health checks and why do they fail? The fact is that many are

just telling people what they already know and clients aren’t getting any real value from these exercises according to Richard Mort.

32 CanfuzzingbeAgile? Agileenablesearlytesting,buthassofarnotbeenfocusedonfindingcriticalsecurity

flaws.TESTreportsfromAri Takanen’s recent webinar where he highlighted automated security testing techniques based on fuzzing, with examples of how they can be integrated into the Agile development process.

34 Conqueringapplicationcomplexity The complexities and interdependencies introduced by new technologies add

extra pressure to get applications to market in quicker time, in an increasingly competitive market. How can testers cope in this ever more complex environment? Michael Allen reports

38 Behind the mask While Dynamic Data Masking can seem a simple and attractive option for sensitive data,

are its limitations understood? Huw Price highlights some of the more common pitfalls to watch out for with dynamic masking.

42 Managing international distributed test teams in Scrum How do test teams deal with the problems of building, running and managing scrums?

DavidCotterell, European region director of Qualitest explains how his company managed it on an international scale.

45 DesignforTEST–Testingthetestprocess Most organisations have a test process but how good it is and how up it date it is can

vary enormously – with the expected consequences. Mike Holcombe reports.

48 LastWord–Afishytale Why did DaveWhalen become a software tester? It was an epiphany really.

Contents...DECEMBER 2012

10

22

6

38

TEST | December 2012 www.testmagazine.co.uk

Automated regression testing – a legal necessity

T estingspecialistSQSSoftwareQualitySystemshashelpedlawfirmIrwinMitchellintroduce automated regression testing

intoitssystemsdevelopmentprogrammes. ThefirstprojecttobenefithasbeenIrwinMitchell’s legal case management system, where SQS’ expertise, especially in automated testing,hashadadirect,positiveimpactonprojecttimelinesandqualityaccordingto the company.

Since 2009, Irwin Mitchell has been developing and improving its legal case management systems. These business- critical systems involve complex workflows, updating documents, diaries and other business systems. Accuracy and reliability of the information is critical – not only could a system error lead to corruption in data quality in multiple systems, but failure to send the right information at the right time could have serious legal ramifications.

Testing and quality assurance were critical considerations when updating the case management systems and Irwin Mitchell asked SQS to help develop and implement test automation strategies to support its development work. The focus of the test automation was on regression testing: maximising test coverage of each system release and reducing the risk of defects being introduced into production systems.

SQS and Irwin Mitchell developed a test automation Proof-of-Concept (PoC) to demonstrate that new case management system releases could be quality assured reliably, quickly and with little manual intervention, freeing up internal Irwin Mitchell resource to concentrate on other strategic inititiatives.

Following the success of the PoC, automated regression testing was expanded to cover Irwin Mitchell’s full case management system

development and SQS delivered a fully scalable and maintainable test automation framework within just three months.

Ian Fowles, head of programme and portfolio delivery at Irwin Mitchell comments: “Irwin Mitchell is dedicated to implementing best-of-breed testing practices so we selected SQS due to its depth of expertise in testing and test automation. Testing throughout our systems lifecycle, from beginning to end, plus identifying and resolving any issues before they arose, was just one benefit of working with SQS. Another was SQS’ ability to run tests out of hours and distribute any changes over multiple machines, implementing the system’s release into our existing operation. As we continue to broaden the scope of this test automation project to other areas of the business, there will no doubt be even more potential to add value as a result of SQS’ automation work.”

Compuware Corporation has announced that Experian has implementeditsdynaTracefor

applicationperformancemanagement(APM).CcordingtoCompuware,Experianis now able to reduce time-to-market and improvethequalityofnewapplications andservicesbyquicklypinpointingthesourceofperformanceissuesduringtheearlystagesofdevelopmentandqualityassurancecycles.Thisenablesittolaunchnewservicesin shorter time periods, which is a key businessobjectivefortheirdata and analytics specialist.

“We need to deploy the latest and most advanced services for our customers as quickly as possible, and this is where the information delivered by dynaTrace offers the highest value,” said Paul Hill, head of support and quality assurance at Experian. “By implementing it, we have the ability to dig down into application code on a granular level during the quality assurance (QA) stage, identifying the source of any performance problems picked up during load testing.”

“With our software architects, development and testing teams based in a number of locations across different time zones, it’s important that communication is as clear and as detailed as possible,” said Hill. “With dynaTrace, we are able to identify exactly where performance issues originate and share transaction snapshots that different teams can utilise consistently. This helps us

to remedy the root cause of any problems quickly and effectively, allowing us to bring our latest services to market much faster, and ultimately deliver the best customer experience possible.”

“For organisations like Experian, reducing time-to-market while improving the user experience of online applications is a key objective,” said David Cooper, UK regional director for Compuware’s APM business unit. “In order to maintain a competitive edge, companies now require more information about exactly where performance issues are occurring during the testing and development stage—not just after the service has been deployed. With dynaTrace’s end-to-end approach, Experian gets actionable insight into where its applications can be improved, ensuring their customers receive the best user experience possible.”

Quality assurance for development

Christmas shoppers turn to smartphones

Newresearchfromhasrevealedthatchanges in consumer purchasing behaviourintheUKthisChristmas

willimpactretailerswhodonothavemobile enabled websites. Approximately one in ten shoppers who used a PC to do their Christmas shopping last year will use their smartphones in 2012. With 33 million smartphones and tablet computers in the UK,forretailerswhoarealreadyworkingontighter and tighter margins, this nine percent couldbethedifferencebetweenaHappyChristmas and a miserable one.

Regional data revealed that London sees the highest number of people, one in six favouring smartphones to do their Christmas shopping. Wales came up top with 93 percent of people favouring their smartphone or PC to do their Christmas shopping. Interestingly, people in the North West of England are one of the least likely (one in five) to buy anything on the internet, whilst nearly one in five consumers in Scotland and the South of England plan to use both a PC and a smartphone to do their Christmas shopping.


NEW

S

ITprofessionalshaveseenpayriseby 4.2percentoverthecourseoftheeconomic downturn, according to

analysisofONSdatacarriedoutbyRandstad Technologies, a specialist IT recruiter.

Since 2006, the average technology professional’s salary has increased by £1,534 per year to £38,262. But, pay rises in the tech sector have lagged behind the national average. The average annual salary of a UK employee has risen by 11.4 percent since the last boom. Inflation has outstripped pay by 7.9 percentage points, and the average pay rise would have

needed to be 19.3 percent simply to keep track with the rising cost of living over the period. As a result of inflation, the average technology professional has seen pay fall by 15.1 percent in real terms in spite of the nominal improvement.

However, the rate of pay inflation has not been consistent across the industry. Information and communication technology managers have seen their pay increase the most, with salaries increasing by 7.3 percent to £47,254 in the last six years, while software professionals have seen their average pay rise by 5.7 percent to an average of £37,000.

Mike Beresford, managing director of Randstad Technologies, said: “It’s tempting to think that salary rises have been hit across the board through the downturn as companies look to keep costs down. But actually it’s been more like the board game snakes and ladders, with different skill sets commanding different premiums. While pay rises as a whole haven’t matched the rate seen in the wider labour market, those in management positions, such as project managers, have seen the biggest pay rises in spite of the downturn.

“The IT jobs market shows signs of building momentum, but future salary rises will not be uniform, and will be driven by employer demand for specific niche skill sets,” added Beresford. “From a development perspective candidates with expertise across the Microsoft technology stack, and those with proven capability in UI UX and data analytics are seeing their salaries increase as they are in such short supply.”

IT INDuSTRy’S SALARy SNAKES&LADDErS

McAfeehasreleaseditsThreatsReport: Third Quarter 2012, whichexplorestechniquesin

cybercrimeaswellastheglobalevolutionofcyberexploits.Thelatestreportuncoversnewdetailsof“OperationHighRoller,” tracks that mobile malware almost doubledthepreviousquarter’stotal,andrevealsanall-timehighindatabasebreaches.McAfeeLabsalsosawjumpsinsomecategoriesofmalware,includingransomware and signed binaries. Rootkits and Mac malware continue to rise, while password-stealing Trojans and AutoRun malware also trended strongly upward.

“Cybercrime exhibits few signs of slowing down,” said Vincent Weafer, senior vice president of McAfee Labs. “Though we tend to highlight the numbers, the fact is that we continue to see increased sophistication of attacks. Cybercrime, hacktivism, and cyberwarfare are in a continual state of evolution. Everyone from governments to large enterprises, small business and home users are facing a wider range of digital threats from these forces, as they gain more actionable intelligence on their victims, and leverage the newest attack platforms and exploits tools to launch their campaigns. We all need to equip ourselves with basic situation awareness to our online risks and how best to prevent and combat these threats.”

Cybercrime’s global expansion

Flow simulator and trafficgeneratorfirstA rgonDesign,aleadingdeveloper

ofhighperformancesoftwareapplicationsformanycore

communications processors has launched Argon Blaster which it claims is the industry’s firstflowsimulationsolutionforgeneratingrealistic,Internetscaletrafficloadsandapplications to test networking and security equipment.

According to Argon, Blaster delivers a line rate, timing accurate, flow simulation application on an affordable PCIe acceleration card for use in standard x86 platforms. This enables OEMs to cost effectively distribute a high performance simulation and traffic generation solution throughout the engineering organisation,

significantly reducing development time and cost, while simultaneously increasing product quality.

It is also designed for enterprise and carrier network operators for performance testing of flow based cyber security and network analytics applications. It enables network managers to verify that these systems are designed and deployed in a manner to match expected network loads.

“Flow processing has become pervasive in cyber security, network analytics and software defined networking designs,” said Steve Barlow, CTO at Argon Design. “As a result, new, cost effective testing and simulation tools are required to stress these products in ways not previously available.”

6 | Feature

Is it justified to keep investing in testing and maintaining features that are not used enough? Software delivery consultant Gojko Adzic has seen far too many teams that suffer from huge maintenance costs of their legacy test suites, but never ask if the features tested by those suites are still actually useful.

Are we testing the right things?

6 | Cover story



Cover story | 7

My assumption, not based on any serious industry research but just on

experience working with many differentteamsindifferentmarkets, is that as an industry wewasteahugeamountoftime and money testing and maintaining the wrong things. Withthepressureofshorterdeliveryphases,ubiquitousforthe IT industry at the moment, teams that make better decisions aboutinvestingtimeandmoneyindeliveryandqualityrelatedactivities,willbeabletogainasignificantcompetitiveadvantage.Butwewillfirsthaveto change our understanding ofquality,andstopdeclaringvictorytooearly.

FogofconfusionIn 2007, Nokia was on the top of the world, one of the most recognisable and valuable brands. Wanting to move into the internet services space, where it had to face established players on their home turf, the company bet big on an innovative service offering called Ovi. According to Tim Brown, author of the seminal book on Design Thinking, Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation (http://amzn.to/WlvNGV), Ovi was a key success story to showcase design thinking. The book was published in 2009, and Brown declared Ovi a huge victory: “Design thinking had enabled Nokia not only to explore new possibilities but also to convince itself that these possibilities were sufficiently compelling to move away from its strongly entrenched and previously successful approach… Today Ovi is one of the operating business divisions of the company, and Nokia – a technology leader – has reinvented itself as a service provider.” Yet history will remember Ovi completely differently, or most likely not remember it at all.

If ever there was an IT equivalent of harakiri, this was it. Nokia – a

technology leader – has “reinvented itself as a service provider” but the service was struggling to find enough customers to gain a serious foothold in the market. Meanwhile, Apple and Google took over the mobile world. Not giving up, Nokia almost bet the company on this in 2010, and started rebranding other services to push the Ovi brand.

This led to ridicule in media, such as an article by Andrew Orlowski in The Register (http://bit.ly/Y5RtMw): “People often ask, ‘What's a Nokia? – is it some new kind of yoga or a fashionable new diet?’ Then you remind them – it's the platform for the Ovi mobile services experience – and the fog of confusion quickly clears”. In 2011, the feedback loop finally closed and someone decided, to quote another Orlowski article from The Register, to give Ovi a mercy bullet.

FulfilmentofpotentialOnce a leader in the market, Nokia is now a follower in a pack led by iPhone and Android device manufacturers. I can’t really speak about the quality of their production process, as I was never an insider, but as a former occasional user of their devices I have the utmost respect for the technical quality of Nokia’s products. I’m sure that design thinking brought in usability and customer engagement to technical quality, but what good is all that if customers don’t end up using it?

The readers of this magazine are those whose role is primarily to assure quality of software products and services, and this story shows how dangerous it is to declare victory too early and ignore the feedback from the market. The key thing missing from the Nokia Ovi story is the fulfilment of the potential. Current quality assurance processes prove that functionality is present, performant enough, usable and so on, but that all proves potential for someone to actually use our products, not the fulfilment of the potential. If nobody is using our stuff, should we really care that all tests passed?

Most of the teams I worked with apply quality related activities only during delivery. We declare victory when our unit tests, acceptance tests and exploratory tests pass. But those things are just telling us that we did what was asked for, not that the results actually deliver quality. Even usability testing with real users shows that they can use our stuff, not that they are actually doing it in the market. If all the tests for a software feature pass but not enough people use it, can we really claim that quality was assured?

Is it justified to keep investing in testing and maintaining features that are not used enough? I’ve seen far too many teams that suffer from huge maintenance costs of their legacy test suites, but never ask if the features tested by those suites are still actually useful.

Scoring business (own)goalsEven proving that something is used in the field is still too early to declare victory. End-users might love something dearly and use it frequently, but it might be for the wrong reasons or not in line with overall business goals. The early 1990s marketing campaign of vacuum cleaner manufacturer Hoover (http://bbc.in/UUHDH7) is one of the most famous cases that illustrates that point. As a way to clear up old products from warehouses, Hoover offered anyone who spent more than £100 two free return flights, initially only to Europe.

With an overwhelming response from customers and travel agents unable to keep up with the demand, the promotion was extended to US flights. It doesn’t take a math genius to figure out that people were soon buying Hoover products only to get a surreal discount on intercontinental flights, and the whole thing imploded. The campaign left Hoover with a costing of roughly £50 million and resulted in six years of court proceedings with disgruntled customers, remembered in history as one of the worst marketing disasters. Hoover’s promotion was used, but was a business failure.

Another Brinkerhoff’s suggestion is linking those impacts to actual company business goals, and measuring the contribution of individual impacts. Why do we want to help settlement team members to process exceptions faster? How does that contribute to the vision of our product? Discussing those things up front also allows us to measure such higher-level impacts and decide if the software change is actually successful from a business perspective.

8 | Cover story

www.testmagazine.co.ukTEST | December 2012

A potential solution for this conundrum is in the work of Robert Brinkerhoff, in particular his book The Learning Alliance, on applying system thinking to HR development (http://amzn.to/T6PSQu). Trying to answer why large organisations waste a huge amount of money and time on training programmes that do not make a big difference at the end, Brinkerhoff suggested in 1994 that the way companies measure training programmes was wrong. This message should have been translated to software two decades ago, but it was completely ignored. Much in the same way, many companies waste a huge amount of money on IT programmes that do not really make a big difference in the end. For example, according to a research published by the BCS (http://bit.ly/TZSxfE) commercial organisations across the European Union lost 142 billion EUR on failed IT projects in 2004 alone, mostly because of poor alignment with business objectives or business strategies becoming obsolete during delivery. We measure the wrong things.

Brinkerhoff suggests measuring success of training programmes against a change that the training was intended to create in someone’s work or behaviour, an impact on someone’s way of working. Translated to software, this would mean by measuring quality against an intended impact on the behaviour or work of our users and stakeholders. The problem many readers will no doubt spot with this is that very few organisations actually define those impacts upfront. The knowledge about this exists in some senior stakeholders’ heads but is rarely communicated and almost never evaluated. At the same time, having this information up front would enable us to take the test-driven concepts much further - not just to software but to actual business impacts.

For example, instead of declaring the role and purpose in a user story (“As a settlement team member, in order to process important exceptions, I want…”) I started asking people to declare the change in someone’s behaviour. Don’t tell me just “in order to process important exceptions”, tell me how would the processing differ from the current situation. From my experience, this opens up a fantastic discussion on measuring those impacts. Say that the difference is “process important exceptions faster”, we can start discussing how much faster, over

what period, and what slows down the process at the moment. Then we can test for those impacts once a user story is delivered, and see if it actually ended up leading to the expected impacts.

Another Brinkerhoff’s suggestion is linking those impacts to actual company business goals, and measuring the contribution of individual impacts. Why do we want to help settlement team members to process exceptions faster? How does that contribute to the vision of our product? Discussing those things up front also allows us to measure such higher-level impacts and decide if the software change is actually successful from a business perspective.

Just to ensure that there is no misunderstanding, I think that unit, acceptance and exploratory tests, usability studies and all the other types of checking, inspections and testing done today, are important. But I also think that they are insufficient to assure quality, without considering if something is really useful and if that usage leads to business success. If we really want to assure quality, we should look for ways to close those two additional feedback loops and inform decisions on where to invest our time in the future, both from design, delivery and quality assurance perspectives.

Gojko Adzic is a strategic software delivery consultant who works with teams

to improve the quality of their products and processes. He won the Jolt Award for the best book of 2012. In 2011, he was voted by peers as the most influential Agile testing professional, and his blog won the UK Agile award for the best online publication in 2010. His new book Impact Mapping: Making a Big Impact with Software Products and Projects is now available from http://www.impactmapping.org

Gojko AdzicSoftware delivery consultant

10 | Feature10 | Feature10 | Regression testing

Smart decisions have to be made about what to perform regression testing on to reduce the risk of releasing software with potential defects. Faisal Qureshi has some formulae that could help.

Determining regression tests via defect probability



Regression testing | 11

There should be a systematic mechanism to easily and accurately determine which features of software should be tested for regression. Thus, using a formula that incorporates multiple criteria would provide a better and more efficient solution for the determination. The formula is probabilistic due to the uncertainty of whether or not a defect will actually be discovered as it is based on future and unknown events.

Sometimes, it is not so easy to determine what to test forregressionwhenanewbuildofsoftwareresolves

certaindefects.Also,inmanycases,itisnotviabletoretesteverythingduetolimitedtimeand/or resources. Thus, smart decisionshavetobemadeastowhattoperformregressiontestingontoreducetheriskofreleasingwithpotentialdefects.

One solution for determining what to perform regression testing on is based on what parts of code the defect fix most closely couples with or interacts with. Another solution would be to base the regression testing on the feature’s likelihood of failure and its impact of failure, which should be determined prior to starting any test execution. There are other criteria, but any one criterion on its own, is inefficient and ultimately inaccurate to the reality of what other areas are truly impacted by the defect resolution.

There should be a systematic mechanism to easily and accurately determine which features of software should be tested for regression. Thus, using a formula that incorporates multiple criteria would provide a better and more efficient solution for the determination. The formula is probabilistic due to the uncertainty of whether or not a defect will actually be discovered as it is based on future and unknown events. The proposed formula to aid in making a smart decision as to what to regression test is:P(defect(FX))=1–((LFX(F0)+CFX)/(dFX+6TFX))The formula provides the probability of finding a defect in the feature (FX) potentially to be regression tested.The variables are:FX: Feature X (feature to potentially be regression tested). X is to be replaced with a feature number.LFX(F0): The code linkage level between feature X and feature 0, where feature 0 is the area of code where a fix was implemented. This variable can take one out of three possible values; 1 – High Linkage level, 2 – Medium linkage level, 3 – Low linkage level.CFX: The code complexity of feature X. This variable can take one out of three possible values; 1 – High Linkage level, 2 – Medium linkage level, 3 – Low linkage level.dFX: The number of defects found for feature X in all previous test cycles combined.TFX: A multiplier value for whether or

not feature X was tested in the previous iteration. There are two possible values; 1 – Tested in the previous cycle, 2 – not tested in the previous cycle.When calculating the probability of finding a defect for a feature based on this formula, we will end up with values in the range of [0,1) where 0 means there is a theoretical probability of not finding a defect, and the higher the values, there is a higher probability of finding a defect for that feature. If the software has 10 features, we can use the probabilistic formula to find out the probability for finding a defect for each feature. Then, we can select the features with higher values to perform regression testing.

ExampleFeature 1:LF1(F0) =2 (medium code linkage)CF1 = 2 (medium complexity)dF1 = 3 (three defects for F1 found so far in testing)TF1 = 1 (was tested in the previous cycle)P(defect(F1))=~0.55

Feature 2:LF2(F0) =2 (medium code linkage)CF2 = 2 (medium complexity)dF2 = 3 (three defects for F2 found so far in testing)TF2 = 2 (was not tested in the previous cycle)P(defect(F2))=~0.73

Feature 3:LF3(F0) =1 (high code linkage)CF3 = 1 (high complexity)dF3 = 5 (five defects for F3 found so far in testing)TF3 = 1 (was tested in the previous cycle)P(defect(F3))=~0.82

Feature 4:LF4(F0) =1 (high code linkage)CF4 = 3 (low complexity)dF4 = 2 (two defects for F4 found so far in testing)TF4 = 1 (was tested in the previous cycle)P(defect(F4))=0.5

From the calculations, if only two features could be tested for regression due to limited time and/or resources, then the results from the formula tells us we are most likely to find future defects in features 2 and 3. However, it is important to note that this is meant to serve as a quantified statistic and human/tester judgment should override theoretical probability.

Faisal QureshiSenior test engineerMotorola Solution [email protected]


12 | Testing in Russia

EvgenyTkachenkoAutomation test engineerInnova Systems www.innovatesting.info

Currently Russia controls three percent of the offshore software development market and is the third leading country (after India and China) among software exporters. EvgenyTkachenko is a Russian test automation engineer with three years experience working for Russian testing specialist Innova Systems. Always happy to share his experience Evgeny gives his opinion on how the Russian testing industry is rising to cope with this growth.

Rising in the east

Overthelasttenyearsthe Russian testing industry has produced manyimpressive

results.OurITcompanieshavemanaged to do much in the wayofadvancement,utilisingmodern approaches in the implementationofsoftwarequalityandeveryonerecognisestheimportanceoftestingas ameansofachievinganexcellentcompetitiveposition in the market.

Only a relatively short time ago, software testing was viewed as a necessary evil for development in many dev teams. Part of this ill feeling can be attributed to poor planning in teams where the Waterfall model is used. Delaying testing until late in the software development cycle contributes to this unfavourable view of testing as developers and testers struggle to meet schedule. Waterfall development isn't new (it has been around since 1970), but some Russian development teams do still use it in its pure form because being a linear model, it is very simple to implement, and the amount of resources required to implement the Waterfall model are minimal.

Test earlyNow most people involved in software development in the Russian IT industries understand the value of testing from the beginning of the development cycle rather than later. It is a truism that the earlier that a fault is detected and removed, the cheaper it is to fix. The benefit of early testing in the development cycle is the early removal of faults and potentially fault causing situations so that they never appear in the delivered product.

It becomes obvious for everyone in the Russian Software Industry that the

common goal is to start the testing process as early in the development cycle as possible. But It is not an easy undertaking and surely not for the faint of heart.

Conferences and exhibitionsThere are a lot of IT conferences that take place in Russia. For many Russian experts and managers the QA conference is a major opportunity to raise the profile of software quality, and also to listen to other software quality experts. It is a great platform to share experience and discus best practise with other professionals involved in the software testing sphere. For the beginners it can also provide the perfect platform for them to network and meet other professional testers.

Some of the major cities in Russia, like Moscow, Kazan, St. Petersburg and Nizhny Novgorod etc, already have thriving communities of software testers. And it has become a tradition that has grown out of the QA conferences that new communities of testers from other cities across the country have been created.

While the level of IT conferences perhaps isn't yet as high as it is in Europe and the USA – the presentations are often not very technical and the speakers are mainly consultants and trainers seeking publicity for their courses – the situation is improving. But to get the latest information about technical advances we still have to make the effort to travel a little further afield and attend international conferences like EuroSTAR or the Selenium Conference.

EducationRussian universities do not offer speciality ‘software testing engineer’ courses. Every specialist in this sphere

is still pretty much self-taught. But this can be a very tricky path for those wishing to get involved because foreign software testing books and manuals are very rarely translated into Russian. As a result, we have a lot of IT companies, but we don't have enough good Russian software testing specialists.It is like the great Russian poet Fyodor Ivanovich Tyutchev said:You can't understand Russia with your brain, You can't measure it with the standard instruments,She has a particular status,In Russia you can only believe.

Now most people involved in software development in the Russian IT industries understand the value of testing from the beginning of the development cycle rather than later. It is a truism that the earlier that a fault is detected and removed, the cheaper it is to fix.


14 | TESTProductprofile

Borland Product Manager with Micro Focus, Becky Wetherill explains what’s happening with its open test management solution Silk Central.

Great tools make better tests

TEST: What is Borland’s background in this area?Becky Wetherill: Borland testing products are made for testers, by testers. Borland’s pedigree is built on years of creating imaginative products that improve testing effectiveness, usability and visibility. We understand the testing environment because we live it every day. That’s why Borland is the brand of choice for testers and test managers, QA Managers, developers and other testing professionals.

TEST: What is Borland’s specialist product?BW: Silk Central is the flagship product of the Borland testing suite. It’s the

central hub that sustains a quality program across all projects and teams, and through the test management cycle, right from development to QA.

Silk Central is Open Test Management software that will work with your source. It unites quality tools and offers full traceability: it is a digital ‘tick list’ that ensures that everything you need to test is tested, leaving a full audit trail. Integrate your Requirements, Source Control, Automation, Defect Tracking and cross-reporting capabilities to boost project quality.

As well as driving effective Open Test Management, it works on every level throughout the development lifecycle. This is the program for central users who want to manage everything from unit test through to performance test.

Silk Central brings disciplined test management to the SDLC and delivers better software with immediate business results, including:•Reducedtimetomarket

through faster delivery;•Improvedproductivityby

reducing rework;•Betteranalyticsthroughincreased

visibility and dashboard control;•OpenTestManagement:geta

complete view of all test activities regardless of the tool.


TESTProductprofile | 15

TEST:WhatsetsSilkCentralapartfromthe competition – and why?BW: As far back as late 2008, a Forrester poll of more than 2,225 software decision-makers at Enterprise and SMB companies revealed that 55% were looking at open source applications. Just three percent were going to decrease or cease their use of open source. With IT budgets getting ever-more stretched it is likely that this number is increasing.

Open Source Test solutions are an interesting proposition but not the complete answer. QA and test teams may need to incorporate information from open source and enterprise systems, so need the assurance of complete visibility. Requirements may come from Word, ReqPro, Doors, Automation from JUnit, MSTest through to QTP, and so on. Achieving the complete view – and managing full control – are significant challenges.

So managing the disparate strands of testing activity can be a concern. If control is the problem, then Silk Central is the answer. It uses plugins and integrations to bring together the information. Need to develop your own plugins? Then use the Open API.

Silk Central enables users to define complex test scenarios and enable teams to test software against multiple configurations – including operating system versions – to track progress and view statistics and detailed reports on recent test activity. Visibility is control: built-in full end-to-end traceability of quality ensures testers see the quality of their work evolve.

Silk Central uses automatic collaboration across the organization to collect, prioritize and control all aspects of tests and test management. It’s the central hub that links processes, requirement management systems, source control systems, test types and issue tracking systems. It is where organizations control and maintain their test assets, from manual tests through to automation, while maintaining functional and non-functional testing.

TEST:What,inaword,defines Silk Central?BW: More than anything else, Silk Central means control. It is underpinned by four key principles:Connect it: Link your tools and processes to create a smoother

delivery. This powerful, open software test management solution includes requirements, automated tests, manual tests executions and issues. It works for both agile and traditional development projects. It directly relates requirements to tests and delivers full traceability to the process. Open test management can link tools together and collate test requirements.Test it: Use Silk Central with your tools and processes to incorporate reports, dashboards and code coverage to improve productivity, and visibility. It provides an integrated framework for high quality software projects. Plan and execute your tests based on needs, time, effort: get the results and address the defects. Track it: Silk Central works with your processes and your view. It delivers continual quality across your business, satisfying the demands of the development and quality testing teams. Have you done what you need to do? Full metrics and analytics are the digital paper trail that prove you’ve met your quality goals. Silk Central delivers the traceability and visibility back to your requirements to ensure you test what you need to – and crystallise everything into a simple go/ no-go decision.Control it: Prioritise and control all aspects of your test by automating collaboration across the organization and use quality goals and risk-based testing to deliver higher quality software, faster.

Silk Central delivers against both the traditional, ‘waterfall’ development approach and the more iterative, agile version. It underpins successful test management by delivering visible metrics, sound analytics and robust quality checks executed against the application under test (AUT).

TEST: How is Silk Central better than other Open Test Management tools? BW: We call Silk Central ‘environmentally friendly’ because it has a flexible approach that works with mobile, cloud and Software as a Service (SaaS) environments. This is cutting-edge tooling that enables developers to implement test processes and practices that meet industry best practice and processes, such as Test Maturity Model Integration (TMMi).

TEST:Anynewfeaturestolookoutfor?BW: Quality goals: apply variables,

including risk and priority to your requirements and tests to achieve better ‘What if’ planning. Quantifying the time and effort required informs better decision-making throughout the process. Quality reporting shouldn’t be left until QA. You cannot achieve 100 percent of your testing: time and cost will affect your decision-making when assessing what to test and Silk Central provides the insight you need to set achievable quality goals.Manual through to automated tests: Silk Central unites assets from Requirements, Manual Tests, Unit Tests, Functional Tests and Performance tests through to execution and defect tracking. Between 50-70 percent of testing is manual and testers working with Silk Central can create, control and manage their manual tests. Testers linking with Caliber can add visualisations (Business Process Flows) to the mix and ‘jump start’ testing by positioning the business process flow as a manual test. Testbook: An exciting new dashboard, delivers real-time visibility to those users working in manual testing. Silk Central users see when changes to scheduled tests occur real time. Posting messages to the dashboard from manual testing increases team collaboration and communication.Multi-tenancy: Silk Central now allows individual clients can manage projects securely within dedicated areas within a single installation.Localisation: This has now been extended to support French and Chinese, adding to the existing German, Japanese and English facilities.

Silk Central has the structure to determine what and how to test. Use it to prioritise, execute and demonstrate how tests respond to initiatives such as risk-based testing. It uses your terminology to define each individual quality goal.

TEST:What’sthefutureforSilkCentral?BW: We always back a winner. Borland’s Agile approach to development enables a swift and effective response to customers’ needs and encapsulate them within a product release and Borland will continue investing in Silk Central. Evolution is driven by customer feedback and every new iteration of Silk Central, the leading test management solution, reflects the needs of client organisations.

Becky WetherillBorland product managerMicro Focus [email protected]

16 | Crowd testing


Dieter Speidel and Mithun Sridharan of PASS Technologies explain why crowdsourcing can offer a rapid and cost-effective approach to software testing and provide a remedy for the apps quality crisis

Many hands make software work

Crowd testing | 17

In today's rapidly growing and trulyglobalsoftwareindustry,success is now more than everdependentonthequality

ofthesoftware.Agoodqualityproductorserviceleadstosatisfiedandloyalcustomers.Thechallenge that most IT managers andexecutiveshaveisacommonone:howtoeffectivelytestsoftwareapplicationsontime and within budget while balancing resources?

Traditionally, software testing was a tedious chore for developers once their real work was done and was based on a set of formal verification mechanisms that were costly to apply. A formal testing approach requires highly specialised talent that is limited or often even unavailable in most organisations. Furthermore, formalised testing cannot be scaled to cover the scope of complex software deployed in the modern IT landscape.

However, several trends are changing the perception in favour of crowd sourced testing. The emergence and massive adoption of the Agile software development methodology that advocates shorter ’sprints’ or iterations of development brings software testers closer to the creative development process. Also, with the advent of web applications, whereby software is deployed on the Internet, bugs become apparent very quickly. Set with a budgetary limit – testers are paid a certain rate per valid bugs found – crowd sourced testing is a highly compelling innovation that can be used to complement existing test cycles, used in pre-release deployments and even post-deployment testing.

TheMicrosoftexperienceWhen it was outlining its new Office 2010 product strategy, Microsoft knew that the stakes were extremely high. Months before it released its new Office 2010 productivity suite, nine million people downloaded the beta version to test the software and to provide feedback. Through this programme, Microsoft collected two million valuable comments and insights from those testers.

According to Denise Carlevato, a Microsoft’s usability engineer, “That's just what you have to do to cater to as broad an audience as possible.” Carlevato and her colleagues from Microsoft's Virtual Research Lab observed how people used new features. Their objective was to make Microsoft Office fit the way millions of people used the product. It was in fact a massive, controlled crowdsourcing project.

Developing a new software product is always exciting, especially to watch ideas take shape and truly become a reality. However, when it comes to testing, we often find ourselves in unchartered waters wondering if the product will actually work in the diverse customer landscape. It is virtually impossible to test the vast number of devices and configurations that web-based software can run on today. Truly robust testing is time consuming and ensuring that every possible permutation and combination of features, localisations and platforms work as intended is nearly impossible.

Too often, buggy code is delivered to the customer who withstands the worst of inadequate testing, especially when faced with the escalating costs of software maintenance and performance. For the software development company, ramifications include distress around brand image, perceived quality, relationships, trust and potential future projects.

Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform towards software quality assurance and control. With this new form of software testing, the product is put to test under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all, bug-free.

It differs from traditional testing methods – generally a resource-limited and therefore time-consuming activity – in that testing is outsourced to a large number of different testers from across the globe. This enables small start-ups to use ad-hoc, quality-assurance teams that they could not afford to resource in-house.

When it was outlining its new Office 2010 product strategy, Microsoft knew that the stakes were extremely high. Months before it released its new Office 2010 productivity suite, nine million people downloaded the beta version to test the software and to provide feedback. Through this programme, Microsoft collected two million valuable comments and insights from those testers.


Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform towards software quality assurance and control. With this new form of software testing, the product is put to test under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all, bug-free.

Why does crowd sourced testing work?To understand why crowd sourced testing works, it is important to understand the set of biases that are common in all testers and test managers. This phenomenon is called, ‘The Curse of Knowledge’, a phrase used in a 1989 paper in The Journal of Political Economy. It means that for a particular subject expert, it is nearly impossible to imagine and look beyond the knowledge the tester has acquired; the set of concepts, beliefs and scenarios that the tester knows or predicts. As a result, it is particularly challenging to think outside the box and conceive the various ways a typical end user would use particular software.

Most testers conduct a battery of tests that they feel is representative and that captures the set of end-user scenarios. The reality is far from this. Any expert tester would accept that it is impossible to capture the complete set of use cases. As a result, critical paths of code under certain scenarios go untested, which leads to problems such as software malfunctioning, production system crashes and lengthy debugging.

Crowd sourced testing circumvents all these headaches by bringing a comprehensive set of code coverage mechanisms and end user scenarios during the design and development stages of software engineering, when the cost of modification is insignificant. This results in identifying critical use cases early on and providing for those contingencies, which reduces software maintenance costs later on during and after product deployment. Besides progressive code coverage, the quality and depth of software testing among various vital software modules is achieved, which ultimately results in higher code quality.

Crowd sourced testing–theframeworkAt the heart of crowd sourced testing is the community that tests a given software product. The community encompasses people from diverse backgrounds, geographies and languages – all with different approaches to software usage. The

community tests any given software under realistic scenarios, which a tester in the core test team may not be able to envision.

Thus, it is easy to observe the broad set of usage patterns that put the software under intense scrutiny. Crowd sourced testing draws its benefits from delegating the task of testing a mobile, web or software project, while in development, on to a large number of internet users working in parallel for comparably short test cycles.

The spectrum of issues that can be uncovered within a short lead-time and within reasonable costs is particularly noteworthy. Testers usually get paid for deliverables, such as test cases and valid reported bugs. Additional small contribution rewards often help to achieve higher project participation rates. Crowd sourced testing providers offer their services based on flat rates for certain test capacities and durations or they charge fixed rates per reported bug, or a combination of both. In general, crowd sourced testing provides a much higher Return on Investment (ROI) compared to traditional means of software testing.

How does it work?Most vendors of crowd sourced testing services provide the platform for managing their crowd testers and the testing projects. Clients specify the type of tests that they wish to have performed and the devices and configurations that the software product must be tested on.

Testers complete a profile, indicating the skills and domain knowledge they have; the devices to which they have access to, and the countries where they reside. Usually, the testers are provided with or are required to submit their inputs for a test plan, which outlines both high level test cases and detailed test scenarios. The plan may also include whether or not the test can be automated and the expected results. A qualified project manager designs or reviews such plans and approves or amends them to cater for the client’s specific requirements.

Each project includes an explanation and access to a forum where bugs and issues are discussed and additional questions can be asked. Testers submit

18 | Crowd testing


documented bug reports and are rated based on the quality of their reports. The amount the testers earn increases as their rating increases.

The community combines aspects of collaboration and competition, as members work to find solutions to specific problems. Forums facilitate networking and discussion of bugs or relevant issues, while rating systems recognise a job well done, which helps participants gain credibility and improved career status.

The crowd sourced testing team ideally works in addition to the organisation's testing team, and not as a replacement.

Checks & balancesSecurity is a crucial element to crowd sourced testing. More often than not, confidential customer information is exposed during application testing. Any breach of this data can lead to serious damage, both to the brand and the business. Test data management ensures the availability and security of data by obfuscating sensitive information for large-scale testing engagements. Masking such information or creating ‘test-only’ data helps maintain privacy and security while using crowd sourced testing services.

In almost all cases, the testers are required to sign a Non-Disclosure Agreement (NDA) when they join the community. Beyond that, the customers can upload a customised NDA, which testers must sign before viewing the project. For projects that require a high level of security, pre-screened lists of ‘white hat’ engineers that have a long professional relationship with the platform company are selected.

By constantly filtering the network of testers to accept only experienced professionals, applicants without formal training and significant professional experience are eliminated. This ensures the quality and the validity of the bugs reported. Last but not least, tests are dispatched to individual testers based on their experience, available material and languages. The testers and test project exposure are continually monitored to ensure both quality and integrity, not only of the test results, but also of the associated environment.

Whattolookoutfor?Crowd sourced testing is best when the product under development is consumer-centric rather than enterprise-centric, such as mobile or web driven consumer applications. A global user base to test the product should exist and the product should be relevant to the community at large. This is also a test for the application’s potential success in the marketplace.

Paul Herzlich, a software-testing analyst, who oversees crowdsourcing services at independent analyst firm Ovum, stated, “If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it.”

The product company should be committed to working with a large group of people and understand that it involves some degree of overhead in such a decentralised test environment. It also requires certain subject matter experts to mentor and monitor various testing efforts as well as offer support and relevant guidance to the testing teams.

With normal employment contracts, employees receive a salary for their contribution and the firm owns any intellectual property developed by the employee during their tenure with the organisation. In a crowdsourcing constellation, people are participating voluntarily. Therefore, it is essential to explicitly state the position on Intellectual Property (IP). For example, a condition of the right to participate is the acceptance that Intellectual Property transfers to the client, to avoid potential risks of IP infringement by the contributors.

A crowd sourced project requires skills in designing the compensation structure, both in monetary and non-monetary terms. The testers are usually paid a certain amount of money for the deliverables provided and for approved bug/issue reports. In some cases, testers prefer non-monetary aspects such as recognition and personal satisfaction rather than monetary compensation, so it is vital to understand the motivations.

In cases where participants are compensated on a per task basis, an incentive for participants to choose speed over accuracy exists. Therefore,

Security is a crucial element to crowd sourced testing. More often than not, confidential customer information is exposed during application testing. Any breach of this data can lead to serious damage, both to the brand and the business. Test data management ensures the availability and security of data by obfuscating sensitive information for large-scale testing engagements. Masking such information or creating ‘test-only’ data helps maintain privacy and security while using crowd sourced testing services.

Crowd testing | 19

www.testmagazine.co.uk December 2012 | TEST

20 | Crowd testing


Thoroughly planned, organised and applied, crowd sourced testing is a great tool for software producers to ensure compatibility of their applications on most major platforms and configurations and it helps to discover and eliminate harmful defects, which survived traditional software QA and testing efforts. The emerging global crowd testing community will specialise into different industry domains, software product categories and customer relationships, most probably materialising into various quasi-vertical categories of on-demand, self-service software testing workbenches.

Mithun Sridharan Marketing consultant PASS Technologies www.passbrains.com

Dieter Speidel CEO PASS Technologies www.passbrains.com

eBay goes crowd testing

The giant auction house eBay has implemented crowd sourced testing

into its testing strategy to further improve software quality. "Managed

Crowd Testing is an excellent tool to further extend our quality assurance

strategy by adding test resources with the most heterogeneous skills,

professional backgrounds and cultural user behaviors," says Michael

Palotas, Head of Quality Engineering Europe at eBay. "Crowd testers think

and work in different ways than internal testers, this is why crowd testing

can still find bugs which haven't been detected through traditional

software testing methods."

Passbrains conducts exploratory functional and usability tests for multiple

national eBay sites. Hundreds of screened and registered professional

software testers are continuously hunting bugs and usability flaws on the

frequently updated online auction platform. Through passbrains.com, eBay

is able to leverage its external test capacities. Many critical bugs occur

under random conditions and in most exotic use cases, that's why the

crowd should be involved before software products are released.

robust governance mechanisms need to be continually monitored and policies regularly updated to reflect the changing trends.

Advantagesofcrowdsourcedtesting:1. Representative scenarios from the

real user base;2. Tight feedback loop;3. Comprehensive use cases, platforms,

tools, browsers, testers, etc; 4. Cost efficiency;5. Diversity among the pool of testers; 6. Reduced time to test, time to market

and total cost of ownership as most defects can be identified prior to deployment;

7. Better productivity and improved product development focus;

Disadvantagesof crowd sourced testing: 1. Governance efforts around security,

exposure and confidentiality when offering a community project to a wide user base for testing;

2. Project management challenges that stem from the testers’ diverse backgrounds, languages and experience levels;

3. Quality assurance efforts to verify and improve bug reports, identify and eliminate bug duplicates and false alarms;

4. Equity and equality constraints in the reward mechanism with

remuneration as a function of the quality of contributions that meets a prescribed minimum standard.

Whatdoesthefuturehold?Crowd sourced testing clearly has its advantages and limitations. It cannot be considered as a panacea for all testing requirements and the power of the crowd should be diligently employed. The key to avoid failure in crowdsourcing would be to use it prudently depending on tactical and strategic needs. It is important for the organisation to embrace the correct model, identify the target audience, offer the right incentives and have a suitable workforce to manage the task results.

Thoroughly planned, organised and applied, crowd sourced testing is a great tool for software producers to ensure compatibility of their applications on most major platforms and configurations and it helps to discover and eliminate harmful defects, which survived traditional software QA and testing efforts. The emerging global crowd testing community will specialise into different industry domains, software product categories and customer relationships, most probably materialising into various quasi-vertical categories of on-demand, self-service software testing workbenches.

TEST | December 2012

22 | Feature


22 | Specification-drivendevelopment

TechExcel explains why SpecDD (specification-driven development) is leading the way down the path to successful Agile software development.

The path to successful Agile development

Agile gains its popularity not by replacing other methods but by integrating with best

practicesfromothermethods.According to market research donebyleadinganalystfirms, 75percentoftheAgileteamsmixpracticesfrommultiple Agile and traditional methods.UseofasingleAgiledevelopmentmethodistheminority at 25 percent.

Welcome changes and delivering working software early are perhaps the two most important Agile principles that have helped its success and wide adoptions. But pure Agile methods

in the real world projects have also been proved to be insufficient in many areas including, no support for total traceability, no model to integrate QA teams with development teams, no clear model for multi-team large Agile project, and no systematic way of improving a team’s innovation as requirement level understanding and communication are often neglected in a pure Agile process.

Today’s applications are becoming more complicated, having to represent complex business processes and end-user logic. Developers have more automation and powerful software design and programming tools. However, as the tools grow more


Specification-drivendevelopment | 23

powerful, it is easier for developers to misunderstand their work assignments. It has become a challenge to sufficiently represent end-user knowledge and business rules so development teams can effectively convert requirements to working software.

So while development technology has advanced to a state that it is relatively easy to develop software prototypes, communicating complicated application logic is still a challenge, especially with globally distributed teams. In this environment, iteratively developing software and incrementally improving it is easier and more feasible than having product managers attempting to provide complete requirements at the start of a project.

Because requirements are so dynamic and ever-changing, working software is an ideal way to bridge the product design team with development teams. Developers can demo the software to and how it satisfies a set of requirements, while the product management team can see how the requirements are modeled by the software. If needed, they can easily modify and enhance the requirements or create new requirements based on how the software functions.

Many of today’s Agile methodologies rely on working software. Development processes are organized into development iterations (or sprints.) The purpose of the iteration is to improve the software with a set of new features improvements or development items. Working software is the only deliverable of the iteration, and at the end of all iterations, the software becomes the final production release.

This approach reduces the risk of project failure, and helps to improve the software to meet the needs of the software’s users. This approach also introduces a potential risk: the software logic is not well documented. Even worse, the decision-making process of how the requirements evolve and are understood is not documented. There is also the loss of process intelligence- the steps of human interaction taken to choose or modify a set of features.

Since the process of development is not documented the reasons why certain actions were taken or certain decisions made can be lost in just a few years of a product’s life.

SpecDD was created to improve two areas of Agile development:•Developmentcanbedonewithboth

software and the product design in parallel. Design is not only about documenting the requirements of the software, but also tracking human interactions and creating a process for improving design.

•Incrementaldevelopmentcanbeapplied not only to the software but also to the design. Change should be managed and tracked with a conceptual understanding of the requirements and not by merely changing the software.

risksofundisciplined AgiledevelopmentThe lack of formal design processes, one of the key benefits of Agile methods, means that it is hard to get meaningful metrics for implementation teams. This lack of measurement is compounded by the global nature of modern teams, who have geographic, language, and cultural barriers. These barriers can lead to longer projects if programmers are disconnected from the business logic of the application and do not correctly implement customer requirements. The constant communication that is required to keep the pace of Agile projects is also more difficult to maintain for these dispersed teams. The entire evolutionary development model can quickly breakdown into chaos.

Balancing Agile with business needs: SpecDDIn order to mitigate these risks, a design process needs to be added on top of the development process. This conceptual framework allows customers, product managers, and implementation teams to interact in a measurable way. Instead of implementation occurring isolated from customers and designers, specification-driven development (SpecDD) provides the needed process, without adding

SpecDD was founded by Dr. Tieren Zhou to provide a hybrid Agile methodology to manage both Agile and non-Agile projects with the same set of development principles. SpecDD teaches you how to integrate Agile practices with requirement, quality, and project management best practices.


24 | Feature


24 |Specification-drivendevelopment

overhead to the project timeline. SpecDD was founded by Dr. Tieren

Zhou to provide a hybrid Agile methodology to manage both Agile and non-Agile projects with the same set of development principles. SpecDD teaches you how to integrate Agile practices with requirement, quality, and project management best practices.

SpecDD aims to achieve two primary goals: increasing the team’s ability to development working with quality quantified and measured, and improving a team’s capacity of understanding requirements for sustainable project success and innovations.SpecDD states:•Businesslogicalwaysdictatesthe

correct way for software to function.•Requirementsmustbeformally

understood and translated into specifications.

•Thesespecifications,alongwiththeir related requirements and other collateral (called knowledge) form a “conceptual product.”

•Theconceptualproductguidesengineering and QA testing.

•Aconceptualproduct,asthecomplete blueprint of an application, should be compared to the implemented product.

•Anydeviationsbetweentheimplemented and conceptual products must be corrected, since the conceptual product is an accurate representation of the application.

BenefitsofSpecDDThe conceptual framework provided by SpecDD provides immediate benefits. First, it enables distributed development teams to interact with product managers and design teams. A core team can take a customer’s requirements and reinterpret them into developer-friendly language as a specification. Developers only need to worry about implementing the specification. If the customer’s requirements change, this can trigger a change to the specification. However, unlike our previous example where this could dramatically alter the course of the project, the impact of a change can be measured. Since the specification works as a parent item to all development tasks, project managers, product managers, and

customers are always aware of the potential impact these changes can have.

SpecDD represents a development process as a continuous set of development iterations that are organised into one or multiple development milestones. At any time of the development, changes are always welcomed, but requirements must be re-specified and documented to drive the development change. Quality standards are managed by independent but coordinated QA teams, where test case templates are defined based on the requirements, QA test plan are created and executed for each development sprint and release.

In SpecDD, requirement management is the foundation for practicing Agile with total traceability built into the process. Requirements are represented as epic and specs. Specs are the normalised and quantified requirements that can be used to drive development and QA testing.

Stories are used to drive both the implementation for the working software, as well as the requirement improvement and refinement. Requirement often cannot be fully understood and specified without the working software. Working software helps to the end users to generate real world usage experience, and often triggers the requirement improvement.

Change ManagementSpecDD also makes it possible to quantify these changes. For changes to specifications or requirements that the implementation team is currently working on, an approval process can be used to analyse the change request. This analysis can look at each specification’s linked development tasks or test plans and flag items that are related to outdated specs and requirements. This allows for a living, parallel design process to take place even during development – providing a great degree of agility while still maintaining a consistent vision for the application. Change requests in SpecDD can be used to perform cost and impact analysis.

TraceabilityFrom a specification, all relevant data should be communicated. This information includes related

SpecDD represents a development process as a continuous set of development iterations that are organised into one or multiple development milestones. At any time of the development, changes are always welcomed, but requirements must be re-specified and documented to drive the development change.


Specification-drivendevelopment | 25

requirements, development work items, QA test plans, and customer support issues. Viewing all these elements together gives a full matrix of traceability for each specification. This empowers decision makers by presenting all relevant information f rom a single view point.

Scalability and collaboration through specificationsThe specification also provides a way for product managers to collaborate with developers. Each specification can allow team members to voice their opinion of the specification through a voting concept. The votes for a particular specification’s ease of implementation can be compared to its’ cost and revenue estimates. This allows product managers to forecast the success of a specification, and alter it if needed. Specifications can also be broken into multiple pieces and assigned to different teams if needed.

Specification- drivensuccessSpecifications create a way for designers to communicate a customer’s intentions to developers. The framework enables Agile development by introducing process and traceability to changes in requirements. This in turn allows the developers to respond quickly to design changes and deliver better increments of the software. The project moves forward at all times, even while exact details of specifications are evolving.

SpecDD maintains the efficiency of Agile development, while also maintaining the quality and vision of the business behind the project. It also provides a way for businesses to adopt a scalable Agile process without adding additional risk to their process.

HybridAgiledevelopmentSpecDD is a requirement driven and quality integrated hybrid Agile development framework. It provides a unified conceptual framework to manage both Agile and non-Agile projects.

Requirements are represented as epic, document epic, and specs. Specs are the quantified requirements that can be used to drive development and QA testing.

For an Agile team, SpecDD supports all Agile functions, and provides clear guideline on integrating requirement and quality management. A development sprint has two deliverables, the improved working software, and the improved and better documented product requirements. With the independent and coordinated QA test process model, SpecDD provides a scalable framework for large size Agile projects.

For teams practicing traditional development, SpecDD provides an easy to use platform to add Agile functions without changing the team structure and development culture. It allows a team to quickly adopt the Agile practice without losing the total traceability, documentation, and project management best practices. www.techexcel.com

SpecDD maintains the efficiency of Agile development, while also maintaining the quality and vision of the business behind the project. It also provides a way for businesses to adopt a scalable Agile process without adding additional risk to their process.

Dr Zhou’s SpecDD blog

Curious about hybrid Agile

software development or

learning about tricks and tips

to help your teams achieve

success; check out Dr Tieren

Zhou’s new blog.

Dr. T and the Developers. After

working for 20 years with Fortune

500 companies, Dr. Zhou has

witnessed and developed best

practices for implementing a

hybrid Agile approach. In the

past seven years, he has focused

his research on developing

a hybrid Agile development

methodology. He founded

SpecDD in 2009 as a hybrid

Agile method to manage both

Agile and none Agile processes

with a same set of principles.

SpecDD aims to enable

development teams to practice

Agile development that is fully

integrated with requirement and

quality management. Dr. Tieren

has provided speaking and

training on SpecDD at numerous

Agile conferences and industry

expert gatherings.

www.specdd.com

26 | Feature


26 | The testing world

It’s Christmas time, there’s no needtobeafraid.Howlongsinceourworldofplenty?Inthis morning’s news a wise

person said that the important thingsinlifearestillwithinourgrasp. We need to look at what wehave,notwhatwehavenot.Apparently,theolderfolkget this, but possibly not the younger ones. They want it all. Today’s child does not want to understand austerity. I don’t want toteachthemiteither.Ihaveno problems with capitalism, providedithasheart,soulandgood cheer.

The saying goes that those who fail to study history are doomed to repeat it. For me, those who fail to study the authors of history are doomed to believe it. Not every account of days gone by is fact. Many belong in paperback novels – novel spin on things. Like why bonus payments need to be that high.

The spin is that they have earned it. The testability question is very Clintonesque – how do we define ‘it’ (his famously was how you define ‘is’ – being the Pres has its perks). ‘It’ as a bonus of millions of pounds gives me no indication of real value. Millions paid to an individual sounds like a lot, actually it sounds like it has crashed through many boundary values, into some

other-worldly partition. No danger that I will become equivalent anytime soon. The test of reasonableness appears to be against some fear of a brain drain or having earned it, based on increased revenues. Let’s try to put these in a testability context (the test being of reasonableness).

The UK government has decided that those earning over £50k will no longer be entitled to some child care benefits. Thus, we can assume that £50k per annum is a reasonable wage for a person. Thus if someone’s performance or attendance every day attracts a bonus of £50k, then they have earned the equivalent of a year’s reasonable wages, on top of their salary. If someone’s performance attracts a bonus of 10 times that amount (£0.5m), then that’s 10 years’ good wages, and 40 times that amount (£2m) buys you 40 years’ extra good living in one year. If the £0.5m was awarded after only 3 months in the job, then the test of reasonableness is going to be a very difficult one to pass. If the pay-out is to avoid costly legal action then the wrong system is under test. Someone higher up has decided that effective risk mitigation is to just pay up if it goes wrong. If it’s their personal money then it’s a cat 3 defect (annoying), if from a private enterprise then cat 2 (employees and shareholders are investors so live with

In this most festive of seasons, Angelina Samaroo is applying the logic of Agile to work out where all the money has gone...

Count your blessings


it), if a public enterprise then it’s a cat 1 (showstopper) and the BBC is all over it, even if it is the BBC.

What if however we’re not talking bonuses, but losses, then the effect is still the same – here today, went yesterday, tomorrow approaching fast. A trader has yet again lost some money. Not a fiver you dropped on your way back from the train station because you didn’t notice the note falling out of your purse, but £1bn you watched and watched until it lost all meaning. After all, if you can trade it how precious is it? In the latest banking scandal at UBS, apparently at least £1bn was ‘lost’. Not entirely sure how you lose this much money without your boss, his boss, his boss’ boss or the big big cheese asking some relevant question in some timely manner. But we do have to blame someone else, and much like testing, you were the last to see it, so you’re culpable. Losses in the financial sector make me think of pension pots. If we assume that 50 percent of average earnings is a reasonable pension then in our UK government approved figure of £50k, a pension payment of £25k per annum is reasonable. Thus, a loss of £1bn is the equivalent of 40,000 years of pension pay-outs, or more dramatically the annual pension contributions of 400,000 people, assuming an average contribution of £2,500 per annum. There is only so much money to go

round until they print some more, so someone receiving more means someone else going without. No test needed here, basic maths – one minus one really does leave you with zero. That someone else may have no idea or interest in the causal link. They have just been advised that their pension funds have fallen short of the mark and contributions need to be topped up due to ‘current economic conditions’.

In the Agile world we talk user stories to make things clearer. This for me is the user story – instead of saying ‘we lost £1bn’, it becomes easier for me to understand if we say ‘we lost the equivalent of annual pensions for 40,000 pensioners living ordinary lives and happily so (until now).’ Maybe that will focus minds, not sums of money we cannot comprehend and thus ignore, but real people like our Mums and Dads who we can see trying to manage in the ‘current economic conditions’.

And that brings us to the time of year, when family seems to matter just that little bit more. After the glorious summer games in the UK, Christmas has a lot to live up to (in our commercial world). My two wishes are James Bond drops in on the Queen’s speech and I don’t have to cook the turkey.

Merry Christmas and a Happy New Year to you all.

The testing world | 27

In the Agile world we talk user stories to make things clearer. This for me is the user story – instead of saying ‘we lost £1bn’, it becomes easier for me to understand if we say ‘we lost the equivalent of annual pensions for 40,000 pensioners living ordinary lives and happily so (until now).’

Angelina SamarooManaging director Pinta Educationwww.pintaed.com


28 | Feature


28 | Feature28 | Process improvement


Process improvement | 29

Why don’t companies like health checks and why do they fail? The fact is that many are just telling people what they already know and clients aren’t getting any real value from these exercises according to Richard Mort.

The external question

Some years ago, when I was a test consultant, I attended a sales call along withafresh-facednew

salesman. Our prospect was the headoftestingatawaterutility:a busy man who, as soon becameobvious,hadalreadybeenvisitedbyanumberoftestingservicecompanies.

The meeting kicked-off well enough with my super-keen colleague outlining our company’s capabilities, assisted by a variety of vividly coloured slides. Things took a turn for the worse, however, when the young buck clasped his hands together, stared into the face of our would-be client and asked the classic question, “Where is your pain?” swiftly followed by “Perhaps we could run a health-check for you?”

Our prospect instantly hit the roof. “If I get one more company offering me a health-check for my pain and charging me a fortune for what I already know!” he yelled.

Past its sell-by dateToday, the phrase ‘health-check’ is long past its sell-by-date and has become associated with a service that, more often than not, fails to deliver any real value. Common complaints include:- The health-check just told us what

we already knew;

- The recommendations were never put in place as day-to-day issues and business-as-usual pressures kicked in;

- The health-check focused on pure testing issues but most of our problems derive from issues outside of the testing team’s influence;

- They interviewed thirty of our staff who all told them the same thing – they could have understood our problems after speaking to three of our people;

- Our people have a day job to do – this was just a distraction.So, we’re agreed – hiring an external

party to carry out testing exercises is a complete waste of time, providing little benefit at great cost, using resources that would be better used sorting out day-to-day issues?

Well, no, I don’t actually agree with any of that. I have no doubt that, for many organisations, the focused use of an external specialist to help identify and implement potential test improvements can bring clear and measurable benefits. A company’s approach to testing could be immature, ineffective or inefficient for any number of reasons. There could be issues between individuals, teams, technical versus business units, or between client and supplier. The challenges could be internal to the Testing arena or interwoven within the wider IT and business psyche of the company in question.

Things took a turn for the worse, however, when the young buck clasped his hands together, stared into the face of our would-be client and asked the classic question, “Where is your pain?” swiftly followed by “Perhaps we could run a health-check for you?”


30 | Process improvement

In my experience, the best approach to testing is often when an external party is engaged to work closely with an internal team on a short, sharp improvement campaign. Not a ‘health-check’ of old, but rather a Process Improvement drive that delivers genuine results and focused business solutions.

Then there are the companies who hurtle down the tracks like an express train trying to meet specific goals and objectives, knowing that they need to improve but believing that they do not have the time or spare capacity to change, so they keep on going the same old way. That approach can only work for so long before they hit the buffers.

Having said that, the capability of internal test teams has grown considerably in the past 10 to 15 years. The title of ‘Head of Testing’ within large organisations is now commonplace and many of these testing champions are now undertaking their own test maturity studies and initiatives to identify process improvements. Whether they’ll then have the necessary support within their organisation to implement any recommendations, however, is another matter entirely.

Test maturityI once undertook a Test Maturity review for a company looking to make improvements to its overall test approach. On reviewing all the collateral, interviewing the relevant parties and collating my response, I presented my findings to the Senior IT Management team, which were well received and my recommendations noted to be acted upon.

Afterwards the Head of Testing came up to me and showed me

a presentation she had done to the Senior Management Team some eight weeks beforehand. It was almost identical in its findings and actions but the consultant stamp on my forehead had made the difference in terms of implementation. The external voice seems to carry more weight as: a) it’s being paid for; b) it is genuinely impartial in its recommendations; and c) ‘they have done it many times over’.

In my experience, the best approach to testing is often when an external party is engaged to work closely with an internal team on a short, sharp improvement campaign. Not a ‘health-check’ of old, but rather a Process Improvement drive that delivers genuine results and focused business solutions.

For example, Edge Testing’s own Process Improvement Model (PIM) focuses on assessing the maturity of testing within an organisation by working in close partnership with the client team. The results from the review stage are then used to identify appropriate initiatives which aim to enhance in-house testing capabilities and the overall delivery of the organisation’s software projects in line with business objectives.

Times have moved on for the testing industry. We no longer need to ‘feel the pain’ of our clients. We need to help them improve the key processes that are vital to their success.

Richard MortOperations director Edge Testing Solutionswww.edgetesting.co.uk


32 | Fuzz testing

Thesoftwaretestingindustryis slowly waking up to security threats. When we lookatsoftware,every

time we see it crash we can start thinking whether or not this represents a security-related flawornot.Typically,ifyoucan reproduce the crash then itisdefinitelysecurity-related.Whether or not hackers could execute code or use malware withthatspecificflawrequiresmuch deeper analysis which is oftenoutofthescopeoftestingorganisations.

Fuzzing and fuzzingtechniquesFuzz testing is the way unknown threats are found if you don’t have access to source code. It is a technique where you send unexpected broken, invalid input data to any interface into the software in order to find structures and messages that the programmers didn’t expect to come from that communication. It is generating or simulating an infinite amount of attack patterns in order to find those ones that have an impact on the software.There are two fuzzing techniques:Mutation/template-basedfuzzing: This is based on having a recording of communication traffic or files that you

seed into the generator and it will then generate an infinite amount of test cases from those template or use cases. Generational/specification-basedfuzzing: This is based on the interface specification, so when your software is using industry standard communications then you can use that specification also as the basis for the test case generation.

There are benefits from both the methods. Mutation/template-based fuzzing is probably the only choice when you are testing proprietary interfaces whereas if you are doing industry standard communication based software then it is best covered with generational/specification- based fuzzing.

FuzzingefficiencyWhen we look at the metrics of fuzzing we find that if you use smart model-based generational fuzzers they tend to find more bugs and they find them faster than mutational fuzzers, but both of them have their own value. Generational fuzzers execute in matter of hours, and can easily be integrated to daily or weekly test cycles. Mutation-based fuzzers need days to complete, so it is much harder to integrate them within an Agile loop, for example a nightly test execution – they are much easier to

do in parallel to the development or in a dedicated sprint.

Looking at security problems in general, it is a never ending process. You are constantly analysing for the changes in the interfaces and the threats in the customer deployments; you are constantly doing tests for ever single release of the product, your reporting goes back to the developers and then you have to mitigate, fix or prevent those attacks from happening. And you have to keep on going around this loop until the software is no longer used in the marketplace because there are constant changes in the threat landscapes and the attack landscapes, so it shouldn’t ever stop. Software is never ready in that sense.

WheretousefuzzingTo understand where fuzzing is used in Agile we have to first find out where it is used in Waterfall. If we look at Microsoft SDL, fuzz testing is used in the verification phase, so it is almost always done by testers even though quite often security specialists are advising and may be taking part. But developers may be involved in fuzz testing, testing each build of the software. Fuzzing can also be part of the release review, a security gate before software is approved for release. Even after

The Agile model was introduced to allow faster product releases and more interleaved software development, realising the aim to test early and often. Agile enables early testing, but has so far not been focused on finding critical security flaws. TEST reports from Ari Takanen’s recent webinar where he highlighted automated security testing techniques based on fuzzing, with examples of how they can be integrated into the Agile development process.

Can fuzzing be Agile?


Fuzz testing | 33

software is released, it is fuzz tested as part of regression testing. Fuzzing is not just restricted to verification.

The focus areas for fuzzing are first of all identifying where the attacks can come from because you have to choose the interfaces that need testing; Then you need requirements for fuzzing, for example, as a minimum deciding between generational and mutational because this has a big impact on how much confidence you need in software confidence and security.

The interface design and specification impacts fuzz test development, and observability is also important, if you try to hide failures this will have an impact on the fuzz testing. Fuzz testing can also take place in unit testing, even before entering the verification phase.

During the verification phase the main issue is prioritisation because you don’t have time to run all the fuzz testing. You have to prioritise which interfaces are the most critical and you have to plan execute and automate fuzzing for these, reporting the failures. Also you have to plan for future re- runs of the test, whether this is in regression testing or some other test automation place.

After testing is complete you still need to be prepared for vulnerabilities to be discovered by third parties using fuzzing tools and there have to be people able to reproduce those problems – this is often the role of the product security organisations. Someone reports a bug and they have to re-run those fuzzing tools in their own environment and then feed those bugs back to the developers for fixing.

Quite a lot of fuzzing takes place in customer deployment configurations because when you test in a white room environment you don’t find all the bugs, but customer deployment configurations change the test results, significantly sometimes.

In traditional development you have a dedicated team for development and a dedicated team for testing and fuzz testing constantly interacts with these two teams. Every time there’s new code to be tested. The test team needs to run tests, create logs, create the test environment, identify bugs and reproduce them. Then quite often the developers will have to run the same fuzz tests in order to catch the bugs and identify the line of code that is responsible for the vulnerability.

AdaptingfuzzingforAgileAdapting this process for Agile is not easy and not that well documented anywhere. If you look at the Microsoft SDL for Agile process, the interesting division – which is common to many Agile processes, not just Microsoft’s – you have some practices that you do once, then you have activities that are related to security which are done in every sprint. And then there are bucket practices, security practices that have to be completed every now and then, but not during every sprint, and fuzz testing is usually located in these practices.

In Agile, fuzz testing is typically built in to the Continuous Integration (CI) ‘platform’ and maintained by the tool team that operates across all Agile teams. The product security organisation is often the ‘owner’ of fuzz testing across product teams and can in training and even integration of tools. Security specialists need to help create the fuzz test requirements (misuse cases) for process integration.

The challenge is to explain fuzz testing as a requirement. Fuzz testing is not a functional requirement, it’s non-functional requirement related to robustness, you need to somehow functionalise it so that you can put it into your pipeline for implementation in the Agile process.

One way to do this is to create negative user stories, attacker stories or misuse cases which can be thought of as positive user stories for an attacker that should not succeed.

Agile/fuzzingintegrationThe most common ways of integrating fuzzing into Agile include the following. They are not mutually exclusive, but you have to choose which ones you integrate first and which work best for your process:Fuzzingbydeveloperseg,innightly build: Fuzzing is part of the development toolkit, available to use when needed. Those involved in Agile development are deciding how to and when to use fuzz testing. When fuzzing is deeply integrated into Agile then it is built in a similar way to any automation or any test cases. Quite often it is available either in the integration platform or as something built during the Agile process.Per sprint eg, integration build test: Fuzzing is part of Continuous Integration (CI), automated unattended process, but integration is handled in the Agile

process (as a backlog task). It might have been built in an earlier software project, but it is something that can be used and integrated and adapted into the process. Often it is a dedicated process that happens in parallel with development or it’s a phase in the Agile process when the focus is on security testing. When you use generational fuzzing you can do it in every sprint and when you do this the number of bugs for each fuzz test rapidly declines.Indedicatedsprint(s)eg,inverificationbucket: It is a choice whether you integrate fuzzing into every sprint or confine it to dedicated sprints. In the Microsoft SDL method fuzz testing is its own bucket, or part of bigger test and verification bucket. InparallelwithAgiledevelopment,overseveralsprints: It can be integrated as a separate process, performed by the product security team. When you have the first build that can be fuzz tested, then you immediately start fuzz testing. Developers don’t have to worry about it, it happens in parallel. When the number of bugs and the automation improve then you can tie it in to the sprints themselves and at some stage it will become a natural part of the process.Inregressiontestingorasafinalacceptance test/gate: In a similar way to regression testing, or as part of regression testing, fuzzing is not taken care of during development, but is part of release and maintenance processes. Here all the bugs will be found very late in the process though.

EssentialfuzzingCurrent testing methods in Agile do not ensure software is as secure and bug free as it could be. Negative testing is also required in addition to fuzzing for this to happen. However, if fuzzing is properly integrated in the Agile project, it allows early testing and makes it possible to find and fix bugs quickly and cost-efficiently, but there has to be a conscious effort to include negative testing/fuzzing in the project requirements, otherwise it will not happen.

The full webinar recording is here: www.codenomicon.com/resources/webcasts/20121002.shtmlA poll to assess how testers integrate fuzzing and security testing into their Agile projects is here: www.codenomicon.com/agile

Ari TakanenCTO Codenomiconwww.codenomicon.com


34 | Application testing


Application testing | 35

The complexities and interdependencies introduced by new technologies add extra pressure to get applications to market in quicker time, in an increasingly competitive market. How can testers cope in this ever more complex environment? Michael Allen reports.

Conquering application complexity

It’sfairtosaythatITisbooming.Technology is no longer a supportfunctionofbusiness,butactuallyformsthecore

ofhowcompaniesdelivertheirofferingstotheircustomers.Checkthe headlines: more and more companies are competing to bring out the latest and greatest applications. In the mobile space alone,appdevelopmenthassoared;withApplereportingover30 billion downloads to date. SaaS applicationslikeSalesforce.comhaveunseatedinternally runandhostedapplicationsforsomebusinessfunctions.Bigdataandvirtualisationrepresentaseismicchangeevenforapplications that don’t make it outsidethefirewall,andareonlyused by company employees.

While this is great news for developers, the complexities and interdependencies introduced by these technologies add extra pressure to get applications to market in quicker time, in an increasingly competitive market.

Performance testing the applications of a few years ago now seems pretty straightforward: the application lived on its own hardware and was self-contained. So, any problem that was found could be linked back to specific code or situations. As long as you had the same hardware available for both test and production environments, checking that the code would run well was easier.

Now, instead of single, stand-alone applications, users consume services that are put together from multiple web services, applications you don’t own, or third party code and frameworks, to deliver what the user requires. This approach allows businesses to solve problems faster and better by recycling existing code and applications that have already been proven to work (at least in some circumstances). New development approaches like Agile are encouraging and supporting IT as a whole, to respond faster to evolving project deliverables. This added complexity though brings with it increased risks – on top, of course, of all the risks you’ve dealt with for years.

The emphasis on speed and versatility masks the impact of the complexity of modern applications on the testing process. Hybrid Java and .Net, distributed and virtualised (from private to public cloud) environments; integration with third party services (from geo-location to card payment services) and the use of frameworks exponentially increase the complexity of applications, leading to errors that may not be seen until production. At this point it's rather too late, as anyone who has ever closed a non-responding application in frustration knows. It’s clear that these issues must be addressed, but the question is how?

Many organisations have been left wondering how they can solve these problems and whether or not writing performance tests is worthwhile in an

environment where project timelines are forever being squeezed. Quite often these pressures lead them to cut corners in the development process, abandon testing entirely or restrict testing to functional unit testing but not end-to-end performance testing for example. This isn’t the answer but it is often the reality. There needs to be an understanding that added complexity means that as organisations search for increased Agility, performance testing cannot be sacrificed.

As end user demand and expectations increase, it’s now more important than ever for organisations to not only produce more applications, but test them and ensure they perform the best they can. However, a different approach is now required. Organisations must look at the overall end-to-end performance across an application from start to finish and recognise that being able to find problems is a necessary step for testing programs covering these complex applications.

Performancetestingin an increasingly complex worldThe first problem that testers face is finding where any problem in performance really is. If a service is performing poorly, this can almost be worse than straight service failure, as at least diagnosing the fault would be easier. Load testing has become more affordable due to a growing list of available commercial and open source tools, while the ability to host software in virtual machines or in the cloud has also reduced the need for expensive or dedicated test hardware. Creating and maintaining test scripts has improved as testing tools provide good scripting support in IDEs like Eclipse or Visual Studio too.

However, despite all the improvements on the testing tool side, analysing and solving the identified performance problems takes most of the time in the overall testing phase of a project. According to various analyst reports, most application problems found during testing come from

inside the application – not from the underlying system. So knowing more about what the application is doing at certain points in time is essential.

When a problem arises, testing reports do not include enough diagnostic information for development to get straight to the root-cause of application issues – they more or less help identify system issues. In order to get to the root-cause of application problems, more test cycles are needed to capture additional code-level performance metrics such as logging, custom performance metrics, etc.

These additional cycles require time and effort to both run the tests and for the developers to refine data capturing. Not only does this process involve additional logging, use of debug code and special analysis tools, it also changes the performance characteristics of the tested application. This can lead to performance problems not being identified in testing because the tested application doesn’t resemble the application eventually deployed.

To solve this problem, load testing should always be combined with an Application Performance Management solution that provides rich, actionably in-depth data for developers to identify and fix problems without needing to go through extra cycles and in order to stay within project schedules. In reality, you shouldn’t be asking whether to load test, but how best to do it.

Pinpointing persistent problemsDespite extensive testing, some problems will always escape to production and will first be identified when real users push the application. Not every single possible real-world test scenario can be tested - however, most problems that “escape” testing could be prevented.

An example of a common problem is individual failing components that manage to get by during the testing phase, but cannot handle real-world production loads and how these

A different approach is now required. Organisations must look at the overall end-to-end performance across an application from start to finish and recognise that being able to find problems is a necessary step for testing programs covering these complex applications.


36 | Application testing



Application testing | 37

scale up. Having a component-level performance and regression analysis in place will show how certain components scale compared to others. Analysing response times with increasing load during load testing down to component level enables predicting when components are likely to fail. They may not fail during the load test, but knowing that a certain component is almost at the breaking point allows this issue to be handled right away instead of waiting until the problem surfaces in production.

Architectural issues – such as too many network roundtrips or database queries per transaction – are another example of component-related problems that may pass testing but will eventually result in a production failure. Analysing individual transaction execution paths can identify these problems in testing by highlighting such architectural shortcomings. If you can situate transactions in a wider business context – based on say, how much revenue they generate – then you can prioritise your problem-solving. If this process is automated, so much the better.

A transaction-centric approach will also highlight where third party services are the source of poor performance. Even though you might not be able to get the level of detail from these that you can on your own code or applications, it can still help to pinpoint the issue for resolution. If you are relying on an API or web service from a third party, then this should be figured in at the beginning of testing. Equally, any commercial relationship with the third party should be considered – if there is a service level agreement in place, it can be used as the basis for tests.

One last example is the impact of additional logging, debug libraries or other environmental differences on the performance characteristics of the tested application. These differences compared to the real production environment hide or shift the real performance problems.

Some of these problems will be the same ones that you are used to

experiencing, but just experienced in new ways due to the interconnected nature of the application. Some of these can also be solved by better communication between the development team or other teams involved. Specifically, developers and testers need to work better together across the entire application lifecycle to ensure success. The main challenge is to get that overview of overall performance, and what the user experience will be ahead of deploying into production.

The key to successOnce deployed in production, the yardstick of success for web applications is usability: commercial success depends on people using the application for the task that it was designed for. Testing for this outcome therefore requires an understanding of how these development projects align with the business requirements that were set out at the beginning.

For testers, the best advice that I can give is to look at how they drill down into the data that they collect, and then how they can automate processes for diagnosis across multiple sources. Recording application transactions on code-level enables component-based performance analysis, eg, based on individual layers of the application such as Data Access Layer, Web Service Layer, Business Logic and Presentation Layer. Analysing the performance of individual components across different testable application builds as well as across different workload scenarios reveals performance and scalability issues in individual application components.

While the application development environment is getting more complex, the fundamental principle remains the same: seeing the whole service through from the initial user request to the completion of the request and delivery of the service. Without a complete overview of the steps involved in this whole process, it will be more difficult to test for both functionality and user experience.

While the application development environment is getting more complex, the fundamental principle remains the same: seeing the whole service through from the initial user request to the completion of the request and delivery of the service. Without a complete overview of the steps involved in this whole process, it will be more difficult to test for both functionality and user experience.

Michael AllenApplication performance solutions director Compuware dynaTracewww.compuware.com


While Dynamic Data Masking can seem a simple and attractive option for sensitive data, are its limitations understood? Huw Price highlights some of the more common pitfalls to watch out for with dynamic masking.

Behind the mask

Developingandtestingsoftwareapplicationsrequirestestdata.Inmost cases, this still

involvesworkingwithcopiesofproduction.However,currentdata protection guidelines and regulations state that no sensitiveorpersonallyidentifiableinformation(PII)isoutsourcedor used in non-production environments.Asorganisations

havecometobetterunderstandtherisksofdatabreaches, data masking – known elsewhereasdataobfuscation,datade-identification,datadesensitisation and data scrambling–hasevolved fromafterthoughtintoamandatory operation.

The most common approach to masking sensitive data has been ‘in-place’ or ‘static data masking’.

38 | Data masking


Data masking | 39

As organisations have come to better understand the risks of data breaches, data masking – known elsewhere as data obfuscation, data de-identification, data desensitisation and data scrambling – has evolved from afterthought into a mandatory operation.

This involves discovering the sensitive data, then masking full or subset copies of production before releasing the data for development and testing. One downside to this method is that working with copies of production can be cumbersome; it can involve considerable manual effort, is prone to human error and leads to high storage costs. There is, however, more than one way to skin a cat.

Dynamic data maskingThe latest buzz in the data security world revolves around the concept of ‘dynamic data masking’; an approach which involves masking sensitive data in production each time it is enquired against. This means that every time a request for data is made, the response is masked according to pre-defined information on what data is sensitive, which masking rules to use and who is able to access the data, before being returned to the user. This enables data to be masked in production without changing the production database, eliminating the need to make copies.

As a result, ‘dynamic data masking’ has been presented by some as an easy-to-use, cost-effective panacea to cure all the traditional aches and pains associated with data security. No wonder it is being viewed as an attractive alternative to ‘static data masking’. Yet, little is understood about the impact of its limitations. This article aims to highlight some of the more common pitfalls to watch out for with ‘dynamic masking’.

Transacting against productionOne of the most commonly cited strengths of ‘dynamic data masking’ is that it obviates the need to have at least two copies of production for testing. Ironically, this is also perhaps its biggest Achilles Heel. Therefore, before adopting ‘dynamic data masking’ as part of your data security strategy, it is important to consider whether or not you really want testers transacting against production data?

Firstly, ‘dynamic data masking’ is only reliably safe for reporting systems. The fundamental point behind masking is to use a one-way masking scheme so that any obfuscation cannot be reversed. However, transacting against dynamically masked systems typically requires a two-way masking scheme, which tends to offer a weak encryption method that fails to meet regulatory compliance.

The risk is exacerbated by the opacity of most ‘dynamic data masking’ methods when masking a single copy of production, which has the effect

of hiding the true nature of what lies beneath, and pass through, the process. The risk of ‘live’ data leaking into non-production environments is well-known; indeed, it is probably the main driver behind the decision to implement some measure of data security. However, masking data directly from production also introduces the risk of test data creeping into the ‘live’ environment. This not only has repercussions for performance, but can have some nasty side-effects if the system has any concept of relations between transactions, for example, a system that computes the price of securities based on frequencies of buying and selling. Unsurprisingly, this can adversely affect genuine transactions; quite the headache for a panacea.

However, in many cases, the result of this opaque process is to instil a false sense of security. The chief culprit here is the idea that you can mask a single copy of production differently, according to the levels of access afforded by your organisation’s security policies. The risk here is that you expose the underlying, raw, sensitive data through the obscurity of the process. Remember, if you can’t see what’s happening, how can you tell if it’s worked properly? The simpler, and more effective, solution here is to mask the entire production database in one go, eliminating the risks of such complexities.

Inserting and masking new recordsEmploying ‘dynamic data masking’ on a transactional database results in a curious dual-state which poses questions about what happens when new data is inserted into the application? One of the purported strengths of ‘dynamic data masking’ is that it only masks the sensitive data, in theory improving the efficiency and cost-effectiveness of the process. However, this requires watertight pre-defined rules to be established. If there are any gaps or errors here, newly inserted sensitive data may not be masked. As the production data is not masked either, it soon becomes difficult to tell what if all the sensitive records have been obfuscated. This can prove problematic when it comes to conducting data security audits – how do you show that all the sensitive data has been masked?

The flip side of the coin occurs when the data is masked. Let’s say that you insert the record ‘John Smith’ into the database – how do you find that particular record again? At some point between insertion and re-selection,


40 | Data masking

Dynamic data masking can be a useful part of any data security policy. However, it is important to look through the hyperbole and understand the limitations of masking data ‘on the fly’. Once you are aware that dynamic data masking is more Curate’s Egg than panacea, and may not be the solution to all your masking requirements, it can provide more useful in securing sensitive data.

Huw PriceFounder and chief technical architect Grid-Tools Ltdwww.grid-tools.com

the record should have been masked at least once and will subsequently be displayed as a different name, ie, ‘Dave Jones’. Without any other means of identifying the record, it is almost impossible to retrieve the inserted data. This may well defeat the point of testing.

Ensuring consistency in masked dataAs modern enterprise architectures become larger and more complex, organisations need to ensure that any masking performed is consistent across multiple applications. For example, if you are using a Siebel app with a back-end Oracle database, this must match with your DB2 application. In short, the same value must be returned on all applications. The reason for this is simple: the whole point of testing is to have a set of meaningful, repeatable process that verifies the application works.

Let us suppose there is a trading system and we want to look at the types of trades inside the system. Now suppose that your masking routine selects randomly from a list of valid trades. This process is only repeatable if one, and only one, value is selected for any given trade. If not, this completely eliminates the possibility of a repeatable process, and therefore, the premise of testing.

There are two ways to ensure that consistent, repeatable results are provided: using deterministic, mathematical algorithms or by storing old and new values across multiple systems, which requires the creation of a central store to hold previously masked data. This store then needs to be maintained dynamically – i.e. the old and new values need to be written to a central table so that, when retrieving large volumes of data, you can build the cross-references in real-time. However, building cross-reference tables for dynamic masking routines is inherently fraught with difficulties.

Say you are planning to run a dynamic masking routine on two tables: a customer account and an order items table, and that the customer account table has an aggregate column that sums up the

total number of order items purchased by each customer. There are two ways to mask this dynamically, both requiring a cross-reference table: 1) call the dynamic mask on individual orders and calculate the aggregate; or 2) mask the aggregate and somehow transform the individual orders to match. In this example, we know the relationship between the two columns, but often not all the relationships are known. The effect of implementing a dynamic masking policy that ignores any unknown relationships is potentially profound; bottlenecks emerge in your testing, which impacts on downstream data consumers and can require a complete re-implementation of the masking routine. It is, therefore, essential that you understand your data, which can be difficult in an opaque process. So, why not do things properly and provide the necessary transparency through a one-time routine?

Maintainingperformancewhen maskingAlmost by default, any new solution or approach should offer greater efficiency and cost-effectiveness, as well as higher quality, than what existed before. One of the purported strengths of ‘dynamic data masking’ is that it eliminates the need to undertake large initial masking runs which could take several days. However, by its very nature, ‘dynamic data masking’ is incredibly performance–intensive; particularly if cross-reference lookup tables need to be processed ‘on the fly’. The challenge here is that ‘static data masking’ requires a one-off constraint on performance, which is quickly rendered irrelevant by the ongoing performance issues provided by dynamic masks.

ConclusionDynamic data masking can be a useful part of any data security policy. However, it is important to look through the hyperbole and understand the limitations of masking data ‘on the fly’. Once you are aware that dynamic data masking is more Curate’s Egg than panacea, and may not be the solution to all your masking requirements, it can provide more useful in securing sensitive data.

How do test teams deal with the problems of building, running and managing scrums? DavidCotterell, European region director of Qualitest explains how his company managed it on a international scale.

Managing international distributed test teams in Scrum

42 | Agile testing


These days test teams are forcedtodealwithmanychallenges when they adoptoneofthedifferent

Agile methods; among them is ‘Scrum’. There are a many differentpossibleactivities– daily meetings, Sprints etc. Therearealsoavarietyoftermsthattheaveragetesterandtestmanageratdifferentlevelsmustcontend with.

Let's assume that we understand a particular activity. We have grasped what the managers are looking for and what concerns the Scrum Master. Moreover, let's assume that the test engineers, TLs and the test managers understood why this Scrum approach is appropriate (If there are still unclear areas here, this is a whole different article). Now we are facing a totally different set of problems:


Agile testing | 43

- How are the scrum teams built, and how is distributed team communications handled?

- How does the team run in everyday life, when parts of the team (both developers and testers) are in different parts of the world?

- What do we do when several test teams test multiple different products during the same time frame?

AbitofbackgroundOur test team was assembled from teams in the US, Israel, and China while development was divided between teams in the US, Israel, China, and Ukraine (outsourced).1. The product line which the QA

department was in charge of, included four different ‘off the shelf’ software products, where each one had a distinct life cycle, it's own targets and goals, release versions etc. Moreover, there were different one time projects that occurred during the year (between five and six a year).

2. Between the different products, there were many overlapping areas, meaning test engineers could efficiently test more than one product.

3. The tests were divided into three major sections:

- Manual tests (including requirement analysis, test designs etc);

- Automation development ‘on the go’ for newly added functionality;

- Customer simulation labs, which were established to simulate the main customer's environment, working procedures, configuration and up to date data from these customers.

4. All of the test engineers were divided amongst five scrum teams. History shows that ‘team formation procedures’ happened once a year on average. (Team formation consisted of the assignment of all developers/testers/analysts etc, to scrum teams in order to fulfil the backlog as efficiently as possible).

5. Company standards – each scrum team should include at least one QA engineer (Most of the time it was more than one). No engineer could be part of more than two different scrum teams at once (Due to the overlap between products, it made sense that some engineers might test two different products in these overlapping areas).

Building the teamsWhen we met in order to form the scrum teams according to the product’s future needs, we could either disregard geography and form teams based on capabilities and knowledge only, or position geography as a limitation to guide the team's structure. The second alternative basically means that a given scrum team won't have members from two different countries.

We chose the first option as our model, meaning capabilities were the leading factor. Multiple scrum teams were formed with engineers from different parts of the globe. We did however manage to assemble the teams with people residing in no more than two different time zones; to facilitate communications.

The major disadvantages of this approach are the time zone challenge of daily communication among different team members and understanding the team's status along the sprint. We overcame these problems by using a few techniques (most of them easy to use):Daily meetings: At first the local team (where most of the team members were) held the daily meeting alone. The other remote parts of the team would send daily updates by email and receives updates the same way after the daily meeting. As time passed it was understood that despite the comfort associated with this offline approach, moving to an online meeting with all team members, prevented dragging issues along, and improved communication and relationships between team members. Plus, issues were raised more frequently. Hence, with the absence of a sophisticated video conference mechanism, most daily meetings were done by phone; parts of them on Skype and webcam. The teams suffered some difficulties during the first few weeks, but after two to three weeks, meetings were held as if all were at one geographical place.Personalised communication: The team members tried to communicate face to face by using video, chat and phone, and less by documents and email. This made information transfer significantly easier despite language barriers. Beyond that, there are a number of difficulties to consider:1. Time differences.2. Cultural differences. At once, both

the QA engineers as well as the developers are required to work

smoothly with their peers who may belong to an entirely different culture. The difference can be in terms of anything; the way they talk (even if all sides speak English, different accents may make it impossible to understand at first), to the way they think, working habits, and much more. The way we approached this issue was by going through a cultural differences workshop which stressed the key differences and where everyone needed to be careful. Over time it became much easier, although it did take a significant amount of time.

3. Exercise. At first communication between the team members is problematic. A simple message which is supposed to be conveyed readily by speaking or writing requires a relatively a long time. The effect of this is that many team members avoid making online contact. The name of the game here IMHO is exercising it as much as possible by talking directly on video/phone/chat rather than by email.

Worldwide scrumBeyond online communications, scrum teams need to maintain some helpful aids to manage their joint information.Scrum board: Initially on an Excel spreadsheet, which was at a shared point, and thus accessible for the entire team regardless of their location. Today it is managed by scrum management software with which the teams can view and update the tasks / tests / User stories easily. This prevented the need for multiple scrum boards for each team in every site.Scrummanagementsoftware:Examples for those are Rally, Version One, Green Hopper etc. By using this software, the teams have online access to all of the relevant data for managing their tasks. Every detail that is being updated by the product owner or scrum master or by one of the team members is seen on the spot, and some of the daily meetings are being done in front of the boards within the software. One of the advantages of this is that all the material regarding the different user stories is in one centralised place and convenient to use.

Real world experienceWhen the company went through this transition, we knew we wanted to conduct the functionality tests as close to the development stage as possible,


44 | Agile testing

During the first few months it was hard for the team members (developers and testers), to move away from the traditional phases of software. Even in traditional methods there are parallel processes and yet it seemed that despite trying to make the processes as parallel as possible in order to reach the finish line faster, we were still under the influence of the ‘old’ way of thinking. Step by step, our methods changed.

and at the same time provide relevant automatic tests while the first cycle of tests were done. Up until that point there were many cases where there was a time gap between end of development and the first testing cycle period.

We realised we had a gap to bridge in the first few sprints (We should have fully tested the functionality that was developed during the months before, plus the new functionality which would be developed during the first few sprints). To overcome this, a special task force was created with testers, CS engineers, test oriented developers and more. The purpose was to eliminate the gap. After two sprints the goal was achieved. Today the testers in the different scrum teams provide tests covering approximately 80 percent of the functionality developed within the same sprint. Within that sprint the testers develop automated tests for about 50 percent of that functionality. Naturally, if a given function is supposed to be potentially shippable at the end of a certain sprint, it is developed, tested and fixed within that same sprint.

During the first few months it was hard for the team members (developers and testers), to move away from the traditional phases of software. Even in traditional methods there are parallel processes and yet it seemed that despite trying to make the processes as parallel as possible in order to reach the finish line faster, we were still under the influence of the ‘old’ way of thinking. Step by step, our methods changed.

A year after we adopted Scrum, a complex project was established for the leading product. A dedicated scrum team was assembled. That project required extensive infrastructure work, new complex algorithms, wide and complex GUI, and there was a very short timeframe. The team was assembled from engineers in the US and China.

Only one of the team members had relevant application knowledge, so there was a learning curve for everyone. The team got to work and made a list of development tasks and tests, where every module in this project would be testable from the user level (meaning through its dedicated GUI). Hence, from the moment the planning phase ended until the end of that project, it seemed that during every daily meeting, someone from the team asked: “just finished the last thing I have done, what is the best thing to do now?” The decision regarding what should be done next, was made by the

entire team. It was clear that in order to prevent bottlenecks, this was a necessity.

In many cases, developers helped in testing (under control and support of the testers and test TLs), in part based on a written STD, and in part exploratory, according to the knowledge and expertise of that developer. Testers also helped developers perform better unit tests where it was found necessary. That actually meant that almost anyone could do anything, with a bit of learning when necessary.

The primary goal that the team took upon itself was to complete the project on time, and with the desired quality, and to reach the milestones within every sprint.

Testing in parallelThere were many test engineers that by their capabilities and the company's products, were required to take part in more than one scrum team during a given sprint, ie, where we have two different products, with an identical functionality to be developed for both. When the testers and TLs got to planning a sprint they faced a few dilemmas:- Given two products and therefore

two different backlogs, how do you prioritise between every task within one of them and every task in the other?

- What is the required capacity for each product? How do I as a tester or a TL define what it is for every product?

- Which scrum team, given the tasks which are its responsibility, needs a particular tester more for the next sprint?

- When the sprint begins, how can we make sure a tester doesn't move between teams too many times causing unnecessary context switches?

Our way to solve the above dilemmas had three steps:1. Understand the overall picture

in terms of constraints and the developer’s plan: The QA manager publishes the priorities, meaning which versions for which products are about to release and what are the priorities between the products accordingly. In parallel, every tester sits down with his scrum team to understand the developer’s plan, what does the team see they will work on during the sprint, which developments are riskier and their meaning etc. Moreover, identify leftovers from previous sprints to the scrum team, the TL and the QA manager.

DavidCotterellEuropean region director Qualitestwww.qualitestgroup.com


Design for TEST | 45

Testing the test processMost organisations have a test process but how good it is and how up it date it is can vary enormously – with the expected consequences. Mike Holcombe reports.

Ifyourphilosophyisoneofcontinuousimprovementandthisisimplementedeffectivelythen there is a good chance

thatwhatyoudoisfitforpurpose.However,theremaybeissues that could be addressed better. Here are some steps to findingoutifyourtestingcultureand processes are as good as they can be:1. Is the quality assurance process

documented clearly and well disseminated?

2. Is this information widely available and used? It should not be restricted to the test department, all those involved in development, sales and marketing, consultancy etc should know about it and how it relates to their activities.

3. Is the process reviewed regularly in the light of how effective it is? Is information about defect identification and removal available so that problems arising might lead to improvements in the process?

4. Are the staff suitably qualified to use the process? Qualifications are all very well but they can become out of date. New techniques and tools come in from time to time. In IT generally, it is easy to think you know what is current but be missing good new ideas. Meeting new people at events and courses can help to keep you current.

5. Do you bring in third party experts to review your processes and teams? This can give added assurance.

6. Are your tools up to date? Sometimes spending on testing tools is a low priority because managers may feel that they are not directly productive – how wrong they are! There are many free tools available now. Do you have the latest versions?

7. Is your test data and test performance information open and transparent? The more people know about how they compare with others doing a similar job the more incentive there is for improvement.

8. Do you engage with the test and QA community? Do you contribute to it through talking at meetings, sharing tools and techniques?

9. Do you actually test the process? This could be achieved by taking some software, injecting some bugs and putting it through the process. This is best done in an open way – rather than like a mystery shopper! If it was presented as a challenge – possibly with a reward – to see who could find the most bugs then it could be quite fun and need not take a lot of time.

10. Finally, make sure everyone talks about testing, encouraging new ideas, criticisms and greater involvement. Hiding testers in a corner is rarely productive.

Mike Holcombe Founder and directorepiGenesys Ltdwww.epigenesys.co.uk

2. Have each tester / TL prepare an individual plan: Every one prepares a plan as a base for changes, since in the planning things will change. This plan takes into consideration leftovers from the last sprint, relatively risky developments ahead, the different backlogs that need to be considered, and the constraints from phase one.

3. Synchronise all personal plans, providing feedback and planning the tests: In case a given tester belongs to one scrum team, the issue is relatively simple, since only the relevant product's needs and his scrum team's plans affects his sprint's preparation. When the situation is different, we gather opinions from all testers and prepare plans together:

- If there is a product that will have large gaps, which we can foresee prior to the sprint's planning, meaning if due to lack of resources, this product won't get the needed tests during the coming sprint. Needed response means testing every function developed, releasing a version (Release procedure) where needed, and automation developments to ease future tests.

- Is a different resource allocation from the tester's recommendation needed in order to cover for such gaps? After we give the testers this feedback, and making adjustments while having the testers take full part in the decision making, every tester has a general picture as to the tests he will probably do within the sprint. This is not a final plan, but a base plan since during the sprint planning things will probably change, at least to some extent.

The purpose of these three phases was to make sure that the resource allocation between the products was sufficient for the coming sprint including the following sprint's sniffing period.4. Timing of the tests that were taken

within the sprint: During the planning you can pay attention to when each test can be done within the sprint. We tried to put risky tests first and of course tests which depend on others. In addition, we tried to unify tests that required similar actions to improve the entire process. The main idea is to map the sprint such that you won't get stuck with half of the tests in the last few days of the sprint.

The process above didn't always look like this. It took us some time to get the people abroad to follow this procedure, and till recently, there was still much room for improvement.


46 | TESTcompanyprofile

Formoreinformation,pleasevisitwww.microfocus.com/solutions/softwarequality

Deliverbettersoftware,faster.Softwarequalitythatmatchesrequirementsandtestingto business needs.Making sure that business software delivers precisely what is needed, when it is needed is central to business success. Getting it right first time hinges on properly defined and managed requirements, the right testing and managing change. Get these right and you can expect significant returns: Costs are reduced, productivity increases, time to market is greatly improved and customer satisfaction soars.

The Borland software quality solutions from Micro Focus help software development organizations develop and deliver better applications through closer alignment to business, improved quality and faster, stronger delivery processes – independent of language or platform.

Combining Requirements Definition and Management, Testing and Software Change Management tools, Micro Focus offers an integrated software quality approach that is positioned in the leadership quadrant of Gartner Inc’s Magic Quadrant.

The Borland Solutions from Micro Focus are both platform and language agnostic – so whatever your preferred development environment you can benefit from world class tools to define and manage requirements, test your applications early in the lifecycle, and manage software configuration and change.

requirementsDefining and managing requirements is the bedrock for application development and enhancement. Micro Focus uniquely combines requirements definition, visualization, and management into a single '3-Dimensional' solution, giving managers, analysts and developers precise detail for engineering their software. By cutting ambiguity, the direction of development and QA teams is clear, strengthening business outcomes.

For one company this delivered an ROI of 6-8 months, 20% increase in project success rates, 30% increase in productivity and a 25% increase in asset re-use.

Using Micro Focus tools to define and manage requirements helps your teams:

•Collaborate, using pictures to build mindshare, drive a common vision and share responsibility with role-based review and simulations.

•Reduce waste by finding and removing errors earlier in the lifecycle, eliminating ambiguity and streamlining communication.

•Improvequality by taking the business need into account when defining the test plan.

Caliber® is an enterprise software requirementsdefinitionandmanagementsuite that facilitates collaboration, impact analysis and communication, enabling software teams to deliver key project milestones with greater speed and accuracy.

SoftwareChangeManagement StarTeam® is a fully integrated, cost-effective softwarechangeandconfigurationmanagement tool. Designed for both centralized and geographically distributed software development environments, it delivers:

•Asinglesourceofkeyinformationfordistributedteams

•Streamlinedcollaborationthroughaunifiedviewofcode and change requests

•Industryleadingscalabilitycombinedwithlowtotalcost of ownership

TestingAutomating the entire quality process, from inception through to software delivery, ensures that tests are planned early and synchronize with business goals even as requirements and realities change. Leaving quality assurance to the end of the lifecycle is expensive and wastes improvement opportunities.

Micro Focus delivers a better approach: Highly automated quality tooling built around visual interfaces and reusability. Tests can be run frequently, earlier in the development lifecycle to catch and eliminate defects rapidly.

From functional testing to cloud-based performance testing, Micro Focus tools help you spot and correct defects rapidly across the application portfolio, even for Web 2.0 applications.

Micro Focus testing solutions help you:

•Aligntestingwithaclear,sharedunderstandingofbusiness goals focusing test resources where they deliver most value

•Increasecontrolthroughgreatervisibilityoverallquality activities

•Improveproductivitybycatchinganddrivingoutdefects faster

Silk is a comprehensive automatedsoftwarequalitymanagement solution suite which enables users to rapidly create test automation, ensuring continuous validation of quality throughout the development lifecycle. Users can move away from manual-testing dominated software lifecycles, to ones where automated tests continually test software for quality and improve time to market.

Take testing to the cloud Users can test and diagnose Internet-facing applications under immense global peak loads on the cloud without having to manage complex infrastructures.

Among other benefits, SilkPerformer® CloudBurst gives development and quality teams:

•Simulationofpeakdemandloadsthroughonsiteand cloud-based resources for scalable, powerful and cost effective peak load testing

•Web2.0clientemulationtotesteventoday’srichinternet applications effectively

Micro Focus, a member of the FTSE 250, provides innovative software that enables companies to dramatically improve the business value of their enterprise applications. Micro Focus Enterprise Application Modernization, Testing and Management software enables customers’ business applications to respond rapidly to market changes and embrace modern architectures with reduced cost and risk.

Micro Focus


TESTcompanyprofile | 47

Topgovernmentsandleadingsoftwarecompanies,operators,serviceprovidersandmanufacturersuseCodenomicon'ssolutions to secure critical networks and toproviderobustproductsandservices.Codenomicon enables its customers tobuildandofferreliablesolutions,andcritical infrastructurecustomers relyonCodenomicon's security solutions when takingpre-emptivesecuritymeasures.

Security testing solutionsIn all types of cyber-attacks, the initial access is enabled by software vulnerability in an open interface. Unknown vulnerabilities are the biggest threat to IT security, because there are no defenses for attacks against them. With Codenomicon’s Unknown Vulnerability Management solutions you can reveal critical interfaces andfindandfixcriticalvulnerabilities,beforeanyonehasa chance to exploit them.

Defensics is the world leading fuzzing solution. It provides fully automated security testing suites for over 200 different communicationprotocols,fileformats,andwebandXMLapplications. Defensics uses model-based systematic fuzzing to provide the best testing coverage, with unparalleledeffciencyinfindingunknownvulnerabilities. http://www.codenomicon.com/defensics/

Fuzz-o-Matic is an automated fuzzing service in the cloud. Especially for Agile projects which require extensive and flexiblefuzztesting,Fuzz-o-Maticistheidealsolution.Accessible anywhere, Fuzz-o-Matic can handle several builds at a time so you can test build by build. Access and share the test results with your development team effciently.Patchverificationhasneverbeenthiseasy.

http://www.codenomicon.com/fuzzomatic/

Abuse Situation AwarenessCodenomicon's Abuse Situation Awareness solution creates interactive visualizations from real-time network traffc and abuse information. See the status of your network and critical resources with one glance. Share the actionable incident information with your stakeholders and get cleaner networks in the process. http://www.codenomicon.com/clarified/

Abuse SA is essential for the CERT and CSIRT organizations of governments, internet service providers and other majorplayersinthefield.AbuseSAisabotnet-inspiredsolution to internet abuse incident handling with automated process for collecting, processing and reporting network abuse.

CodenomiconCodenomiconisaleadingvendor ofproactivesoftwaresecuritysolutions

T:+3584247431F:+3588340141E:[email protected] Ltd. Tutkijantie 4E, FIN-90590 Oulu, Finland


48 | The last word...

Why did DaveWhalen become a software tester? It was an epiphany really...

Afishytale

EversinceIwasaweelad, I wanted to be a fisherman.I'mactuallyfromasmallislandoffthe

coastofMaine.Myfather,bornand raised on the island, would alwaysfillmewithromanticsea-faringstories.Heenlistedinthe Air Force and during his 20- yearcareerwemovedaroundalot.Wewouldgohomeeverynowandthenfor‘OldHomeWeek’-theisland'sFourthofJulycelebration.IfollowedinmyDad'sfootstepsandenlistedintheAirForcerightafterhighschool.Mydream,afterIretiredfromtheAirForce,wastoreturnhomebeadeepseafisherman.

For our honeymoon I took my new bride to Maine for the Fourth. As we approached the island it was dusk, the fog was just lifting, the sun glowed orange off of the old white houses on the hillside. It was picture perfect. The following morning we would be on the fishing boat – I was downright giddy!

We got up early. Even though it was July, the air was crisp and cool. Steam was gently rising off of my cup of coffee. There was a thick blanket of fog on the water. And you could hear the seagulls. It was perfect!

Reality set in about the time the coffee ran out. The fog lifted and it was hot and muggy. The lifting fog also released the stench of dead fish. The seagulls were getting annoying. They smelled the dead herring and were circling looking for food. We spent much of the day dodging seagull poop. The tide was out adding a special aroma to the mix. We were gagging on diesel smoke. I spent the first hour or so helping my Dad bait hooks with rotting, slimy herring. My clothes reeked. My hands reeked. And there were still four hours to go!

The only thing we caught was a squid. My new wife caught it. She didn't really catch it. It had the misfortune of swimming by just as she was reeling

in her line. As my Dad unhooked it, it wrapped its tentacles around his arm and hung on for dear life. He eventually peeled it off and returned it to the bay.

As we sailed past the island I was jealous of the islanders in their cozy cottages, eating a nice hot breakfast. I longed to be back in my soft bed, snuggled up against my new wife, or sprawled out on the couch watching television with the smell of sizzling bacon wafting through the house. I couldn't wait to get off that stupid boat! Needless to say, that was my last fishing trip.

On the flight home I pondered my future career aspirations. Whatever I decided to do was going to be indoors, temperature controlled, and not reek of fish. I hate fish! I don't eat fish. It's been almost thirty years and I haven't been fishing since. So here I am – a software tester. I couldn't be happier! Is it glamorous? No. Will I get rich doing it? No. Will I be a famous tester? Doubtful. Am I enjoying my chosen career? Absolutely!

Most people have absolutely no clue what I do. When someone asks I will proudly state: “I am a software tester!” There is typically a look of confusion on their face. So I explain. Do you have a computer? Yes. Does it have software? Yes. Does it work? Yes... You're welcome!

I've been taking things apart to see how they work as far back as I can remember. Software testing is not all that different. If you're constantly asking yourself questions that start with “I wonder what would happen if...” then you're probably a good tester.

I learned this skill early in my career validating the software used to test avionics. I would look at a circuit diagram of a component and wonder what would happen if circuit X was open. So I'd pull out the circuit board, take it to one of the solder-masters and have them open the circuit for me. Then I'd reinstall the newly defective circuit board, putting it back in the

component and run the test software to see if the software would find it. Sometimes it did. If it didn't I'd bring the developer to my work station, show them what I did and run the test. They would fix the test software, bring it to me and we'd repeat the process until the software found the problem and gave the expected corrective action – replace the circuit board. It was the most fun I've ever had.

As a tester you're really a fisherman. You're trying to find those elusive critters. It helps to know where to look. That's the art of software testing. Finding it is then just a simple process of casting your bait or your net in the right place and hauling it in. The best part of software testing over fishing? It's usually warmer and smells better.

the last word...

DaveWhalenPresident and senior software entomologistWhalen Technologiessoftwareentomologist.wordpress.com

Most people have absolutely no clue what I do. When someone asks I will proudly state: “I am a software tester!” There is typically a look of confusion on their face. So I explain. Do you have a computer? Yes. Does it have software? Yes. Does it work? Yes... You're welcome!

For exclusive news, features, opinion, comment, directory, digital archive and much more visit


The Whole Story

www.31media.co.uk

Print Digital Online


test magazine - december 2012-january 2013

Documents