fostering long-term test automation success

BT9 ConcurrentSession11/12/151:30pm

“Fostering Long-Term Test Automation Success”

Presented by:

Carl Nagle

SAS Institute, Inc.

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·info@techwell.com·www.techwell.com

Carl Nagle SAS Institute, Inc.

At SAS Institute Carl Nagle is a principal software developer, focused for the past seventeen years on the development, integration, and evolution of enterprise software test automation tools and the open source SAFSDEV framework. For more than thirty years, Carl has been responsible for engineering both hardware and software testing systems including the design of advanced electronics for analog and digital subsystem testing with embedded software for high-end power generation and communication systems. This naturally evolved into the development of advanced software testing systems and frameworks. Carl and his team thrive on solving “impossible” test automation scenarios across multiple processes, platforms, and technologies.

Fostering*Long,term*Test*Automa2on*Success

!�Sustainable�!Test!Automa0on!

!with!Carl%Nagle,!!SAS!Ins0tute!

• Why%are%YOU%here?%

• Why%am%I%here?%(Has%it%really%been%9%years?)%

•  How%did%we%achieve%ini@al%success?%

• Where%are%we%now?%

•  Q%&%A%%!

In%A%Nutshell%

If!these!apply!to!YOU:!!•  NEED!or!WANT!easy!test!automa0on!NOW,!•  For!MY!current!team!or!project,!•  Have!NO!concern!about!others,!or!even!future!projects,!•  Hope!to!�fix�!a!failing!automa0on!strategy…!

Then,!you!might!be!looking!at!the!wrong!presenta0on!!!(!But!stay!!!We!might!be!able!to!change!your!mind!;Q)!

Why%Are%YOU%Here?%

If!these!apply!to!YOU:!!•  NEED!test!automa0on!processes!that!last!•  To!be!shared!and!leveraged!across!projects!•  That!don�t!change!with!technology!or!tools!•  That!might!�replace�!a!�struggling�!automa0on!strategy…!

Then,!you!might!be!looking!at!the!right!presenta0on!!!!

Why%Are%YOU%Here?%

!!!!!!!!!!!!!Over!18!years!ago,!SAS!Ins0tute!tasked!me!to:!!•  Provide!a!common!test!automa0on!frameworkQQeliminate!silos!•  Make!test!automa0on!accessible!to!developers!*and*!testers!•  Handle!all!product!and!everQchanging!technology!domains!•  Sustain!test!automa0on!success!over!years,!not!cycles!•  Develop!it!in!an!openQsource!environment!

!!!!!!Today!I!report!that!this!prospers!s0llQQeven!aWer!18!years!!

Why%am%I%Here?%

1997% 2002% 2004% 2006% Today%

1.  We!listened!to!the!industry!experts!�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�!

2.  We!listened!to!poten0al!users!�We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.�!

3.  We!collaborated!with!professional!peers!�Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.�!!

4.  We!implemented!it,!!shared!it,!!and!it!con0nues!to!evolve.!

How%did%we%achieve%ini@al%success?%

The!headlines!read:!!

!“In%today’s%environment%of%plumme@ng%cycle%@mes,%test%automa@on%becomes%an%increasingly%cri@cal%and%strategic%necessity.%%%Assuming%the%level%of%tes@ng%in%the%past%was%sufficient%(which%is%rarely%the%case),%how%do%we%possibly%keep%up%with%this%new%explosive%pace%of%webUenabled%deployment%while%retaining%sa@sfactory%test%coverage%and%reducing%risk?%%%The%answer%is%either%more%people%for%manual%tes@ng,%or%a%greater%level%of%test%automa@on.%AVer%all,%a%reduc@on%in%project%cycle%@mes%generally%correlates%to%a%reduc@on%of%@me%for%test.”%

The!headlines!read:!!!

%“With%the%onset%and%demand%for%rapidly%developed%and%deployed%web%clients,%test%automa@on%is%even%more%crucial.%%%Add%to%this%the%cold,%hard%reality%that%we%are%oVen%facing%more%than%one%ac@ve%project%at%a%@me.%%%Perhaps%the%team%is%finishing%up%Version%1.0,%adding%missing%features%to%Version%1.1,%and%prototyping%some%new%technologies%for%Version%2.0!”%

IBM!Ra0onal!Robot,!Func0onal!Tester!

HP!WinRunner,!QuickTest!Pro!

AutoIt!

“We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.”%

DIFFERENT!Technologies!

HTML!3.2,!4.0,!5,!SAP,!DOJO,!GWT!

Flex!

Java!

iOS,!Android!

VB,!.NET,!WPF,!Office!

DIFFERENT!Tes0ng!Tools!

SmartBear!TestComplete,!TestExecute!

EggPlant,!Sikuli,!ImageQBased!Tes0ng!

Mobile!SDKs,!Robo0um!

Selenium!1.0,!WebDriver!

Require%

C/C++!

•  Separate!the!test!design!process!and!assets:!!•  Completely!independent!of!the!test!execu0on!“engines”!•  DataQdriven,!ac0onQbased,!keywordQdriven,!behaviorQdriven!•  Support!testQfirst!principles!where!desirable!

!

�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%

PseudoQTypical!Example!Test!Records:!!

Google%%%%SignIn%%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%"Coords=50,200�%%%%(tsk,%tsk,%tsk)%Explorer%%FolderTree%%Click%%My%Documents%%%%%%%%(beher)%

•  Separate!the!test!design!process!and!assets:!!•  Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance!

�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%

[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%

[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%

The!Map:!

The!Test:!

Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance.!

•  Separate!the!test!design!process!and!assets:!!•  Common%API!despite!years!of!evolving!technologies!and!tools!

�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%

•  Doesn’t!maler!what!the!run0me!execu0on!engine!is:!!SQA%Robot,%IBM%Ra@onal%Robot,%XDE%Tester,%Func@onal%Tester%HP%WinRunner,%QuickTest%Pro,%SmartBear%Test%Complete,%AutoIt,%EggPlant,%Sikuli,%IBT,%Selenium,%Mobile%SDKs,%Robo@um%…%

Google%%%%SignIn%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%My%Documents%

�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%

hlp://safsdev.sourceforge.net/sqabasic2000/SAFSReference.php!

•  Separate!the!test!design!process!and!assets:!!•  Test!IDE!can!be!different,!depending!on!skills!and!preferences!•  CSV%Text%Files,!Spreadsheets,!Test!Composer!(CSV!Exporters)!•  Java!IDE,!Eclipse,!SeleniumPlus!(Java!API!Drivers)!•  Custom!IDE,!or!programs!(COM,!Ruby,!Perl…)!invoking!the!API!

�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%

Java!API!Examples:!!

Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%

Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%

Java!API!!Test:!

CSV!Test:!

The!Map:!

[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%

Then%do%safs%ac@on%Click%on%SignIn%in%Google%Then%do%safs%ac@on%Click%on%FolderTree%in%Explorer%using%�My%Documents�%

Cucumber!Test:!

•  We!need!to!be!able!to!move!our!people!around!easily.!•  A!person!tes0ng!.NET!can!easily!switch!to!tes0ng!HTML.!

!

•  They!can’t!be!bogged!down!constantly!learning!new!tools.!•  We!just!want!to!focus!on!test!design!using!the!Common%API.!

!

•  They!want!to!support!each!other!across!divisions.!•  Experience!applies!to!all!current!and!future!projects.!

“We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.”%

Applica@on%Map(s)%%(Vola0le!SPOM)!

SAFS%Services%and%Engine(s)!!

Java,!Web,!Flex,!Android!

Applica@on%Under%Test%%Java,!Web,!Flex,!Android!

Logs%

Func@onal%Tester%TestComplete%SeleniumPlus%Robo@um%ImageUBased%Tes@ng%

Executable%%Test%Designs%

%(CSV,!Cucumber,!

!Java,!SeleniumPlus)!

What%this%looks%like%in%a%complex%graph?%

•  Implement!independent!Engines!for!test!execu0on:!!•  Completely!independent!of!the!applica0on!being!tested!•  Engines!interpret!and!implement!the!Common%API%

•  Whether!the!domain!is!Java,!.NET,!Flex,!HTML,!SAP,!DOJO…!•  Whether!the!tool!is!Adobe,!IBM,!Selenium,!SmartBear…!

•  Engines!can!be!services!available!to!any!machine!process!•  Engines!don�t!know!and!don�t!care!what!is!driving!them!•  Mul0ple!engines!can!run!concurrently,!on!the!same!machine!•  Support!engines!centrally!for!ALL!users,!and!ALL!applica0ons!

“Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.”%

What%are%those%internal%Services?%

Logging!

Maps!

Variables!

Drivers!

Engines!

(But%you%really%don’t%have%to%care%about%this.)%

Where%are%we%now?%•  We!have!grown!to!4!fullQ0me!inQhouse!framework!developers.!•  Internally,!our!userQbase!grows!in!the!US,!Canada,!India,!and!China.!•  We!just!completed!development!migra0on!to!SAFSDEV!on!GitHub.!•  New!discussion!forums!at!safsdev.freeforums.net.!•  Older!forum!archives!at!sourceforge.net/p/safsdev/mailman/!

!•  We!have!doc!and!updated!binaries:!SAFSDEV!on!SourceForge.!

!•  We!have!always!welcomed!contributors!for!new!engines!or!

features.!•  Frankly,!we!have!been!VERY!BUSY!since!our!last!public!talk!in!2006.!

Fostering*Long,term*Test*Automa2on*Success

!�Sustainable�!Test!Automa0on!

!with!Carl%Nagle,!!SAS!Ins0tute!Q*&*A