towards further automation of the q uality a ssurance cycle
DESCRIPTION
Towards Further Automation of the Q uality A ssurance Cycle. Alan Berg: University of Amsterdam Central Computer Services (IC) Group Education and Research Services. But first an advert. QA’ ing makes you sexy. QA or the Sakaiger gets it. QA’ing helps you get the toy. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/1.jpg)
Towards Further Automation of the Quality Assurance Cycle
Alan Berg: University of Amsterdam Central Computer Services (IC) Group Education and Research Services
![Page 2: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/2.jpg)
But first an advert
![Page 3: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/3.jpg)
QA or the Sakaiger
gets it
![Page 4: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/4.jpg)
QA’ing helps you get the toy
![Page 5: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/5.jpg)
We have only 30 minutes, so only a brief biased personal snapshot is possible
![Page 6: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/6.jpg)
Agenda
• The potential for a QA event horizon• What does the QA WG want?• Current hints to the developer
– Where are we
• The Sash Tool• Jameleon• Conclusions
![Page 7: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/7.jpg)
The QA event horizonRisk losing a deterministic QA cycle
http://amazing-space.stsci.edu/resources/explorations/blackholes/teacher/grabbag.html
![Page 8: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/8.jpg)
To remain deterministic we will need more automation
![Page 9: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/9.jpg)
The QA event horizonRisk losing a deterministic QA cycle
The Obvious• Code base increasing rapidly• Not just Java, but also XML, Javascript,
Python,CSS and many Java frameworks• Iterative process that requires defined
response times• Requires manpower/resources on time• Would love QA as part of Continuous Integration
![Page 10: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/10.jpg)
What does QA WG want?
• A "stack" of open source tools for load testing and nightly integration testing
• A load-testing repository and "cookbook" for open source tools
• Regression testing work flow
• QA Process - no tool makes it into the core (stable) set of tests
• QA Process - no bug in JIRA is closed without an automated test that checks for that bug.
http://bugs.sakaiproject.org/confluence/display/QA/QA+Test+Automation
![Page 11: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/11.jpg)
Hints to the developer
![Page 12: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/12.jpg)
Supporting developers• Nightly
– builds – Continuum [Claret]– Javadocs - [nightly]– Static code reviews – [UvA]– Internationalisation - [LOI ]– Unit testing [ developers ] • – Potentially through Continuum
• -Mock objects from Josha Holtzman
• Knowledge capture [Confluence, community, conferences]
![Page 14: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/14.jpg)
Random Example
![Page 15: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/15.jpg)
Duplicate Code
![Page 16: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/16.jpg)
Do we wish to use the reports more aggressively?
• Are the developers taking note?– Bugs reported are not always correct– Bugs found not always important– But duplicate code, unit test coverage, and failing
to deal properly with exceptions hint strongly at quality.
• Answer probably not, but thankfully trends in the marketplace will improve the quality of results overtime.
![Page 17: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/17.jpg)
The Sash Tool
![Page 19: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/19.jpg)
Realistic Proof of Concept• Provision tool
– add/delete users,sites,members,tools,permissions
– upload files from file system to dropbox/resources
– Controlled by a property file in site resources
• liveTest• Diagnose• Report• But have we enough API’s to do the job?
![Page 20: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/20.jpg)
SASH TOOL (Steven Giffens)
• Live• Brilliant for learning• Environment that targets sys_admin• Easy to extend commands• Consistent use of cover api's allow services
to be easily guessed.• Use to write provision, diagnose, live test
tools.
Just requires imagination and Sakai specific knowledge.
![Page 21: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/21.jpg)
import org.sakaiproject.tool.cover.ToolManager;import org.sakaiproject.tool.api.Tool;import org.sakaiproject.component.cover.ComponentManager;import org.sakaiproject.content.api.ContentHostingService;
ident="....";
all = ToolManager.findTools(null, null);counter=1;for ( Iterator i = all.iterator(); i.hasNext(); ) {
tool= i.next();print("["+counter+"] "+tool.getTitle());print("ID : "+tool.getId());print(ident+tool.getDescription());print (ident+ident);counter++;
}counter=1;print ("Interfaces");inters = ComponentManager.getRegisteredInterfaces();for ( Iterator i = inters.iterator(); i.hasNext(); ) {
inter= i.next();print("[: "+counter+"]"+inter);counter++;
}
Trunk 2.4....Tools=104,Interfaces=568 live
Uses covers
Consistent useof java.utils
LIVE
![Page 22: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/22.jpg)
Cover like ...import org.sakaiproject.test.cover.TestManager;import org.sakaiproject.testl.api.Test;
all = TestManager.findTests(TestManager.RUNALL);counter=1;for ( Iterator i = all.iterator(); i.hasNext(); ) {
test= i.next();print("["+counter+"] "+test.getTitle());print(test.getDescription());
try{print(test.doTest());
}catch(FailedTestException e){print(“Error was: “+e.getMessage());
}finally{counter++;
}}
![Page 23: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/23.jpg)
RememberQA Process - no tool makes it into the core (stable) set of QA Process - no tool makes it into the core (stable) set of
teststests
BRAINSTORM ONLY
• Push tests back to the developer
• Runs against a live system
• Can be removed at compile time for production distributions
• Non standard [Yeuch]
• Consistent, and test driven
• Easy to script through Sash
• Only solves a certain significant class of tests
![Page 24: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/24.jpg)
Jameleon
http://jameleon.sourceforge.net/
![Page 25: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/25.jpg)
Regression testing workflow
CRITERIA• Run automatically • Results consistent• Many roads to Rome.
– Selenium RC– Selenium RC wrapper in Maven, Python, Ant
• Cost benefit ratio favourable
![Page 26: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/26.jpg)
Jameleon - Proof of Concept
• Wrapper around Selenium RC and other tools– Junit,Jiffie (Internet Explorer,HttpUnit,generic)
• Configured via XML• Generates reports• Can run headless via cron or continuum• Data driven• Almost self documenting
![Page 27: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/27.jpg)
Data driven• Makes tests server generic• Property file per organization
– uva.properties• name=admin• pass=admin• base=qa1-nl.sakaiproject.org:8380
• csv with variables defined in first line• eid• newbiex• newbie2
![Page 28: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/28.jpg)
Self documenting
http://jameleon.sourceforge.net/
![Page 29: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/29.jpg)
<?xml version = '1.0'?><testcase xmlns="jelly:jameleon"> <test-case-summary >Add user to SakaiQA</test-case-summary> <test-case-author >Alan Berg</test-case-author> <test-case-level >SMOKE</test-case-level> <functional-point-tested >Add User to Sakai ${baseUrl}</functional-point-tested>
<selenium-session startSeleniumProxy="true" application="uva" beginSession="true" seleniumMultiWindowMode="true" seleniumStopProxyServerOnSessionClose="true"><csv name="add_user_sakai" countRow="true"> <selenium-open functionId="Goto Portal" url="http://${base}/portal/tool/!gateway-710?panel=Main" /> <selenium-type keys="${eid}_FIRST" functionId="Type Firstname " locator="name=first-name" /> <selenium-type keys="${eid}" functionId="Type username ${eid}" locator="name=eid" /> <selenium-type keys="${eid}" functionId="Type password" locator="name=pw" /> <selenium-type keys="${eid}" functionId="Type verify" locator="name=pw0" /> <selenium-type keys="${eid}@dev.null" functionId="Type mail" locator="name=email" /> <selenium-type keys="${eid}_LAST" functionId="Type Lastname" locator="name=last-name" /> <selenium-click functionId="Click submit" locator="eventSubmit_doSave" functionDelay="1000" /> <selenium-select-window functionId="Select relevant frame" windowName="null" /> <selenium-click xmlns="jelly:jameleon" functionId="Click submit" locator="link=Logout" /> </csv> </selenium-session></testcase>
XML test cases
![Page 30: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/30.jpg)
Workflow
• From Cron– Download and build Sakai– Copy different scripts to directory– Run Jameleon• (can run against IE, Firefox, Opera etc)
– Generate reports– Move reports to website
• Can be added as a script to run to continuum
![Page 31: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/31.jpg)
Results• Works• Selenium plug-in still beta• Tests are fragile and require maintenance• Will require a significant effort for basic
coverage• Still very promising and should be considered.• HATS OFF TO THE DEVELOPERS
![Page 32: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/32.jpg)
My plans
• I have one day a week free to:– Concentrate on Sash scripts– Wait until the next release of Jameleon.– Beg the big brains to build a realistic test
manager.
![Page 33: Towards Further Automation of the Q uality A ssurance Cycle](https://reader036.vdocument.in/reader036/viewer/2022062314/5681369e550346895d9e410f/html5/thumbnails/33.jpg)
Questions
????
?