predict model for test automation. does it sound familiar to you? organization has procured test...
TRANSCRIPT
PREDICT Model for Test Automation
Does it sound familiar to you?Does it sound familiar to you?
• Organization has procured test automation tools
• Management expectations are high
• Multiple test automation project initiatives started
• Enthusiasm, commitment and motivation declined with
time
• Script maintenance had become a significant challenge
• Test automation tools are back on shelf
• Testing and test automation requires same skills
• Capture/Replay is a successful test automation approach
• Test automation is an integral part of the over all test process
• Test automation is effective in solving regression testing issues
• Every test and test type can be automated
• Test automation development is software development
• Test automation projects can be delivered by personnel with strong
project management skills
Spot QuizSpot Quiz
Essential building blocks for successful test automation solutions
Test automation solution approachTest automation solution approach
PREDICT ModelPREDICT Model
RationalizeRationalize
TransitionTransition EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
PlanPlan
PREDICT ModelPREDICT Model
• Structured methodology• Multi stage process consisting of seven components
• Plan• Rationalize• Estimate• Design• Implement• Consolidate• Transition
• Step-wise approach• Easy to apply
PREDICT Model - PlanPREDICT Model - Plan
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
Need for test automation projectNeed for test automation project
• What do you consider to start a project ?• Test related checks
• Objective• Scope• Documentation• Test ware
• Organization related checks• Investments• Expectations• Resource availability
• Technology related checks• Stability of SUT• Test tool suitability
Need for test automation projectNeed for test automation project
• Are you getting bang for your buck ?• Costs
• Licensing• Framework• Implementation• Maintenance factor
• Benefits• Saved manual effort per test run• Number of test runs• Re-usability factor
• Amortizing benefits• Life cycle of the application/product• Opportunities for test automation
Plan of approachPlan of approach
• Introduction• Objective• Pre-requisites• Set-up pilot project• Activities• Planning• Deliverables• Organization• Resources
PREDICT Model - RationalizePREDICT Model - Rationalize
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
How do we rationalize requirements ?How do we rationalize requirements ?
• 5 P factors
– Project
– Process
– Product
– People
– Price
• Test suite attributes
• Building trade-offs based on project context
ProjectProject
• Objective and scope• Clear objectives in terms of time, money and quality
• Strategy• Test automation strategy fitment with manual testing strategy
• Build• Build frequency
• Configuration management• Bad experiences
• User interface• Stability
• Cycle time• Regression possible
ProcessProcess
• Quality documentation• Manual test scripts, documentation
• Development approach• Phases clearly defined, how early can test automation
start
• Test practice• New builds tested before delivering to QA• Regression moments
• Naming conventions• User Interface conventions
ProductProduct
• Stability• Base platform stability
• Testability• Degree of interaction between the tool and product
under test
• Test tool integration• Integration between test management tool and
configuration management
• Test oracles• Interaction with non UI components,database
• Automation framework• Availability of frame work to reuse
PeoplePeople
• Organization readiness• Management expectations
• Resource availability• Machine, Licenses and people with right skills
• Non- programming resources• Framework inclined towards non-programming resources
• QA population• Ratio of development to QA
• Collaboration• Collaboration between QA and Dev
• Technical skill set• Test engineers with technical skills
PricePrice
• Return on Investment• Opportunities of reusing test automation investment
• Reusability of Investment• Are there any products which are tested in a similar
way
• Business Plan• Is there a business plan to realize benefits of Test
Automation
Test suite attributesTest suite attributes
• Maintainability• Reusability• Reliability• Usability• Portability
Which is best test automation regime ?Which is best test automation regime ?
IT DEPENDS ON THE PROJECT CONTEXT
PREDICT Model - EstimatePREDICT Model - Estimate
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
Test automation estimation challengesTest automation estimation challenges
• Test organization maturity• Scope of requirements• Framework• Skill level• Test case complexity• Domain complexity• Test tool• Testability
Estimation techniquesEstimation techniques
• Delphi• Gather estimates from each team member• Ask high and low estimator to explain estimates• Repeat twice, then use average
• Three point method• Have team estimate best, worst case and expected case• Use expected case• Variances between best and worst useful to gauge
estimate accuracy
Work Break down StructureWork Break down Structure
Risk categoriesRisk categories
• Technical• Test tool• Application/product• Communication
Risk managementRisk management
PREDICT Model - DesignPREDICT Model - Design
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
Test suite design issuesTest suite design issues
• Testability• Test suite design patterns
Testability considerationsTestability considerations
• Tools• Design• Naming Conventions• Exception & Error Handling• Timing issues• Logging• Visual Cue• Processes & Documentation
• Record/Play back
• Functional de-composition
• Key-word Driven
• User Interface-driven
• Model-based
Test automation development approachesTest automation development approaches
• Advantages
• Useful in determining how the tool interacts with the application under test
• Provides initial ideas on how to develop test scripts
• Useful while ‘Playing around’ with the tool
• Disadvantages• Test scripts contains hard-coded values
• Un-reliable scripts : Any small change in the application results in not being able to replay
• Maintenance costs are astronomical
• Not viable & cost-effective
Record & PlaybackRecord & Playback
• Reduces test scripts into their fundamental tasks• Driver Scripts : Initialization, Calling scripts in a desired
order • Generic scripts : General,Application
specific,Navigation,Data verification,Reporting,logging etc…
• Business Function Scripts : ‘Re-usable’ Business functionality,Makes calls to Generic Scripts
• Test Script : Test Logic, Makes calls to Business Function Scripts and Generic Scripts
• Test data and logic are separated• Hierarchical structure is employed :
Modular design• Each test script unique data file
Functional decompositionFunctional decomposition
Test Director
Reports
Test Scripts
Generic Library BusinessLibrary
Test Scripts
EnvironmentConfiguration
Stored Proc ObjectRepository
Technical ArchitectureTechnical Architecture
Functional decomposition: AnalysisFunctional decomposition: Analysis
• Advantages• Modular design : Reduces redundancy and duplication of
efforts • Test data input & output stored in a single data file : Easy
maintenance• Scripts can be developed whilst application development is
in progress : Generic Scripts• Single point maintenance is possible using Business
Function scripts• Scripts can return TRUE (or) FALSE to the calling scripts.
This makes it easy for ‘Error Recovery’ & un-attended test execution
• Assemble tests on demand
• Disadvantages• Proficiency in the TSL• Complex test data file management
• The test script is represented as a set of key-words in a spread sheet• Examples of Key-words are start_test, Initial
window,Url,Input,Verify etc..
• Driver Script Performs the initialization as required • Driver Script Reads and processes the file name(s)
of the test scripts• Driver Script Matches the ‘Key Words’ contained in
the file• Driver Script invokes Business Function & Generic scripts• Report the return values back• The entire process is data-driven
Key-word drivenKey-word driven
• Advantages• Preserves most of the advantages of the ‘Functional
decomposition’ method• Very similar to documenting manual test script• Minimal programming knowledge is required to write and
execute test scripts• Return on investment can be achieved quicker than other
methods in most of the cases• Re-usability significantly increases over a period of time• Test script development is possible with out a functioning
application
• Disadvantages• Investment in initial frame work• Training might become tedious
Key-word driven - AnalysisKey-word driven - Analysis
GenericScripts
Keyword Driven Test Cluster
Business FunctionScript4
Reports
Driver Script
Business FunctionScript1
Business FunctionScript2
Business FunctionScript3
GUI Map
Technical ArchitectureTechnical Architecture
• Abstraction of test structure: Test set Test case Test step Test action
• Key-words are identified based on the application under test
• Test steps are created using user interface• Test set management (Creation/execution)
Test set
Test case
Test step
Test action
User interface drivenUser interface driven
Automation Framework
RecordingEngine
Script Generation
Module
APPLICATION
User Interface
Test
Test Script
XML Object Repository
Object Creation Module
Execution Engine
Win Runner/QT
P/Silk Test/…
Library Structure
Technical ArchitectureTechnical Architecture
• Advantages• Preserves most of the advantages of the previous
methods• Insulates tests both from the changes of the application
and failings of test automation tools• Zero-programming required• Return-on-investment is very high• Re-usability significantly increases over a period of time• Robustness and maintainability is high• Partly application independent• Minimal learning curve
• Challenges• High initial investment in creating framework• Functioning application needed
User interface, Model driven - AnalysisUser interface, Model driven - Analysis
Test Case Generator(In key words)
Keyword Translator engine(Test tool specific)
Automated Test Scripts
Test drivergenerator
Behavior modeler
Requirements (UML)
Win Runne
r
Win Runne
r
Silk Test
Silk Test
Other test tools
Other test tools
Model DrivenModel Driven
PREDICT Model - ImplementPREDICT Model - Implement
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
Test automation implementation issuesTest automation implementation issues
• Team structure• Work allocation• Coding guide-lines and check lists• Peer reviews• Test runs• Integration• Configuration management
Work allocationWork allocation
Coding guidelines and checklistCoding guidelines and checklist
ReviewsReviews
Building integrationsBuilding integrations
Test Script1
Test Script2
Test Script3
Call Test Script1()Call Test Script2()Call Test Script3()
Configuration management : VSSConfiguration management : VSS
Test executionTest execution
• Test suite execution check-lists are used• Test Execution
• A separate test environment is prepared • Fresh test data in test DB• Fresh test data in Application DB
• Tests are executed • Test suite execution issues & root cause
analysis is documented• Report Test Results• Update test scripts if needed
Test execution logTest execution log
PREDICT Model - ConsolidatePREDICT Model - Consolidate
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
• Update scripts and test data• Complete documentation (User Manual, installation
Guide and maintenance guide)• Freeze & archive test suite• Professionally packaged shippable test suite• Easy distribution mechanism
Activities in consolidationActivities in consolidation
User manualUser manual
Professional packagingProfessional packaging
PREDICT Model - TransitionPREDICT Model - Transition
RationalizeRationalize
TransitionTransition
PlanPlan
EstimateEstimate
DesignDesign
ImplementImplement
ConsolidateConsolidate
• Acceptance test runs performed• Knowledge transfer to the user groups• Gain feedback from the field• Prepare for maintenance and sustenance• Up-gradation plans• Discussion for next project
Transition processTransition process
• Successful test automation projects require mature test automation leadership skills
• Test automation projects should be treated like a development project
• Understanding PREDICT Model• Challenges, issues, tips and tricks in applying
PREDICT model
Wrap upWrap up