fusion testing - maximizing software test execution
Post on 05-Dec-2014
2.301 Views
Preview:
DESCRIPTION
TRANSCRIPT
Fusion Testing
Maximizing Test Execution
By: James Tischart
About Me• Director - Product Delivery SaaS at McAfee • 15+ years experience in testing and engineering
• Multiple certifications in Agile and Testing
• Passionate about Testing as an eng discipline, a science and an art
• Continue to challenge the status quo, test new approaches and always strive to improve
• Support the legitimacy of the testing professional in the broader engineering world
What is Fusion Testing?
An occurrence that involves the production of a union
• Organization of structured testing • Freedom of exploratory testing • Rigors of automated testing • Combined into one test methodology
To achieve
• Maximize code execution• Increased test coverage • Reduced test artifact and documentation• Higher quality for users• Improved data for organization
Fusion
Focus – start your day with 15 minutes of thought
Usage – how will your users work with the system
Scope – decide on the scope of everything
Initiate – just go and explore
Organize – create a plan & be ready to deviate from it
Note – keep track of your exploration to retrace steps
Fusion Testing GuidanceMaximize test execution with Fusion by:
1. Identify Goals for Testing
2. Choose the right mix of methodologies
3. Utilize Tests Lists to guide exploration
4. Automate the Right Things
5. Document at the Right Detail
Fusion Implementation
1. Guide your team through the change
1. Create a Change Message
1. Mentor your team members
2. Train the team in testing and engineering
3. Create Metrics to measure success & failure
Fusion & PlanningRelease Test Plan
• Set the goals for the overall release
• Plan the test environment
• Organize the metrics and performance
• Plan a budget, resources and a time frame
• Plan your automation and tool set
• Plan at a high level
Iterative Test Plan• Very effective in Agile implementations
• Break up the testing into iterations
• Low level details for the time frame• Keep to two weeks or less• Detail the testing to accomplish• Detail the performance tests• List what will not be tested• Identify risks to the plan• Plan what Regression Testing• Identify what test-lists need to be written
Daily Plan• Individually created each morning • Provides guidance and goals for the day
• Details the execution plan
Test Lists• A grouping of test ideas
• No specifics in the test list
• Priority, variations & guide for exploration
The Power of Many• Utilize different users of your system to bring fresh perspectives
- Support personnel - Sales staff- Developers - Training- User groups - Product Management
• Help guide the testing by providing checklists, environments and goals
• Don’t give exact details or specific steps this minimizes innovation
• Make the event fun and you will have many people continuing to help
• Have testing experts available to help the volunteers for problems
Identify Defect Traits• For new products use historical traits from similar projects
• Analyze Severity 1 Customer reported incidents
• Determine the trends to those defects in coding & testing
• Constrain the review to a calendar year of defects
• Next analyze Sev 1 internally found and fixed issues and determine the traits
• Inform the developers & testers of the traits
• Create test lists or test ideas to cover these traits
• Repeat the process with each severity level you use
Fusion & AutomationDetermine your layers for automation:•Server
• Databases, server processes• Machine interaction points • Process interaction with other
layers
•Middleware/API layers
•User Interface
•Optimize your test artifacts • Don’t duplicate test-cases• Use automation for
traceability.
Test Results & Metrics
There will always be interest in the test data no matter the test structure.
Q: How can you accurately assess quality when the testing combinations exceed the particles of the universe?
A: You can’t! We need to present what was tested, what has not been tested and support the assessment used to make this prioritization.
• # of Test Ideas Executed based on Priority
• # of Test Cases passed/failed versus total coverage that the tests represent
• # of Automated Tests Passed/Failed by # of executed times
• Performance/Benchmark comparisons by build/iteration/release • Team Quality Satisfaction Rating – get the gut feeling of the team
• Open defects to highlight potential issues that your customers may find
• Go to fusiontesting.blogspot.com for more details on these metrics
Since you can’t test everything, here are some ideas of results to report:
Traceability Matrices
Positives Negatives
Can trace tests to requirement coverage
Time consuming to create and maintain
Displays what has been executed Often out of date and misleading
Shows relationship between tests and features
Duplicates information from test cases and requirements
Provides defect traceability to features
Dedication to frequent updates needed
Do Don’t
Tie your test lists to requirements or stories
Trace to the test case or test idea
Constrain your traceability to the feature level
Try to trace to the functional level
Build traceability into the automated tests
Duplicate information in many documents
Map automation to test lists or test specs
Over complicate maintenance of multiple sources
Implementation Challenges
Challenge: Management
Identify the Challenges: • Reliance on historical metrics
• Understand the current processes and practices
• Decision-making timelines
• Need traceability to feel confident of data
Respond to the Challenges:
• Provide better metrics
• Show how new process improves quality
• Prove how decision can be made faster • Review traceability needs and support them
Challenge: PMOIdentify the Challenges: • Reliance on historical metrics
• Struggle with the details of testing
• Need predictability for costs and schedules
• Require improved time to market
Respond to the Challenges:
• Debunk misleading metrics
• Show how more testing provides better data
• Prove reduced costs with efficiency gains • Track lost opportunity cost on test artifacts
Challenge: EngineeringIdentify the Challenges: • Takes time away from coding
• Testing can be tedious
• Not their specialization
• Rely on a serial approach to testing
Respond to the Challenges:
• Test first design increases new coding time
• Automated & exploratory tests are less tedious
• Understanding testing improves code writing • Less defects will be logged with up-front tests
Challenge: Regulation
Identify the Challenges: • Documentation Requirements
• Formal or Standards Approval
• Full traceability
• Rigorous Automation & detailed test results
Respond to the Challenges:
• Lean towards more structured testing
• Use exploratory but document results
• Plan for and work in shorter iterations • Automate more, document more
Challenge: Outsourcing
Identify the Challenges: • Require more instruction and detail
• Lack expertise in the product and markets
• Multi-cultural communication
• Formal process & document to support work
Respond to the Challenges:
• Phase in Fusion and train on exploratory
• Provide more details in documentation
• Implement multiple communication paths • Support and train new processes
Ensuring Success
Implement in stages
1.Measure the impact to customers (before & after)2.Collect metrics for a project for test execution time before implementation3.Introduce Fusion to cover about 25% of testing4.Increase Fusion usage to 50% to further improve execution time5.Continue to phase in until 75 to 90% of testing is Fusion
Monitor
1.Increases or decreases in execution time 2.Review documentation size closely reviewing the test artifacts 3.Measure the amount of direct automation4.Review with the team the amount of time spent on maintenance5.Continuously try to improve exploration time and increase efficiency
Adjust based on the needs of the project
1.One project may require more precise documentation 2.Another project may require less structure3.Identify these requirements at the start of projects and adjust4.Reduce the amount of time spent on artifact management5.Inform people of the time impacts of data collection
Five Keys to Fusion1. Detail – the right level of detail at the right time
2. Planning – consistently plan, execute and adjust
3. Automate - spend time on specifics in automation
4. Report – measure what truly represents the team and customer
5. POM – get more people involved to expand the breadth of test
Conclusion
top related