r&d testing
TRANSCRIPT
R&D testingVenturing under the surface of testing
Ondrej Tulach, Innovation Week 2018 Get slides: tulon.cz/QA
Testing: what does it mean officially?
„Procedure of submitting a subject of test to such conditions or
operations as will lead to its proof or disproof or to its acceptance or
rejection” (Merriam-Webster)
„Process concerned with planning, preparation and evaluation of SW to
determine that it satisfies specified requirements; to demonstrate that it's
fit for purpose; and to detect defects.” (ISTQB)
Questions, more questions
„Subject to conditions”? What ones? How to find out?
„Satisfies requirements”? Which? Where to get them?
„Is fit for it‘s purpose”? How do you prove that?
„Find defects”? How many is enough? How to find them? Should we count them?
Testing is a process, but far more is required to properly test
What is testing for in R&D practice?
Making sure the product works as it should
– by running scenarios similar to what the customer would do
– by ensuring that customer‘s use of product doesn‘t leave bad impression
(it‘s not cumbersome, annoying...)
Making it possible to fix defects in the product
– by discovering errorenous implementation, bad assumptions and
unhandled problems
Increasing confidence in product
– by successfully trying enough scenarios that we know the products is
reasonably robust
Quality Assurance isn‘t just „testing”
QA is interdisciplinary field consisting of many activities:
1. Testing the actual execution of testing scenarios
2. Test automation tests run&evaluate themselves→regression
3. Test analysis systematic creation and selection of scenarios
4. Risk analysis determining probability and impact of failures
5. Review (static m.) critical feedback to SW in each lifecycle phase
6. Quality management adjusting processes and tools to ensure quality
„Testing” itself is just a tip of the iceberg
Testing maturity (=quality) levels
1) Verification testing– demonstrate it works (in at least 1 way)
2) Destructive testing– 1) + try to break it, directly and indirectly; test
sanitization of inputs, error messages, invalid data, ...
3) Lowering overall risks– 1) + 2) + use Test Analysis tools to increase scenario
coverage of product and environment
4) Preventive Quality Assurance– 1) + 2) + 3) + Integrity levels and FMEA etc.
What is needed for QA?
1) Ability to actually subject the system to tests
– Access to an existing System Under Test in testable state
– All the authorizations and clearances neccessary for testing
– Appropriate testing data
2) Tools
– for testing, test results management, and automation
– for identifying and/or managing risks
3) Clear criteria what is correct and incorrect (no ambiguity!)
– functional specifications
– independent correct results for cross-checking
4) QAs
– Testers with enough attention and dutifulness – or Test Automation
– Test Analytics, QA a Risk managers, Test Automators, Reviewers
5) Time
But we don‘t need...
...testing (above Testing Maturity Level 1)
– Really? Because all the users will always use the product only the way you expect, never „their own way”? Because they‘ll only ever use your data? And because outages of power or HW affecting your products never happen?
...test analysis and risk analysis
– Really? Because all the scenarios which could trigger a failure you‘ll just suck from your thumb, not
forgetting any important one? Because all assumptions of developers, conscious and sub-
conscious, were correct, and the resulting code super-robust?
...test automation
– Really? Because you‘ll never ever change once finished code? Your changes will never break
anything, hence you could do without regression tests?
Practice: how did and does look QA in R&D?
1) Debugging: „fixing until it works”
2) Waterfall: split the development to phases per skills
3) Contemporary competing methodologies
a) „V-model”: freeze of specs, test horizontally as well as vertically
b) „Agile/Iterative”: develop piece-present to customer-rework
1) is used by newbies until they burn themselves, 2) uses industry
wherever quality is required, 3) is avalanching through IT
How do Debugging, Waterfall, V-model and Agile look from the QA‘s
perspective?
R&D and QA in „testing-by-debugging”
Prehistoric era of IT development
No differentiation between developers and QA
No advanced QA methods, tools or metrics used
Cycle of writing code-execution-modifications
When the code finally works to the developer with his data/scenario, it‘s considered OK
Problem: what if anyone runs the code in a different way than dev assumed, handled and sanitized? With different data? Or an unexpected state of system happens?
R&D and QA in „Waterfall”
Developers, architects, testers, test analysts, reviewers are all distinct specializations, jobs, departments
Semi-formalized process, product moves through isolated phases of design, development and testing
Problem: all testing activities occur at the end of development cycle, when the QAs get their hands on final code and documentation
User requirements
• Business analysis
• Requirements
• Functional specifications
Software development
• SW Specifications
• SW design
• SW coding
Testing activities
• Test analysis
• Testing and validation
• Regression test suites creation
Waterfall
R&D and QA in „V-model”
QA and Dev specializati-ons are distinct professions, but collaborate
Formalized process, but
Each phase does critical review of previous && is verified by independent tests on same level
Analysis and creation of tests happens parralel with coding, from same specs, instead of at the very end
R&D in „Agile/Iterative”
„Agile” = manifesto of priorities: – people > processes
– working SW > documentation
– feedback > specifications
– responding to change > following plan
„Iterativne” = deliver spall pieces often
Developed product changes during development per customer‘s feedback
What does that mean for QA?
„Agile development”:When you‘re halfway through developing a locomotive and the customer realizes he actually wanted airplane, and you should do and test it
QA in „Agile/Iterative” – challanges & solutions
1. No time for Functional Spec in iteration/PI
– Solution: do TA instead of Specs; ambiguities are resolved with the PO = „TDD”
2. No time for Test Analysis in iteration/PI
– Solution: testing has phases 1) basic & 2) advanced; TA is done parralel with testing 1); 2) is done later when TA is finished
3. „Agility” → „no difference between QAs and Devs” = regression 40 yrs back before Waterfall; no Test Analyst or QA mgmt. role
– Solution: QAs in the team must take on all the specialist roles: Tst, TAn, Rev, Risk mgmt,TAut!
R&D example: where is CA test-maturity-wise?
Entry training „ASE“ gives employees QA skills between level 1 and 2
Advanced opt-in training „Advanced QA: Test Analyst“ gives employees QA skills between levels 3 and 4
CA offers employees ISTQB exam and certification, a world standard in Quality Assurance
Frequent mistakes
„Testing could/should ensure defect-less product”
– More possible states in the Solaris Server than molecules in universe!How do you cover that with tests? → impossible to fully test nontrivial SW
„Random testing could substitute targeted”
– won‘t cover „low probability-high impact“ defects
– won‘t test errors in chain-of-inputs, usually not even boundary values
„Testing is a low-qualification job” (anyone could test)
– Only testing on maturity level 0, not higher – and you do want higher
– Test Analyst or Test Automator require skills on-par with Developer
Questions and answers
Go on, ask...