value based testing
DESCRIPTION
TRANSCRIPT
Your mileage may vary, void where prohibited 1
ODTUG
Value-Based Testing
Your mileage may vary, void where prohibited 2
Wikipedia.com: Quality assurance (QA) refers to the planned and systematic activities implemented in a quality system so that quality requirements for a product or service will be fulfilled. It is the systematic measurement, comparison with a standard, monitoring of processes and an associated feedback loop that confers error prevention.
ODTUG
Your mileage may vary, void where prohibited 3
What do they really do?
• Ensure proper functionality• Look for “big picture” issues• Large-scale testing (hopefully)• Identify edge cases• Advocate for user experience• Environment is volatile
ODTUG
Your mileage may vary, void where prohibited 4
What do they really do?
DEV QA OPS
Old school process:• Throw code over the
wall• And run• Its Ops problem now
Your mileage may vary, void where prohibited 5
What do they really do?
• Responsibility lines blurred• Collaboration required• Dev should be in escalation
path
ODTUG
DEV
QA
OPS
Your mileage may vary, void where prohibited 6
ODTUG
• New tools, scripts, jobs must follow release procedures
• Patches, PSUs, environment settings
• Can combine to test all at once• Monitoring must be tested
(verified) too
Ops changes also need QA
Your mileage may vary, void where prohibited 7
But how?• Make identical environments
(i.e. not a laptop VM!)• Automation ensures repeatable
results, fewer mistakes• Two-way communications
– Ops takes input from QA– QA takes input from Ops
ODTUG
Your mileage may vary, void where prohibited 8
Conducting Tests…Better• QA needs *real* info about
production• Session counts, workload
stats, response time• Apply load identical to prod
load.• When identical isn’t
possible, scale back, note risks
ODTUG
Your mileage may vary, void where prohibited 9
More Tests• Operational tests – HA
failures, backups, stats gathering
• Avg load, peak load, crash load
• Load test with missing nodes, disks, NICs
ODTUG
Your mileage may vary, void where prohibited 10
What did you do?• Record tests completed,
especially parameters• Keep performance
reports, AWR, capture ASH data, key exec plans
• Baselines = key, especially around upgrades
ODTUG
Your mileage may vary, void where prohibited 11
Typical Challenges, Issues• Spend all the money on
prod, no $$$ for QA• Our staff is so skilled, we
don’t need a “real” QA• Our schedule slipped, so
we cut QA time down to 10% of the original plan
ODTUG
Your mileage may vary, void where prohibited 12
Typical Challenges, Issues• We don’t have
requirements• Can’t you test it faster?• We don’t have a way to
simulate 500 users
ODTUG
Your mileage may vary, void where prohibited 13
Answer to Typical Challenges• Me: What is the cost of downtime?• Mgmt: We aren’t sure…• Me: Let me take production down, then we can
measure.• Mgmt: On second thought, I think I can probably
estimate it pretty closely without an outage.
ODTUG
Your mileage may vary, void where prohibited
War Stories
ODTUG
Your mileage may vary, void where prohibited 15
Q & A• Any questions or comments, please contact:– Gwen Shapira• Email: [email protected]• Twitter: @gwenshap
– Dan Norris• Email: [email protected]• Twitter: @dannorris
ODTUG
Your mileage may vary, void where prohibited