stupid metrics - rbcs, incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfstupid...

Click here to load reader

Post on 27-Apr-2018

216 views

Category:

Documents

2 download

Embed Size (px)

TRANSCRIPT

  • Stupid Metrics TricksAnd How to Avoid Them

  • I t d tiIntroductionEver seen someone do something Ever seen someone do something stupid with metrics?Sure ya have!T d l k t t id Today, were gonna look at stupid stuff people do with metrics

    What are the most common metrics i t k ? mistakes?

    Why are they mistakes? Why do people make these mistakes? A ki h i k ? Are you making these mistakes? Why use metrics at all, when there are so many mistakes?

    And, of course, well look at ways to implement

    Stupid Metrics Tricks Page 2

    metrics that arent stupidwww.rbcs-us.com

    Copyright (c) RBCS 2016

  • Y Li ht Up M LifDo you find this slide

    You Light Up My LifeDo you find this slide easier to read when the font is lighter or darker?gOr do you find that the changing brightness of the font attracts your g yattention?You are a victim of the Hawthorne effectWhat gets measured gets doneg g

    Stupid Metrics Tricks Page 3www.rbcs-us.com

    Copyright (c) RBCS 2016

  • A d Wh t D t G t M d D t!And What Doesnt Get Measured, Doesnt!As this figure shows, As this figure shows, people can get distractedfrom important factsSome people measure their worth in followers, likes, retweets, etc.likes, retweets, etc.How many count whether they cross the street without

    i hi b ?getting hit by a car?What happens if you measure things that are less important than what you dont measure?less important than what you don t measure?

    Stupid Metrics Tricks Page 4www.rbcs-us.com

    Copyright (c) RBCS 2016

  • S ti Y P hi St iSometimes Youre Pushing on a StringWhat if you attach yrewards or punishments to a metric?Thatll cause it to move in the right direction, yes?Well, it might just , g jfrustrate the hell out of people and cause no movement at allOr, you might mistake random variation for movement

    Stupid Metrics Tricks Page 5www.rbcs-us.com

    Copyright (c) RBCS 2016

  • D i R d B d E p i tDemings Red Bead ExperimentNeed six volunteer operators, three teams of one p ,bead selector and one bead counter eachOperators are issued a paddle with nine bead gaps and a bucket with mixed beadsand a bucket with mixed beadsOperators are to separate white beads from a mix of red and white beadsOperators are to submit one paddle every 15 Operators are to submit one paddle every 15 seconds, 10 paddles totalBead counters are to note any red beads or gaps in the paddle submitted by the selectorthe paddle submitted by the selectorScore per paddle is 1 point per white bead, 0 points per empty gaps, and -2 points per red beadT ith th t i t i iTeam with the most points wins a prize

    Stupid Metrics Tricks Page 6www.rbcs-us.com

    Copyright (c) RBCS 2016

  • P t th Li i th C tPut the Lime in the CoconutQuick which shows a lower

    3030

    Quick, which shows a lower temperature?Are you sure?

    10 10

    2020

    3030Are you sure?This illustrates a common mistake made by many

    0 0

    10 10y ytesters, test managers, and other managersOne example: taking the test pyramid literally rather than metaphorically

    Stupid Metrics Tricks Page 7www.rbcs-us.com

    Copyright (c) RBCS 2016

    Blue Redthan metaphorically

  • Ri f th M hiRise of the MachinesIts not unusual for people to have a goal of 100% test automationDoes this make sense?Does this make sense?Does this mean you dont test areas that arent automatable?automatable?Does this mean you automate tests that run rarely?Is testing that consists entirely of Is testing that consists entirely of verification (with no validation) really complete?Whos wagging whom?Who s wagging whom?

    Stupid Metrics Tricks Page 8www.rbcs-us.com

    Copyright (c) RBCS 2016

  • I t bl ? I l t?Inscrutable? Irrelevant?Two questions from Two questions from assessments:

    Do you understand the test results?test results?Do the test results help you do your job?

    Common answers:No and noY b t nYes, but no

    Always be able to answer the so what questionq

    Stupid Metrics Tricks Page 9www.rbcs-us.com

    Copyright (c) RBCS 2016

  • Th W t C f R D tThe Water Cannon of Raw DataSome tools make it easy to deliver too much data yand low/no informationHow often have you seen 20 page test reports, slicing and dicing test case and bug metrics?slicing and dicing test case and bug metrics?Does that actually answer key questions like:

    Whats been tested and what hasnt?and what hasn t?What is still at risk?If we release now, what can we have confidence in and what cant we?

    These are the kinds of questions management q gactually cares about

    Stupid Metrics Trickswww.rbcs-us.com

    Copyright (c) RBCS 2016 Page 10

  • S Wh B th ith M t i ?So Why Bother with Metrics?So, with all these potential mistakes, why bother?, p , yBecause its better to make decisions based on facts, not opinions, first impressions, opinions, first impressions, and common senseHuman cognition and humanpsychology do not lend psychology do not lend themselves to good choices without the aid of factsIgnorance and bad ideasIgnorance and bad ideasthrive in low-fact settings Pick one: proof by assertion or proof by evidence?or proof by evidence?

    Stupid Metrics Tricks Page 11www.rbcs-us.com

    Copyright (c) RBCS 2016

  • Wh t N d d t G th G d M t iWhats Needed to Gather Good MetricsStart with the objectives, then derive metrics for Start with the objectives, then derive metrics for effectiveness, efficiency, and satisfactionKeep the metrics set concise, relevant, and actionableEnsure proper tool setup to gather the metricsP id t i i i th f th t lProvide proper training in the use of the toolsRemove any direct or indirect incentives or nudges to corrupt datanudges to corrupt dataTailor the metrics and their presentation to each audience

    Stupid Metrics Tricks Page 12www.rbcs-us.com

    Copyright (c) RBCS 2016

  • C l iConclusionsThe very act of measurement can changeThe very act of measurement can changebehaviorMeasurement also re-directs attentionEven well-intentioned metrics can be misused, with dire resultsB f th i t k id Be aware of the mistakes so you can avoid themBut dont reject metrics simply because its But don t reject metrics simply because it s hard to use them rightInstead, work diligently and methodically

    d i i lto put good metrics in placeStupid Metrics Tricks Page 13

    www.rbcs-us.com Copyright (c) RBCS 2016

  • T C t t RBCSFor over 20 years, RBCS has delivered consulting, training, and expert services to clients, helping them with software and hardware testing Employing the industrys most

    To Contact RBCShelping them with software and hardware testing. Employing the industry s most experienced and recognized consultants, RBCS advises its clients, trains their employees, conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development decreased tech support clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.Address: RBCS, Inc.

    31520 Beck Road31520 Beck RoadBulverde, TX 78163-3911USA

    Phone: +1 (830) 438-4830E il i f @ bE-mail: [email protected]: www.rbcs-us.comTwitter: @RBCS, @MisterSDET, @LaikaTestDog Facebook: @TestingImprovedbyRBCS

    Stupid Metrics Tricks Page 14

    LinkedIn: https://www.linkedin.com/in/rex-blackYouTube: https://www.youtube.com/user/RBCSINC

    www.rbcs-us.com Copyright (c) RBCS 2016