quality - search for courses | courses...quality ”the proof of the pudding is in the eating”...

QualityAnd Software Product Management

Autumn 2017 CSM14104 Software Product Management 1

Lecture Outline

• What do we mean by ”Quality”?• What are the factors that influence quality?• How is quality measured?


What is ”Quality”?


Quality”The proof of the pudding is in the eating”


"Christmas pudding" by Musical Linguist at the English language Wikipedia. Licensed under CC BY-SA 3.0 via Wikimedia Commons– http://commons.wikimedia.org/wiki/File:Christmas_pudding.JPG#mediaviewer/File:Christmas_pudding.JPG

Definitions

• Definitions from two pioneers of industrial Quality Control andManagement

• P. Crosby[1]: ”Quality means conformance to requirements”• J. Juran[2]:

1) ”Quality consists of those product features which meet the needs of customers andthereby provide product satisfaction

2) Quality consists of freedom from deficiencies”


[1] Crosby, P. B.: Quality is Free. McGraw-Hill, New York, 1979.[2] Juran, J. M.: Juran’s Quality Control Handbook, 4th ed. McGraw-Hill, New York, 1988.

Definitions

• Crosby’s definition expects that a product has a defined set ofrequirements that is unambiguous, extensive, objective and thatcovers the needs of all stakeholders

• The quality of a product can be determined by comparing its true (measured)attributes to what is the expected level stated in the requirements

• But it does not take into account the possibility of errors in the requirements• A missing or incomplete feature (from user’s perspective) does not lower the quality of

the prodcut, if it still conforms to the requirements - but the user is unhappy!

• A software product almost never has a complete and unambiguousdefinition of requirements


Definitions

• Juran’s definition places the needs and satisfaction of the customerand the user in the center, as in ”Fitness for use”

• The real needs vs. (documented) requirements• The features and characteristics of the product that respond to those needs• The second important aspect is freedom from deficiencies• 1 +2 = the product has the right features and they work as the customer expects• Puts the emphasis on the customers and the users instead of defined

requirements• Requires some other way to find out the true needs than a binding set of

requirements defined at the beginning of a project• Changes are to be expected and must be prepared to respond to them along the way



Stakeholders and quality

• A product or service has many stakeholders to whom quality meansdifferent things

• The end-user (consumer, employer)• Operator, in charge of deploying to production and running it• Developer, maintainer, contractor• Tech support• Customer service• Sponsor, financer, investor• Financer of maintenance and further development• Public authorities, regulator• … and so on!!

Is quality subjective?

• Different stakeholder groups have different needs and expectation but there canalso be variation within a group

• Professional vs. casual users• Enthusiasts and early adopters vs. conservatives, etc.

• Weinberg, G.[3]: ”Quality is value to some person”• Value = what someone is ready to pay to get her need fulfilled• A key question is whose needs are taken into account - and whose not

• A software product manager needs to• Understand what quality means to the different stakeholders• Balance conflicting views and make trade-offs

Autumn 2017 CSM14104 Software Product Management 9[3] Weinberg, G. M.: How Software is built. Leanpub, 2014.

What’s a ”bug”? (a.k.a defect, problem)

• [Good quality means] “The absence of defects that would makesoftware either stop completely or produce unacceptable results.Defects can be traced to requirements, to design, to code, todocumentation, or to bad fixes of previous defects.”[5]

• ”A Bug is an attribute of a software product• That decreases its value to a favored stakeholder• Or increases its value to a disfavored stakeholder• Without a sufficiently large counterveiling benefit.”[6]

[5] Capers Jones, Applied Software Measurement. McGraw-Hill, 2008.[6] C. Kaner & J. Bach, Black Box Software Testing Foundations, www.testingeducation.org/BBST, 2010.Autumn 2017 CSM14104 Software Product Management 10

http://www.testingeducation.org/BBST

A quality modelStandards’ view



ISO/IEC:n definition of Software Quality

• ISO/IEC* definition• Degree to which a software product satisfies stated and implied needs when

used under specified conditions• Thus providing value to its stakeholders

• A very general definition but one that emphasis the needs of thestakeholders

• “Individual or organization having a right, share, claim or interest in a systemor in its possession of characteristics that meet their needs and expectations”

*) ISO/IEC 25010:2011 Systems and software engineering--Systems and software Quality Requirements and Evaluation (SQuaRE)--System and software quality models, 4.3.13.

A copy available form course materials via a link to cs intranet (requires university user id and password)

ISO/IEC 25000 series of standards, ”SQuaRE”

• Software Quality Requirements and Evaluation• Development of Standards and Technical Reports for Software Product and

System Quality Requirements, Measurement and Evaluation

• Includes a quality model, defines how quality is measured in principleand provides some examples of possible measures

• Available as paper copies from the Campus Library• Some draft versions available electronically from course web page (only to be

used for studying purposes)


ISO/IEC 25000 series of standards, ”SQuaRE”

• Key standards• Quality model division

• ISO/IEC 25010: System and software quality models• (ISO/IEC 25011 quality model for IT-services is under development)• ISO/IEC 25012: Data quality model

• Quality measurement division• ISO/IEC 25020: Measurement reference model and guide• ISO/IEC 25021: Quality measure elements• ISO/IEC 25022: Quality in use measures• ISO/IEC 25023: Measurement of system and software product quality• ISO/IEC 25024: Measurement of data quality


ISO/IEC 25010 – Quality Model

• “Defined set of characteristics, and of relationships between them,which provides a framework for specifying quality requirements andevaluating quality”

• Common vocabulary and a frame of reference for thinking about softwaresystem quality

• Gives a superset of quality characteristics that can be important for softwaresystems and products

• Covers a wide range of systems – not all characteristics are relevant to allsystems

• Every product/system does not have to invent their own model but adapt thegeneric model


ISO/IEC 25010 – Quality Model

• Is divided into two sub-models1) Quality in use2) Software product quality

• ISO/IEC 25012 Data Quality Model is a closely related but a separatestandard


ISO/IEC 25010 – Terminology

• software quality characteristic = category of software quality attributes that bearson software quality

• may be refined into multiple levels of subcharacteristics and finally into software quality attributes.

• quality attribute1 = measurable component of quality• stakeholder = individual or organisation having a right, share, claim or interest in

a system or in its possession of characteristics that meet their needs andexpectations (e.g. software developers, system integrators, acquirers, owners,maintainers, contractors and end users).

• user = individual or group that benefits from a system during its utilization.• direct user = person who interacts with the product


1attribute: inherent property or characteristic of an entity that can be distinguishedquantitatively or qualitatively by human or automated means [ISO/IEC 15939:2002]

Quality in use

• Is composed of five characteristics (some of which are furthersubdivided into subcharacteristics) that relate to the outcome ofinteraction when a product is used in a particular context of use

• Applicable to the complete human-computer system, including bothcomputer systems in use and software products in use

• The subcharacteristics are representative of typical concerns for each maincharacteristic – not necessarily an exhaustive list


Quality in use – characteristicsEffectiveness

Accuracy and completeness with which users achieve specified goals

Efficiency

Resources expended in relation to the accuracy and completeness withwhich users achieve goals

Satisfaction

Degree to which user needs are satisfied when a product or system isused in a specified context of use

Freedom from risk

Degree to which a product or system mitigates the potential risk toeconomic status, human life, health, or the environment

Context coverage

Degree to which a product or system can be used with effectiveness,efficiency, freedom from risk and satisfaction in both specified contexts ofuse and in contexts beyond those initially explicitly identified


EffectivenessEfficiencySatisfaction

UsefulnessTrustPleasureComfort

Freedom from riskEconomic risk mitigationHealth and safety risk mitigationEnvironmental risk mitigation

Context coverageContext completenessFlexibility

Software product quality

• Is composed of eight characteristics (which are further subdividedinto subcharacteristics) that relate to static properties of software anddynamic properties of the computer system

• Applicable to both computer systems and software products


Software product quality – characteristics


Functional suitabilityFunctional completenessFunctional correctnessFunctional appropriateness

Performance efficiencyTime behaviourResource utilizationCapacity

CompatibilityCo-existenceInteroperability

UsabilityAppropriateness recognizabilityLearnabilityOperabilityUser error protectionUser interface aestheticsAccessibility

ReliabilityMaturityAvailabilityFault toleranceRecoverability

SecurityConfidentialityIntegrityNon-repudiationAccountabilityAuthenticity

MaintainabilityModularityReusabilityAnalysabilityModifiabilityTestability

PortabilityAdaptabilityInstallabilityReplaceability

ISO/IEC 25012 Data Quality Model

• Data quality is a key component of the quality and usefulness ofinformation derived from that data, and most business processesdepend on the quality of data

• A common prerequisite to all information technology projects is thequality of the data which are exchanged, processed and usedbetween the computer systems and users and among computersystems themselves

• The data quality model categorizes quality attributes into fifteencharacteristics considered by two points of view: inherent and systemdependent


Inherent data quality

• From the inherent point of view, data quality refers to data itself, inparticular to:

• data domain values and possible restrictions (e.g. business rules governingthe quality required for the characteristic in a given application);

• relationships of data values (e.g. consistency);• metadata


System dependent data quality

• System dependent data quality refers to the degree to which dataquality is reached and preserved within a computer system whendata is used under specified conditions.

• From this point of view data quality depends on the technologicaldomain in which data are used; it is achieved by the capabilities ofcomputer systems' components such as:

• hardware devices (e.g. to make data available or to obtain the requiredprecision),

• computer system software (e.g. backup software to achieve recoverability),and

• other software (e.g. migration tools to achieve portability).



Characteristic Inherent System dependent

Accuracy X

Completeness X

Consistency X

Credibility X

Currentness X

Accessibility X X

Compliance X X

Confidentiality X X

Efficiency X X

Precision X X

Traceability X X

Understandability X X

Availability X

Portability X

Recoverability X

Using quality models

• The product quality and quality in use models are useful for specifyingrequirements, establishing measures, and performing qualityevaluations

• The defined quality characteristics can be used as a checklist forensuring a comprehensive treatment of quality requirements, thusproviding a basis for estimating the consequent effort and activitiesthat will be needed during systems development

• The characteristics in the quality in use model and product qualitymodel are intended to be used as a set when specifying or evaluatingcomputer system or software product quality


Using quality models

• It is not practically possible to specify or measure all subcharacteristics for allparts of a large computer system or software product

• Similarly it is not usually practical to specify or measure quality in use for allpossible user-task scenarios

• The relative importance of quality characteristics will depend on the high-levelgoals and objectives for the project

• Therefore the model should be tailored before use as part of the decompositionof requirements to identify those characteristics and subcharacteristics that aremost important, and resources allocated between the different types of measuredepending on the stakeholder goals and objectives for the product


Influencing factors


Four views to quality

29

"Gran Canyon USA" by Ot Pi - Own work.Licensed under CC BY-SA 3.0 via Wikimedia

Userexperience

"Blaues Fahrrad mit Achter" by 4028mdk09 - Own work.Licensed under CC BY-SA 3.0 via Wikimedia Commons

Externallyvisible andmeasuableattributes ofthe product

"Disassembled Campagnolo Centaur cassette - side view" by Ximeg - Own work.Licensed under CC BY-SA 3.0 via Wikimedia Commons

Internalattributes

"Bike mechanic at a local bike shop" by Andrew Dressel - Taken byAndrew Dressel at en.wikipedia.Licensed under CC BY-SA 3.0 via Wikimedia Commons -

Developersand manufacturers

Autumn 2017 CSM14104 Software Product Management

Quality influences

• The needs of users and the context of use are the starting point• Example: getting timetables for public transport with a mobile device• User’s needs and expectations (quality in use):

• Effectiveness, Satisfaction – The information delivered must be accurate andup to date

• Efficiency, Satisfaction – It must be intuitive and fast to find the timetable fora certain line and a certain stop

• User’s needs and expectations (data quality):• Accuracy, Currentness, Availability, Efficiency


Quality influences

• The external product quality characteristics that influence the userecxperience (looking at the system as a black-box)

• Usability – for example• How many interaction steps does it take to get to the information (taps, selections, button

presses)?• How quick and clear is the feedback from the UI to user’s actions?

• Performance efficiency (tehokkuus) – for example• How long does the whole use case take?• How long does it take for a new view to render?• How fast does the application start?• How fast data connection does the application need?


Quality influences

• Internal product quality enables (or inhibits!) achieving the desired externalproduct quality (looking at the system as a glass-box)

• Performance efficiency (response time, through put) is largely determined by the design andarchitecture of the application as well as from the technologies used and the correctness oftheir use

• Usability is influenced directly by the UI design and indirectly by the ablity to monitor theusage of various features for logging and statistical inference


Quality influences


Measuring quality


”Quality is free”

• P. Crosby• “It is always cheaper to do the job right the first time”• “Quality is free, but no one is ever going to know it if there isn't some sort of

agreed-on system of measurement”• http://www.philipcrosby.com/25years/read.html• http://en.wikipedia.org/wiki/Philip_B._Crosby

• ”You can’t control what you can’t measure” – Tom DeMarco, 1982.


http://www.philipcrosby.com/25years/read.html

http://en.wikipedia.org/wiki/Philip_B._Crosby

Measurements

• The quality goals of an organization can be defined by setting target levels forquality indicators

• A measurement program is set up to define the indicators and needed measurements

• E.g. measuring customer satisfaction and loyality• Opinion polls/surveys• ”Net Promoter Score”• Number of failure reports/complaints and support requests as a function of time and relative

to the number of (active) users

• We want to assess, estimate and collect statistics about the products, processesand resources for management and development purposes


Measurements

• Establishing a measurement program can be used for communicating the qualitytargets and engaging the organization even if the program is not applied in full oronly for a limited

• It is more important to make people aware of the goals and get their commitment rather than to implement theprogram in full scale

• N.B.! – Using measurements as the basis of promotions/bonuses or in assessingpersonal performance can have unwanted, counterproductive side-effects

• There can be many ways to make the metrics ”look right” when in fact the situation is quitedifferent (gaming the metrics)

• Developers do not like personal measurements and vote with their feet• You get what you measure – are you sure that really is what you want?



Direct, indirect and surrogate measures• Measures can be direct or indirect

• Direct measures indicate immediately some aspect of the state of theobject of measurement (valid by nature). E.g. the temperature of aphysical object. f(a ribute) → value

• Indirect measures are obtained as a function of direct measures. E.g. theaverage monthly temperature of some place. h( f(o1),g(o2),… ) → value

• Care must be exercised with software related metrics to declare somemetric as ”Direct”!

• A Surrogate metric is used when we do not know how to directlymeasure an attribute that we are interested in

• Using the size of a software module as an indication of how error prone itis

• Using the number of open bug reports to gauge the product’s readinessfor release (can be very questionable!)

”Onks kymppi paljon vai vähän?”



Choosing measures

• We can’t measure everything, and we do not even know how to measure manyinteresting things

• How do we then choose what to measure?• ISO 25000 series gives some ideas…• The Goal - Question - Metric –method is one of the best known and widely used

approaches (GQM, Basili & Rombach, 1988).


Goal - Question - Metric

• According to GQM[7], in order to develop a sensible and effective measurement program, theorganization needs to

1. Define the company- and project-level targets or goals2. Determine the product and process level indicators/information that bear on those goals3. Plan a framework for collecting and interpreting the data in practice

• In other words:1. What do we want to achieve? What are the business and organization level goals that we want to

follow?2. What information do we need about the products and processes to see whether we are reaching

the goals or?3. How do we collect the data in practice and how do we interpret the data?

• GQM defines a process how to do this

[7] http://www.cs.umd.edu/~mvz/handouts/gqm.pdf

Example (from [7], not really about quality!)


quality - search for courses | courses...quality ”the proof of the pudding is in the eating”...

Documents