iwsm2014 exponentially weighted moving average prediction in the software development process...

October 6–8, 2014

MotivationWarm Up: Shewhart Control Charts

Short-Run Control ChartsEWMA Q Control Charts

Appendix

Exponentially Weighted Moving Average(EWMA) Prediction in the Software

Development Process

Thomas M. Fehlmann1 Eberhard Kranich2

1Euro Project Office AGZurich, Switzerland

[email protected]

2Euro Project OfficeDuisburg, Germany

[email protected]

T. Fehlmann, E. Kranich (Euro Project Office) EWMA Prediction in the Software Development Process



Appendix

Project Managment Task

Task

Design and implement a tool for monitoring and controlling thesoftware development process, especially its test phase.

Key Requirements

1 Visualization of concurrently running projects in one graphic.

2 Enable a graphical comparison of detected defect types.

3 Monitoring/Controlling asap after process/test phase start-up.

4 Implement a forecast functionality for controlling the process.




Appendix

Historical RemarksA Typical Control ChartSubgroup Control Chart

Historical Remarks




Appendix


X – Control Chart

UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL

CLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCL

LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL

observations = 21mean = 22.43std.dev. = 6.5

UCL = 41.93CL = 22.43LCL = 2.93

beyond limits = 0violating runs = 0

10

20

30

40

0 5 10 15 20day

# de

fect

s

X − Chart for Category: defects




Appendix


Category Control Chart

UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL UCL

CLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCLCL

LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL LCL

observations = 21mean = 2.24std.dev. = 3.48

UCL = 12.67CL = 2.24LCL = −8.19

beyond limits = 1violating runs = 0

−5

0

5

10

15

0 5 10 15 20day

# al

gorit

hm

X − Chart for Category: algorithm




Appendix

IntroductionQ-Control ChartsMerging Q Control ChartsHandling Outliers

Why Short-Run Control Charts ?

1 A sufficiently large data set is not available to construct aShewhart control chart.

2 A process has to be monitored and controlled within a shorttime after its start-up.

3 To stabilize an individual unit producing process as soon aspossible, the process must be analyzed automatically inreal-time.




Appendix


References

Monitoring the software development process usinga short-run control chart

Chih-Wei Chang • Lee-Ing Tong

Published online: 22 July 2012� Springer Science+Business Media, LLC 2012

Abstract Techniques for statistical process control (SPC), such as using a control chart,

have recently garnered considerable attention in the software industry. These techniques

are applied to manage a project quantitatively and meet established quality and process-

performance objectives. Although many studies have demonstrated the benefits of using a

control chart to monitor software development processes (SDPs), some controversy exists

regarding the suitability of employing conventional control charts to monitor SDPs. One

major problem is that conventional control charts require a large amount of data from a

homogeneous source of variation when constructing valid control limits. However, a large

dataset is typically unavailable for SDPs. Aggregating data from projects with similar

attributes to acquire the required number of observations may lead to wide control limits

due to mixed multiple common causes when applying a conventional control chart. To

overcome these problems, this study utilizes a Q chart for short-run manufacturing pro-

cesses as an alternative technique for monitoring SDPs. The Q chart, which has early

detection capability, real-time charting, and fixed control limits, allows software practi-

tioners to monitor process performance using a small amount of data in early SDP stages.

To assess the performance of the Q chart for monitoring SDPs, three examples are utilized

to demonstrate Q chart effectiveness. Some recommendations for practical use of Q charts

for SDPs are provided.

Keywords Software development process � Statistical process control � Control chart �Short production run � Q chart

C.-W. Chang (&) � L.-I. TongDepartment of Industrial Engineering and Management, National Chiao Tung University,1001, Daxue Rd., Hsinchu City 300, Taiwan, ROCe-mail: [email protected]; [email protected]

L.-I. Tonge-mail: [email protected]

123

Software Qual J (2013) 21:479–499DOI 10.1007/s11219-012-9182-y Short-Run Control Charts for SDP

Introducing Short-Run Control Charts forMonitoring the Software Development Process

Thomas M. Fehlmann1), Eberhard Kranich2)

1)Euro Project Office, Zurich, Switzerland2)T-Systems International GmbH, Telekom IT (PQIT), Bonn, Germany

1)[email protected], 2)[email protected]

Abstract

It is common practice in manufacturing industries that Statistical Process Control(SPC) methods are applied for monitoring, controlling and improving processesover time. One of the prominent SPC tools is the classical Shewhart control chartwhich gives valuable insight into the sources of variation of a process, wherebya variation is caused either by in-process interactions such as a man-machine in-teraction, or by out-of-process events that deteriorate the process and thus makethe process unstable. Shewhart control charts are well suited for long run pro-cesses so that actual in-process parameters or actual control limits can be es-tablished on the basis of historical data gathered during a large number of theprocess runs. Short run processes lack a sufficiently large set of historical data, sothat Shewhart control charts cannot be constructed. But short-run control charts,also termed self-starting control charts, enable the monitoring and controlling ofa process within a short time after its start-up, when a large amount of historicaldata are not at hand, and update the in-process parameters with each new processrun. Hence self-starting control charts such as Tukey’s control charts and Quesen-berry’s Q-charts are highly appropriate for controlling and monitoring a softwaredevelopment process and its various phases, for instance, when the software testprocess is monitored by the faults-slip-through measurement process.

Keywords

Software Development Process (SDP), Statistical Process Control (SPC), short-run control chart, self-starting control chart, Tukey’s control chart, Q-chart, faults-slip-through measurement process

1 Introduction

The Six Sigma methodology with its DMAIC phase model is an accepted ap-proach to improve an existing process systematically. In contrast, if an existingprocess is to be redesigned completely or a new process is to design, the Designfor Six Sigma (DFSS) methodology with its mostly applied DMADV phase

MetriKon 2013




Appendix


Specific References

SPC Q Charts for Start-Up Processes and Short or Long Runs

CHARLES P. QUESENBERRY North Carolina State University, Raleigh, North Carolina 27695-8203

Classical control charts are designed for processes where data to estimate the process parameters

and compute the control limits are available before a production run. For many processes, especially

in a job-shop setting, production runs are not necessarily long and charting techniques are required

that do not depend upon knowing the process parameters in advance of the run. It is desirable to

begin charting at or very near the beginning of the run in these cases. We present here the needed

formulas so that charts for both the process mean and variance can be maintained from the start of

production, whether or not prior information for estimating the parameters is available. These Q charts

are all plotted in a standardized normal scale, and therefore permit the plotting of different statistics

on the same chart. This will sometimes permit savings in the chart management program.

Introduction

THERE has recently been considerable interest in II using SPC charting techniques in the job-shop environment. Classical SPC charting methods such as X, R, and S charts assume high volume manufacturing processes where at least 25 or 30 calibration samples of size 4 or 5 each can be gathered to estimate the process parameters before on-line charting actually begins. However, the job-shop environment involves low-volume production and there is often a paucity of relevant data available for estimating the process parameters and establishing control limits prior to a production run. In many applications there is no really reliable data available for this purpose. Another important practical problem in many job shops is that there are so many different types of measurements (i.e. part numbers) that a multitude of charts is required. Thus standardized charts that permit different statistics to be plotted on the same chart can be used to simplify the chart management problem.

Specifically, we consider the following model setting. Let

(1)

represent measurements that may be from a sequence of consecutively produced parts. If the values Xr are

Dr. Quesenberry is a Professor of Statistics. He is a Senior

Member of ASQC.

Vol. 23, No. 3, July 1991 21 3

stochastically independent with the same distribution, then this common distribution is called the process distribution and its mean Il and variance 0-2 are called the process mean and process variance. It should be kept in mind that when the measurements are on consecutively produced parts that the independence assumption may be invalid due to autocorrelation, and that the techniques given here are appropriate only for independent observations. Methods of checking for autocorrelation given in books such as Box and Jenkins (1976) and Fuller (1976) can be used to detect autocorrelation. The test of Durbin and Watson (1950, 1951, 1971) can be used to make a test for autocorrelation. Marr and Quesenberry (1989) recently gave a test for autocorrelation that can also be used in this context and is particularly convenient for some types of data encountered in applications in SPC. Also see Montgomery and Mastrangelo (1991) in this issue.

Most presently available SPC charting methods such as Shew hart charts, CUSUM charts, or geometric moving average charts assume that data to estimate the process parameters are available before the run of parts giving the data in (1) is made. In effect, these charting techniques assume that the values of the process mean Il and process variance 0-2 are known when the run of parts for (1) is begun. However, in many cases the process mean and variance cannot be known before the production run is begun, because they change from run to run. This makes it difficult to construct valid charts using presently available

Journal of Quality Technology




Appendix


A Q Control Chart

UCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCL

LCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCL

−2

0

2

0 5 10 15 20day

Q−

Sta

tistic

Defect Type

defects

Q Control Chart




Appendix


Merging Q Control Charts



−2

0

2

0 5 10 15 20day

Q−

Sta

tistic

Defect Type

algorithm

Q Control Chart

UCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCL

LCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCL

−2

0

2

4

0 5 10 15 20day

Q−

Sta

tistic Defect Type

functions

algorithm

Q Control Chart

UCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCL

LCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCL−2.5

0.0

2.5

5.0

0 5 10 15 20day

Q−

Sta

tistic

Defect Type

functions

interface

algorithm

Q Control Chart

UCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCLUCL

LCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCLLCL−2.5

0.0

2.5

5.0

0 5 10 15 20day

Q−

Sta

tistic

Defect Type

defects

functions

interface

algorithm

Q Control Chart




Appendix


Handling Outliers



−2

0

2

0 5 10 15 20day

Q−

Sta

tistic

Defect Type

algorithm

Q Control Chart



−2

0

2

0 5 10 15 20day

Q−

Sta

tistic

Defect Type

algorithm

Q Control Chart




Appendix

An EWMA Q Control ChartFast Initial Response (FIR)Forecasting

An EWMA Q Control Chart

−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

Defect Type

defects

EWMA Q Control Chart: λ = 0.25, ρ = 2.998




Appendix


Fast Initial Response (FIR)

−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

(fir

)

Defect Type

defects





Appendix


EWMA vs. FIR EWMA

−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

Defect Type

defects


−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

(fir

)

Defect Type

defects





Appendix


EWMA vs. Modified FIR EWMA

−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

Defect Type

defects


−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

(m

fir)

Defect Type

defects





Appendix


FIR EWMA vs. Modified FIR EWMA

−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

(fir

)

Defect Type

defects


−2

−1

0

1

2

0 5 10 15 20day

EW

MA

Q−

Sta

tistic

(m

fir)

Defect Type

defects





Appendix


Q-Statistic Forecasting

One-Step Forecast Procedure

1 Utilize the R package forecast.

2 Apply the function ses to the actual Q-Statistics Qk(xk),

3 in order to obtain a forecast of Q-Statistic Qk+1(xk+1).

4 Check summary of ses for 80% and 90% prediction intervals.

5 If the forecast is not located in the intervals, then REACT,

6 else calculate the forecast xk+1 by the inverse Q-Statistic.




Appendix


A Brief Example

Predicting Q18(x18)

fnct. forecast Q18(x18) 80% CI 95% CIses 0.348 2.078 [−0.64, 1.33] [−1.16, 1.85]holt 0.182 2.078 [−0.86, 1.22] [−1.41, 1.77]

Predicting x18

fnct. forecast x18 x18 80% CI 95% CIses 24 36 [17, 31] [14, 35]holt 23 36 [16, 30] [12, 35]




Appendix

Standard Normal vs. Student’s t DistributionThe Q-StatisticThe Q-Statistic: Updating FormulasFIR/MFIR EWMA Calculations

Appendix




Appendix


N (0, 1) vs. t-Distribution

0.0

0.1

0.2

0.3

0.4

−4 −2 0 2 4

density N(0, 1) t(df= 1) t(df= 6) t(df=10) t(df=20)

N(0, 1) vs. Student's t−distribution (df=1, 6, 10, 20)




Appendix


The Q-Statistic

Case UU: µ and σ2 unknown

Qk(xk) = Φ−1

{Gk−2

[√k − 1

k

(xk − xk−1

sk−1

)]}, k ≥ 3.

Properties

1 Each statistic Qk(xk) produces a sequence of independent,N (0, 1) distributed variables.

2 Consequently: UCL = +3, CL = 0, LCL = −3.




Appendix


Sequential Updating Formulas

1 Mean:

xk =1

k

k∑j=1

xj = xk−1 +1

k(xk − xk−1)

with k ≥ 2 and x1 = x1.

2 Variance:

s2k =1

k − 1

k∑j=1

(xj − xk)2 =

(k − 2

k − 1

)s2k−1 +

1

k(xk − xk−1)

with k ≥ 3 and s22 = 12 (x2 − x1)2.




Appendix


Calculating the Parameter a

1 FIR definition:

FIRadj =(

1− (1− f)1+a(k−1))b, f ∈ (0, 1]

2 Calculation of a (b = 1):

a = − 1

k0 − 1

(1 +

log(1− FIRadj)

log(1− f)

)with k0 is user defined so that the impact of FIRadj on theEWMA control limits vanishes for all iterations k > k0.

3 Example: FIRadj ≈ 0.99, k0 = 20, f = 0.5, b = 1 ⇒ a = 0.3.


iwsm2014 exponentially weighted moving average prediction in the software development process...

Software

shewhart control chartsshort

ashewhart control chart

conventional control

fixed control limits

valid control limits

wide control limitsdue

q chart effectiveness

software industry