Software Quality Journal 5, (1996) 235-242

Software quality programmes: a snapshot of theory versus reality T R A C Y H A L L l a n d N O R M A N E. F E N T O N 2

1School of Computer Science, University of Westminster, 115 New Cavendish Street, London W1M 8JS UK. 2Centre for Software Reliability, City University, Northampton Square, London EC10HB, UK

Received February 1996

We analyse the software quality programmes of two major UK companies whose software development departments, we feel, are typical of many hundreds in the UK. The analysis is based on an almost exhaustive survey of relevant practitioners at these companies (123 out of a possible 149 individuals) using questionnaires and interviews. We discovered significant differences between senior managers' perceptions and the reality of what quality practices took place. In particular, use of reviews and inspections, which are widely believed to be an extremely cost-effective QA procedure, was very limited even when safety critical software was being dev- eloped. On the other hand there was widespread use of development and coding standards whose value is questionable. We also discovered significant differences between practitioners' and managers' views of what the company quality goals really were. While no generalizable conclusions can be drawn from the results of just two companies, the observations do confirm the informal results arising from the more extensive nationwide survey of which this particular study forms a part.

Keywortls: quality, software, project management, inspections

1. Introduction

As part of an ongoing and extensive investigation into software quality programmes throughout the UK IT industry we have been building up at least an informal picture of the real state-of-the- art. We have been struck by the gap between the supposed best practice (as recommended and discussed in books and journals such as this) and the actual practice of even the country's most prestigious companies. Although the nature of our study is such that no scientifically 'valid' con- clusions can be drawn, we have previously published a number of results which at least indicate the real problems that exist [1]. In this paper we focus on a detailed snapshot from two major com- panies to emphasize the nature of the problem. Their software development departments are, we feel, typical of many in the UK. Although both companies are large neither have quality pro- grammes that are fully matured, and each is probably typical of the programmes within many ordinary companies. As such their experiences are likely to be more useful to a greater number of ordinary software practitioners. What makes this study particularly interesting is that it is based on a uniquely comprehensive survey. At both companies we were able to get the input of almost all of the staff directly involved in, or affected by, the quality assurance programme (123 out of a possible 149 individuals). The breakdown is shown in Table 1 (for reasons of commercial con- fidentiality, throughout this paper we refer to the two case study companies as 'Embedded Systems' (ES) and 'Information Systems' (IS)). This very high response rate was due primarily to a desire

0963-9314 © 1996 Chapman & Hall

236

Table 1. Response rate.

Hall and Fenton

Staff Number responding to questionnaire Response

Organization targeted Total M a n a g e r Non-manager rate %

Embedded systems 24 20 10 10 83 Information systems 125 103 48 55 82

(from both companies) for independent evaluation and the opportunity for improvement of their quality programmes. Quality was a hot issue and practitioners had a lot they wanted to say about it, while managers were keen to encourage their contribution.

While we stress that no generalizable conclusions can be drawn from our observations [2], what we can say with confidence is that the results present an extremely accurate picture of the state-of- practice at the two companies. We believe the results presented here give many interesting insights that will be relevant and applicable to other organizations.

We analysed the two case study quality programmes in terms of the penetration of best practice. We also investigated the organizational infrastructures (which must be sound for a company to have an optimized quality programme). Since companies are under significant commercial pressure to be seen to have implemented a well-constructed quality system, we were interested in whether the quality claims of the companies stand scrutiny. Thus we aimed to answer the following ques- tions for each of the companies:

1. Are they doing what they should be in terms of quality? 2. Are they doing what they say they are doing in terms of quality?

The paper is organized as follows: Section 2 examines the two case study quality programmes. This section concentrates particularly on the quality infrastructure; the penetration of quality controls; goals for quality. Section 3 analyses the organizational infrastructure within which our two case study quality programmes are set. Section 4 provides the main conclusions of the study.

2. The quality programmes

The infrastructure of both quality programmes seemed adequate (although neither was ISO certi- fied). Both organizations said that they had:

lifecycle coverage of standards, reviews and procedures tool support for software development and project management configuration management a measurement programme a quality manual a quality manager

We focus here on the penetration of standards, reviews and procedures (see [3] for a discussion of the other quality mechanisms, most notably the measurement programmes). We also consider the issue of quality goals.

2.1 The penetration of quality controls

Despite the dearth of conclusive empirical evidence [4], it is widely believed that basic quality control mechanisms have an important influence on product quality improvement. The most

Software quality programmes." theory versus reality 237

Work that is never reviewed or inspected

: ;!iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii i :::::iiii iii;iii:ii : : : 2j

Work that is frequently reviewed or inspected

Fig. 1. Inspection and review penetration.

convincing evidence relates to reviews and inspections; systematic use of these significantly improves the quality of the final software product [5]. The evidence in support of the use of standards is less clear-cut [6].

We examined the quality control penetration levels at both case study organizations. We found that actual use of quality control procedures was generally poor, and was lower in practice than the organizations claimed. Even the procedures known to have significant cost benefit had low pene- tration.

For example, Fig. 1 shows the penetration of reviews and inspections at three phases of the lifecycle (we produced analogous results, not shown here, for use of standards). Practitioners were asked how much of their work was subjected to review or inspection. The 'Activity' bar of Fig. 1 shows the involvement of practitioners at each phase in the lifecycle and the 'Review' bar shows the penetration of reviews at those lifecycle phases. The difference between the two bars illustrates the review 'shortfall'.

We discovered the following about quality control penetration at the two organizations:

1. Quality control shortfalls. A significant amount of work at every lifecycle phase is not inspected or reviewed. Furthermore, the review shortfall gets wider further up the lifecycle, so that more work is actually reviewed at specification than coding.

2. Ineffective deployment of quality controls. Despite empirical evidence demonstrating that reviews and inspections are a highly cost-effective quality assurance mechanism neither

238 Hall and Fenton

.

organization used them widely. On the other hand standards were widely used in both organizations throughout the lifecycle. For example, 86% of the practitioners at ES who spent most of their time coding, said that they 'frequently' used coding standards; 70% in IS said the same. Furthermore 75% of ES practitioners whose main task was testing, 'fre- quently' used test standards; 50% in IS said the same. The basis of either organization's preference for standards is not clear. We suspect that organizations consider standards to be easier and cheaper to implement, even though this is not borne out by the empirical evidence. Deviation from the 'official' position. We found a significant shortfall between the official and the actual use of quality controls. Managers at both organizations said that standards were always used at all stages of the lifecycle. Managers at IS said that reviews were always per- formed on all lifecycle deliverables. Managers at ES said that reviews were almost always performed on all lifecycle deliverables. Figure 1 shows that when we asked the practitioners themselves, the actual penetration of quality controls was far poorer than the official position led us to believe. This deviation from 'the official' quality system is probably common; in the SMARTIE project [6] we discovered objective evidence of this. The underlying reasons for this deviation are, however, unclear.

2.2 Quality goals

It is widely believed that in order for an organization to have an effective quality improvement initiative it is important that they set clearly defined goals [7]. Practitioners must at least be aware of their organization's quality goals and at best be committed to those goals.

We looked at how our two organizations were performing in terms of quality goals. We analysed practitioners' own software quality goals and compared them to what practitioners thought the goals of their organization were. Practitioners were asked to rank five quality goals (see Table 2) according to what they thought their organization's quality goals were and according to their own personal goals (1 = high priority; 5 = low priority); a mean score for each goal was then calculated. As we now report, there was a disturbing level of confusion and variance about goals in both organizations.

Organizational goals at ES. Table 2 shows there was very little agreement among practitioners about organizational goals, although there is more agreement among managers than non-man- agers. This confusion about the organization's goals is not surprising given that 60% of practi- tioners in this organization said organizational goals were not made clear to them (Fig. 3).

Table 2 also shows that, overall, practitioners in ES think that low costs are the company's

Table 2. Embedded Systems: goals - ranking and mean score

Goal

Organizational goals Practitioners' goals

Overall Manager Non-manager Overa l l Manager Non-manager

Rank Mean Rank Mean Rank Mean Rank Mean Rank Mean Rank Mean

Low costs 1 Conformance 2 Speed 3 Reliability 4 User satisfaction 5

2.6 1 2.2 2 2.8 5 4.2 5 4.4 4 3.9 2.9 2 2.6 4 3.2 2 2.3 2 1.9 3 2.6 3.0 3 3.0 3 3.0 4 4.2 4 4.1 5 4.3 3.1 4+ 3.6 1 2.6 1 2.0 1 1.8 1+ 2.1 3.5 4+ 3.6 5 3.4 3 2.6 3 2.9 1+ 2.1

Software quality programmes: theory versus reality

Table 3. Information Systems: goals - ranking and mean score

239

Goal

Organizational goals Practitioners' goals

Overall M a n a g e r Non-manager O v e r a l l Manager Non-manager

Rank Mean Rank Mean Rank Mean Rank Mean Rank Mean Rank Mean

User satisfaction 1 2.6 1 2.6 1 2.6 1 1.8 1 1.8 1 1.8 Speed 2 2.9 2 2.9 2 2.8 4 3.9 4 4.0 4 3.8 Reliability 3 3.0 3 3.0 3 3.0 2 2.1 2 2.0 2 2.1 Conformance 4 3.1 4 3.2 4 3.1 3 2.6 3 2.7 3 2.6 Low costs 5 3.4 5 3.3 5 3.5 5 4.5 5 4.4 5 4.6

highest priority. This is a disturbing result from a high profile, safety critical equipment supplier. Interestingly, non-managers think that the organization rates reliability more importantly than do managers (a 2.6 mean score as opposed to a 3.6 mean score). Whatever the explanation for this manager/non-manager disagreement, it is worrying that managers themselves think that reli- ability should have the highest priority and yet perceive reliability to have the lowest organiza- tional priority.

Practitioners in ES all agree that the organization ranks user satisfaction as least important. Although this potentially alarming result may be mitigated by the embedded nature of the appli- cation area, it must be noted that practitioners themselves rate user satisfaction as a very important goal.

Practitioners' goals at ES. Table 2 shows that there is a great deal more agreement on personal quality goals among practitioners. Nevertheless there are some interesting differences in personal goals between managers and non-managers. For example, non-managers rate user satisfaction higher than managers; managers rate conformance to requirements more highly than non- managers; managers rate low costs as a less important goal than non-managers.

It is very clear that in ES there is a huge gulf between what practitioners think are the organ- ization's quality goals and what practitioners own quality goals are (they are almost inverse to each other). Such goal incongruence is likely to make quality improvement difficult.

Organizational goals in IS. There was a great deal more agreement among practitioners on what their organization's software quality goals were. Indeed managers and non-managers here ranked their perceived organizational goals identically. Overall, practitioners thought user satisfaction was an organizational priority. Practitioners also agreed low costs had a low organizational priority.

Practitioners'goals in IS. There was a high level of agreement here about practitioners' own quality goals. They had a very similar pattern to perceived organizational goals (agreement being evident between highest and lowest priorities). In addition there was almost complete agreement on per- sonal goals between managers and non-managers.

3. The organizational infrastructure

We believe that the efficacy of the organizational infrastructure has an impact on the quality of the software produced. Since practitioners are ideally placed to comment on the true state of their organizational infrastructure, we asked them what they felt about it.

Overall we found that the practitioners in IS were much happier than the practitioners in ES

240 Hall and Fenton

Embedded Systems Information Systems

Fig. 2. How often individuals think that they work efficiently.

with their organizational infrastructure. Indeed morale at IS was significantly higher with 41% of practitioners selecting the lowest morale indicator compared to 60% of practitioners at ES.

Figure 2 shows that practitioners in both organizations felt that they were unable to work efficiently, although practitioners at ES were more negative than practitioners at IS.

Figure 3 shows that both organizations have a problem in communicating to practitioners what the organization is about. Again, ES has the biggest problem. This is further evidence of what we saw in Tables 2 and 3.

We also found that practitioners were very unhappy with feedback levels (this was significantly more of an issue at ES). This is confirmed in Fig. 4 which considers feedback at three different levels.

4. Summary and conclusions

We have described some of the results of an exhaustive investigation into the software quality programmes at two major IT companies. This work forms part of a more extensive survey of UK industry, where we are attempting to discover the real state of practice and to identify those particular methods that are bringing the most benefits. The results for the two companies here are a more detailed version of what we are trying to find out generally. We interviewed almost all the staff in the two organizations involved in, or affected by, their quality programmes and hence have been able to present a definitive assessment of these two organizations' quality pro- grammes. Although no generalizable conclusions can be drawn from this particular study, our wider study suggests that the quality problems highlighted here are probably widespread.

Embedded Systems Information Systems

Fig. 3. How often organizational goals are made clear to individuals.

Software quality programmes: theory versus reality 241

Fig. 4. Satisfaction with levels of feedback.

Despite strong evidence that the systematic use of reviews and inspections is cost-effective, the two organizations here (one of which produces safety critical software) were not incorporating them effectively into their development process. Moreover, the penetration of these basic quality controls was much lower than senior managers believed and claimed. It is not clear if this shortfall is a result of poor technology transfer (companies not knowing of the benefits to be gained), or whether companies do not believe that there really are benefits. Either way the companies are deciding not to allocate effort to reviews or inspections while they are happy to allocate effort to implementing low-level standards. However, senior management in both companies were keen to be seen using reviews and inspections. This dichotomy of theory and reality could prove difficult for customers of software produced by suppliers not third-party accredited. Such custo- mers are likely to find it impossible to establish how much of the official quality programme is actually practised during production.

The results also show that each organization had varying degrees of misalignment between organizational and practitioner quality goals. This confusion about what the quality of the final software should be will make producing software at the fight level of quality difficult. Again the particular goal incongruence difficulties that ES exhibited makes compromised safety critical soft- ware more likely.

We discovered that, contrary to popular belief, practitioners do not have a stereotyped 'wish list' for software development. In neither organization did practitioners cite a clichtd list of quality goals, but rather seemed to tailor their own goals to their specific environment. This is important for managers. Practitioners are often excluded from influential goal setting initiatives because it is generally thought they have unrealistic goals. This study suggests that they do not.

The organizational goal analysis also suggests that organizations want everything to be a prior- ity. In neither organization was there a clear consensus on organizational goals. Practitioners cited a variety of goals as their organization's priority. This suggests that the organizations were trying to emphasize too many goals, instead of being clear about real priorities.

The context in which a quality programme is set is important. A quality programme does not operate in isolation and so getting the organizational environment right, before even starting a quality programme, is the ideal. The organizations in this study seemed to have some way to go

242 Hall and Fenton

in getting their organizational environments right. The results suggest that the organizations are failing to provide a constructive working environment. Many basic tenets of management science are broken by both organizations. In particular, basic communication within the two organ- izations seems very poor. As a result of this it is clear that many practitioners feel negative and de- motivated about their company. Mass staff de-motivation will make it difficult to optimize quality mechanisms. It seems that senior managers think that a successful quality programme can be insti- tuted without considering the effectiveness of the company's overall framework. While practi- tioners in both of the companies believed that a number of deep organizational problems existed, senior management were primarily interested in developing the quality system rather than the organizational system. Such a strategy is only likely to have limited success.

References

1. T. Hall and N.E. Fenton. Software practitioners and software quality improvement, 5th International Conference on Software Quality, pp. 313-323 (ASQC, Austin, TX, 1995).

2. B. Kitchenham, L. Pickard and S. Pfleeger. 'Case studies for method and tool evaluation' IEEE Software, 12 (1995) 52-63.

3. T. Hall and N.E. Fenton. Implementing effective software metrics programmes, IEEE Software (to appear February 1997).

4. N.E. Fenton, S.L. Pfleeger and R. Glass. Science and substance: a challenge to software engineers, IEEE Software, July (1994) 86-95.

5. T. Gilb and D. Graham. Software Inspections (Addison Wesley, Reading, MA, 1993). 6. S.L. Pfleeger, N.E. Fenton and P. Page. Evaluating software engineering standards, IEEE Computer,

September (1994) 71-79. 7. V.R. Basili and H.D. Rombach. The TAME project: towards improvement-oriented software environ-

ments, IEEE Transactions on Software Engineering, 14 (1988) 758-773.