this actualgoings-on in results research, wehave themvz/pub/j21.pdf · 2008-11-20 · this...

This in-depth survey of 30 companies reveals actual goings-on insoftware production. Results show that, while practice is 10 years behind

research, we have the tools to narrow the gap.

Software Engineering Practices

in the US and Japan

Marvin V. Zelkowitz, Raymond T. Yeh, Richard G. Hamlet, John D. Gannon, and Victor R. Basili,

University of Maryland

The term software engineering first appeared in the late1960's to describe ways to develop, manage, and maintainsoftware so that resulting products are reliable, correct, ef-ficient, and flexible. I The 15 years of software engineeringstudy by the computer science community has created aneed to assess the impact that numerous advances havehad on actual software production. To address this need,IBM asked the University of Maryland to conduct a surveyof different program development environments in in-dustry to determine the state of the art in softwaredevelopment and to ascertain which software engineeringtechniques are most effective. Unlike other surveys, suchas the recent one on Japanese technology, 2 we were lessinterested in recent research topics. Journals, such as theIEEE Transactions on Software Engineering adequatelyreport such developments; we were more interested indiscovering which methods and tools are actually beingused by industry today. 3 This report contains the resultsof that survey.The goal of this project, which began in spring 1981 and

continued through summer 1983, was to sample about 20organizations, including IBM, and study their develop-ment practices. We contacted major hardware vendors inthe US, and most agreed to participate. Several other soft-ware companies and other "high-technology" companieswere contacted and agreed to participate. While we ac-

knowledge that this survey was not all inclusive, we didstudy each company in depth, and based on discussionswith others in the field, we believe that what we found was

typical.We were not interested in R&D activities in these com-

panies. Most had individuals engaged in interestingdevelopments, and most knew what was current in thefield. Our primary concern was what the average program-mers in these companies did to develop software projects.

Data was collected in a two-step process. A detailedsurvey form was sent to each participating company.When the form was returned, a follow-up visit was madeto clarify the answers given. We believe that this process,although limiting the number of places surveyed, allowedus to present more accurate information than if we hadrelied on the returned forms alone.Each survey form contained two parts. Section one asked

for general comments on software development for theorganization as a whole. The information typically repre-sented the standards andpractices document for the orga-nization. In addition, several recently completed projectswithin each company were studied. Each project leadercompleted the second section of the survey form, whichdescribed the tools and techniques used on that project.

Several companies were concerned that the projects wewere looking at were not typical of them. (Interestingly,very few companies claimed to be doing typical software.)However, since the companies selected the projects theydescribed on the form, we believe we saw the betterdeveloped projects-if there is any bias to our report, it isthat the average industry project is probably worse thanwhat we describe here.

Thirty organizations in both the US and Japan par-ticipated in the study: five IBM divisions, 12 other UScompanies, and 13 Japanese companies. About half theJapanese companies were not interviewed, while the otherhalf were interviewed to varying degrees of detail. All UScompanies were interviewed. The "Acknowledgments"section at the end of this article lists the US participants.Some of the Japanese participants never responded to ourrequest for permission to use their names, so only a fewJapanese companies are listed.

Table I characterizes the companies visited, divisionswithin a company, and the projects studied, arbitrarily

0018-9162/84/0600-0057$01.00 (11984 IEEE 57June 1984

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 10, 2008 at 13:32 from IEEE Xplore. Restrictions apply.

classifying projects and teams into four groups accordingto sizes: small, medium, large, and very large. Projects areclassified according to the number of staff-months neededto complete them, and teams according to the number ofmembers. All companies listed with zero projects wereJapanese companies that submitted part one of our formonly. We interviewed at least one manager in depth in allsurveyed US companies, in addition to general projectmanagement personnel.

After reviewing the basic data, we recognized thefollowing three software development environments:

(I) contract software. Department of Defense andNASA aerospace systems;

(2) data processing applications. Software produced byan organization for its own internal business use;and

Table 1. Companies surveyed. The size of the project is in staffsmonths where (S)mall=cc10, (M)edium=10-100, (L)arge=100-1000,and (V)ery (L)arge = >1000. Team size is in staff members whereS = <10, M = 10*25, L = 25*50, and VL => 50.

NO OF NO. OFDIVISIONS PROJECTS

2 32 71 11 33 41 31 21 71 91 41 81 11 31 21 11 12 01 01 11 41 01 11 11 11 11 22 51 11 11 7

INTERVIEWEDYesYesNoYesYesYesYesYesYesYesYesYesYesNoYesNoNoNoYesYesYesYesYesNoNoYesYesYesYesYes

PROJECTSIZELVLSLVLVLLLVLLVLLMSVLMM

VLLMLLLMVLMLVL

TEAMSIZE

LVLMLVLVLLMVLVLMVLVLSVL

L

SVLVLSS

VLSSVL

Figure 1. Typical organization structure.

(3) systems software. Operating system support soft-ware produced by a hardware vendor as part of atotal hardware-software package of products for a

given operating system.A single company might have projects in more than one

of these categories. For example, one aerospace companywas involved in several DoD-related projects and one in-ternal data processing application.

General observations

This article is a series of general observations about eachenvironment. Two of our first observations were that thedata collected by each organization is insufficient and in-terpretations for similar concepts (e.g., phases of the lifecycle, job descriptions from similar sounding titles, whatcertain automated tools should or did do, etc.) differ. Wecould have generated a survey consisting of 50 to 100techniques and proceeded to tabulate them in a report.However, as we found out, the detailed interview processwas much more informative. In addition, we would notwant others quotinz such numbers, since they would besubjective and imprecise. We believe that the structure wechose gives a better idea of software development today.

Every company had either written guidelines or unwrit-ten folklore as to how software was developed, and majordeviations were rare. Differences in projects within a com-pany were less than the differences in projects among com-panies. But more significant was the wide gulf betweenpractices in industry and those documented in currentsoftware engineering literature.The literature contains many references to software

engineering methodology, including tool supportthroughout the life cycle, specification and designlanguages, test data generators and coverage metrics,measurement and management practices, and other tech-niques. We found surprisingly little use of softwareengineering practices across all companies. No organiza-tion fully tries to use the available technology. Althoughsome companies had stronger management practices thanothers, none used tools to support these practices in anysignificant way.We are not implying that the companies do not have

talented personnel. Most have individual projects that tryto keep abreast of current technology, but within eachcompany these projects are relatively rare, and the re-sulting experience is rarely applied to a different project.

Organizational structure. Most companies had anorganizational structure similar to the one in Figure 1. Thesoftware technology group typically has one to five in-dividuals collecting data, modeling resource usage, andgenerating standards and practices documents. However,this group has no direct authority to mandate softwareengineering practices even within a single division. As aresult, standards often vary within a single organization.We believe that this structure explains a current anoma-

ly in the use of software engineering techniques.Developers of real products often think that members ofthe software technology (research) group (who attend na-

COMPUTER

CODEABC0FFGH

JKLMN0p0RSTUVwxy

AABBCCDD

58

DIRECTOR OF SOFTWARE

II -1

SOFTWARE AREA 1 AREA 2 AREA NTECHNOLOGY MANAGERS MANAGERS"'

MANAGERSGROUPJ I L I I I


tional conferences and write most of the research papers)are too optimistic about the effects of these techniques,since they have not applied them to software products.Managers know their personnel often lack the educationand experience needed to apply these techniques suc-cessfully. Even techniques that have been adopted are fre-

quently misused. For example, although many companiesused the term chiefprogrammer to describe their program-ming team organizations, most descriptions bore little

resemblance to the technique outlined in the literature.Many projects had two to three levels of managers whohandled staff and resource acquisition but did not activelyparticipate in system design.A further problem in many organizations is that

generally no one person is the Director of Software (seeFigure 1), responsible for making software decisions. Insome companies, software activities span several divisions.So even if such a person exists in each division, standardsvary across the company. Interestingly, organizationsoften have one person making hardware decisions.

Tool use. Tools are not widely used in the industry. Nottoo surprisingly, the use of tools varies inversely from howfar the tools are from the code development phase. Toolsare most frequently used during the code and unit testphase of software development (e.g., compilers, codeauditors, test coverage monitors, etc.). Tools are less fre-quently used in the adjacent phases of the software lifecycle -design and integration (e.g., PDL processors andsource code control systems). Few requirements ormaintenance tools are used. Table 2 gives some indicationof which techniques and tools are used. In classifying com-panies, we were somewhat subjective because we countedtools used on most projects, but not necesarily all, within acompany. For example, every company used high-levellanguages on some projects, but they also used assemblylanguage quite frequently.Companies tend to adopt methods relatively early

because their "capitalization" cost is relatively low, buttool use takes longer, since development or purchase costs

are higher.Although the percentages in Table 2 are not exact, the

trends seem clear. Tool use generally has the flavor ofvintage-1970 timesharing. Jobs have a batch flavor in thatruns are built and then compiled. Tool support isminimal-mostly compilers and simple editors.

Timesharing computer systems and compiler writingbecame practical in the late 1960's and early 1970's; thus,on-line access and high-level languages can probably belabeled the successes of the 1960's. Similarly, since reviewsand pseudocode or program design language (PDL) were

so widely used, we can call them the successes of the1970's. It is disappointing that few other tools have beenadopted by industry. Testing tools are used by only 27 per-cent of the companies, and most are simply test-datagenerators. Only two companies used any form of unit testtool to measure test-case coverage. In a few cases, toolswere developed but deemed too expensive to use. In one

company, the quality assurance group had developed a

testing tool and was trying to get it used on various proj-ects. However, each project manager with whom we spoke

praised the tool but claimed it was too expensive to run, in-creased the size of the system too much, etc.PDL is frequently used, but it is not automated. Some

PDL processors are simply manual formatters, while somedo a pretty print and indent the code. Often the PDL isonly a coding standard and not enforced by any tool. Onlyone location had a PDL processor that checked interfacesand definition/use patterns for variables.Our sample identified two general classes of companies.

One class had strong management control over develop-ment with little tool support, while the other had relativelylax control over programmers, who generally built theirown tools. In one extreme case, three project managers weinterviewed expounded the virtues of their individual texteditors.The problems in using tools can be attributed to several

factors. Corporate management, particularly of hardwarevendors, tends to have an engineering background. Man-agers have little, if any, software background and are notsympathetic with the need for tools. No separate corporateentity exists whose charter includes tools, so we have nofocal point for tool selection, deployment, and evaluation.Tools must be funded from project development budgets,so there is a fair amount of risk and expenditure for a proj-ect manager to adopt a new tool and train people to use it.Since project management is rated according to whethercurrent costs and schedules are met, tool use must beamortized across several projects to be effective. Conse-quently, a project manager building and using a new toolwill almost always stand out as unproductive. Companiesoften work on different hardware, so tools are nottransportable, limiting their scope and their perceived ad-vantage. The most striking example of this handicap wasone system in which one million dollars was spent buildinga database, yet no one ever thought of using that databaseon another system. The need to maintain large existingproducts (written in the past in assembly code) makes ithard to introduce a new tool that processes a new higherlevel langauge. Finally, many of the tools are incompleteand poorly documented. Because such tools fail to live upto promises, project managers are justifiably reluctant toadopt them or consider subsequently developed tools.

Review process. At the end of each phase, the evolvingsoftware product is subject to a review process to try to un-

cover problems as soon as possible. A review might beeither an inspection or a walk-through, without regard to

the distinctions made in the software engineering

Table 2. Method or tool use.

METHOD PERCENTAGE OFOR TOOL COMPANIES

High-level languages 193On-line access 93Reviews 73Program design languages 63Some formal methodology 41Some test tools 27Code auditors 18Chief programmer team 7Any formal verification 0Formal requirements or specifications 0

59June 1984


literature. 4 Nearly everyone agrees that reviews work, andnearly everyone uses them, but the ways reviews are con-ducted differ greatly. Most agree that software projectscan be routinely completed within time and budget con-straints that only a few years ago could be managed onlyby luck and sweat. Reviews were instituted first for code,then extended to design. Extensions to requirements andtest-case design are not universal, and some feel that thetechnique may have been pushed beyond it usefulness.Managers would like to extend the review process, whilethe technical people are more inclined to limit it to the bestunderstood phases of development.Two aspects of reviews must be separated: managerial

control and technical utility. Managers must be concernedwith both aspects, but technical success cannot be assuredby insisting that certain forms be completed. If the tasksassigned to the reviewers are ill-defined, or the form of theproduct reviewed inappropriate, the review will waste thetime of valuable people. Lower level managers prefer touse reviews only when they think reviews are appropriate.The technical success of the review process rests on the

expertise and interest of the people conducting the review,not on the mechanism itself. The review process must becontinually changed to reflect past successes and failures,and much of this information is subjective, implicitlyknown to experienced participants. Some historical infor-mation is encoded in review checklists, which newcomerscan be trained to use. However, subjective items like thecompleteness of requirements are of little help to a novice.New and old companies differed considerably in their

approaches to reviews. New companies were less commit-ted to reviews, treating code reviews as training exercisesfor junior employees or as verification aids for particularlydifficult modules. Since the newer companies did not havea large existing software base, they emphasized rapiddevelopment rather than maintenance. However, as com-panies grew and aged, accumulating software, reviewsseemed to take on added significance as an importantverification aid.

Figure 2. Amount and type of data collected.

Data collection. Every company collects some data, butnot much of it becomes part of the corporate memory tobe used beyond the project on which it was collected. Datagenerally belongs to individual managers, and they decidewhat to do with it. Data is rarely evaluated and used in ananalysis to see if the process could have been improved.The opposite is true in Japanese companies in which'"postmortem" analysis is frequently performed.

Several companies are experimenting with resourcemodels, such as Price S. Slim. 5'6 No company seems totrust any model enough to use it extensively; instead themodels are used to check manual estimates. Figure 2 showsthat little data is being collected across all companies. Thelevels of Figure 2 are somewhat arbitrary; high representsthe amount of data collected by the NASA SoftwareEngineering Laboratory (resource use by programmer ineach module, detailed error reports including causes, etc.),while low represents a minimal amount of data (e.g., thenumber of major and minor errors detected in reviews,outstanding error reports per time period, etc.). 7 Ingeneral, more error data than resource data is collected,and resource data is typically limited to hours spent byeach programmer on the project to facilitate project bill-ing.

It is extremely difficult to compare data across com-

panies. First of all, quantitative data is quite rare withinmost companies. Error data is rarely tied to causes of er-rors, and the process of counting errors is never fed backinto the development process. Knowing how many errorsoccurred does not necessarily improve the programmingprocess if you don't know why they occurred. To keep thereview process open, results are sometimes limited to thereview group and the quality assurance manager. In addi-tion each company has different definitions for most ofthe measured quantities, such as

* lines of code, which is defined as source lines, com-ment lines, with or without data declarations, ex-ecutable lines or generated lines;

* milestone dates, which depend on the local softwarelife cycle used by the company (whether re-quirements, specification, or maintenance data areincluded will significantly affect the results);

* personnel, which might include programmers,analysts, typists, librarians, or managers.

The differing definitions prevent any meaningful com-parison. It is quite evident that the computer industryneeds more work on the standardization of terms before itcan address these quantitative issues. Also, it is not clearthat management believes in a need for such data exceptfor budgeting, so it is rarely collected.

Software development environmentsin the US

In describing the general characteristics of software en-vironments, we limit discussion, for the most part, to the13 companies in the US where we made site visits to over 20different locations and interviewed approximately 60 proj-ect managers. Comments on Japanese software develop-

60COMPUTER

TYPE OF DATA | ] RESOURCEuu COLLECTED l ERO

C.3 ~~ ~ ~ ~ ~ ~ UERRORi HIGH

CD

C MEDIUMCZ

LU>. LOU-J

AVERAGE SPEC DESIGN CODE TEST MAINTENANCEPHASE PHASE PHASE PHASE PHASE


ment and comparisons with the techniques used in the USare presented in the subsequent section.

General life cycle. The life cycle of a project consists ofthe requirements and specification phase, the designphase, the code and unit test phase, and the integrationtest phase.

Requirements and specification phase. At all placescontacted, requirements were in natural language text.Some projects had requirements documents that could beprocessed by machine, but tool support was limited toscreen-oriented text editors. No analysis tools (like SREMand PSL/PSA were used except on toy projects. 8,9 Proj-ects were either too small to justify the use of a processoror too large to make such use economical.

Reviews determine if the system architecture is com-plete, if the specifications are complete, the internal andexternal interfaces are defined, and the system can beimplemented. These reviews are the most difficult to per-form, and their results depend greatly on the quality ofpeople doing the review because the specifications are notformal. There is little traceability between specificationsand designs.

Design phase. Most designs are expressed in some formof PDL or pseudocode, which makes design reviews effec-tive. Tools that manipulate PDL vary from editors to sim-ple text formatters. Only one company extended its PDLprocessor to analyze interfaces and the data flow in theresulting design.

While the use of PDL seems to be accepted practice, itsuse is not particularly effective. For example, we considerthe expansion of PDL to code a reasonable measure of thedetail in a design. A PDL-to-source-code expansion ratioof 1:1 may indicate that the design has the same detail as itseventual code. With design and code being separate tasks,this expansion indicates that the two concepts are notseparated. The expansion ratios of PDL to source codewere 1:5-10 at one location, 1:3-10 at another, and1:1 .5(!!)-3 at a third. Customer involvement with designvaried greatly even within installations. Some producedvolumes of PDL with an attitude similar to that for olderprojects that produced many detailed flowcharts: Nobodycares, but it's in the contract.

Code and unit test phase. Most code was in higher levellanguages-Fortran for scientific applications or somelocal variation of C, Pascal or PL/l for systems work. Inthe aerospace industry, Fortran was the predominantlanguage. People who normally worked in assembly lan-guage thought that Fortran and PL/l signficantly en-hanced their productivity. Historical studies have shownthat programmers produce an average of one line ofdebugged source code per hour regardless of the language.(Brooks contains a concise review of this work. 10)

Despite claims that they used chief programmer teamsin development, very few first- or second-line managersever wrote any PDL or code themselves. We heard com-plaints that chief programmer teams worked well onlywith small groups, six to nine people, and on projects in

which a person's responsibility was not divided among dif-ferent groups.Much of the code and unit test phase lacks proper

machine support. Code auditors could greatly enhance thecode review process. We studied one code review form andfound that 13 of 32 checks could be automated. Manualchecks are currently performed for proper indentation ofthe source code, initialization of variables, interface con-sistency between the calling and called modules, etc.Most unit testing could be called adversary testing. The

programmer claims to have tested a module and themanager either does or doesn't believe the programmer.Almost no unit test tools are used to measure how effec-tively the tests devised by a programmer exercise the sourcecode. While a test coverage measure like statement orbranch coverage is nominally required during the review ofunit test, mechanisms are rarely available to ensure thatsuch criteria have been met.

Integration test phase. Integration testing is mostlystress testing-running the product on as much real orsimulated data as is possible. The data processing environ-ment had the highest level of stress testing during integra-tion testing. Testing in the systems software environmentwas not as rigorous compared to integration testing in thedata processing environment.

Resources. Office space for programmers varied fromone to two programmers sharing a Santa-Teresa style of-fice with a terminal to large bullpens divided by low,movable partitions. I1l Terminals were the dominant modefor computer access. Some sites had terminals in offices,while others had large terminal rooms. The currentaverage seems to be about two to seven programmers perterminal. Newer companies had two terminals per pro-grammer and some were replacing terminals with personalcomputers. Within the last two years most companies haverealized the cost-effectiveness of giving programmers ade-quate computer access via terminals but have still not pro-vided adequate response time. A response time of 10 to 20seconds was considered good at some places, where asubsecond response time was possible. 12Most companies are willing to invest in hardware, such

as terminals, to assist their programmers but are reluctantto invest in software that might be as beneficial.

Education. Most companies have agreements with alocal university to send employees for advanced training,and have their own seminar series. However, there is littletraining for project management. Only one company has afairly extensive training policy for all software personnel.Many companies had two problems with their educa-

tional program: (1) Programmers were sent to courses with

Table 3. Six lines of source code per staff-month.

APPLICATIONLINES OF CODE AND LANGUAGE

75 OS in Assembly91 1/0 controller in HLL

142-167 OS in HLL182-280 Assembly applications

61June 1984


little or no follow-up experience, so what they learned wasrarely put into practice and often forgotten; and (2) Somesites were far away from any quality university, and theisolation caused problems.

Data collection efforts. The data typically collected onprojects includes the number of lines of PDL for each levelof design, the number of lines of source code produced perstaff-month, the number and kinds of errors found inreviews, and a variety of measures on program troublereports. As Table 3 shows, the range in productivity wasfrom 75 to 280 lines of code per month for different prod-ucts using relatively similar development methods. Thisdiscrepancy illustrates that using lines of code as a measureof productivity is unwise and that more refined productivi-ty measures are needed. Because of the differing applica-tion areas, we cannot really compare numbers in Table 3.However, it does seem obvious that the difficulty of theapplication area has more impact on productivity than theimplementation language used (operating systems andother real-time programs being the most difficult).One location reported that two major and five minor er-

rors per 1000 lines were found during reviews in the designphase, and five major and eight minor errors per 1000 lines

US software developers areprimarily producing applications, systems,

or data processing software.

were found during reviews in the code phase. Realistically,though, the classification of errors into categories like ma-jor and minor may be useful for quality control in productdistribution, but it sheds little light on the causes andpossible treatments of these errors and their prevention infuture systems.

Development environments. The development en-vironments centered on three types of projects: applica-tions software, systems software, and data processing.

Applications software. We studied 13 projects in fourcompanies that produce applications software. In thisarea, software is contracted from the organization by aFederal agency, typically the Department of Defense orNASA. Software is developed and "thrown over the wall"to the agency for operation and maintenance. Typically,none of the organizations we surveyed were interested inmaintenance activities. All believed that the payoff inmaintenance was too low and that smaller software housescould do whatever maintenance was necessary.

Since contracts are awarded after a competitive biddingcycle initiated by a "Request for Proposal," and re-quirements analysis is typically charged against companyoverhead, analysis is kept to a minimum before the con-tract is awarded. Requirements are written in English, andno formal tool is used. In addition, since the goal is to wina contract, there is a clear distinction between cost andprice. Cost is the amount needed to build a product-atechnical process that most companies feel capable of

handling. On the other hand, price is a marketing strategyneeded to win a bid. The price has to be low enough towin, but not so low that either money will be lost on theproject or the company will be deemed "not responsive"to the requirements of the RFP. Thus, many ideas of soft-ware engineering developed during the 1970's on resourceestimation and workload characterization are not mean-ingful in this environment because of the competitive pro-cess of winning bids.

In addition, two distinct types of companies emergedwithin this group-system developers and softwaredevelopers. The system developers would package bothhardware and software for a government agency intoproducts such as a communications network. In this case,most of the costs were for hardware, and software was notconsidered significant. On the other hand, the softwaredevelopers simply built systems on existing hardwaresystems. DEC's PDP/l series seemed to be the mostpopular with system builders that were not hardware ven-dors.

All companies surveyed had a methodology manual;however, they were either out of date, or were just in theprocess of being updated. In this environment, Depart-ment of Defense MIL specifications were a dominant driv-ing force, and most standards were oriented to govern-ment policies. The methodology manuals were oftenpolicy documents outlining the type of information to beproduced by a project but not how to obtain that informa-tion.

Tool use was relatively sparse. Fortran was the domi-nant programming language. Two tools did seem to beused. In compliance with DoD specifications, most hadsome sort of management reporting forms on resourceutilization. However, these generally did not report onprogrammer activities. PDL was the one programmingtool that many companies did depend on, probablybecause the cost was low.

Staff turnover was uniformly low, generally five to 10percent a year. Space for programmers seemed adequate,with one to two per office being typical. All locations ex-cept one used terminals for all computer access, and thatone site had a pilot project to build private offices con-nected to a local minicomputer.

Systems software. We studied 18 projects produced by11 vendors. Most of the projects were for large machinesalthough some projects for microprocessors were in-cluded. Operating systems for those machines were themost important projects studied. The other projects,mostly compilers and utilities, did not follow the develop-ment rules for operating systems projects because theother projects were considered small, and hence theirdesigns would be well understood.Many companies are heading towards a policy of never

building a large product. Development effort is limited tono more than two years and 10 programmers on any par-ticular product. A great deal of effort is expended in thedesign of traditionally large pieces of software likeoperating systems to segment them into pieces of this size.Japanese companies also seem quite proficient at design-ing and assembling small projects only.

62 COMPUTER


Software is generally written on hardware similar to thetarget machine. Terminals are universally used and theratio of programmers to terminals varies from 1:2 to 3:1.Getting a terminal is frequently less of a problem than get-ting CPU cycles to do development.

In most places, software support is generally limited totext editors and interactive compilers. High-level develop-ment languages exist, and in most cases, the policy is thatthey be used; however, a substantial portion of operatingsystems remains in assembly language (20 to 90 percentdepending on the company). The reasons are partly good(such as the prior existence of assembly code) and partlythe usual: alternatives have never been considered at thetechnical level. Text formatting programs are in wide use,but analysis of machine-readable text other than sourcecode is virtually nonexistent.

We studied 18 projects produced by11 vendors. Most of these were for largemachines, with operating systems being

the major product.

Most testing is considered part of the development ef-fort. There may be a separate test group, but it reports tothe development managers. Only a final field test may beunder the control of an independent quality control group.One company assigned the quality control person directlyto the development group, but group members believedthat the independence of that individual in testing thesystem had been compromised as a result.

Maintenance is usually handled by the developmentstaff. A field support group obtains trouble reports fromthe field and forwards them to the development organi-zation for correction. In most cases, the developers, evenif working on a new project, handle errors.Programmers are usually organized into small teams by

project, and usually stick with a project until it is com-pleted. The term chiefprogrammer team is used incorrect-ly to describe conventional organizations: a chain ofmanagers (the number depends on project size) who donot program, and small groups of programmers with littleresponsibility for organization.

Staff turnover is relatively high (up to 20 percent peryear) compared with that in the applications softwarearea. Most programmers typically have private cubiclesparceled out of large open areas. The lack of privacy isoften stated as a negative factor.

Software engineering practices vary widely among theprojects we investigated. Not surprisingly, the older thesystem, the fewer software engineering techniques used.

Data processing. We studied seven data processing proj-ects at five locations, although every location had some

data processing activities for internal use. Most data pro-cessing software that we studied was developed in Cobol,although some systems were written in Fortran. There is aneed to maintain the code throughout the life cycle.

Requirements were mostly in English and unstructured,although one company structured specifications by userfunction. Designs, especially for terminal-oriented prod-ucts, were similar-a prototype set of simulated screendisplays and menus to which the user could respond. Themost striking difference in the data processing environ-ment was the heavy involvement of users in the twodevelopment steps. The success of the project dependedon how much the user was involved before integrationtesting. One site clearly had a success and a failure on twodifferent projects that used the same methodology. Thecompany directly attributed the success to the high level ofinterest on the part of the user assigned to the team duringdevelopment.

All data processing was done at terminals. Office spacewas more varied than in the other two environments weobserved. Some places used one- and two-person offices,while others partitioned large open areas into cubicles.Stress was often high in that overtime was more common,and turnover was the highest in this environment-oftenup to 30 percent per year. One location had a low turnoverrate, which they attributed to their salaries being higherthan those offered by comparable companies.

Data processing environments often use a phased ap-proach to development, and quality control is especiallyimportant. One company, which had had numerous fail-ures, attributed its recent successes to never attemptingany development that would require more than 18months. Since these systems often managed thecompany's finances, the need for reliability was mostcritical, and stress testing was higher than in other areas.

Japan and US comparisons

Unlike the recent survey by Kim 2 which emphasized theintegrated tool sets and artificial intelligence techniquesbeing employed by Japanese industry, we found the levelof technology used by the Japanese to be similar to USpractices, but with some important differences. Program-mers in both countries complain about the amount ofmoney going toward hardware development and the lackof resources for software. However, we found thatJapanese companies typically optimize resources acrossthe company rather than within a single project. One ef-fect of this policy is that tools become a capitalized invest-ment paid for or developed out of company overheadrather than project funds. The cost of tools is spreadamong more projects, knowledge about tools is known tomore in the company, and project management is more

willing to use tools since the risk is lower. Thus, tooldevelopment and use is more widespread in Japan. Oursurvey does reinforce some of the conclusions of theearlier survey, namely:

(1) Japanese often use techniques developed in the USor Europe;

(2) Emphasis seems to be on practical tools.

Two successful techniques used by the Japanese are keep-ing projects small and relating failures to their causes

through postmortem analyses of error data.

63June 1984


Improving software development in the US

While tool use is important for software development,the most important factor that we saw was the quality ofthe personnel on various projects. We firmly believe thattools cannot replace good people or good methods forsoftware production. However, tool use can help a goodprogrammer be even better.Some very small changes can improve productivity in

many installations. While there is no empirical evidencethat will permit us to forecast gains, there is a general con-sensus in the software community (like that for the use ofhigh-level languages) that supports these ideas.

* More and better computer resources should be madeavailablefor deuelopment. The computer systems be-ing used for development are comparable to thoseavailable in the late 1960's or early 1970's providingtimesharing on large machines. The use of screeneditors at some locations has been a major improve-ment, but other tools seem limited to batch compilersand primitive debugging systems. Response timeseems to be a major complaint at many developmentinstallations.

* Methods and tools should be eualuated. A separateorganization with this charter should be established.As of now, it does not appear that any one grouip inmost companies has the responsibility to study theresearch literature and try promising techniques.Since the most successful tools have been high-level-language compilers, the first tools to be developedshould be integrated into compilers. Thus, these toolsshould concentrate on the design and unit test phasesof development, which have formal languages andrelatively straightforward compiler extensions. Thisorganization could both acquire and evaluate thetools by looking at case studies and/or conducting ex-

periments.

v Tool support should be builtfor a common high-levellanguage. The tools we would pick first include a

PDL processor, a syntax-directed editor-an editorthat knows the underlying language syntax andprevents such errors from entering the program 13 a

code auditor, and a unit test coverage monitor. ThePDL processor should at least check interfaces. Un-fortunately, commercially available processors do lit-tle more than format a listing; however, interfacechecking is nothing more than 20-year-old compilertechnology. The processor should also constructgraphs of data flow through the design and extract

PDL from source code so that while both are main-tained together they can be viewed separately. Codeauditors can be used to check that source code meets

accepted standards and practices. Many checks, suchas those to verify whether begin-end blocks are

aligned, are boring to perform manually and are

therefore error prone. Unit testing tools can evaluatehow thoroughly a program has been exercised. Thesetools are easy to build and should be accepted quick-ly, since many managers requiire statement or branchcoverage during unit test.

* PDL processors should support an automated set ofnetrics that cover the design and coding process. Themetrics in turn can monitor progress; characterize theintermediate products, such as design and sourcecode; and attempt to predict the characteristics of thenext phase of development. Possible metrics includedesign change counts, control and data complexitymetrics for source code, and structural coveragemetrics for test data. 14

* In syntax-directed editors, the grammar of theunderlying language is built in, so the programmerneeds to include only valid constructs at any point inthe program. SLuch systems facilitate top-down pro-gram development and often permit interactive con-text switching between program editing and programexecution. 13315 The programmer can then think moreabout algorithm design than keyboarding the pro-gram text.

* The review process should be improved. Reviews orinspections are a strong part of current methodology.The review process can be strengthened by the toolsmentioned earlier. Manual labor in design and codereviews could be reduced with more effective tools.Such tools would permit reviewers to spend moretime on the major purpose of the review the detec-tion of logical errors and avoid the distractions offormatting or syntactic anomalies.

* Incremental development such as iterative enhance-ment should be used. 16 Many successful companieslimited development to under two years and 10people. One data processing company, afterrepeatedly failing to deliver software, decided neverto build anything that required more than 18 staff-months. Since then they have been successful. Severalother companies reported similar experiences. Largeprojects tend to have several layers of managementand their success seems to depend on a strongerreview process that comprises requirements, design,and code reviews. Smaller projects need less manage-ment and can succeed with only design reviews.

* Data should he collected and analyzed. Most of thedata being collected now is used primarily to schedulework assignments. Measurement data can be used toclassify projects, evaluate methods and tools, andprovide feedback to project managers. Data shouldbe collected across projects to evaluate and helppredict the productivity and quality of software. Thekind of data collected and the analysis performedshould be driven by a set of questions that needanswers rather than by what is convenient to collectand analyze. For example, classifying errors into ma-jor and minor categories does not shed any light ondevelopment activities. A more detailed examinationof error data can determine the causes of common er-rors, many of which may have remedies. Projectpostmortems should be conducted.

Observations

We identified several approaches for improving soft-ware productivity althoLugh they are not strictly supported

COMPUTER64


(or contradicted) by the data we collected. We offer theseto stimulate discussion on this important topic.

* Maintain compiler technology. Many companiesseem to contract out compiler development to smallersoftware houses because the nature of building mostcompilers is pedestrian. While compiler technology isrelatively straightforward and perhaps cheaper tocontract to a software house, the implications are farreaching. Software research is heading toward an in-tegrated environment that covers the entire life cycleof software development. Research papers are beingwritten about requirements and specificationlanguages, design languages, program complexitymeasurement, knowledge-based Japanese fifth-generation languages, etc. 17 All of these depend onmundane compiler technology as their base.

* Try prototyping. Prototyping is one of the hot topicsin software engineering literature today. 18 It is alsocrucial for all other engineering disciplines, but it wasnever mentioned on any survey form or during ourvisits. A form of prototyping in building data pro-cessing application software was common: the crea-tion of screen displays during the design process. Thistechnique and others should be expanded.

* Develop a test and evaluation methodology. Testdata has to be designed and evaluated. While the cur-rent software development process provides for thedesign of test data in conjunction with the design ofsoftware, there is little tool support for this effort. Asa result, almost every project builds its own test datagenerator, and a few even build test evaluators. Con-cepts like attribute grammars may provide the basisfor a tool to support test data generation. 19

* Examine the maintenance process. Surprisingly,maintenance was rarely mentioned in our interviewprocess, even though it is an expensive activity thatmost companies engage in. A Japanese project is tobuild maintenance workstations; their view is thatdevelopment is a subset of maintenance. This impliesthat the successful methods and tools used in develop-ment should be adapted for use in this stage of theprocess.

* Encourage innovation. Experimental softwaredevelopment facilities are needed. Managementshould be encouraged to use new techniques on smallfunded-risk projects.

In preparing this article for publication, we weregrateful for the thoughtful insights and comments fromthe reviewers. However, two issues kept on creeping intotheir reviews, and we suspect that the reader might havesimilar opinions.

(1) The comment was made that the reviewer (or hiscolleagues) used more or better tools, thus thesurvey was not representative. However, as westated at the beginning, the goal was to look at pro-duction programmers-not research laboratories.We suspect that most reviewers and probably many

readers of Comnputer also represent the researchcategory.

(2) Names were sometimes given to demonstrate thatindustry is doing something about the problem.However, every person mentioned by the reviewerswas interviewed by us and is included in this article.They either represent a research environment, andare not involved in "revenue-producing" software,or are considered an anomaly within their own com-pany.

We are not saying here that software practices aredismal in the US. Technology transfer takes time, and itappears that the current level in industry represents theresearch environment of the mid-1970's-a delay of only10 years. However, certain practices we mentioned dohinder technology transfer. We hope that this article is animpetus to address those issues so that discussions can startwithin companies to improve the process and shorten-still further-the time it will take to adopt good practicesin industry. *

Acknowledgments

This project was sponsored by the IBM Corporation.We also acknowledge the cooperation of the followingorganizations for allowing us to survey their developmentactivities: Bankers Trust Company, AT&T Bell TelephoneLaboratories, Digital Equipment Corporation, Hewlett-Packard, Honeywell Large Information Systems Division,Kozo Keikaku Kenkyujo, Japan Information ProcessingService, Microsoft, Nomura Computer Systems Ltd.,Software Research Associates (Japan), Sperry Univac,System Development Corporation, Tandem Computers,Tokyo Electric Power Company, Toshiba Corporation,TRW, Wang Laboratories, and Xerox Corporation.Several Japanese companies did not respond to our re-quest to use their names, so they are not listed here. Thisproject would not have been possible without the help ofall these companies.

References

1 Software Engineering, P. Naur and B. Randell, eds., NATOScientific Affairs Division, Brussels, 1969.

2. K. H. Kim, "A Look at Japan's Development of SoftwareEngineering l1echnology," Computer, Vol. 16, No. 5, May1983. pp. 26-37.

3. R. C. Houghton, "Software Development Tools: AProfile," Computer, Vol. 16, No. 5, May 1983, pp. 63-70.

4. M. E. Fagan, "Design and Code Inspections to Reduce Er-rors in Program Development," IBM Systems J., Vol. 15,No. 3, Mar. 1976, pp. 182-211.

5. F. Freiman and Park, PRICE Software Model Overview,RCA, Cherry Hill, N.J., Feb. 1979.

6. L. Putnam, "SLIM Software Life Cycle ManagementEstimating Model: User's Guide," Quantitative SoftwareManagement, (July 1979).

7. Collected Papers: Volume 1, tech. report 82-005, NASAGoddard Space Flight Center, Software EngineeringLaboratory, Code 582. 1, Greenbelt, Md., 1982.

June 1984 65


8. M. W. Alford, "A Requirements Engineering Methodologyfor Real-time Processing Requirements," IEEE Trans.Software Engineering, Vol. SE-3, No. 1, Jan. 1977, pp.60-69.

9. D. Teichroew and E. A. Hersey III, "PSL/PSA: AComputer-Aided Technique for Structured Documentationand Analysis of Information Processing Systems," IEEETrans. Software Engineering, Vol. SE-3, No. 1, Jan. 1977,pp. 41-48.

10. F. P. Brooks, Jr., The Mythical Man-Month., Addison-Wesley, Reading, Mass., 1975.

11. G. M. McCue, "IBM's Santa Teresa Laboratory-Ar-chitectural Design for Program Development," IBMSystems J., Vol. 17, No. 1, Jan. 1978, pp. 4-25.

12. A. J. Thadani, "Interactive User Productivity," IBMSystems J., Vol. 20, No. 4, Apr. 1981, pp. 407-423.

13. T. Teitelbaum and T. Reps, "CPS-The Cornell ProgramSynthesizer," Comm. ACM, Vol. 24, No. 9, Sept. 1981, pp.563-573.

14. V. R. Basili, Models and Metricsfor Software Managementand Engineering, IEEE-CS Press, Los Alamitos, Calif.1980.

15. M. V. Zelkowitz, "A Small Contribution to Editing with aSyntax-Directed Editor," Proc. ACM Sigsoft Symp. Prac-tical Software Development Environments, Apr. 1984, pp.1-6.

16. V. R. Basili and A. J. Turner, "Iterative Enhancement: APractical Technique for Software Development," IEEETrans. Software Engineering, Vol. SE-I, No. 4, Dec. 1975,pp. 390-396.

17. H. Karatsu, "What Is Required of the Fifth GenerationComputer-Social Needs and Its Impact," Fifth Genera-tion Computer Systems, North-Holland, Amsterdam, 1982.

18. "ACM SIGSOFT Software Engineering Symposium:Workshop on Rapid Prototyping," ACM Sigsoft SoftwareEngineering Notes, Vol. 7, No. 5, Dec. 1982.

19. A. Duncan and J. Hutchinson, "Using Attributed Gram-mars To Test Design and Implementation," Proc. IEEEFifth Int'l Conf. Software Engineering, 1981, pp. 170-178.

Martin V. Zelkowitz has been with theComputer Science Department of the Uni-versity of Maryland since 1971 where he isan associate professor and, since August1982, an associate chairman for education.His research interests include compiler andlanguage design and the building of in-tegrated tools for software development.He has also worked with the Institute forComputer Sciences and Technology of the

National Bureau of Standards since 1976 and is the chairman ofthe Computer Society's Technical Committee on SoftwareEngineering. He is past chairman ofACM Sigsoft and is now onthe Sigsoft Executive Committee.He received a BS in mathematics from Rensselaer Polytechnic

Institute in 1967 and an MS and PhD in computer science fromCornell University in 1969 and 1971.

Raymond T. Yeh is professor of computersciences at the University of Maryland. Hehas served as chairman of the ComputerScience Departments at the Universities ofMaryland and Texas at Austin. He wasalso director of the Center for InformationSciences Research at Maryland. Yeh re-ceived a BS in electrical engineering, anMA in mathematics, and a PhD in mathe-matics from the University of Illinois. He is

the founding editor-in-chief of IEEE Transactions on SoftwareEngineering.

Richard G. Hamlet has been with theUniversity of Maryland since 1971 and isnow associate professor. He will move tothe Oregon Graduate Center in July 1984.His research interests include computabilitytheory, programming languages, and soft-ware engineering, particularly testingtheory. Hamlet received a BS in electricalengineering in 1959 from the University ofWisconsin in Madison, an MS in engi-

neering physics in 1964 from Cornell University, and a PhD incomputer science in 1971 from the University of Washington inSeattle.

John D. Gannon has been with the Univer-sity of Maryland since 1975 and is nowassociate professor. His research interestsinclude human-factors approaches to pro-gramming language design, formal specifi-cation and validation, and distributed com-puting. Gannon received an AB in mathe-matical economics in 1970 and an MS in

-,~i; :- 5 computer science in 1972 from BrownUniversity in Providence, R.I., and a PhD

in computer science in 1975 from the University of Toronto.

Victor R. B#Ail is professor and chairmanof the Computer Science Department at theUniversity of Maryland in College Park,where he has been involved in the designand development of several software proj-ects, including the SIMPL family of pro-gramming languages. He is currentlymeasuring and evaluating software devel-opment in industrial settings and has con-sulted with many agencies and organiza-

tions, including IBM, GE, CSC, Naval Research Laboratory,Naval Surface Weapons Center, and NASA. Basili has publishedover 50 papers on the methodology, the quantitative analysis, andthe evaluation of the software development process and product.In 1982, he received the Outstanding Paper Award from IEEETransactions on Software Engineering. He serves on the editorialboards of the Journal ofSystems and Software and IEEE Trans-actions on Software Engineering. He is a member of the ACMand the IEEE-CS Executive Committee of the Technical Commit-tee on Software Engineering.

COMPUTER66


this actualgoings-on in results research, wehave themvz/pub/j21.pdf · 2008-11-20 · this...

Documents