journal of operations management volume 9 issue 2 1990 [doi 10.1016%2f0272-6963%2890%2990098-x]...

8/18/2019 Journal of Operations Management Volume 9 Issue 2 1990 [Doi 10.1016%2F0272-6963%2890%2990098-x] Barbar…

http:///reader/full/journal-of-operations-management-volume-9-issue-2-1990-doi-1010162f0272-69632890299009… 1/35

JOURNAL OF OPERATIONS MANAGEMENT

Vol. 9, No. 2. April 1990

Empirical Research Methods in OperationsManagement

BARBARA B. FLYNN*

SADAO S.~KAKIBARA**

ROGER G. SCHROEDER**

KIMBERLY A. BATES**

E. JAMES FLYNN*

EXECUTIVE SUMMARY

This paper discusses the need for more research in operations management which is based on datafrom the real world. Tying operations management theory in with practice has been called for over a longperiod of time, however, many PlOM researchers do not have a strong foundation in gathering and usingempirical data. This paper provides a starting point that encourages operations management researchersto use empirical data and provides a systematic approach for conducting empirical studies.

Empirical research can be used to document the state of the art in operations management, as well as toprovide a baseline for longitudinal studies. It can also be invaluable in the development of parameters anddistributions for mathematical and simulation modeling studies. A very important use for empirical datais in theory building and verification, topics which are virtually ignored in most P/OM research.

Operations management researchers may be reluctant to undertake empirical research, due to its cost,both in dollars and time, and the relative risk involved. Because empirical research may be considered“soft,” compared with mathematical modeling, it may be perceived as risky. This paper attempts toprovide a foundation of knowledge about empirical research, in order to minimize the risks to researchers.It also provides a discussion of analytical techniques and examples of extremely rigorous empirical PiOMresearch.

Although operations management researchers may not recognize it, all research is based on theory.The initial step in conducting empirical research deals with articulating the theoretical foundation for thestudy. It also includes determining whether the problem under investigation involves theory building ortheory verification.

In the second step, a research design should be selected. Although surveys are fairly common inempirical P/OM research, a number of other designs, including single and multiple case studies, panel

studies and focus groups, may also be used, depending on the problem being studied. Third, a datacollection method should be selected. One method, or a combination of several data collection methods,should be used in conjunction with the research design. These include historical archive analysis,participant observation, outside observation, interviews, questionnaires and content analysis.

The implementation stage involves actually gathering the data. This section of the paper focuses onusing questionnaires as the method of data analysis, although some of the concepts discussed may beapplicable to other data collection methods, as well. A brief overview of data analysis methods is given,along with documentation of the types of data analysis which have been used in various types of

Manuscript received March 1, 1990; accepted September 17, 1990, after two revisions

*Iowa State University, Ames, Iowa SO01 1**University of Minnesota, Minneapolis, Minnesota 55455

250 Vol. 9, No. 2



empirical research conducted by operations management researchers over the past ten years. Potentialoutlets for publication of empirical P/OM research are discussed, and their history of publishing suchresearch is documented.

Underlying every step of the process are considerations of reliability and validity. Conductingempirical research without considering its reliability and validity is pointless, because the researcher willnot be able to generalize from the results. This should be considered in each of the four stages listed in theapproach described above.

A number of conclusions are discussed. These include the need for more empirical research and theneed for PiOM researchers to become more critical readers of the empirical research done by others.Colleagues in the social sciences can bc a valuable source of information about conducting empiricalresearch. Industry contacts can be useful, as well, in pilot testing, finding industry sites and determiningconsensus on the definition of terms. Finally, researchers in operations management need to be moreaware of the theory which underlies their work. Empirical research can be highly useful in both theorybuilding and theory verification.

INTRODUCTION

This paper provides a foundation for P/OM researchers who seek to incorporate real world

data into their research. The term “empirical,” which means “knowledge based on real worldobservations or experiment,” is used here to describe field-based research which uses datagathered from naturally occurring situations or experiments, rather than via laboratory orsimulation studies, where the researchers have more control over the events being studied.

A substantial amount of empirical operations management research has been published in theacademic and practitioner-oriented journals in recent years. Although the proportion ofempirical P/OM research is increasing, relative to P/OM modeling research, empirical P/OMresearch with a strong conceptual and methodological base is less common. This may be due tothe fact that P/OM researchers lack exposure to the variety of data collection and analysismethods used in empirical studies. The purpose of this paper is to provide a justification for theuse of empirical research in operations management, an overview of empirical research designsand data collection methods, and a tutorial in the mechanics of empirical data collection andanalysis. Where appropriate, selected examples from the operations management literature areprovided.

EMPIRICAL P/OM RESEARCH

Why Use Empirical Research in P/OM?

The gap between operations management theory and practice has been noted for some time(Buffa (198 l), Hax (198 l), Groff and Clark (198 l), Miller and Graham (198 l), Amoako-Gyampah and Meredith (1989)). Meredith, Raturi, Amoako-Gyampah and Kaplan (1989) arguethat P/OM research is not very useful to operations management managers and practitionersbecause it fails to recognize the applied nature of production/operations management. In recentmonths, the Journal of Operations Management has explicitly announced its receptiveness topapers which use empirical and field-based methodologies (Ebert 1990)). Information derivedfrom actual practice can enhance PiOM research in a number of ways. Gathering systematicinformation about practices in P/OM provides information about the state of the art in P/OM.Anecdotal articles may describe current practices at a single firm, however, systematic datagathering can provide more generalizable evidence about trends and norms in specific

populations of firms. This may be used to make inferences about firms in general, such as theHayes and Clark (1985) study on the effect of managerial policies on productivity, or about a

Journal of Operations Management 251



specific group, such as the Hyer (1984) survey of adopters of group technology. Systematic datagathering efforts also provide a baseline for longitudinal studies of P/OM practices (see Harmanand Tuma (1979) and Martin (1983)) before some anticipated change in an industry or in generalpractice. The Boston University manufacturing futures study (Ferdows, Miller, Nakane andVollmann (1986)) is an example of the establishment of a cross-industry baseline, withlongitudinal follow-ups, designed to examine changing manufacturing technology and strategy.

Empirical data can also be used in conjunction with simulation and mathematical program-ming research. In 1980, Buffa called for mathematical modelers to attempt to developobjectives, criteria and parameters from real systems. However, mathematical model distribu-tions and parameters are still frequently chosen to correspond with current modeling practice orare selected for computational convenience. Too often, it is forgotten that the results ‘ofmathematical modeling are only as valid as the assumptions upon which the model is based. It isvital that mathematical models be based on realistic, rather than simply convenient, assump-tions. For example, simulation studies frequently assume that job interarrival times are Poissondistributed and that processing times follow an exponential distribution, often without verifyingthis through examination of the actual values. Using empirically-based distributions as inputs tosimulation models can yield findings with greater external validity.

A very important use for empirical data in P/OM is theory building and verification. Theoryshould be developed from a careful, consistent documentation of actual practice and thesubsequent discovery of relationships between actual practice and plant performance. Anderson,Schroeder, Tupy and White’s (1982) MRP study provides an example of this. Theory can also beverified through the collection of empirical data, as illustrated by Roth’s (1987, 1989)manufacturing strategy studies. Causal relationships can be refined and explored throughsubsequent mathematical and simulation modeling research.

Why Isn’t There More Empirical P/OM Research?

There are a number of factors which may bias P/OM researchers against undertakingempirical studies. Such studies often require significant financial and time resources for sitevisits and data gathering. For example, researchers conducting survey studies must designreliable questionnaires based on valid constructs, before data collection begins. Lead times fordata acquisition can be substantial when several rounds of mailings or telephone contacts arerequired. Obtaining the commitment of respondents to participate can require as much time asinstrument design. Without such agreements, risks increase for inadequate response rates,jeopardizing the credibility of the research. Because they require less time and money,

traditional mathematical formulation and simulation studies are “safer” (Chase (1980)),particularly in the “publish or perish” environment of many research-oriented academicinstitutions. However, Chase (1980) states that, “ . . . we cannot avoid some high-risk researchif we are to capture the critical characteristics which are contained in the managementcomponent of the operations management field.”

The P/OM community has tended to view empirical research as less esteemed than researchbased on mathematical modeling. Part of the reason for this perception may lie in the fact thatsome OM researchers do not realize that empirical research can produce reliable insights tooperations management research issues, when conducted using existing data collection methodsand statistical analysis techniques.

252 Vol. 9, No. 2



A SYSTEMATIC APPROACH FOR EMPIRICAL RESEARCH

This paper describes a systematic approach for conducting empirical research which shouldhelp prevent P/OM researchers avoid the potential problems described above. Its components aredrawn primarily from the social sciences, where empirical research is the predominant researchmode. Our colleagues in organization behavior, psychology, marketing, anthropology and

sociology can be very valuable resources on conducting empirical research (see Weinberg(1983)). Although the methods are not unique to PIOM, the approach described in this paper canbe very useful in addressing P/OM issues.

Figure 1 provides an overview of the approach to conducting empirical research describedherein. In the first stage, the theoretical foundation for the research is established. Based on thetheory which underlies the problem being studied, either a theory-building or a theory-verification approach will be pursued. Next, a research design, which is appropriate to both theproblem and theoretical foundation, is selected. An overview of a number of research designswhich may be appropriate for empirical P/OM research is provided. The third section describesseveral data collection methods. One of these methods, or a combination of them, is used, in

conjunction with the research design. Implementation includes selecting an appropriate sample,designing and administering data collection instruments. The fifth step is processing andanalyzing the data. The final step is preparing the research report for publication. Underlyingeach of these steps are considerations of reliability and validity. Following some simplereliability and validity procedures at every step of the process provides more assurance that theresults of the study will be generalizable and will merit publication as a contribution to research.

ESTABLISHING THE THEORETICAL FOUNDATION

Theory provides the foundation for all scientific research. Although some may perceiveoperations management as being virtually atheoretical, in actuality, P/OM theories are all toooften implicit or difficult for the researcher to articulate. Empirical studies can be used to eitherbuild theory or to verify theory. It is important to determine, in advance, which of these is beingdone, since both cannot be accomplished in the same study.

Theory verijcation is the most widely understood approach. It is based on the scientificmethod, in which many OM researchers are grounded. Hypotheses are generated in advance ofthe study, and they are tested by the data collected. The origin of the hypotheses has historicallybeen of little concern to the researcher. Hypotheses may have been generated from prior studies,from the literature, or literally picked from “thin air.” Classical inferential statistics andsignificance tests are used to either accept or reject the hypotheses. The focus of theoryverification is on testing the hypotheses to within specified confidence levels, not on the originof the hypotheses.

A theory-building study is based upon a different origin and uses data in a different way.However, even in theory building, a priori theory or constructs provide the foundation (Glaser &Strauss (1967)). Without a conscious attempt at theory building, research can degenerate intosimply “data dredging,” or the ad hoc collection and analysis of data for whatever conclusionscan be found. Generally speaking, the origin for a theory-building study is not a hypothesis, butrather, some assumptions, frameworks, a perceived problem or perhaps, very tentativehypotheses. Proponents of the theory-building methodology (see, for example, Glasser and

Strauss (1967) and Yin (1989)) argue that a stronger theory will result if it has been grounded in




FG

ASY

MA

C

A

H

FO

EMPRC

R

E

H

Snec

su

b

Se

aDa

C

eo

Meh

Hsoc

acv

ay

s

P

cp

o

v

o

O

sd

o

v

o

Inevew

Q

o

e

C

e

ay

s

Im

em

ao

P

ao

se

o

Sm

ese

o

S

e

d

om

Q

o

e

c

u

o

Po

en

Man

A

yso

n

ep

c

aesc

Dae

y

R

a

y

a

v

dy

c

dao

u

e

a

sa



data, rather than if the origin of the theory is of little concern. They argue that data should alsobe used to build theories, not only to verify them.

For example, assume that the researcher begins with an initial theory, called Theory A. Thisinitial theory includes some suppositions and variables of possible interest. The researcher thenproceeds to collect data which would elaborate Theory A or suggest modifications to Theory A,but would not confirm it or deny it in any statistical sense. The notion of hypothesis testing isinappropriate for theory building, since probability distributions and even random selection ofthe sample points are not used. Rather, theory building is an interpretative exercise designed toproduce a theory for later testing. Theory building uses extensive questioning and strategicpositioning of the sample, in order to enrich the initial Theory A and to suggest modifications toit. At the end of the data collection phase, a new Theory B, which is grounded in data, isproposed. If appropriate, the new Theory B can be subjected to traditional theory verificationmethods, or it can be further enriched to Theory C, before verification.

An example of theory building might be helpful to further illustrate this discussion. Suppose,for example, that we start with Theory A, which suggests that there is a connection between theproduct and the production process used. We further posit that a high volume product, withlimited variety, will be made in a highly automated, assembly line process, while a low volumeproduct with high variety will be made in a process with less automation and a process layout.This Theory A is, of course, the familiar product-process matrix (Hayes and Wheelwright(1979)). This is all the theory which is needed initially, however, the theory could be somewhatmore elaborate, or even somewhat less.

The next task is to think about what data could help to elaborate this theory. First, we mightspeculate that the industry in which plants are located could affect this theory, so, a singleindustry, containing both extremes of products and processes, different levels of volumes, anddifferent levels of automation, would be selected. The next step is to select a sample which

appears to contain characteristics which would permit the researchers to elaborate this theory.These would be comprised of some companies with high volume products and automatedassembly lines and others with the opposite situation. Access to these companies allows testingand refinement of the definition and measurement of what constitutes high and low volumes,what constitutes high and low automation, and other measurements. It also enables examinationof other variables that might affect the product and process match, such as intensity ofcompetition, company strategy, etc.

The next step is to select some companies which appear to violate the proposed theory. Thesecompanies could include those who were operating on the margin of the industry or which hadgone bankrupt. They could also include some companies which were not aware of the

importance of product and process match. The use of companies which disprove the theorypermits enrichment of the theory, in order to explain when it does not apply. The goal of thistheory development is to gain an understanding of the processes in the organization whichproduce the observed effects, in terms of the proposed theory. The next step in theory building isto extend the theory, for example, by moving to another industry to further develop the theory orto provide support for the theory via hypothesis testing. This involves much larger samples andmuch more structured data collection methods, for hypothesis testing.

One caution in theory building is that researchers must consider that their theories will beunderstood by their subjects and others; the problem of “reactivity” occurs during and aftereach study. This is especially true in business-related research, such as action research, where

many academics may perform a consulting, as well as an investigative, role. This relationship




with subjects makes sound observation and careful theory development, based on empiricalresearch methods, of utmost importance.

Theories are never proven or disproven, but rather, are constantly evolving (Glasser andStrauss (1967), Strauss (1987)). Because of this, even theory verification is limited and must befollowed by further verification and elaboration. The value of theory building lies in itspermitting a wider range of observation and inquiry than the more traditional theory testingdoes. Theory building is very useful in fields, such as P/OM, which lack an established base oftheory and measurement methods.

SELECTING A RESEARCH DESIGN

The next step, after determining whether the study will build or verify theory, is to select aresearch design. Many designs can be used for empirical P/OM research. Although the survey isprobably most frequently used by P/OM researchers, there are a number of other designs,developed primarily in the social sciences, which may also be used to structure empirical P/OMstudies.

The Single Case StudyThe single case study documents, in detail, the operations of a single plant. It is different from

the familiar anecdotal “success story” article, in that it provides a careful and detaileddocumentation of practices, to be used as the basis for research. This may be used in conjunctionwith survey research, or some other type of comprehensive data gathering effort, to developexplanations for some of the findings on a more comprehensive basis. It is very different fromthe single case study which is used as the basis for MBA discussion classes.

Probably the most widely cited single case study in the P/OM held is Monden’s (1983) casestudy of just-in-time at Toyota. Examination of the rich details of this study has generated a largenumber of testable hypotheses about JIT, which provided the basis for a number of studies byother researchers. In another single case study of JIT, Schonberger (1982) provided a detailedanalysis of the Kawasaki plant in Lincoln, Nebraska. His work was a milestone in providingevidence that JIT could work with U.S. workers in a U.S. plant, and was not a culturally-limitedphenomenon.

Eisenhardt (1989) provides an excellent, step-by-step guide to using the case study researchdesign. Yin (1981, 1989), Glasser and Strauss (1967) and Glasser (1981) also provide someuseful guidelines. Although these sources focus primarily on multiple case studies, eachdiscusses single case studies, as well, and provides insights which are relevant to either single ormultiple case study research.

Multiple Case Studies

In multiple case studies, detailed information is gathered at each of several sites, although thesame information may not necessarily be gathered at each. Whatever is available at each site isdocumented, in as much detail as possible. In analyzing the data, similarities and differencesbetween the sites are noted and documented, to the extent possible.

Miles and Huberman (1984) provide some useful guidelines to the analysis of qualitative data,such as that obtained from multiple case studies. It may be possible to develop simple summarytables and figures, which show hypothesized relationships. If there are enough cases, limitedstatistical testing can be done. With only a few cases, no attempt is made to perform a statisticalanalysis of data, other than perhaps calculating a few descriptive statistics. However, when the

256 Vol. 9, No. 2



purpose of the multiple case study is theory building, confirmatory statistical analysis is notexpected or desired.

Multiple case studies can also be used for theory verification. Large samples are notnecessarily required; in theory verification, one case can falsify a hypothesis. If there are enoughcases, however, some forms of inferential statistical analysis are possible. For example, Pesch(1989) used 12 factory sites and 23 plant-within-a-plant units. This allowed a modest applicationof regression analysis and limited statistical testing. On the other hand, studying a population,rather than a sample, eliminates the need for inferential statistical analysis. In the case ofGarvin’s (1983) room air conditioner study, data was gathered at all of the major companies inthe industry. Thus, there was no need for statistical analysis because there was not a need. tomake inferences about the population; Garvin already had complete information about thepopulation.

Field Experiment

In a field experiment, the researcher manipulates some aspect (an independent variable) of thenatural setting and systematically observes the resulting: changes (Stone (1978)). For example, aresearcher might administer an attitude survey to workers in a plant before and after a pay-for-skill program is implemented. Field experiments have much greater external validity than labexperiments, because they take place in the natural setting. Because of their richness, fieldexperiments are useful in both building and verifying theory. However, the researcher’s limitedcontrol of the natural setting may preclude accurate conclusions about causality.

Panel Study

A panel study obtains the consensus of experts. It can be very useful in defining terms andmaking predictions. The Delphi method is probably the most frequently cited technique forpanel studies. Experts respond, in writing, to a series of questions. Their anonymous responses

are distributed to all members of the panel, who are permitted to revise their own responses insubsequent rounds. The rounds continue until consensus is reached. Panel studies areoccasionally used in operations management research, for example Groff’s 1989) study ofgroup technology practices and problems, Ettlie’s (1989) development of a P/OM researchagenda and Pesch’s study of the definition of the term “factory focus.”

Focus Group

A focus group is similar to a panel study, however, the group is physically assembled and eachresponse is given to the entire group orally, rather than in written form. Thus, the group is awareof the origin of the response. The group is given a set of questions, often prior to its gathering. It

meets with a facilitator, who asks the questions, allowing every member a chance to express hisor her opinion. Discussion is permitted, with the goal of reaching consensus. Topics appropriatefor a focus group are similar to those which are appropriate for a panel study.

Surveys

The survey is undoubtedly the most commonly used research design in operations manage-ment. It relies on self-reports of factual data, as well as opinion. One approach is to administer asurvey to a group which is homogeneous with respect to at least one characteristic, such asindustry or use of a common technology. Hyer’s 1984) group technology research provides anexample. She sampled only users of group technology, since the goal of this study was to define

the state of the art in group technology. Sampling manufacturing firms at random did not make




sense, because the likelihood of selecting more than a few users of group technology was low. Incontrast, when the focus of the research is generalizability to an entire population of firms,administering a survey to a large sample is a more appropriate approach, as in the BostonUniversity manufacturing futures research (Ferdows, et al. (1986)), which used samples inexcess of 500 respondents.

SELECTING A DATA COLLECTION METHODThe methods described in this section may be used alone, or in tandem, with most of the

research designs described above, to document what is being observed. A combination of datacollection methods to study the same issue, or “triangulation” (Jick (1979)), may be veryuseful. For example, a combination of questionnaires, structured interviews and archivalfinancial data could be used to determine the impact of JIT implementation on plantperformance. By providing several sources of verification, triangulation improves the re-searcher’s judgmental accuracy. Useful references on data collection methods include Emory(1980), Best 1970), Converse and Presser (1986), Stone (1978) and McGuigan (1978).

Historical Archive AnalysisHistorical archive analysis uses unobtrusive measures, including physical traces and archives

(Bouchard (1976)), often in conjunction with a single or multiple case study design. Abernathyand Wayne (1974) used archival analysis in their single case study of how the productionprocesses of the Ford Model T developed over time, demonstrating the limits to using thelearning curve for cost reduction. Archival data is unbiased, because the providers of it have noawareness of being observed. However, since the researcher does not control the environment, itmay be impossible to obtain the type of data desired. Therefore, the collection of archival data issometimes used in conjunction with a survey or panel study, to gather historical factual datafrom respondents.

Some archival sources exist because firms are required to file reports, which become publicinformation. For example, airlines file a great deal of operating information with the federalgovernment. Other firms also make operations data available to the public, however, little usehas been made of it in P/OM research to date. University reference librarians can be excellentguides to identifying untapped archival sources.

Participant Observation

Although not fully explored in an operations management context, participant observationhas been used occasionally. Participant observers become part of the process being observed, inorder to record what participants experience. Clearly, this is very valuable in theorydevelopment and hypothesis formulation. The participant observer is usually known to subjectsas a researcher. Only in cases where subjects would not accept researchers is participantobservation used covertly. For example, Runcie (1980) worked on an automobile assembly linefor five months and documented his experiences, using covert participant observation as the datacollection method in a single case study design. Participant observation is appropriate for singleand multiple case studies, as well as panels. Participant observation may also be used as acomponent of action research, where a person within an organization collects and analyzes dataregarding an ongoing change (Dubin (1976)).

Outside Observation

Outside observation uses an unbiased observer to collect data, often employing some methodsfor ensuring that data is collected systematically. Industrial engineering techniques, which are

258 Vol. 9 No. 2



typically used in the design of jobs, may be used for structuring the collection of research data.For example, unbiased observers could develop flow diagrams of traditional and JIT shops, inorder to determine whether JIT shops have simplified flows. A variety of charts may be used todocument interactions between workers in different types of industrial settings. Stopwatchstudies can also be useful, for example, to compare the amount of time a particular task requireswhen it is done repetitively, with the same task done as part of an enlarged job. Research designsappropriate for outside observation include single and multiple case studies, as well as panels.

Interviews

Interviewing involves mar? than talking with members of an organization and, perhaps,taking notes. There are several methods of interviewing which permit the use of an organizedapproach, without compromising the richness of conversation (see Bradburn (1983)).

Structured Interviews. A structured interview involves the use of a script, which specifiesquestions to be used. Other questions may be asked, as well, based on the direction of theconversation, however, certain questions are standard. Structured interviews permit somecomparison between interviewees, without sacrificing the depth of the personal interview.

Ethnographic Interviews. Ethnographic interviewing facilitates discovery of what is meant byspecific concepts. A hierarchy of questions is asked, beginning with a general question. Furtherquestions are framed based on the respondent’s answers to previous questions. Used inconjunction with pilot testing of a survey, ethnographic interviews can be used to validateresponse categories in questionnaires, or to indicate where improvement is needed. They arevery useful for discriminating between the myriad definitions of popular concepts, anddetermining when such categories coincide with concepts used in the hypotheses. Spradley(1979) and others (Pelt0 (1978), Narroll and Cohen (1973), Gregory (1983), Schall (1983)Sanday (1979)) provide good references on the ethnographic interview process.

Both structured interviews and ethnographic interviews are greatly enhanced by transcrip-tions. Most respondents do not object to being taped, and the quality of the interview is raisedsignificantly if the researcher does not have to take meticulous notes. Transcriptions can be usedby the research team to improve interviewing techniques, to detect the presence of leadingquestions on the part of the interviewer and to guard against selective memory. They may beused in conjunction with content analysis. The interview is taped, and a transcript is prepared.Content analysis then codifies the transcript, noting recurrent usage of a phrase or concept ofinterest. Hypotheses may be developed or tested, based on content analysis of the transcript.

Questionnaires

The questionnaire is most commonly used in survey research, however, it may also be used insingle and multiple case studies, panels and focus groups. Although P/OM researchers have usedquestionnaires as a data collection method with some frequency, many P/OM questionnairesappear to have been thrown together hastily, with little thought of reliability, validity orgeneralizability. For example, many P/OM surveys use questionnaires which were constructedwithout first articulating the hypotheses of the study. This paper, in subsequent sections,provides a foundation for the construction and administration of questionnaires so as tomaximize their reliability and validity.

IMPLEMENTATION

Since survey designs with questionnaires are the most commonly used approach in empiricalP/OM research, most of the remainder of this paper will be devoted to a discussion of




questionnaire construction and use. Some of this discussion may also be applicable to otherdesigns and data collection methods.

The foundation for questionnaire construction is the theory which underlies it. A question-naire should not merely consist of a series of “interesting questions,” but should be designed todevelop or test a theory. The theory should be carefully defined by reference to the literature andby logical thought. The resulting theory (a set of variables and relationships among thosevariables) can be depicted by a flow chart or diagram which shows the key relationships in thetheory. Useful comprehensive guides to questionnaire design and construction include McIverand Carmines (198 1) and Alreck and Settle (1985).

Population Selection

Much empirical research uses the corporation or the individual as its level of analysis. It maybe easier to obtain data at these levels, and many important problems deal with corporate orindividual issues. The plant level may also be appropriate for P/OM studies. For example, theMinnesota-Iowa State research on World Class Manufacturing (see Flynn, Bates, Schroeder,Sakakibara and Flynn (1989)) used the plant as the level of analysis because, although WorldClass Manufacturing is a strategic approach, many of its measurable improvement initiativeshave occurred at the plant level. Whether the individual, plant, division or corporate level isselected as the unit of analysis depends upon the research questions and hypotheses. SIC codesare commonly used to define industry classifications. However, SIC codes were not designed forP/OM research, and may be somewhat misleading. For example, process technology can varyconsiderably between two related SIC codes (e.g., computers are classified with machinery).There are other justifiable ways of choosing a sample, such as based on the use of a commonprocess technology. SIC codes can provide a useful starting point, however, their classificationsmay need to be modified, as appropriate to the needs of the P/OM researcher.

Dun’s Metalworking Directory 1986) is one of only a few sources which gives plant, ratherthan corporate, level information. It can be invaluable in obtaining addresses of plants and otherplant level information, such as products made and number of employees at that plant. Despiteits somewhat misleading title, this is a comprehensive reference which deals with most types ofmanufacturing.

Sample Selection

The sample should be selected as randomly as possible, in order to help control against bias.Convenience samples, for example, the sample of students in an executive MBA class, are highlybiased. Even when the sample is drawn from a specific group, such as a given industry or usersof a specific technology, the actual sample should be drawn randomly, once the master set ofnames has been obtained. Using SIC or Dun’s Metalworking Directory listings in conjunctionwith a random number table is a good way to ensure the randomness of the sample.

Controlling for industry effects can compensate for variability between industries, in terms ofprocess, work force management, competitive forces, degree of unionization, etc. This can bedone through the use of an experimental design which includes several industries. Within anindustry, the type of production process can vary widely between job shop, batch production andline production. Including the process as an independent variable will permit controlling for theprocess during data analysis. If the level of analysis is the plant or division, corporation effectsmay also be important. Company size is another critical variable. The number of employees and

total sales are widely available figures, which can be incorporated into the sample selectionprocess.

260 Vol. 9 No. 2



Scale Development

Appendix A contains a brief description of some potentially useful scales. Appendix Bcontains P/OM examples of each scale described in Appendix A. These tables are basedprimarily on information contained in Alreck and Settle (1985). The sophistication of dataanalysis is highly dependent upon the type of data obtained, thus, using Appendix A requiresconsideration of the type of data which each scale gathers.

Nominal data assigns observations to categories (Best (1970)). For example, respondents maybe asked to check the quality techniques they understand. Their choices cannot be placed in aspecific order. Ordinal data indicates relative rank, or order, among the categories. For example,respondents may be asked to rank their strategic manufacturing goals. Ordinal measures have noabsolute values, and the differences between adjacent ranks may not be equal. Interval data canbe ranked, and the differences between the ranks are equal. The widely used Likert scale is anexample of an interval scale. Interval measures may also be added or subtracted. For example,Likert scale responses are frequently added to form a summated scale. However, since a Likertscale has no true zero, responses cannot be related to each other’as multiples or ratios. Finally,

ratio data has all of the properties of the three types of data mentioned above, as well as a truezero and all of the qualities of real numbers. Thus, ratio data can be added, subtracted,multiplied or divided. It is mostly gathered from factual, archival sources; ratio scales designedto gather opinion data are not readily available. Because of their mathematical properties,making an attempt to obtain interval or ratio data, as much as possible, opens up a host ofanalytical techniques to the researcher.

Using items which are worded to assure comparability of responses greatly simplifies datainput and analysis. For example, rather than asking, “What is the defect rate of your primaryproduct?“, more comparable responses will be obtained by asking, “What is the defect rate, inparts per million, of your primary product?” Pilot testing, in conjunction with structured

interviews, panel studies or ethnographic interviews, can be extremely helpful in assuringcomparability of responses.

Summated Scales

A summated scale score serves as an index of attitudes towards the major issue or a generalconstruct. They are used because individual items tend to have a low statistical relationship withattributes. Summated scales permit averaging of the relationship with other items, and allowmore exact distinctions to be made between respondents. Their use also enhances the reliabilityof the responses.

In the initial development of a summated scale, the Thurstone approach (Alreck and Settle

(1985)) can be very useful. This method allows the inclusion of a small number of items whichhave high discriminating power, combined with high reliability. A list of a large number ofstatements which are related to the same construct or dimension is generated. They are thenrated by a group of respondents, known as judges, who are experts, or very well informed aboutthe construct. The judges are asked to rate each statement on an 1 l-point scale, ranging from“Strongly favorable statement” to “Strongly unfavorable statement.” Ten to 20 of thestatements are ultimately selected for inclusion in a summated scale, based on the criteria thatthey should each have a relatively low standard deviation of responses from the judges and therange of average responses for the ten to 20 items selected should be distributed evenly across all11 choices.




Questionnaire Construction

A portion (one-fourth to one-half) of the items in each summated scale should place thepreferred choice at one extreme, while the remainder places the preferred choice at the otherextreme (Alreck and Settle (1985)). This helps to keep the respondents interested in the itemsand prevents them from being lulled into marking “Strongly agree” for every item. Intermixingitems from a given summated scale with items from other summated scales will preventrespondents from guessing the construct which is being measured.

Record keeping is facilitated by the development of a master copy of the questionnaire, witheach summated scale identified. Of course, the questionnaires which are distributed to therespondents should not identify items by summated scale. Development of a master list of allsummated scales is useful in both developing the summated scales and analyzing the data. Thislists each summated scale in its entirety, its source and a list of the questionnaires which containthe summated scale.

Pilot Testing

Pilot testing is an integral part of questionnaire construction. It provides feedback on howeasy the questionnaire is to complete and which concepts are unclear or out of the respondents’range of knowledge and/or responsibility. For example, quantitative accounting data can be veryuseful to P/OM researchers, yet most plants exhibit a marked reluctance to divulge it. Throughpilot testing, the researcher may learn that respondents are more likely to provide accountingdata when they are instructed that giving rough estimates is preferable to leaving items blank.

Pilot testing consists of administering the preliminary questionnaire to a small group oftypical respondents. There is no need to select the respondents randomly; a convenience sample,such as students in an executive MBA class or members of the local APICS chapter, is quiteacceptable. By administering the pilot test in person, the researcher can determine whether there

are systematic differences between the way the researcher views specific measures versus therespondents.After pilot testing, questionnaires typically require revision, in order to help ensure the

validity and reliability of the measures, as well as making it more user-friendly. If the pilot testindicates that the questionnaire contains sensitive questions or that key variables are measureddifferently by most respondents, it may be necessary to consider using site visits to administerthe surveys to all respondents, rather than using a mail survey.

Mail Surveys

Mail surveys are very effective for well-defined research topics with a fairly narrow scope.

Such topics permit the use of a short questionnaire, which is more likely to be completed andreturned. A typical approach is to send questionnaires to a relatively large, randomly selectedsample and hope that an acceptable number are returned. Researchers in the social sciences lookskeptically at any survey with less than a 40% to 60% response rate. While studies have beenpublished in the operations management literature with response rates as low as 10% to 20%,such studies are highly unreliable, even if non-respondent bias has been checked. A higherstandard should be established, perhaps in the 50% response range, together with nonrespondentchecking, in order to ensure more reliable and, therefore, more generalizable results.

Researchers who send questionnaires to respondents who are to remain completely anony-mous can enhance their response rate by sending a reminder letter to all recipients of the

questionnaire, instructing them to disregard it if the questionnaire has already been completedand returned. However, because relatively high response rates are virtually required for

262 Vol. 9 No. 2



publication in high-quality journals, the researcher, despite his or her best efforts in sampleselection and questionnaire design, is left very vulnerable when a random, anonymous mailsurvey is used.

One effective means for increasing the response rate is to contact potential respondents andobtain their commitment to completing the questionnaire, prior to distribution. When therespondents understand the purpose of a study, lack of anonymity may not be as bothersome.This also facilitates provision of feedback to respondents, which may serve as an incentive forparticipation. This approach is being used by the Minnesota-Iowa State study of World ClassManufacturing (Flynn, et al. (1989)), which provides each plant with a profile of its responses,relative to the sample of other plants in its industry. In the case of a battery of questionnaires, theinitial contact can assign a site research coordinator to oversee the distribution and return of thequestionnaires. For example, Saraph, Benson and Schroeder (1989) appointed a researchcoordinator at each location to serve as the liaison between that location and the researchers,assisting with questionnaire distribution and collection. This may also be useful in gatheringmissing data when incomplete questionnaires are returned.

NonrespondentsIf those who chose not to respond are systematically different than those who did respond, the

generalizability of the results is compromised. For example, respondents from firms with qualityproblems may exhibit a marked reluctance to return a questionnaire on quality. Generalizingfrom those who chose to return this survey may not be truly reflective of the characteristics ofthe entire population. Analyzing important characteristics of nonrespondents can help todetermine whether they are systematically different from respondents. However, there is anobvious dilemma in attempting to identify characteristics of nonrespondents. Since they havechosen not to respond, there is little detailed data for determining their characteristics.Furthermore, when questionnaires are distributed anonymously, it will be impossible to evenidentify the nonrespondents.

If nonrespondents can be identified, they may be willing to answer a few brief questions onthe telephone. Another useful approach is demographic matching, using archival data todetermine whether there is a difference between respondents and nonrespondents. However,sampling the nonrespondents using a site visit, mail survey or telephone contact is the preferredmethod of nonrespondent analysis, especially when there is a large non-response rate (greaterthan 50%).

Data Entry

Careful examination of completed questionnaires, prior to data entry, can prevent subsequentdata analysis problems. Things to look for include incomplete or blank items, handwrittencomments about difficulty or interpretation of specific items and inappropriate responses, forexample, “Too many to count,” as a response for “How many suppliers did your plant dobusiness with last year’?”

It is unwise for members of the research team to perform the actual data entry task. Theintegrity of the data is vital to the generalizability of the conclusions of the study. Experienceddata-entry personnel assure input accuracy through the use of such devices as a “mask,” ortemplate, which makes the screen emulate a page of the questionnaire. Responses are enteredfrom the questionnaire, and the software positions the responses in the appropriate records in thedatabase. For large and complex databases, it is recommended that the research team also seekthe help of experts in database design; a well-designed database can save countless hours ofagony during data analysis.




DATA ANALYSIS

It is difficult, if not impossible, to draw conclusions from empirical data and to generalizethem, without the assistance of statistical evidence. Perhaps part of the reason why empiricalresearch in operations management is not held in the same esteem as other types of operationsmanagement research is that, with a few notable exceptions, its data analysis is relatively

unsophisticated. This is probably due to the predominance of nominal and ordinal data.Table 1 summarizes the characteristics of several methods which are useful in empirical data

analysis. Descriptive statistics are appropriate for any type of data. Statistical analysis ofnominal data is limited to categorical methods of data analysis. A number of nonparametrictechniques are also useful with ordinal data. Many other techniques of statistical analysis, usinginterval or ratio scales, are listed in Table 1. Since these methods are well known, no furtherdiscussion is provided here.

TABLESTATISTICAL TECHNIQUES FOR EMPIRICAL RESEARCH

M ethod

Descriptive Statistics

T-Test

Chi-Square Test

F-Test

Regression/Correlation

Path Analysis

Cluster Analysis

Factor Analysis

Purpose

Makes data moreintelligible

Compares twovariables

Goodness of fit, Testof homogeneity

ANOVA; 2-way, 4-way,n-way

Study relationshipsbetween variables

Establish causalinference

Define groups

Classify data

Samp l e Si ze

Small to large

Small to large

Small to large

Small to large

Small to large

Medium to large

Medium to large

Medium to large

Comments

Used to describeindustry practice.

Can compare only twovariables at a time.

Good for categoricaldata (e.g., Yes/No).

Yields interactions, aswell as main effects.

Allows specification ofdependent andindependent variables.

New to many OMresearchers. Powerfulfor theory verification.

New to many OMresearchers. Powerfulfor theory building.

Useful in establishingreliability and validity

It is tempting to statistically test for a number of differences, when analyzing survey results.This is appropriate, as long as it was predetermined in the design of the study and the nature ofthe Bonferroni problem of multiple comparisons is understood. As more comparisons are made,the likelihood increases that some of them will be statistically significant, based solely onchance. For example, if 100 comparisons are made at the 5% level of significance, an expectedvalue of five of them should be statistically significant, due to chance alone. This problem can be

dealt with by using 5% as the overall error probability, rather than the probability for individual

Vol. 9, No. 2



comparisons. The relationship between the overall significance and individual comparisons is asfollows:

P = 1-(1-P)”where: P = overall significance level

P = significance level of each test

n = number of tests madeFor example, if each test is made at the .05 level of significance and there are 50 tests made, thenthe overall significance of the results is 0.923. Conversely, in order to arrive at an overallsignificance level of .OS, an individual test level of about .OOl is needed for 50 tests.

RELIABILITY AND VALIDITY

The data collected by surveys and other empirical designs is of little use unless its reliabilityand validity can be demonstrated. Measurement papers, which discuss strictly the reliability andvalidity of survey instruments, have been very limited in P/OM research and are just beginning

to appear. Saraph, Benson and Schroeder’s (1989) quality measurement paper provides a goodexample. Measurement papers are important in disseminating reliable and valid instruments foruse by other researchers. Useful references on measurement, including concerns aboutreliability and validity, include Nunnally (1978), Mehrens and Ebel (1967), and Carmines andZeller (1979).

Reliability

Reliability measures the extent to which a questionnaire, summated scale or item which isrepeatedly administered to the same people will yield the same results. Thus, it measures theability to replicate the study. A non-reliable measure is like an elastic tape measure; the same

thing can be measured a number of times, but it will yield a different length each time. One ofthose measurements may, indeed, be the correct length, but it is impossible to determine which.Reliability is a prerequisite to establishing validity, but not sufficient (Schwab (1980)). If ameasure yields inconsistent results, even very highly valid results are meaningless.

There are a number of methods for measuring various aspects of reliability. When a measureis administered to a group of individuals at two different points in time and their scores at thetwo times are correlated, that correlation coefficient is a measure of test-retest reliability. Whileappropriate for physical measures, such as machine speeds, there are obvious problems withadministering the same questionnaire items to the same group of people at two different points intime. During the retest phase, the respondents will be answering questions which they havepreviously seen. Reflection or discussion with coworkers after the first phase may alter the way inwhich they answer questions during the retest phase. In addition, if a substantial period of timehas passed between test and retest, differences in scores may reflect actual changes which havetaken place, rather than unreliability. Parallelforms reliability can be established by constructingtwo equivalent (in terms of means and variances) forms of the same measure and administeringthem to a common set of subjects at different points in time. The correlation between the scoresis known as the parallel forms reliability estimate. This is particularly appropriate when bothforms of the measure will actually be used in data collection, such as in assessing jobsatisfaction before and after a technological change in the plant.

Internal consistency is important when there is only one form of a measure available, such asmost P/OM questionnaires. There should be a high degree of intercorrelation among the items




that comprise the measure or summated scale. Using the split-half technique, the items within asummated scale are divided into two subsets and the total scores for the two subsets arecorrelated. To the extent that the measure is internally consistent, the correlation between totalscores for the two subsets will be high. The most widely accepted measure of a measure’sinternal consistency is Cronbach’s Alpha (Cronbach & Meehl (1955)). Alpha is the average ofthe correlation coefficient of each item with each other item (Nunnally (1978)). It is popularbecause it incorporates every possible split of the scale in its calculation, rather than onearbitrary split, such as the split-half measure. Cronbach’s alpha is part of the standard reliabilitypackage in SPSSX and is quite easy to use. The minimum generally acceptable Alpha value is.70, however, Nunally suggests allowing a somewhat lower threshold, such as .60, forexploratory work involving the use of newly developed scales.

Validity

In general, validity measures two things. First, does the item or scale truly measure what it issupposed to measure? Second, does it measure nothing else? If either of these questions can be

answered “no,” the item or scale should not be used. Using an invalid item or scale is like tryingto measure inches with a meter stick; precise quantitative data can be collected, but it ismeaningless.

Content validity is a judgement, by experts, of the extent to which a summated scale trulymeasures the concept that it intended to measure, based on the content of the items. Contentvalidity cannot be determined statistically. It can only be determined by experts and by referenceto the literature. The Delphi method is a very useful means for establishing the content validityof items (for example, see Pesch (1989)). An extended literature search, particularly notingrecurring concepts, should also be used.

Content validity is, of course, very critical. If the content of a construct, or theory, is faulty,

no amount of reliability or construct validity will suffice. Researchers must carefully considercontent, in advance of data collection, by informed logical analysis, insight and theoryformulation. As noted above, this should be based on literature searches and expert opinion.Content validity can also be improved, over time, by theory building and theory verification.Well-done empirical studies should lead to evolving knowledge and a more sophisticatedunderstanding of content.

Criterion-related predictive) validity investigates the empirical relationship between thescores on a test instrument (predictor) and an objective outcome (the criterion). For instance, ifthe researcher is developing a quality measurement instrument, the instrument should accuratelypredict objective quality outcomes. The most commonly used measure of criterion-relatedvalidity is a validity coefficient, which is the correlation between predictor and criterion scores.A validity coefficient is an index of how well criterion scores can be predicted from theinstrument score. Therefore, computing the multiple correlation coefficient between theinstrument score and performance or outcome, and obtaining a high value, is an indication thatthe measurement instrument has criterion-related validity. Two techniques are generally used.These are simple correlation, for testing a summated scale with a single outcome, and canonicalcorrelation, for testing a summated scale, or a battery of summated scales, with multipleoutcomes.

Construct validity measures whether a scale is an appropriate operational definition of anabstract variable, or a construct. For example, job satisfaction is a familiar construct which isrelevant in operations management research. It cannot be directly assessed, but instead must be

266 Vol. 9 No. 2



inferred from scores on summated scales which purport to measure job satisfaction. Thesesummated scales comprise the operational definition of the construct, job satisfaction.Establishing construct validity is a difficult process (Schwab (1980)). Since the construct cannotbe directly assessed empirically, only indirect inference about construct validity can be made byempirical investigations. Appropriate construct validity procedures depend on both the nature ofthe construct specified and the hypothetical linkages which it is presumed to have with otherconstructs.

In examining the hypothetical linkages which a construct has with other constructs, it ishelpful to construct a “nomological network,” or a framework, to illustrate the proposedlinkages. Based on the nomological network, hypothesized linkages to other valid constructs canbe empirically tested. This is also helpful in providing clarification of the definition of theconstruct, itself. To demonstrate the linkages between the construct and other constructs, aseries of assessments should be made, examining the measure against criteria established frommultiple hypotheses, measures of alternative constructs and samples (Schwab (1980)). Thehypotheses should be a logical outgrowth of the proposed linkages illustrated by the nomologicalnetwork.

Factor analysis can also be useful in establishing construct validity. It may be used in twoways (Schwab (1980)). First, factor analysis is helpful in identifying tentative dimensions, aswell as suggesting items for deletion and places where items should be added. Conducting afactor analysis on a single summated scale will show whether all items within the summatedscale load on the same construct, or whether the summated scale actually measures more thanone construct. The results of such an analysis are likely to be sample specific, however, having alarge sample (ratio of respondents to items of IO: 1) can ameliorate this problem. Also, within-sample heterogeneity (age, sex, job level, etc.) can influence the resulting factor structure; thus,generalizations should only be made to similar populations. A second use for factor analysis in

establishing construct validity is in testing hypotheses in an already-developed scale. In thiscase, the factor analysis is conducted on the scales which comprise several summated scalessimultaneously. The researcher should specify both the number of dimensions in the constructand the specific items or scales which are hypothesized to load on those dimensions a priori.Comparing this with the dimensions and loadings from factor analysis will help in establishingconstruct validity of a previously-developed summated scale.

Because of the difficulty of establishing reliability and validity, it is very useful forresearchers to use summated scales whose reliability and validity have already been demon-strated. These are difficult to find for the technical aspects of operations management research,however, there is a wealth of behavioral measures available in such handbooks as Cook,

Hepworth, Wall and Warr (1981) and Price and Mueller (1986). Many of these are relevant totopics which are of interest to researchers in operations management.

PUBLICATION

Appendix C shows that there is a substantial amount of empirical research in operationsmanagement which has been published. In addition, a surprisingly large number of doctoralstudents have used empirical research for their dissertations, which, for the most part, are notreferenced here. Based on our search of recent volumes of relevant journals, Table 2 lists thepublication of empirical operations management articles, by journal. The largest amount of

empirical operations management research was published in Production and Inventory Manage-ment Journal and the tnternational Journal of Operations and Production Management. These




journals have a predominantly practitioner audience, although they are cited by academics, aswell. It is not surprising that empirical work would be of interest to their readers. The empiricalwork published in these journals tends to be highly descriptive, documenting current OMpractices. It is encouraging to note, however, that some more academic journals are contained onthis list, as well. The empirical research published by these journals has moved beyonddescription into the inferential mode. There is more hypothesis testing, and some of thesearticles use highly sophisticated and powerful statistical analysis. The fact that the moreacademic journals have not published as much empirical operations management work in thepast may be reflective of the relative sophistication of the studies which they have received andreviewed. It does not appear that academically-oriented journals are systematically biasedagainst empirical operations management research. This is very encouraging to researchers whoare committed to conducting empirical research and yet are reluctant to sacrifice the opportunityto publish in high quality academic journals.

TABLE 2

RECENT EMPIRICAL RESEARCH IN POM BY JOURNAL(1980-1989)

Journal Number of Articles

Producti on and Inv entor y M anagement JournalInt ernati onal Journal of Operations and Production M anagement.Journal of Operati ons M anagement.Decision Sciences.I nt erfaces..................................................

Int ernat ional Journal of Producti on Research.

Har vard Business Revi ew .

Other.....................................................

.......

.......

.......

.......

.......

......

......

191814I643

576

TABLE 3TOPICS OF RECENT EMPIRICAL POM RESEARCH

(1980-1989)

Topic Number of Articles

Manufacturing Strategy .........................Manufacturing Technology (FMSiCAMIMISISoftware)MRP ........................................Manufacturing Management .....................Quality/Quality Control/Quality Circles ............Just-in-Time ..................................Use of OR Techniques in POM ..................Productivity ..................................Production SupervisionInnovationFuture Manufacturing SystemsReliability and ValidityTeaching POMOther.........................

10

8

...... I

...... 5

...... 5

...... 5

...... 43

. 32

.,.............. 222

1876

266 Vol. 9, No. 2



Table 3 lists the topics of recent operations management publications, across journals.Manufacturing strategy and technology are the most popular topics. This is not surprising, sincethese are relatively new areas to the field of operations management and provide fertile groundfor theory building. However, this table also shows that a wide variety of topics has been studiedempirically. It illustrates that the use of empirical research is not restricted to a limited set ofoperations management topics, but, rather that virtually any of the traditional and morenontraditional areas of operations management can be enhanced by the collection and analysis ofempirical data.

CONCLUSIONS

This paper makes a number of points. In general, the development of the field of operationsmanagement will be enhanced by empirical work. All types of empirical research are needed.While questionnaires can be very useful, more in-depth site studies are needed, as well.Conducting empirical research is not any more difficult than other types of operationsmanagement research, once the researcher is familiar with how to conduct it.

In keeping with this conclusion, it is important for all of us to become more critical readers ofoperations management research. Empirical research can provide a strong foundation formaking realistic assumptions in mathematical and simulation modeling research in operationsmanagement. Models which are based on unsupported assumptions are no more justified thanempirical studies with weak methodology. When the assumptions are not realistic, the resultscannot be generalized.

Development of the theory base in operations management has long been neglected. In orderfor the field to develop and advance, careful attention needs to be paid to building and verifyingtheory. Empirical research provides a powerful tool for building or verifying theory. Furtherefforts are now needed to improve the quality of the empirical OM work published, particularlyas it relates to P/OM theory development and testing.

This paper presents a logical process for designing and conducting empirical research inoperations management, which was summarized in Figure 1. An important feature of Figure 1 isthat it makes a distinction between research designs and data collection methods. Selecting anappropriate data collection method to support the chosen research design is vitally important.An overview of a number of research strategies and data collection methods is presented here,along with a rationale for their use.

We have argued that the use of empirical research is well established in the literature oforganization behavior, sociology, marketing and other areas. Researchers in operations manage-ment should not be afraid to learn from their colleagues in other areas, where empirical researchis the norm. Many of the references for this paper are standard sources on empirical research inother fields. It is not necessary for operations management researchers to reinvent the wheel.

When questionnaires are used, selecting summated scales with established reliability andvalidity will make the findings much more credible. Operations management researchers do nothave the advantage of numerous sources of reliable and valid summated scales, as researchers insome other areas do. Thus, it is vital that operations management researchers share data onreliable and valid summated scales. There is a need for P/OM researchers to begin to publishmeasurement papers, which share summated scales, providing detailed data on their reliabilityand validity, as well as sharing the models and their underlying theory. This will facilitate the

establishment of a body of reliable and valid measurements for the field, as a whole.Unlike some types of mathematical research in operations management, empirical research




cannot be conducted in isolation from the real world. Industry contacts can be invaluable indoing empirical research. Keeping in touch with former students, attending APICS or OMAmeetings, etc., can be vital in finding industry experts, pilot testing and determining consensuson the meaning of terms. Having the support of an industry group, in writing, can greatlyenhance the response rate of a large sample questionnaire. Industry groups may also be willingto provide some financial support to help defray the out-of-pocket costs of conducting empirical

research.Finally, the distinction between the exploratory and confirmatory mode of research in

operations management is useful to operations management researchers. The focus of most OMresearch to date has been on confirmation. We need to do much more exploratory research, inorder to lay a foundation for our confirmatory research. In the long run, the results ofconfirmatory research will be greatly enhanced when effort has been initially put into theorybuilding.

It is hoped that this paper will generate more empirically based and high quality empiricalstudies in P/OM, as well as measurement papers. The trends in this direction are alreadypositive. We hope to help accelerate these trends.

ACKNOWLEDGMENT

This research was supported, in part, by grants from the Japan-U.S. Friendship Commission and the McKnightFoundation.

REFERENCES

Abernathy, W. J.. and K. Wayne. “The Limits of the Learning Curve.” Harvard Business Review September-October1974, 109-I 19.

Alreck, I , and R. Settle. The Survey Research Handbook. Homewood, IL: Richard D. Irwin Co., 1985.Amoako-Gyampah, K., and J. R. Meredith. “A Journal Review of Operations Management Research.” In Proceedings

Decision Sciences Insfitute. Vol. 2. Atlanta, GA: Decision Sciences Institute, 1989, 1021-1023.Anderson, J. C., R. G. Schroeder, S. E. Tupy, and E. M. White. “Material Requirement Planning Systems: The State of

the Art.” Production and Inventory Management Journal vol. 24 no. 4, Fourth Quarter 1982, 51-66.Ansari, A., and 9. Modarress. “The Potential Benefits of Just-in-Time Purchasing for U.S. Manufacturing.” Production

and Inventory Management Journal vol. 28 no. 2.. Second Quarter 1987, 30-35.Bahl, H. C. “Teaching Production and Operations Management at the MBA Level-A Survey.” Production and

Inventor? Management Journal vol. 30 no. 3, Third Quarter 1989, 5-E.Benson, I? G., A. V Hill, and T. R. Hoffmann. “Manufacturing Systems of the Future-A Delphi Study.” Production

and Inventory Management Journal vol. 24 no. 3, Third Quarter 1982, 87-105.Bessant, J., and 9. Haywood. “Experiences with FMS in the U.K.” International Journal of Operations and Production

Management vol. 6, no. 5, 1986, 44-56.Best, J. W. Research in Education. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1970.Blackstone, J. H., and J. E Cox. “Selecting MRP Software for Small Businesses.” Production and Inventory

Management Journal vol. 26 no. 4, Fourth Quarter 1985, 42-49.Blalock, H. M. Theory Construction: From Verbal to Mathematical Formulations. Englewood Cliffs, NJ: Prentice-Hall,

Inc., 1969.

Bouchard, T. J., Jr. “Unobtrusive Measures: An Inventory of Uses.” Sociological Methods and Research vol. 4 no. 3,1976.

Bradbard, D. A., E N. Ford, J. E Cox, and W. N. Ledbetter. “The Management Science/Operations Research Industrial-Academic Interface.” Interfaces vol. 17 no. 2, March-April 1987, 39-48.

Bradburn, N. M. “Response Effects.” In Handbook of Survey Research PH. Rossi, J. D. Wright, and A. 9. Anderson(eds.) New York: Academic Press, 1983.

Buffa, E P “An Empirical Study of Inbound Consolidation Opportunities.” Decision Sciences. 19. 1988 635-653.Buffa, E. S. “Commentary on ‘Production/Operations Management: Agenda for the ‘EOs’.” Decision Science vol. 12,

no. 4, 1981, 572-573.

270 Vol. 9, No. 2



Buffa, E. S. “Research in Operations Management.” Journal of Operations Management vol 1, no. 1, 1980, 1-7.Campbell, D. T., and J. C. Stanley. Experi mental and Qua si-Experim ent al Designs for Research. Boston: Houghton

Mifflin Company, 1963.Celley, A. E, W. H. Clegg, A. W. Smith, and M. A. Vonderembse. “Implementation of JIT in the United States.”

Journal of Purchasing and M at eri al s M anagement vol . 22 no. 4 1986 9-15.Chase, R. B., G. B. Northcraft, and Cl. Wolf. “Designing High-Contact Service Systems: Application to Branches of a

Savings and Loan.” Decision Sciences vol. 15, no. 4, 1984, 543-556.

Chase, R. B. “A Classification and Evaluation of Research in Operations Management.” Journal of OperationsM anagement vol. 1, no. 1, 1980, 9-14.

Chervany, R. F , and L. W. Scott. “A Survey of MRP Implementation.” Producti on and Inv entor y M anagement Journal

vol. 30, no. 3, Third Quarter 1989, 31-34.Clark, K. B. “Project Scope and Project Performance: The Effect of Parts Strategy and Supplier Involvement on Product

Development.” M anagement Science vol. 35, no. 10, October 1989, 1247-1263.Cleveland, G., R. G. Schroeder, and J. C. Anderson. “Production Competence: A Proposed Theory.” Decision

Sciences vol. 20 no. 4 Fall 1989, 655-668.Converse, J. M., and S. Presser. Survey Q uesti ons: Handcraft ing the Standar dized Questi onnai re. Beverly Hills: Sage

Publications, 1986.Cook, J. D., S. J. Hepworth, T. D. Wall, and l? B. Warr. The Experi ence of W ork : A Compendium and Revi ew of 249

M easures and Their U se. New York: Academic Press, 198 1.Crawford, K. M., J. H. Blackstone, Jr., and J. I? Cox. “A Study of JIT Implementation and Operating Problems.”

Int ernat ional Journal of Producti on R esearch vol. 26, no. 9, 1988, 1561-1568.Cronbach, L. J., and P E. Meehl. “Construct Validity in Psychological Tests.” Psychological Bull eti n vol . 52 1955,

281-302.Dale, B. G., and J. Lees. “Factors Which Influence the Success of Quality Circle Programmes in the United Kingdom.”

Int ernati onal Journal of Operations and Producti on M anagement vol . 5 no. 4, 1985, 43-54.Darrow, W. P. “An International Comparison of Flexible Manufacturing Systems Technology.” Interfaces vol. 17, no.

6, 1987, 86-91.De Meyer, A., and K. Ferdows. “Managerial Focal Points in Manufacturihg Strategy.” International Journal of

Production Research vol. 25, no. 11. 1987, 1551-1562.De Meyer, A., and K. Ferdows. “Integration of Information Systems in Manufacturing.” International Journal of

Operations and Producti on M anagement vol. 5, no. 2, 1985, 5-12.Dubin, R. “Theory Building in Applied Areas.” In Handbook of Industri al and Organizati onal Psyhology. M. D.

Dunnette. New York: Rand McNally, 1976.Dun’s Industri al Guide: The M etal w orki ng D irectory . New York: Dun and Bradstreet, 1986.Ebert, R. J. “Announcement on Empirical/Field Based Methodologies in JOM.” Journal of Operat io ns M anagement

vol. 90 no. 1, 1990, 135-137.Eisenhardt, K. M. “Building Theories from Case Study Research.” Academy of M anagement Revi ew vol. 14. no. 4.

1989, 532-550.Emory, C. W. Busi ness Research M ethod s. Homewood, IL: Richard D. Irwin. Inc., 1980.Ettlie, J. Personal communication, 1989.Feldman, H. D. “The Senior Production Manager Job: Comparisons Through an Assessment of Demands.” Production

and Inv entor y M anagement Journal vol. 27, no. 3, Third Quarter, 1985, 88-100.

Ferdows, K., and I? Lindberg. “FMS as Indicator of the Strategic Role of Manufacturing.” International Journal ofProduction Research vol. 25, no. 11, 1987, 1563.1571.

Ferdows, K., J. G. Miller, J. Nakane, and T. E. Vollmann. “Evolving Global Manufacturing Strategies: Projections intothe 1990s.” Int ernati onal Journal of Operations and Producti on M anagement vol. 6, no. 4, 1986, 6-16.

Flynn, B. B., K. A. Bates, R. G. Schroeder, S. Sakakibara, and E. J. Flynn. “World Class Manufacturing in the UnitedStates.” Proceedi ngs Decision Sciences I nsti tu te. New Orleans: Decision Sciences Institute, 1989.

Ford, E N., D. A. Bradbard, W. N. Ledbetter, and J. E Cox. “Use of Operations Research in Production Management.”Producti on and Inv entor y M anagement Journal vol. 28, no. 3, Third Quarter 1987, 52-59.

Gaither, N., and D. R. Fraser. “Setting Aggregate Inventory Levels: An Operations Management Decision‘?”Int ernati onal Journal of Operations and Producti on M anagement vol. 4, no. 1 1 1984, 18-33.

Garvin, D. A. “Quality on the Line.” Harvard Business Revi ew September-October 1983, 64-75.Geisler, E., and A. H. Rubenstein. “The Successful Implementation of Application Software in New Production

Systems. Interfaces vol. 17, no. 3, 1987, 18-24.




Gergen, K. J. “Correspondence versus Autonomy in the Language of Human Action.” In Methodology in SocialScience, D. W. Fiske and R. Q. Schweeler (eds.) Chicago, IL: University of Chicago Press, 1986.

Gilgeous, V “Aggregate Planning in UK Manufacturing Companies.” Int ernati onal Journal of Operations andProducti on M anagement vol. 7. no. 1, 1987. 50-61.

Glaser, B. Cl., and A. L. Strauss. The Di scovery of Grounded Theory : Str at egies for Q uali t ati ve Research. Chicago:Aldine, 1967.

Gregory, K. “Native View Paradigms: Multiple Cultures and Culture Conflict in Organizations.” Administrative Science

Quarterly vol. 28 1983 359-376.Groff, G. Personal communication, 1989.Groff, G. K., and T. B. Clark. ‘Commentary on ‘Production/Operations Management: Agenda for the ‘~OS’.” Decision

Science. vol. 12, no. 4, 1981, 578-581.Hamilton, S., and R. G. Schroeder. “Computer-Based Manufacturing and Accounting Systems for Smaller Manufactur-

ing Firms.” Producti on and Inv entor y Management Journal . vol. 25, no. 4, Fourth Quarter 1986, 92-105.Hannan, M. T., and N. B. Tuma. “Methods for Temporal Analysis.” Annual Review of Sociology, vol. 5, 303-1983,

328.

Hax, A. “Comment on ‘Production/Operations Management: Agenda for the ’80s’: Decision Sciences Vol. 12, no. 4,198 1, 574-577.

Hayes, R. H., and K. B. Clark. “Explaining Observed Productivity Differentials between Plants: Implications forOperations.” Interfaces vol. 15, no. 6, 1985, 3-14.

Hayes, R. H., and S. C. Wheelwright. “Line Manufacturing Process and Product Life Cycles.” Harvard BusinessRevi ew January-February, 1979.

Hitt, M. A., R. D. Ireland, B. W. Kats, and A. Vianna. “Measuring Subunit Effectiveness.” Decision Sciences vol. 14,no. 1, 1983, 87-102.

Horte, S. A., I Lindberg, and C. Tunalv. “Manufacturing Strategies in Sweden.” Int ernati onal Journal of Producti onResearch vol. 25, no. 11, 1987, 1573-1586.

HSU, J. I. S. “Equipment Replacement Policy-A Survey.” Producti on and Inv entor y M anagement Journal vol. 29, no.4, 1988, 23-26.

Huber. V L., and N. L. Hyer. “The Human Factor in Cellular Manufacturing.” Journal of Operations M anagement vol.5, no. 2, February 1985, 213-228.

Huck, S. W., W. H. Cormier, and W. G. Bounds. Readi ng Stat i sti cs and Research. New York: Harper and Row,Publishers, 1974.

Hyer, N. L. “The Potential of Group Technology for U.S. Manufacturing.” Journal of Op erat ion s M anagement vol. 4,no. 3, May 1984, 183-202.

Im, J. H., and S. M. Lee. “Implementation of Just-in-Time Systems in U.S. Manufacturing Firms.” InternationalJournal of Operat ions and Producti on M anagement. vol. 9, no. 1, 1989, 5-14.

Inman. R. A., and S. Mehra. “The Transferability of Just-in-Time Concepts to American Small Business.” Interfacesvol. 20 no. 2 1990 30-37.

Instone, E J., and B. G. Dale. “A Case Study of the Typical Issues Involved in Quality Improvement.” InternationulJournal of Operat ions and Producti on M anagement vol. 9 no. 1, 1989, 15-26.

Jick, T. D. “Mixing Qualitative and Quantitative Methods: Triangulation in Action.” Administrat ive Science Quarterly

vol. 24, no. 4, 1979, 602-611.Kerlinger, E N. Foundations of Behavi or Research. New York: HoIt, Rinehart and Winston, Inc., 1973.

Khan, A., and V Manopichetwattana. “Innovative and Noninnovative Small Firms: Type and Characteristics.”M anagement Sci ence vol. 35, no. 5, 1989, 597-606.

Kivijarvi, R., I? Korhonen, and R. Wallenius. “Operations Research and its Practice in Finland.” Interfaces vol. 16, no.4, July-August, 1986, 53-59.

Klein, J. A. “Why Supervisors Resist Employee Involvement.” Harvard Business Review September-October 1984,87-95.

Kulonda, D. J., and W. H. Moates, Jr. “Operations Supervisors in Manufacturing and Service Sectors in the UnitedStates: Are They Different?” Int ernati onal Journal of Operations and Producti on M anagement vol. 6, no. 2, 1986,21-35.

LaForge, R. L., D. R. Wood, Jr., and R. G. Sleeth. ‘An Application of the Survey-Feedback Method in a ServiceOperation.” Journal of Operat ion s M anagement vol. 5, no. 1, November 1984, 103-l 18.

LaForge, R. L., and V L. Sturr. “MRP Practices in a Random Sample of Manufacturing Firms.” Production and

Inv entor y M anagement Journal vol. 27, no. 3, 1986, 129-136.

272 Vol. 9, No. 2



LaForge, R. L., and L. R.

journal of operations management volume 9 issue 2 1990 [doi 10.1016%2f0272-6963%2890%2990098-x]...

Documents