rose mann vess ey

8/13/2019 Rose Mann Vess Ey

1/22

Rosemann & Vessey/Improving the Relevance of IS Research

MIS Quarterly Vol. 32 No. 1, pp. 1-22/March 2008 1

ISSUES AND OPINIONS

TOWARD IMPROVING THE RELEVANCE OF INFORMATIONSYSTEMS RESEARCH TO PRACTICE: THE ROLE OFAPPLICABILITY CHECKS1

By: Michael Rosemann

Queensland University of Technology

Brisbane, Queensland 4000

AUSTRALIA

[email protected]

Iris Vessey

The University of Queensland and

Queensland University of Technology

Brisbane, Queensland 4000

AUSTRALIA

[email protected] or

[email protected]

Abstrac t

This paper takes a first step in aiding researchers to improve

the relevance of their research to practice. By proposing that

Information Systems researchers conduct applicability checks

with practitioners on the research objects (for example,

theories, models, frameworks, processes, technical artifacts,

or other theoretically based IS artifacts) they either produce

or use in theory-focused research, our paper presents an

actionable, systematic approach to evaluating, establishing,

and further improving research relevance. Furthermore,

because it is an approach that can be conducted as an addi-

tional step either at the beginning or the end of the traditional

1Carol Saunders was the accepting senior editor for this paper. Ola

Henfridsson served as a reviewer. An additional reviewer chose to remain

anonymous.

research life cycle, it leaves untouched the rigorous methods

used to conduct the study, that is, it does not compromise

traditional research models.

The approach we propose is based on the analyses of three

dimensions of relevance that are critical to practitioners

attempts to internalize IS research findings (importance,

accessibility, and suitability), and a comprehensive set of

solutions that can be used to address them. Our analysis

reveals that the most critical dimension for practice is the

importance of the research to the needs of practice. The

solution we propose to address that need is to conduct an

applicability check on the research objects of interest. Theapplicability check forms an integral part of the research

process, either prior to or following engagement in a typical

research process. We present principles and criteria for the

conduct and evaluation of an applicability check, which is

primarily based on the focus group method, and secondarily

on a modified nominal group technique.

Keywords: Relevance, rigor, academic research, research

process, applicability check, focus group method, modified

nominal group technique

Introduction

Information systems research relevance has long been an

issue in the IS academic community. Research is relevant

when it addresses the needs of one of a number of different

stakeholders. Research could, for example, be relevant to

other researchers, to research and development organizations,

to boundary spanners such as consultants, to IS students, and


2/22


2 MIS Quarterly Vol. 32 No. 1/March 2008

even to society in general (Davenport and Markus 1999;

Davison et al. 2004). Like much of the literature on rigor and

relevance, we address relevance from the viewpoint of IS

professionals (see, for example, Applegate and King 1999;

Benbasat and Zmud 1999; Davenport and Markus 1999;

Kavan 1998; Senn 1998).

Relevance is essential for satisfying ISs practitioner con-

stituents whom academia will educate as the next wave of IS

professionals, as well as those who read reports of the

research, disseminate the knowledge, and provide leadership

to organizations on the effective management and utilization

of information technologies (Kavan 1998). Without research

outcomes relevant to practice, the very existence of a research

discipline could be questioned because the discipline could

well lack impact beyond its own (academic) community.

Establishment of rigor in research, on the other hand, is

essential for satisfying the traditional quality standardsdemanded of academic disciplines: IS academics need to con-

duct rigorous research to guarantee that it is of high quality

in order to establish credibility, to publish in high quality

journals, to attain tenure and promotion, and to compete for

research funding (Applegate and King 1999; Dennis et al.

2006; Galliers 1994; Robey and Markus 1998). Rigor is

manifested largely by using sound methodology (Benbasat

and Weber 1996).

The issue of IS research relevance is raised frequently within

the IS community, most often in the context of the debate on

rigor versusrelevance. Recent commentaries appear inMIS

Quarterly(Volume 23, Issue 1, 1999) and as part of specific

research methodologies (for example,MIS Quarterly, Volume

28, Issue 3, 2004, with its focus on Action Research).

Attention has also been paid to research relevance as an

important issue in its own right, as in the special issue of the

Information Resources ManagementJournal(Winter 1998),

and a number of papers in Communications of the AIS

(Volume 6, 2001). Note, also, that the 2007 European

Conference on Information Systems was subtitled Relevant

Rigour Rigorous Relevance, while the 2007 Australasian

Conference on Information Systems has the theme The 3Rs:

Research, Relevance and Rigour Coming of Age.

Notwithstanding Benbasat and Webers (1996) statements to

the contrary, it is not clear that the IS academic community

has made much progress in recent years in placing more value

on relevance. There appear to be three major issues. First,

there are few incentives for researchers to conduct research

that is relevant to practice due to the perception that it is not

valued by the top IS research journals. While the majority of

researchers most likely will not change the focus of their

research without changes to the incentive system, certain

researchers do choose to conduct research that is designed to

impact practice directly. A further group of researchers

would likely conduct relevant research if they could be

assured of maintaining rigor. While intellectual propertyarrangements could also be a disincentive to working with

practice, emerging research funding models based on assess-

ments of the research quality (rigor) andthe research impact

(on end users) of a research group (for example, the

Australian Research Quality Framework) would increase the

pressure to consider practitioners as relevant stakeholders in

the research (DEST 2006).

Second, there is a perception that it is impossible to attain

both rigor and relevance in research. The IS academic com-

munity often views rigor and relevance as conflicting research

objectives, that is, as the two extreme points of a continuum.

Achieving one is viewed as necessarily compromising theother (see, for example, Davenport and Markus 1999). Hence

the issue is perceived as that of rigor versus relevance. We

argue, however, that it is possible to conduct research that is

both rigorous and relevant (see, also, Benbasat and Zmud

1999; Fllman and Grnland 2002; Kock et al. 2002; Robey

and Markus 1998; Senn 1998).

Third, in addition to arguments internal to the IS academic

community, there is limited demand on the part of practice for

the outcomes of IS academic research. Evidence for this

assertion can be found in the rather low number of practi-

tioners who subscribe to IS journals or who attend IS con-

ferences such as ICIS. The limited exposure of IS research to

the practitioner community suggests that only a small portion

of the body of academic IS knowledge is disseminated and

evaluated for its potential relevance.

As can be seen in the numerous citations referenced here, the

IS community is concerned about the relevance of its

research. The objective of this paper is to propose appli-

cability checksas a way of allowing practitioners to provide

feedback to the academic community on the research objects

it produces or uses in theory-focused research. Applicability

checks are evaluations by practice of the theories, models,

frameworks, processes, technical artifacts, or other theoreti-cally based IS artifacts that the academic community either

uses or produces in its research.

Applicability checks could be conducted on emerging IS

research outcomes as well as on established research models,

for example, models such as DeLone and McLeans IS

success model (1992, 2003), the technology acceptance model


3/22


MIS Quarterly Vol. 32 No. 1/March 2008 3

(Davis 1989), and adaptive structuration theory (DeSanctis

and Poole 1994; Poole and DeSanctis 1990). Further, an

applicability check would improve future research by

incorporating learnings into revisions to theories or models,

as well as sensitizing the practitioner community to a body of

research. Hence applicability checks would aid in improvingthe relevance of research conducted over time.

It is important to emphasize that conducting an applicability

check on research objects leaves intact the normal research

process, thereby addressing relevance while still supporting

a rigorous research process. Further, our suggestion to con-

duct applicability checks responds directly to the statement by

Benbasat and Zmud (1999, p. 13) that the IS field does not

possess the evidence with which to illustrate the impact of its

research.This is an important question that academics

should investigate.

On Research Relevanceand Potential Solutions

While the dimensions that underlie research relevance are

well established in the literature, potential solutions have

received relatively little attention. Hence we present the

dimensions of research relevance briefly and explore the

potential solutions in more detail.

Dimensions of Research Relevance

Numerous publications have addressed the dimensions of

research that render it relevant to the practitioner community.

Perhaps the most visible is Benbasat and Zmuds (1999)

characterization of relevance as an articles content, with

dimensions of interesting, applicable, and current, and the

articles style, with the dimension of accessible. More

recently, Klein et al. (2006) used Benbasat and Zmuds

characterization as the starting point for a comprehensive

literature-based examination of relevance. This examination

resulted in the following three dimensions of relevance:

importance, accessibility, and applicability. We use Klein et

al.s three dimensions of relevance as our starting point for

characterizing research relevance and developing an approach

to improve it.

First, Klein et al.s view of the importance of research to

practice encompasses whether the characteristic or process

under consideration can be controlled within the organization,

whether it focuses on a key management issue, whether it

addresses a real-world problem, and whether it is timely.

Based on these criteria, we view research that is importantas

that which meets the needs of practice by addressing a real-

world problem in a timely manner, and in such a way that it

can act as the starting point for providing an eventual

solution.

Second, accessibility of research to practice encompasses

whether the research is understandable, readable, and focuses

on results rather than the research process (Klein et al. 2006).

Substantial evidence suggests that practitioners often perceive

research publications as being difficult to understand (see, for

example, Benbasat and Zmud 1999; Kavan 1998; Kock et al.

2002; Robey and Markus 1998; Senn 1998). The research

community has its own terminology and expends considerable

effort reporting on methodological aspects (for example,

significance tests, instrument validity, instrument reliability,

etc.) that are often impossible for nonacademics to under-

stand. Furthermore, academic papers refer to and build onrelated work, and often cannot be read in isolation from this

body of knowledge.

Third, applicability of research to practice encompasses

whether the published article is complete, whether it provides

guidance and/or direction, and whether it provides concrete

recommendations (Klein et al. 2006). We believe, however,

that even if practitioners believe that a research object is

important, they may not be able to apply it. Many proposed

models lack, for example, detailed insights into how their

tenets can be achieved in practice (for example, appropriate

top management support in critical success factor models).

We therefore envisage research as going through a step priorto applicability, which we call suitability. If research is

deemed to be importantto practice andsuitablefor meeting

the needs of practice, the research can be further elaborated to

render it applicable to practice.

Using the dimensions of importance, accessibility, and

suitability, we thereby view research relevance as the degree

to which practitioners can readily comprehend research as

promising a solution potentially applicable to a problem

existing in their organization.

Add ressing the Relevance ofResearch to Pract ice

Figure 1 presents a comprehensive characterization of the

ways in which research relevance can be improved based on

a hierarchy of three perspectives. Each of these perspectives

can impact positively the dimensions of relevance to prac-

titioners, that is, importance, accessibility, andsuitability.


4/22



Institutional Perspective

1. Institutional governance approach

2. Clinical approach

Project Governance Perspective

Forming alliances with

1. Companies

2. Research institutes

3. Professional societies

4. National and international funding bodies

Research Process Perspective

1. Identification of the research problem

2. Theoretical development

3. Research methodology

4. Data analysis

5. Communication of the findings

Institutional Perspective

1. Institutional governance approach

2. Clinical approach

Project Governance Perspective

Forming alliances with

1. Companies

2. Research institutes

3. Professional societies

4. National and international funding bodies

Research Process Perspective

1. Identification of the research problem

2. Theoretical development

3. Research methodology

4. Data analysis

5. Communication of the findings

Figure 1. Solutions for Improving Research Relevance

The Institutional Perspective

The institutional perspective, as the name suggests, refers to

mechanisms that institutions put in place to provide an

environment conducive to pursuing research that is relevant

to practice. By institutions, we mean structures and mecha-

nisms of social order and cooperation governing the behavior

of two or more individuals. Institutions are identified with a

social purpose and permanence, transcending individual

human lives and intentions, and with the making and

enforcing of rules governing cooperative human behavior

(en.wikipedia.org/wiki/Institutions, accessed July 10, 2007).

We can identify two major streams of thought based on the

institutional perspective leading to increased research rele-

vance: the institutional governance approach and the clinical

approach.

One example of the institutional governance approach is

found in the 1996 report of the Leadership Task Force of the

AACSB (American Assembly of Collegiate Schools of Busi-

ness), which presented a number of strategies for dealing with

issues surrounding the relevance of business research to

practice. The major focus is on establishing a substantially

broader approach to the conduct of academic research in an

attempt to make business school research relevant to practice.

In Table 1, we present those aspects of the report that apply

to IS/IT research, as presented in Saunders (1998).

Other institutional-level approaches include the Research

Assessment Exercise (RAE) in the United Kingdom and the

Research Quality Framework (RQF) in Australia. The RAE,

introduced in 1986, is a research funding model that takes into

account both the quality of the research per se and the quality

of the research in relation to practice. The RQF, which is due

to be introduced into Australia in 2008, places a greater em-

phasis on research relevance (termed impact[DEST 2006]).

The adoption of such frameworks increases the pressure to

view practitioners as stakeholders in research that academics

conduct.

The clinical approach is based on Moodys (2000) analogy of

IS to medicine. Moody argues that the applied discipline of IS

should be patterned on the applied discipline of medicine.2

The medical discipline is characterized by a high level of inte-

2Note that arguments made in the context of medicine are relevant to other

applied disciplines, for example, architecture.


5/22



Table 1. AACSB Recommendations for Improving the Relevance of Business School Research,Adapted to IT Research (Based on Saunders 1998)

Develop Closer Links to Business and Technology

Sabbaticals in corporations

Industry-based projects for students

Internships for junior faculty Business consulting

Building partnerships and alliances with business groups

Improve Faculty Skill Levels

Re-evaluate tenure criteria

Realign faculty reward processes

Communicate standards for evaluation

Revise Ph.D. Program Requirements

Require at least a minimal level of business experience and managerial involvement as a requirement for

admission or as a supplemental part of doctoral programs

Adopt a strategy that interdisciplinary dissertations and studies of actual business practices are viewed positively

within the dissertation process

Develop business experience of students in doctoral programs

Form Partnerships with Professional and Discipline-Based Organizations Individual schools, and the AACSB, should encourage disciplinary-based academic organizations to include

practitioners in their annual meetings to help define new issues of which the membership would be aware

gration between research and practice that melds the two into

a single community, perhaps via joint universityindustry

appointments. In this instance, the researcher is also a prac-

ticing clinician. Further, the medical discipline does not

distinguish between academic and practitioner journals and

conferences. It places equal weight on the rigor and relevance

of its research, and requires articles to be easily under-

standable to practitioners.3

The approaches presented here recommend several ways of

improving the interaction between academia and business. If

adopted, such approaches could result in major refocusing,

and perhaps upheaval, not only among IS faculty, but also

within universities.

The Project Governance Perspective

The project governance perspective reflects the influence of

business on research projects via the formation of different

kinds of research alliances. This influence appears to be

operationalized almost exclusively via the funding of researchprojects that address the funding sources needs. These

approaches tend to ensure that the research findings are at

least relevant for the industry partner(s). As Table 2 shows,

those alliances may be built at a number of different levels:

(1) specific companies, which provide various types of

research funding; (2) research institutes, either university or

nationally based, which attract a select body of sponsors who

have first right to research findings in return for monetary

support; (3) professional societies, which use a similar

funding model to that used by research institutes; and

(4) national or international funding bodies, which typically

make cash contributions to collaborative research projects that

can then be used to leverage corporate contributions. Table 2

also provides examples of alliances that are intended to be

illustrative rather than comprehensive. Note that there are

numerous regional differences within these types of research

collaborations (see, also, Lyttinen 1999).

The Project Research Process

Because the research process perspective is totally under the

control of the researcher, it can be used to address relevance

directly. We identify a number of ways in which IS re-

searchers may make their research more relevant to the IS

practitioner community based on the research process. Toorganize these potential solutions, we introduce a typical

research life cycle and characterize these approaches in terms

of the phases involved: (1) identification of the problem;

(2) theoretical development; (3) research methodology in-

cluding data collection; (4) data analysis; and (5) communi-

cation of the findings. Figure 2 presents our model of the

research life cycle, while Table 3 uses the model to present

our analysis of suggested solutions.

3See Introna and Whittaker (2004) for an in-depth analysis of why this

approach was tried and ultimately discontinued at theMIS Quarterly.


6/22



Relevance

Identificationof the Research

Problem

Theoretical

Development Data

Analysis

Research

Methodology

Communication

of the Findings

Rigor Time

Relevance


Problem


Problem

Theoretical

Development Data

Analysis

Research

Methodology

Communication

of the Findings

Rigor Time

Table 2. Examples of Types of Research Governance

Level Examples

Companies Scholarships, chairs, projects funded by IT users, IT vendors, IS/IT consultants (Markus

and Robey 1998)

Cooperation with corporate research departments (e.g., IBM, Hewlett Packard, SAP,Infosys, Microsoft)

Research Institutes University-based research centers with close industry relationships

" Babson College: Process Management Research Center; Working Knowledge

Research Center

" MIT: Center for Information Systems Research

" University of Arkansas: RFID Research Center

" University of Memphis: Center for Supply Chain Management

Nationally based research centers with close ties to industry

" NICTA, Fraunhofer Gesellschaft

Professional Societies Society for Information Management Advanced Practices Council

National and

International FundingBodies

National or international funding schemas that leverage the involvement of at least one industry

partner with substantial cash and in-kind contributions in a research project European Framework Programme for Research and Technological Development

Australian Research Council: Linkage Grants and Cooperative Research Centers

Figure 2. The Research Life Cycle


7/22



Table 3. Approaches to Improving Research Relevance Based on the Research Life Cycle

Identification of the Research Problem

Seek cu rrent pract i t ioner issues

Be aware of dominant issues; for example, as presented in the annual report, Critical Issues of Information

Systems Management (CSC), and critical issues studies conducted by IS researchers (Senn 1998) Be aware of topics that are the focus of practitioner initiatives (Lyttinen 1999); for example, the SIM Advanced

Practices Council grants (Senn 1998)

Initiate, moderate, or participate in communities of practice (see, for example: www.bpm-roundtable.com)

Conduct major issues studies using focus groups, interviews, surveys in order to formally derive an industry-driven

research agenda

Focus o f the research

Focus on needs of practitioner audience (Robey and Markus 1998) not only on academic interests

Focus research on a theme (Senn 1998); however, Robey and Markus (1998) question the viability of such an

approach, given rapid technological advances

Avoid moving from issue to issue without producing principles, prescriptions and well formulated theories that

create at best fragments of research not meaningful solutions (Senn 1998)

Theoretical DevelopmentAdop t new mo dels of research (Robey and Markus 1998)

Applied theory research; using an established theory to address a problem in practice (industry-driven, not

curiosity-driven)

Evaluation research, which is not necessarily based in theory, involves examining a practical intervention using a

well-established, rigorous research approach

Policy research, which is commonly found in schools of political science and government, focuses on a broad area

that requires resolution; such studies are rarely theoretically based

Use useful logic and theory (Robey and Markus 1998)

Embrace the role of non-deterministic theories; consider using contradictory logic (Robey and Markus 1998)

Seek ob ject ive f indings rather than on es that conf irm assump t ions and bel iefs (Senn 1998)

Ensure that you have action levers (Robey and Markus 1998)

Research Methodology

Form al l iances with pract i t ioners

Practitioners will serve as critics of theory, research approaches, and findings (Senn 1998)

Involv e the practit ion er directly in the research (Kavan 1998)

Create win-win situations with the practitioner (see, for example, the SIM-APC model referenced above) (Kavan

1998; Senn 1998)

Become sensitive to the nuances of corporate culture (Kavan 1998)

Develop an appropr iate solut ion p rocess

Use pilots, appropriate research methodologies (e.g., case studies, focus groups, action research), prototypes

(Kavan 1998)

Continue trend toward qualitative research (Kavan 1998)

Ensure that the research (particularly data collection) involves sufficient numbers of appropriate practitioners

(Senn 1998)

Data Analysis

Ensure you h ave credible evidence to supp ort claims (Robey and Markus 1998)

Describe evidence and the processes for generating it in ways that can be believed by practitioners (Robey and

Markus 1998)


8/22



Table 3. Approaches to Improving Research Relevance Based on the Research Life Cycle (Continued)

Communication of the Findings

Produce cons umable research reports

Use an accessible style tailored to the target audience (Benbasat and Zmud 1999; Kavan 1998; Kock et al. 2002;

Robey and Markus 1998; Senn 1998)

Produce reports that are shorter and more concise, more clearly written, and much better illustrated than those

prepared for journals (Benbasat and Zmud 1999; Robey and Markus 1998; Senn 1998), have minimal statistical or

mathematical detail, and few citations in the text (Markus and Robey 1998)

Focus on results and their implications more than on process (Senn 1998)

Use a top-down orientation (Kavan 1998)

Use novel, critical, constructive story lines (Robey and Markus 1998)

Ensure that you have useful implications for practice that include action levers for practitioners (Robey and

Markus 1998)

Produce multiple versions of research reports tailored for different audiences (Senn 1998)

Expand publ icat ion out lets

Choose more practitioner-oriented journals such Sloan Management Review, Communications of the ACM,

Harvard Business Review, MIS Quarterly Executive,and journals that are distributed to the members of

professional associations (Robey and Markus 1998; Fllman and Grnland 2002; Kock et al. 2002) Have trade journals carry abstracts of relevant academic research (Kavan 1998)

Forge better partnerships between research outlets and the vendor community (Kavan 1998)

Present a related body of research in a form suitable for practitioners; for example, an edited volume with original

contributions on a particular theme; individual universities have also taken this approach (Robey and Markus

1998)

Use traditional practice reports, management briefs, white papers, and the Internet (Senn 1998)

Produce special issues on topics of interest to IS managers and hold themed conferences with both academics

and practitioners (Kavan 1998)

Use the Internet to shorten publication time (Robey and Markus 1998; Senn 1998); to distribute research findings

to practitioners (Moody and Shanks 2000; Senn 1998); and to make data available (Mandviwalla and Gray 1998)

Write a popular book (Robey and Markus 1998)

Note that Figure 2 presents the research life cycle phases on

a continuum from rigor to relevance along the Y axis: those

phases further from the origin are those that manifest greater

research relevance. The two phases that have the most

significant impact are those in which researchers seek to

reflect the needs of practice. First, researchers may seek the

help of practitioners in identifying research problems that are

grounded in current practice and deal with important issues,

rather than being driven by curiosity or opportunity. Second,

researchers interested in establishing the relevance of their

research communicate their research findings through

accessible channels in a way that is easily understandable to

practitioners. Between these two phases, there are severalphases (theoretical development, research methodology, and

data analysis) that may be largely invisible to practitioners.4

Hence these are the phases in which researchers should seek

rigor.

It is immediately apparent from Table 3 that there is a

plethora of recommendations for how to overcome the per-

ceived problems with IS research from the viewpoint of its

impact on practice in each of these phases. By far the greatest

number is focused on communicating research findings.

These recommendations take two forms: producing con-

sumable research reports and expanding publication outlets.

We also identified a number of suggestions for increasing

research relevance in the identification of the research topic,

theoretical development, and research methodology phases of

the life cycle. Recommendations for increasing the relevance

of data analysis were almost nonexistent, perhaps because

data analysis is not controversial.

Improving Research Relevance UsingApplicability Checks

In the sections that follow, we distinguish between the appli-

cability check approach, which presents the generic solution

4Note that collecting data from professionals does little to improve research

relevance.


9/22



approach we are proposing, and the applicability check

method, which presents a detailed process for conducting and

evaluating applicability checks.

In this section, we first motivate the use of applicability

checks as an additional step at either the beginning or the end

of a research life cycle. Given that applicability checks may

not be appropriate in all circumstances, we then characterize

the situations in which their use is appropriate.

Theoret ical Foundations of

App l icabi l i ty Checks

The dimensions of research relevance and their potential

solutions are the key to using applicability checks. From the

perspective of the dimensions of research relevance, we

sought to identify the dimension that needs to be satisfied for

us to even concern ourselves with the relevance of the

research conducted. It is our belief that the crucial conditions

for producing relevant research are, first, whether the research

is important to practice, and second, whether the research

ultimately could be suitable for applying in practice.

Accessibility, which refers largely to the presentation style

rather than the substance of the research, is not critical to the

research per se, though, of course, it is critical to communi-

cating it. Further, suitability of the research to practice

becomes relevant only when the research has already been

deemed to be importantto practitioners. Hence we focus here

on the importance of research to practice.

From the perspective of choosing a solutionthat evaluates the

importance of research to practice, the institutional and

project governance perspectives presented in Figure 1 are not

under the control of the researcher, and do not play a large

role in current IS research. We therefore consider solutions

that the researcher can control and that can impact research

today. These are solutions based in the research process

perspective.

Perusal of Figure 2 shows that our research life cycle does not

address whether the research is important to practice. We

propose addressing this dimension by evaluating the impor-

tanceof the research object under investigation in a further

step in the life cycle. A major advantage is that such an

approach would leave untouched the traditional research life

cycle, with its emphasis on rigor, thereby avoiding major

discomfort for those researchers reluctant to seek relevance

due to the perceived concomitant reduction in rigor. We

therefore need to develop a way to determine the importance

of research to practice.

In Figure 3, we show an applicability check occurring

between two instantiations of the research life cycle. Such a

check could therefore be conducted as (1) the final step in the

research life cycle; (2) the first step in the life cycle; or (3) at

the end of the first instantiation of the life cycle, leading into

the second instantiation. The first possibility focuses on

research just concluded. The second possibility is based on

the existence of relevant prior research outcomes. The second

and third possibilities both focus on the early phases of a

research project, including the formulation of a new model,

etc., in a complete research life cycle. In these instances, the

applicability check would therefore lead to better research

models, the development of more appropriate hypotheses, and

so on.

Our emphasis on evaluating research objects means that our

primary focus in this context is on the theory development

phase of the research life cycle. Hence, applicability checks

per se may have only an incremental impact on the identi-fication of the research problem, for example.

When to Conduct an App licabi l i ty Check

There are two situations to consider in establishing when to

conduct applicability checks: (1) timing in the context of the

extended research life cycle (process perspective) and (2) the

types of research that lend themselves to evaluation via

applicability checks (cross-sectional perspective).

Timing of an Applicability Check in theExtended Research Life Cycle

From a process perspective, there is a continuum of possibi-

lities for when an applicability check that follows a traditional

research life cycle can be conducted; note, therefore, that this

consideration applies to possibilities one and three as pre-

sented in the discussion of the extended research life cycle in

Figure 3. The continuum is determined by the level of

maturity of the research object at the time the applicability

check is conducted. For example, at one extreme, researchers

could conduct applicability checks on research objects as soon

as their findings are known, and therefore prior to publication,

in which case there might be no publication to be shared withpractitioners. At the other extreme, applicability checks could

be conducted on mature IS models such as DeLone and

McLeans (1992, 2003) IS success model.

The timing of the applicability check has important implica-

tions for the relevance dimensions that researchers can

evaluate. An applicability check conducted on mature, pub-

lished research, can evaluate a research objects importance,


10/22



Relevance

Identification

of the Research

Problem

Theoretical

DevelopmentData

Analysis

ResearchMethodology

Communication

of the Findings

Rigor

Time

Applicability

Check

Identification

of the Research

Problem

Theoretical

DevelopmentData

Analysis

ResearchMethodology

Communication

of the Findings

Research Lifecycle 2Research Lifecycle 1

Relevance

Identification

of the Research

Problem

Identification

of the Research

Problem

Theoretical

DevelopmentData

Analysis

ResearchMethodology

Communication

of the Findings

Rigor

Time

Applicability

Check

Identification

of the Research

Problem

Theoretical

DevelopmentData

Analysis

ResearchMethodology

Communication

of the Findings

Research Lifecycle 2Research Lifecycle 1

Figure 3. The Extended Research Life Cycle

accessibility, and suitability to practitioners. On the other

hand, lack of related publications when conducting an appli-

cability check on a newly established research object would

preclude the practitioner audience from evaluating the

articles accessibility.

Boundaries of Applicability Checks

We now examine the types of research objects that are, or are

not, suitable for examination using applicability checks. First,

in suggesting that applicability checks should be conducted on

research objects, we are making the implicit assumption that

we are interested in examining theory-focused research and

not research that is descriptive in nature or that addresses

meta-issues within the IS community such as research on the

IS academic discipline itself (see, for example, Vessey et al.

2002).

Second, when the research objects are extremely complex (for

example, deeply theoretical or mathematical in nature) or are

not specified sufficiently well, they may not be appropriate to

examine via applicability checks. Hence prior to conducting

an applicability check, the research object may require prior

elaboration by a research and development group or pro-

fessional boundary spanners such as consultants.

Third, applicability checks are unnecessary in studies that are,

in some sense, sanctioned by practice. One such type of

study is action research, which involves the conduct of a real-

world practitioner project by a team comprised of both

researchers and practitioners (see, for example, Avison et al.

1999; Baskerville and Myers 2004; Davison et al. 2004; Kock

et al. 2002). The role of the researchers is to apply and

further develop theory that provides direction for the project.

Hence, the symbiotic nature of the intervention results in

continuous feedback between researchers and practitionersrendering an applicability check unnecessary.

Fourth, we now consider the relevance of applicability checks

in intensive and case study research. In seeking to understand

a real-world phenomenon, research in behavioral IS typically

produces or uses abstractions presented in the form of

theories, models, frameworks, processes, or technical artifacts

that describe that phenomenon. While this may be the tradi-

tional realm of hypothetico-deductive (positivist) research, it

is also true of intensive research (Markus and Lee 1999),

irrespective of whether its origins are positivist or interpretive

in nature.

Intensive studies engage the researcher in an on-going, in-

depth change process using methods such as ethnography,

grounded theory, and participant observation, and thereby

provide the researcher with ample opportunity to augment

his/her confidence in what is to be reported. However, such

studies most often also produce research objects, typically in

the form of models (see, for example, Schultze and Orlikow-


11/22


12/22



(1) Planning the applicability check: The research object

under consideration, objectives of the applicability check,

and information needed are clearly specified. Further,

the research question clearly identifies the research

objective, target population, and specific issues to be

addressed.

(2) Selecting the person to conduct the check: The person

conducting the check has in-depth knowledge of the

research under investigation, as well as having significant

social skills.

(3) Ensuring that participants are familiar with the

research object under examination:Participants in the

check are provided with materials that introduce the

research, the research object under investigation, and the

implications of the object from the viewpoint of practice.

Prior to the check proper, the person conducting the

check ensures that each participant is sufficiently wellinformed to take part in the projected evaluation.

(4) Designing the materials for conducting the check:

The applicability check method adheres to well-

established design criteria. The format of the pretested

questions for the study, the sequence of their presen-

tation, and an agenda that can fit into the time allotted are

specified.

(5) Establishing an appropriate environment for con-

ducting the check: The person conducting the check

creates an environment for running the check that is

conducive to a fruitful interaction.

(6) Conducting the check: The person conducting the

check presents the agenda and the ground rules for con-

ducting the check, then ensures that the check is con-

ducted in a professional manner that results in unbiased

input from all participants.

(7) Analyzing the data: Procedures for analyzing qualita-

tive data are used to analyze the data derived from the

check. Multiple sources of evidence are used, data is

coded, and a trail of evidence is provided from the raw

data to the final outcomes.

Requirements for the ApplicabilityCheck Solution

The focus group method meets all but one of the above

criteria for an applicability check method. We therefore pro-

pose using focus groups as the foundation for conducting an

applicability check (Krueger and Casey 2000; Morgan 1998;

Stewart et al. 2007). The requirement not met by focus

groups is that for initial knowledge sharing between

researcher and practitioner regarding the research object, its

objectives, and implications. We propose that this phase be

accomplished using the nominal group technique modified for

use in combination with focus groups (Stewart et al. 2007, pp.

153-154). It would be implemented as a single step in the

conduct of the applicability check method (Step 3), prior to

the conduct of the focus group proper.

The focus group method, which is a qualitative approach to

behavioral science research, consists of group interviews that

involve a small number of appropriate persons discussing the

topics raised by a moderator who guides the interview process

(Morgan 1998, p. 1). Focus groups have been widely used in

a variety of disciplines such as marketing, clinical psychol-

ogy, sociology and social psychology, and communication,

among others (Stewart et al. 2007, pp. ix). Because partici-

pating group members will be selected IS practitioners with

a common understanding of the issue addressed in theresearch, the groups will be quite homogeneous and the

applicability check can therefore be more focused than the

majority of typical focus groups (Stewart et al. 2007).

The modified nominal group technique (NGT) consists of

either interviews with non-colocated individuals and subse-

quent sharing of summaries of responses and ideas of other

team members with the group (Stewart et al. 2007, pp. 153-

154), or face-to-face discussion followed by individual,

private responses to the questions of interest (Delbecq et al.

1975). Because the modified NGT can be used without direct

interaction among group members, we propose using it to aid

focus group participants in gaining an initial understanding ofthe research under investigation prior to examining the issues

in a face-to-face focus group setting. We also propose eval-

uating the familiarity of the participants selected with the

research object under investigation prior to their participation

in the check proper.

Appendix A presents an initial example of using an

applicability check to evaluate the relevance to practitioners

of a popular IS research model, DeLone and McLeans IS

success model (see, also, Rosemann and Vessey 2005). Our

experiences with this exploratory study, which used focus

groups alone, led us to introduce the modified NGT to allow

applicability check participants to gain exposure to theresearch object of interest.

Specifications for the ApplicabilityCheck Method

Our approach to developing a method for conducting appli-

cability checks is analogous to that of Davison et al. (2004) in


13/22



Table 4. Specifications for the Applicability Check Method

1. Planning the Applicability Check

1a. Was the purpose of the research clearly defined in terms of the outcomes to be achieved and the information that is needed to

examine those outcomes?

1b. Did the research question clearly identify the topic of the research, the population that is relevant to the question, and the specific

issues of interest?1c. Did the group consist of representative, unbiased members of the population under investigation?

2. Selecting the Applicability Check Moderator

2a. Was the moderator a member of the research team?

2b. Did the moderator selected have a supportive leadership style?

2c. Did the moderator selected have empathy, an ability to listen, and to ask appropriate follow-up questions?

2d. Did the moderator selected ensure that s/he did not bias the responses of the participants?

3. Ensuring Familiarity with the Research Under Investigation

3a. Did all group members receive copies of the research summary and associated questions?

3b. Did all group members share feedback on the research under investigation with the applicability check moderator?

3c. Were all group members sufficiently prepared to participate in the face-to-face part of the applicability check method?

4. Designing the Applicability Check Interview Guide

4a. Did the interview questions relate directly to the stated purpose of the focus group?

4b. Were the questions largely open-ended (unstructured), unambiguous, and singular in purpose?

4c. Did the broader questions precede more probing questions?

4d. Did the more important questions appear near the beginning of the interview guide?

4e. Did the interview guide consist of less than 12 questions?

4f. Could the questions be addressed effectively within a two-hour period?

4g. Was the interview guide tested prior to use?

5. Establishing the Applicability Check Environment

5a. Did the available facilities provide an environment appropriate to the conduct of an applicability check (for example, space, noise,

white boards, recording, equipment, and so on)?

5b. Was there an initial meet-and-greet period prior to the formal focus group interaction?

5c. Were the less talkative group members, as observed during the meet-and-greet period, placed directly opposite the moderator

and the more talkative members adjacent to the moderator?

5d. Were the names of individuals displayed on name tents?

5e. Did the focus group members sign human subjects consent forms / ethical clearance forms?

5f. Were the focus group members told that the discussion would be recorded and that all comments would remain confidential?6. Conducting the Applicability Check

6a. Did the moderator present the objectives of the group, the agenda for the discussion, and outline the ground rules to the group

members?

6b. Did the moderator build rapport within the group by asking a factual question, for example, one related to the members place of

employment?

6c. Did the moderator encourage all focus group members to share their thoughts?

6d. Did the moderator manage the time available in such a way that issues were covered effectively?

6e. Did the moderator probe for more specific information without influencing the response when s/he felt that the focus group

member wished to elaborate further on a specific issue?

6f. Did the moderator deal effectively with any issues that arose during the focus group?

7. Analyzing the Applicability Check Data

7a. Assuming multiple data analysis, was the analysis strategy documented, understood, and able to be clearly articulated by each

member of the research team?

7b. Did the analysis procedure follow a sequential process involving field notes and recordings, oral summaries that verified keypoints that arose during the focus group, debriefing with the moderator team immediately following the group, and transcripts (if

used)?

7c. Was there sufficient data to provide a trail of evidence, thereby ensuring that the analysis was verifiable (reliable)?

7d. Was the analysis completed following each focus group so that questions or the approach could be modified prior to the next

focus group?

7e. Was the analysis team sufficiently skeptical with regard to the possibility of social desirability influences, groupthink, and the

dominance of a focus group member or members?

7f. Was the reliability of the analysis established by using two coders to identify major themes and issues?


14/22



their development of a method for conducting canonical

action research.6We present our method as a set of principles

that form the foundations of the method, each of which is

further differentiated in terms of a set of criteria for con-

ducting and evaluating each step. Hence these principles and

criteria should be used both as guidelines by researchers using

applicability checks to help establish in a rigorous manner therelevance of research to practitioners and as evaluation

criteria by reviewers wishing to evaluate the conduct of an

applicability check (see Markus and Lee 1999).

Table 4 presents our method for conducting an applicability

check based on a focus group method with the addition of a

modified NGT to familiarize participants with the research

object under investigation.

Implications

Using applicability checks as a way of improving research

relevance has several implications. First, we envisage that

researchers may need to expend considerable effort in

recruiting appropriate applicability check participants. How-

ever, the fact that the knowledge gained from involvement in

applicability checks would benefit practitioners by exposing

them to new, potentially important research that could be of

direct value to them, research that they would not otherwise

have the opportunity of being exposed to for months or

perhaps even years, should encourage participation. Further,

recruitment would become much easier if the value of appli-

cability checks were to become known to the practitioner

community. A source of valuable knowledge for enrolling

participants is the focus group literature, which places

significant emphasis on the selection and recruitment process

(see, for example, Krueger and Casey 2000; Morgan 1998;

Stewart et al. 2007).

Second, there are two implications for future research on

applicability checks. With respect to the applicability check

approach, we note that it is just one of a number of solutions

that could be used to improve the relevance of research to the

practitioner community. Future research could investigate

other methods for improving research relevance, which, like

applicability checks, are focused on the research process.Other approaches also might target different phases of the

research life cycle rather than the theoretical development on

which we focus here. Finally, institutional and project-level

approaches could also be used to improve research relevance.

Using such approaches would complement the methods based

on the research process that we use here.

With respect to the applicability check method itself, the pre-

liminary applicability check we conducted on DeLone and

McLeans IS success model (see Appendix A) served as the

first step in developing the method. The method we present

here should now be evaluated by conducting checks on

research objects in the context of on-going research. Such

checks would consider every aspect of the check from issues

as diverse as the accessibility or understandability of the

materials used to support the check, the effectiveness of the

moderator, the effectiveness of both the modified nominal

group and focus group processes, among others. Further, it is

possible that, as researchers gain experience with both the

approach and the method itself, the applicability check

method might be better differentiated to present methods

specific to different types of research objects.

Third, from the viewpoint of publication, the findings of

applicability checks should be reported in traditional aca-

demic publications as well as in publications intended for

practitioners (such asMIS Quarterly Executive). To further

motivate researchers to focus on the need for research

relevance, requirements for conducting applicability checks

and other similar approaches to evaluating and/or improving

research relevance could be incorporated into publication

criteria, for example, by modifying both the editorial policies

of journals and the evaluation forms completed by reviewers.

Fourth, from the viewpoint of relationships between academia

and industry, the increased awareness of IS research resulting

from the conduct of applicability checks should ultimately

encourage industry to increase its support for IS research. As

well as further increasing the opportunity to improve the rele-

vance of our research, it would raise the level of industry-sup-

ported research funding. Hence the conduct of applicability

checks would, in this way, have a positive effect on one of the

other potential solutions we highlighted earlier for achieving

research relevance, the project governance perspective.

The notion of applicability checks introduces new and

potentially important avenues for research and, in addition to

improving the relevance of research to practice, could becomea distinct research area in its own right. From a disciplinary

perspective, we also note that applicability checks are gener-

alizable to the work of other academic communities that seek

to be relevant to practice, rather than specifically to the IS

academic community. Nonetheless, our primary hope is that

the IS researchers embrace our suggestions regarding the

conduct of applicability checks in their quest to improve the

practical relevance of their research.

6Note that the applicability check method could also be formalized using

design science principles (Hevner et al. 2004).


15/22



Acknowledgments

The authors wish to thank Izak Benbasat and Lynne Markus, and the

editor-in-chief, Carol Saunders, and the two reviewers for their

insightful comments on earlier versions of this manuscript.

References

Applegate, L. M., and King, J. L. 1999. Rigor and Relevance:

Careers on the Line,MIS Quarterly(23:1), pp. 17-18.

Avison, D., Lau, F., Myers, M., and Nielsen, P. A. 1999. Action

Research, Communications of the ACM(42:1), pp. 94-97.

Baskerville, R., and Myers, M. D. 2004. Special Issue on Action

Research in Information Systems: Making IS Research Relevant

to Practice. Foreword,MIS Quarterly(28:3), pp. 329-335.

Benbasat, I., and Zmud, R. W. 1999. Empirical Research in

Information Systems: The Practice of Relevance,MIS Quar-

terly(23:1), pp. 3-16.

Benbasat, I., and Weber, R. 1996. Rethinking Diversity in

Information Systems Research,Information Systems Research

(7:4), pp. 389-399.

Cohen, J. 1960. A Coefficient of Agreement for Nominal Scales,

Educational and Psychological Measurement(20), pp. 37-46.

Davenport, T. H., and Markus, L. M. 1999. Rigor and Relevance

Revisited: Response to Benbasat and Zmud,MIS Quarterly

(23:1), pp. 19-23.

Davis, F. D. 1989. Perceived Usefulness, Perceived Ease of Use,

and User Acceptance of Information Technology, MIS Quar-

terly(13:3), pp. 319-340.

Davison, R. M, Martinsons, M.G., and Kock, N. 2004. Principles

of Canonical Action Research, Information Systems Journal

(14), pp. 65-86Delbecq, A. L., Van de Ven, A. H., and Gustafson, D. H. 1975.

Guidelines for Conducting NGT Meetings, in Group

Techniques for Program Planning, A. L. Delbecq, A. H. Van

der Ven, and D. H. Gustafson, (eds.), Glenview, IL: Scott,

Foresman, 1975, pp. 40-66.

DeLone, W. H., and McLean, E. R. 1992. Information Systems

Success: The Quest for the Dependent Variable,Information

Systems Research(3:1), pp. 60-95.

DeLone, W. H., and McLean, E. R. 2003. The DeLone and

McLean Model of Information Systems Success: A Ten-Year

Update,Journal of Management Information Systems(19:4), pp.

9-30.

Dennis, A. R., Valacich, J. S., Fuller, M. A., Schneider, C. 2006.Research Standards for Promotion and Tenure in Information

Systems,MIS Quarterly(30:1), pp. 1-12.

DeSanctis, G., and Poole, M. S. 1994. Capturing the Complexity

in Advanced Technology Use: Adaptive Structuration Theory,

Organization Science(5:2), pp. 121-147.

DEST. 2006. Research Quality Framework. Assessing the Quality

and Impact of Research in Australia: Research Impact,

Canberra: Commonwealth of Australia.

Eisenhardt, K. M. 1989. Building Theories from Case Study

Research,Academy of Management Review(14), pp. 532-550.

Fllman, D., and Grnland, A. 2002. Rigor and Relevance

Remodeled, in Proceedings of the 25th Information Systems

Research Seminar in Scandinavia, K. Bdker, M. K. Pedersen, J.

Nrbjerg, J. Simonsen, and M. T. Vendel (eds.), Bautajj,

Denmark, August 10-13.Galliers, R. D. 1994. Relevance and Rigour in Information

Systems Research: Some Personal Reflections on Issues Facing

the Information Systems Research Community, in Business

Process Re-Engineering, Proceedings of the IFIP TC8 Open

Conference on Business Process Re-Engineering, B. C. Glasson,

I. T. Hawryszkiewycz, B. A. Underwood, and R. A. Weber

(eds.), Queensland, Australia, pp. 93-101.

Hevner, A. R., March, S. T., Park, J., and Ram, S. 2004. Design

Science in Information Systems Research, MIS Quarterly

(28:1), pp. 75-105.

Introna, L. D. and Whittaker, L. 2004. Journals, Truth and

Politics: The Case of MIS Quarterly, in Information Systems

Research: Relevant Theory and Informed Practice, B. Kaplan, D.P. Truex III, A. T. Wood-Harper, and J. I. DeGross (eds.),

Boston: Kluwer Academic Publishers, pp. 103-120.

Kavan, C. B. 1998. Profit through Knowledge: The Application

of Academic Research to Information Technology Organiza-

tions, Information Resources Management Journal (11:1),

Winter, pp. 17-22.

Klein, G., Jiang, J. J., and Saunders, C. 2006. Leading the Horse

to Water Communications of the AIS(18), pp. 259-274.

Kock, N., Gray, P., Hoving, R., Klein, H., Myers, M., and Rockart,

J. 2002. IS Research Relevance Revisited: Subtle Accom-

plishment, Unfulfilled Promise, or Serial Hypocrisy?, Com-

munications of the AIS(8), pp. 330-346.

Krueger, R. A. and Casey, M. A. 2000. Focus Groups: A Practical

Guide for Applied Research(3rded.), Thousand Oaks, CA: SagePublications.

Lyttinen, K. 1999. Empirical Research in IS: On the Relevance

of Practice in Thinking of IS Research, MIS Quarterly(23:1),

pp. 25-28.

Mandviwalla, M., and Gray, P. 1998. Is IS Research on GSS

Relevant,Information Resources Management Journal(11:1),

Winter, pp. 29-37.

Markus, L. M., and Lee, A. S. 1999. Special Issue on Intensive

Research Information Systems: Using Qualitative, Interpretive,

and Case Methods to Study Information Technology

Foreward,MIS Quarterly, (23:1), 1999, 35-38.

Miles, M. B., and Huberman, A. M. 1994. Qualitative Data

Analysis(2nded.), Thousand Oaks, CA: Sage Publications.

Moody, D. L. 2000. Building Links between IS Research and

Professional Practice: Improving the Relevance and Impact of IS

Research, inProceedings of the 20thInternational Conference

on Information Systems, W. J. Orlikowski, S. Ang, P. Weill, H.

C. Krcmar, and J. I. DeGross (eds.), Brisbane, Australia,

December 10-13, pp. 351-360.

Moody, D. L., and Shanks, G. 2000. Using On-Line Medical

Knowledge to Support Clinical Decision Making: Towards


16/22



Evidence-Based Practice, inProceedings of the IFIP Working

Group 8.3 Conference on Decision Support through Knowledge

Management, Stockholm, July.

Morgan, D. L. 1998. The Focus Group Guidebook: Focus Group

Kit 1,Thousand Oaks, CA: Sage Publications.

Poole, M. S., and DeSanctis, G. 1990. Understanding the Use of

Group Decision Support Systems: The Theory of AdaptiveStructuration, in Organizations and Communication Tech-

nology, J. Fulk and C. Steinfeld (eds.), Newbury Park, CA: Sage

Publications, pp. 173-193.

Robey, D., and Markus, L. M. 1998. Beyond Rigor and Rele-

vance: Producing Consumable Research about Information

Systems,Information Resources Management Journal(11:1),

Winter, pp. 7-15.

Rosemann, M., and Vessey, I. 2005. Linking Theory and Practice:

Performing a Reality Check on a Model of IS Success, in

Proceedings of the 13thEuropean Conference on Information

Systems, D. Bartmann, F. Rajola, J. Kallinikos, D. Avison, R.

Winter, P. Ein-Dor, J. Becker, F. Bodendorf, and C. Weinhardt

(eds.), Regensburg, Germany, May 26-28.Saunders, C. 1998. The Role of Business in IT Research,Infor-

mation Resources Management Journal(11:1), Winter, pp. 4-6.

Schultze, U., and Orlikowski, W. 2004. A Practice Perspective on

Technology-Mediated Network Relations: The Use of Internet-

Based Self-Serve Technologies,Information Systems Research

(15:1), pp. 87-106.

Sedera, D., Gable, G., and Chan. T. 2004. Measuring Enterprise

Systems Success: The Importance of a Multiple Stakeholder

Perspective, inProceedings of the 12thEuropean Conference on

Information Systems, T. Leino, T. Saarinen, and S. Klein (eds.),

Turku, Finland.

Senn, J. 1998. The Challenge of Relating IS Research to Prac-

tice, Information Resources Management Journal (11:1),

Winter, pp. 23-28.Stewart, D. W., Shamdasani, P. N., and Rook, D. W. 2007. Focus

Groups: Theory and Practice. Thousand Oaks, CA: Sage

Publications.

Strauss, A., and Corbin, J. 1998. Basics of Qualitative Research

Techniques and Procedures for Developing Grounded Theory

(2nded.), Newbury Park, CA: Sage Publications.

Vessey, I., Ramesh, V., and Glass, R. L. 2002. Research in

Information Systems: An Empirical Study of Diversity in the

Discipline and Its Journals,Journal of Management Information

Systems(19:2), pp. 129-174.

Yin, R. K. 1994. Case Study Research: Design and Methods.

Thousand Oaks, CA: Sage Publications, 1994.

About the Authors

Michael Rosemannis Professor of Information Systems and co-

leader of the Business Process Management Group at Queensland

University of Technology, Brisbane. He received his MBA and

Ph.D. in Information Systems from the University of Muenster,

Germany. His main areas of interest are business process manage-

ment, process modeling, enterprise systems, and ontologies. He has

published more than 120 refereed papers including publications in

journals such asInformation Systems,IEEE Transactions on Knowl-

edge and Data Engineering,European Journal of Information Sys-

tems,Decision Support Systems, andInformation Systems Frontiers.

Besides more than 40 journal publications, 70 conference publi-cations, and 35 book chapters, Michael is the author or editor of five

books. He is a member of the editorial boards of six journals and a

member of the Australian Research Council College of Experts.

Iris Vesseyis Honorary Professor at the University of Queensland

and Adjunct Professor at the Queensland University of Technology.

She received her M.Sc., MBA, and Ph.D. in Management Infor-

mation Systems from the University of Queensland, Australia. Her

research interests focus on the evaluation of emerging information

technologies, the management and organization of enterprise

systems, and knowledge management strategies. She serves, or has

served, as an associate editor at Information Systems Research,

Journal of Database Management, Journal of Management

Information Systems,MIS Quarterly, andManagement Science, and

serves on the executive boards ofInformation Systems Frontiersand

International Journal of Information Systems and Management.

During the first 8 years of its life, Iris served as secretary of the

Association for Information System, as well as the International

Conference on Information Systems following its merger with AIS.

She is an inaugural Fellow of the AIS.


17/22



Appendix A

An Exploratory Applicability Check on

DeLone and McLeans IS Success Model

We conducted an applicability check on DeLone and McLeans (1993, 2003) IS success model (see, also, Rosemann and Vessey 2005). This

model addresses an issue crucial to business, is backed by significant research effort, and has been validated by further research following its

initial formulation (DeLone and McLean 2003). Hence it is an effective model for assessing the reaction of the IS practitioner community to

a significant body of mature IS academic research.

DeLone and McLeans Model o f IS Success

DeLone and McLeans (1992) IS success model summarized and structured the extant literature on IS success into an overall model designed

to capture the relevant aspects of the success of IS in practice, as well as the inter-relationships among them. The authors published a revised

model in 2003 (see Figure A1). The second article, which examined just those studies on IS success that appeared following the publication

of the first paper, reported that there are almost 300 articles in refereed journals that have referred to, and made use of, this IS success model.

Hence, a substantial amount of the blood, sweat, and tears of IS research has been expended in trying to determine what constitutes IS success.

We might therefore ask ourselves the question: Is DeLone and McLeans IS success model accessible to IS practitioners? If so, is it perceived

as important for and applicable to the challenges of these IS practitioners?

The Applicabi l i ty Check

The innovative nature of an applicability check required an exploratory approach to examine whether the approach was feasible. We chose

to explore the use of focus groups in this first attempt at conducting an applicability check. This section details the conduct of that applicability

check and presents the major findings.

Use of Focus Groups

The focus group participants we sought all had prior relationships with universities and therefore had a general appreciation for academic

research. Because consultants have considerable influence on corporate IS activities, we performed our applicability check using IS consultants

as well as corporate IS personnel. To increase the homogeneity of the focus groups, we conducted two separate sessions. The focus groups,

which were conducted in Australia, lasted approximately 2.5 hours because the participants had to be exposed to the model under investigation

as well as responding to the applicability check issues per se. The authors served as focus group moderators.

We opened each focus group with the question: How do you measure the success of your most complex information systems? We then

introduced DeLone and McLeans model to the group and answered any related questions. Finally, we compared the criteria identified in the

focus group with DeLone and McLeans established academic model. The final question we addressed was: Would you consider using this

model to measure IS success in your organization? If not, why not?

Data Analysis

We found that the directions taken by our two focus groups were quite sensitive to differences in the responsibilities of the representatives, both

between and within groups. We therefore describe the findings for each of the groups separately and then compare them to DeLone and

McLeans model.

Focus Group I : Corporate Personnel. The three corporate IS personnel who formed the first focus group had quite diverse backgrounds,

coming from the utility and finance industries, and from the public sector.


18/22



Information Quality

System Quality

Service Quality

Intention to

Use Use

User Satisfaction

Net Benefits

Information Quality

System Quality

Service Quality

Intention to

Use Use

User Satisfaction

Net Benefits

Figure A1. DeLone and McLeans Revised Model of IS Success (2003)

The focus group member from the public sector was the manager of IT operations. Not surprisingly, therefore, his focus was on issues such

as continuity measures and service desk response. He also believed that end-to-end project tracking from initiation through operation is an

important success criterion.

The focus group member from the finance industry was the organizations process manager. From a technical perspective, this participant

viewed infrastructure up-time (availability) and service requests (help desk) as important criteria. From a business perspective, he believed

that net business value both from a technical perspective (cost to keep a system running, including licensing) and a business perspective

(developers meeting users expectations), as well as the business view of the performance of a system, should be assessed.

The third focus group member was a Group Process Manager at a utility company. His first focus was on project success (on budget, on time)

and on the benefits derived from projects 3 to 6 months following implementation. Important criteria for the latter were reduction in head count,throughput, time of process execution, as well as net benefits. He also stressed the fact that levels of system availability beyond the essential

are a cost to the company. Other measures on which he focused included business operational support and service for operations, as well as

service for all of the IT components of the technical infrastructure. Finally, he indicated that end-user satisfaction is a measure that his company

monitors.

Following sharing and discussion of the success criteria used in their three organizations, the three participants grouped their criteria and

prepared a model that reflected their view of the success criteria. The group focused on achieving net business value or project benefits as

the ultimate goal for the success of a system. User satisfaction was also perceived as a valid indicator. The conspicuous differences of this

model from those of DeLone and McLean lie, however, in the emphasis on infrastructure, project, and operational issues, factors well beyond

DeLone and McLeans focus on the success of a system. Operational effectiveness, measured using service level agreements (SLAs), was seen

as another measure of system success. The infrastructure has an impact on the SLAs, as well as on the net business value of the project.

Independent of these system-oriented criteria, criteria related to the execution of the IS project were identified as being important, including

delivering on-time, on-budget, and in the requested quality.

Focus Group I I : Consultants. Two of the three consultants involved in the second focus group worked for large, global consulting companies.

The third consultant, who focused on web-based solutions, formed part of a much smaller, global consulting firm. The group of consultants

was surprisingly homogeneous in their comments on how they currently measure the success of information systems. Hence we present the

findings for this group in terms of their major success criteria, which clustered as follows:

Financial measures such as return on investment, net present value, shareholder value, cost savings.

Productivity measures, such as reduction in headcount, etc.

Comparisons of individually defined key performance indicators before and after the project.


19/22


7A referencable client is one that the company can use for promotion purposes.


Client satisfaction with the system; a related issue is the customer/partner/supplier happiness factor, as one participant phrased it.

User-related measures of personal satisfaction such as user acceptance, user empowerment, and improved task skills.

System-related measures such as accessible information and system responsiveness.

During the focus group, it became clear that the participants had difficulty differentiating between the success of an IT application as opposed

to the success of the related project. It appears likely that this focus, which is evidenced in the statement below, is shaped by the consultants

role in conducting implementation projects.

We only engage in such implementation projects if they also provide our own organization with benefits. This includes

not only financial benefits, but also a referencable client;7and the consultants also have to improve their knowledge.

It has to be a true win-win situation.

A further suggested measure was the project-focused delivery upon project milestones.

Compari son of I S Success Model wi th Models from Practice. What can practice learn from DeLone and McLeans model?

First, probably the factor that was addressed least (and/or least well) by practice was intention to use/use, a factor that was all but absent among

those of our six participants. One consultant referred to a related factor as workers support of the systems implemented; not trying to find

a way around it. Nonetheless, the participants appeared intrigued with the notion and showed interest in potentially applying it in their

corporations. However, they also saw potential issues in its measurement.

Second, our participants did not refer to information quality in their initial assessment of success factors. When it was brought to their attention

in DeLone and McLeans model, they commented that the effect of information quality can be compounded across a number of systems.

Hence, the lack of attention to information quality appeared to be an oversight.

Third, service quality was not used by our respondents in the way in which it was intended by DeLone and McLean, that is, as a measure of

the service provided to those employees engaging in end-user computing. It does seem to us that systems and services are different, and distinct,

entities and that their success should be evaluated using models designed specifically for each. Hence, we believe that the inclusion of service

quality in DeLone and McLeans model is problematic.

Next we present some of the factors elicited by our focus group participants that are missing from DeLone and McLeans model. First, both

groups focused to a considerable extent on the project that delivered the systems under discussion, in terms of cost, time, and quality. While

this is not surprising in the case of the consultants, the emphasis given to the project by corporate IS personnel was also substantial. These

observations suggest that DeLone and McLeans model should be extended to include the influence of project quality on the ultimate successof the system.

Second, both focus groups placed considerable emphasis on service delivery based on both systems and technology (the infrastructure), issues

that are largely absent from DeLone and McLeans model. Again, this observation suggests that DeLone and McLeans model needs to be

extended to include operational issues such as these.

Discussion

Here we evaluate the applicability check we conducted on DeLone and McLeans IS success model in terms of its performance on the

dimensions of relevance: importance, accessibility, and suitability. This is followed by the implications of applicability checks and

recommendations for future research.

Di scussion of F indings Based on Relevance Dimensions. With respect to importance, and therefore the fit, of the research to practice, all

of the focus group participants perceived the model as important. Clearly, measuring the success of information systems is a timeless challenge,

and one that has not been addressed satisfactorily to date. Most focus group participants confirmed the fit of factors such as information quality,

system quality, operational service quality (viewed largely as the quality of the help desk), user satisfaction, and net benefits (sometimes using

different terminology) to IS success before we exposed them to DeLone and McLeans IS success model. The notion of intention to use was


20/22



new for all the practitioners and was viewed with interest. Furthermore, participants from both focus groups wanted a stronger recognition

of the characteristics and success of the project leading to the implementation of the system.

With respect to accessibility, none of the participants had heard about the model before or read any of the academic journals in which the model

and its revisions had been published. It was satisfying, therefore, to observe that all participants could see the merit in DeLone and McLeans

model; for example, they all requested the original papers. We also helped the practitioners involved to understand the model, which they

perceived as largely intuitive. There was a certain challenge in both groups in understanding the semantics of the arrows linking the factors.While there might be issues in understanding the presentation of the related models, we do not see major issues in understanding their core.

Nevertheless, we see lack of accessibilityas the first critical hurdle in the transfer of DeLone and McLeans IS success model into practice.

This is a concern because DeLone and McLeans model is one of the most widely discussed models in the IS community and related articles

have been published in a wide variety of academic journals.

With respect tosuitability, some of the consultants were interested in using the model, but it is just an academic paper, as one participant

phrased it, clearly distinguishingsuitabilityfrom applicability. When the model is so successful in academia, why did nobody develop a tool

based on this model that facilitates its application in practice? The consultants perceived as a major downside the fact that a potentially

interesting and mature model could not be easily applied to their current challenges.

The major limitation of our study is the preliminary nature of the applicability check conducted. Our objective was to provide initial feedback

on the feasibility of conducting such checks and to illustrate the type of information that might be forthcoming. The methodology used for

this initial test was therefore exploratory in nature.

Implications for DeLone and McLeans IS Success Model. Based on our findings, we see potential for future research on DeLone and

McLeans success model in several areas.

First, further research needs to be conducted on the link between system and project success, a link identified by all participants and not

addressed in DeLone and McLeans model. There was consensus that a system can be perceived as successful, but not the related project, and

vice versa. This issue is, of course, more relevant in the early phases of the system life cycle. Nevertheless, we believe that the

interrelationships between system and project success deserve further attention.

Second, the feedback from the two types of focus group members (users and consultants) highlighted the different perceptions of these

stakeholders. We see potential for more research into the development of a better understanding of the role of the stakeholder in evaluating

IS success (see, for example, the work by Sedera et al. 2004).

Third, the service quality dimension was ignored by our focus group participants based on lack of relevance to the issue at hand, that of,evaluating systems.

Fourth, there is clearly an opportunity to derive automated solutions based on IS research. A solution for DeLone and McLeans success model

could, for example, be made available via a simple web service, which would guide the user through an assessment of the success of their

systems based on the model. Hence, such a solution would facilitate the immediate application of the model in practice.

Impl icat ions for an Ap plicabi l i ty Check Method

We engaged in an exploratory investigation of the feasibility of conducting an applicability check on the relevance of DeLone and McLeans

model of IS success to practice. In this regard, our research contributes to both the research methodology for conducting such a check and to

illustrating the type of detailed information that an applicability check on the theories or models that are the subject of the check can provide.

We learned from this experience that

1. focus groups are an appropriate technique for conducting applicability checks, but that

2. we needed to seek ways to use the time devoted to the group sessions more effectively. The two focus groups conducted here took

2.5 hours because we both introduced participants to the model and sought their responses to it during that time frame. We believed,

following our experience here, that the group time could be shortened if the participants were exposed to the research object in an

independent intervention prior to conducting the applicability check proper. Using this approach would also allow us to ensure that

each participant fully understood the research object under investigation.


21/22



Appendix B

Details of Steps in the Applicability Check Method

We now present the seven steps of the applicability check method in more detail.

Step 1: Planning the Applic abi l i ty Check

Effective planning is the key t

rose mann vess ey

Documents