Pergamon

S09b'7-4174(96)O0O0~X

E,xp¢~ System~ With Applieafiom, Vol. 11, No. 1, pp. 53--58, 1996 Cop~ght © 1996 ElJevier Science l.td

Printed in Great Ikitain. All gillhu reNrved 0957-4174/96 515.00+0.00

Benefits from Expert Systems: An Exploratory Investigation

BRENDA MART~

AMP Inc., P. O. Box 3608, M.S. 194-020, Harrisburg, PA 17105-3608, U.S.A.

GIRISH H. S ~ ~ AND GAYLE J. YAVERBAUMt

School of Business Administration, Penn State Harrisburg, 777 W. Harrisburg Pike, Middletown, PA 17057, U.S.A.

AbstractmThis study uses data collected from persons in industry and business to assess benefits derived from expert system technology. This work explores variables linked to expert system benefits from the viewpoint of thase working with them. The results show that although expected expert system benefits are often realized, there are differences between benefits expected and those actually realized. Copyright © 1996 Elsevier Science Ltd

1. INTRODUCTION

AN INCREASING NUMBER of expert systems (ES) are being deployed in the commercial environment. Furthermore, by the turn of the century, companies plan to spend 15-20% of their technology budgets on ES applications (Goel, 1994). As financiers and users of expert systems, managers need to know more about the benefits they will receive from an investment in them.

Numerous organizations that have integrated expert systems into every day operation report colossal success and boast that expert systems offer varied benefits to their organization. Paradoxically, there is also skepticism about the acclamations of ES success and, moreover, there is considerable diversity of opinion regarding actual benefits derived from implementation of ES.

This paradox might be explained by several factors. First, although abundant literature analyzing success factors for information system (IS) is available, little evidence is applied directly to expert systems. Yet, because expert systems differ from other categories of information systems (IS), it is clearly important that we study these systems separately. Yoon et al. (1995) state several reasons why the general IS literature is not appropriate when applied to expert systems. Included are the following:

• The determinants of success are not the same for ES as other information technologies and

• The "expert mimicking nature, the domain- oriented problems addressed, the characteristics

"~ To whom all cot~,spondence should be addressed.

of ES shells, unique knowledge acquisition activities, as Well as a unique relationship with end-users make ES distinct from other types of IS.

Second, although it is easy to find opinions about the success or failure of expert systems, most of these opinions are based not upon empirical evidence but rather they are based on mere supposition. These opinions however, though largely unsubstantiated, appear in publications read by many and a practitioner must make a decision about expert system development based upon evidence that may or may not be reliable. Yoon et al. (1995) cite a number of these theoretical reports.

This paper describes an attempt to move from theoretical frameworks found in the expert system literature and explore issues and subsequent benefits of live expert systems that have been used in organizations. Specifically, we address the following questions:

• What are benefits expected from the use of expert systems?

• Are expected benefits from an investment in expert systems realized?

To answer these questions, we share the results of a survey administered to people working with expert systems. Their perceptions of actual benefits derived from using expert systems are compared with anticipated benefits. We then propose a number of suggestions for those involved in making decisions about whether or not to invest in expert system technology.

53

54 B. Martin et aL

2. SYSTEM S U C ~ AND BENEFITS

2.1. System Success

Information system (IS) measurement often centers around system success. Because "success" is such a broad term, its measurement, especially as applied to IS, is complex. According to Pitt et al. (1995), multiple measures are required. DeLone and McLean (1992), stating that the term "success" is in itself an elusive goal, propose six distinct forms of success: (1) system quality, (2) information quality, (3) use, (4) user satisfaction, (5) individual impact and (6) organizational impact. Pitts et al. also propose that service quality is a measure that should also be included.

Within the ES literature, opinions vary about expert system success. Open ended interviews, conducted by Will et al. (1994), of ten expert system developers support five critical factors for successful expert system implementation. These factors include maintenance, verification and validation, case-based reasoning, real- istic user expectations and problem definition.

Duchessi and O'Keefe (1995) agree that these problems are relevant to success but postulate that management, users and organizational concerns are equally as important. Several other researchers stress that non-technical issues are more important to successful systems than technical issues (Coleman, 1993; Tyran & George, 1993). In one of a few empirical studies directly examining ES issues, Yoon et al. (1995) look at user satisfaction as the important measure of implementation success. They find a relationship between expert system success and developer skill, end-user characteristics, desirable impact on end-users, shell characteristics and user involvement.

We agree that non-technical issues are very important but approach expert system success from a slightly different perspective by looking at benefits derived from the use of the system. Benefits, as related to information systems, are the positive qualities that make them useful to an organization and impact the organizations effec- tiveness.

II Benefits

Among the most widely reported impacts that should be realized from expert systems are: (1) improvements in productivity, (2) preservation of knowledge, (3) econom- ical benefits, (4) improved quality of goods and services, (5) training and (6) job enrichment. The discussion of benefits is best illustrated by Fig. 1, a model that summarizes benefits found in the literature and which were the basis for the questionnaire administered to participants in our survey.

The most cited benefits are from productivity improvements (Beckman, 1990; Byrd, 1993; Liebowitz, 1990; Martin & Zickefoose, 1992; Meyer et al., 1992; Van Dijk, 1990). Sharman (1991) proposes that expert

o Productivity o Knowledge Preser-

vation o Economics o Quality o Training o Job Enrichment

FIGURE 1. Benefits of expert system development.

systems save time and reduce work loads. Ansari and Modarress (1990) examine benefits and discover that 28% of organizations report decreased decision-making time as a benefit. Other researchers list benefits that can be associated with productivity, such as effectiveness (Beckman, 1990), efficiency (Feiustein et al., 1990), consistency in decision making (Ansari & Modarress, 1990; Byrd, 1993; Gill, 1988; Turban, 1988), timeliness in decision making (Hauser, 1992; Liebowitz, 1990; Turban, 1988), refiability of decisions (Hauser, 1992; Turban, 1988), accuracy (Byrd, 1993; Hauser, 1992) and enhanced problem solving (Turban, 1988).

Knowledge preservation, the second benefit cited above, is the perpetuation of critical knowledge from experts who may leave a company (Edwards & Counell, 1989; Feinstein et al., 1990; Hellerstein et al., 1990; Liebowitz, 1990; Olsen, 1989; Van Dijk, 1990). They view this as an important advantage and obtaining and keeping scarce or expensive knowledge is also put forth by others (Edwards & Connell, 1989; Hellerstein et al., 1990; Karlinsky & O'Leary, 1992; Liebowitz, 1990; Turban, 1988; Van Dijk, 1990) as a benefit from expert systems. Related is the benefit derived from the distribu- tion of knowledge throughout the organization (Edwards & Connell, 1989; Olsen, 1989).

Economic benefits are supported in the literature by Liebowitz (1990), who states that there is a return on investment benefit, and by Buchanan (1986), Byrd (1993), Croft (1989), Feinstein et al. (1990), Olsen (1989), Sviolda (1986) and Turban (1988), all perceiving that expert systems reduce costs. Liang (1990) and Martin and Zickefoose (1992) view the economic benefits of expert systems as being a consequence of significant competitive advantages that are realized by their use, and Byrd (1993) finds that quantified benefits are derived from reductions in personnel. Ansari and Moclarres (1990) report that 21% of the organizations that they surveyed reported cost savings in their

Benefits from Expert Systems 55

operations. Quality improvements attributed to expert systems are

a result of increased quality found in products and services (Ashmore, 1989; Beckman, 1990; Buchanan 1986; Martin & Zickefoose, 1992; Meyer et al., 1992; Turban, 1988), consistency in decision-making (Ansari & Modarres, 1990; Byrd, 1993; Feinstein et al., 1990), and better compliance to organizational policies and procedures (Beckman, 1990). Sharman (1991) postulates that expert systems improve the quality of decisions. He attributes the improved quality to the reliability and dependability of the expert system, the capability of the expert system to consider a large number of alternatives, and the system's ability to respond to unexpected situations.

Training employees is a function of expert systems that reduces training time (Beckman, 1990; Olsen, 1989; Turban, 1988) and quickly improves skills (Feinstein et al., 1990; Olsen, 1989). Hanser and Herbert (1992) state that expert systems assist in the documentation of decision-making information for future use and training. Increased training and continual training are perceived by both Sharman (1991) and Olsen (1989) as a benefit of explanation facilities, an integral part of an expert system.

Job enrichment is linked to the following: expert systems (1) free the expert's time to do more interesting tasks (Edwards & Connell, 1989; Hauser, 1992; Olsen, 1989; Turban, 1988), (2) permit an employee to have more independence (Turban, 1988) and (3) improve business conditions (Olsen, 1989). Greater flexibility and less complexity (Turban, 1988) are other attributes leading to an enriched job environment.

3. METHODOLOGY

3.1. Survey

Using the claims of the theorists referenced above, we constructed a survey to capture opinions of persons who actually work with expert systems. The areas investi- gated were (1) benefits anticipated by the developer of the system, (2) benefits realized through the use of existing expert systems and (3) the relationship between anticipated and realized benefits.

In the questionnaire, respondents were asked to reflect upon their experience with a specific expert system. They were then instructed to select the top three benefits they had anticipated receiving from the expert system. They were provided a list of sixteen benefits from which to choose and were instructed, should they desire, to add additional perceived benefits.

Participants were also asked to circle benefits they believe to have been realized from the actual use of expert systems in their organization. As with the previous section of the survey, there was space for respondents to add benefits not listed.

The list, shown in Table 1, was constructed from the ES literature reviewed above. Thus, the survey covers a range of benefits that are widely accepted by the ES community. These benefits are studied individually and in the five categories listed above as being the primary classifications of expert system benefits. Benefits antici- pated by the respondents are compared with those perceived as actually realized.

3.2. Sample Group and Surveys

Two hundred surveys were sent to persons selected from two sources: (1) persons from industry and business who are members of the Association for Computing Machin- ery (ACM) Special Interest Group on Artificial Intelligence (SIGART) and (2) industry and business participants in expert system development that were identified from a search of literature in the field.

Although the mailing produced a number of usable surveys, to encourage more participation, a follow-up letter was sent 2 weeks after the initial mailing. Forty- nine (49) usable surveys, i.e. approximately 25%, were returned useable.

Thus the respondents represented a wide range of industry and business persons who actually develop and/ or use expert system technology. Both manufacturing and service industries throughout the world were included, with the largest contingents from educational, public utility and government organizations.

4. RESULTS

Table 2 shows the frequencies of anticipated and received benefits as an aggregate total of all respondents

TABLE 1 Benefits of ES Appearing on Survey

Increased productivity Educational benef'ds Increased quality of decisions Equipment operation Reliability Solve complex problems with narrow domains Faster deo/sion making Use low cost equipment

Flexibility in decision making Capture scarce expertise Operate in hazardous zone Integration of knowledge of several experts Enhances problem solving Working with incomplete or uncertain information Reduced downtime Use in remote locations

56 B. Martin et aL

TABLE 2 ~m~a ted Benef~ from ~ S y ~

Anticipated Received as benefit actual benefit

Faster decision making 19 Increased produvtlvit¥ 23 Increased quality of decision 25 Educational benefit 15 Capturing of scarce expertise 8 Enhanced problem solving 8 Solving of complex problems 7 Integration of several experts 6 Reliability of decisions 12 Rexible decision making 4 System used in remote locations 2 Miscellaneous other benefits 7 Equipment operation 1 Ability to work with uncertainty 4 Reduction of downtime 5

22 19 17 13 11 9 8 8 7 7 3 4 2 1 1

* Note that participants were not limited in the number of choices they could make.

and does not match anticipated and realized benefits. Table 3 categorizes these benefits, according to the literature and as shown in Fig. 1, along with the frequencies by category.

There is a strong positive relationship between the frequencies of anticipated and perceived benefits for both the non-categorized and categorized data. The most frequently cited individual anticipated benefits were, for the most part, the frequently cited actual perceived benefits. For instance, 19 people selected faster decision making as one of the top three benefits anticipated from expert systems and 22 actually believed this to have occurred. Increased productivity and increased quality of

decision making, also often cited anticipated benefits were perceived as being actually received by 19 and 17 people, respectively.

In the case of the categorized benefits, productivity components were anticipated 75 times and perceived as actually received 68 times. Similar results, although checked less frequently, were found for each of the other categories. Therefore, at an aggregated level, participants feel that they are receiving the benefits that they have anticipated from their investment.

Due to the qualitative nature of the data, the means of the frequency numbers are not relevant. Rather the differences in frequencies for each individual benefit or category of benefit is relevant and a t-test was performed to determine if there is a statistically significant differ- ence between anticipated and realized benefits. These results appear in Table 4.

If the mean of the differences between the realized and anticipated frequencies is significantly different than zero, organizations developing and using expert systems are not realizing the benefits they anticipate from the expert system. The calculated t value for individual benefits is 1.05 with a probability value of 0.3081. Thus, the mean of the differences is not significantly different from zero and we, therefore, conclude that organizations developing and using expert systems are realizing the benefits they anticipate from the system.

The calculated t value for categories of benefits is -0 .444 with a probability value of 0.6754. Using similar reasoning, we accept the fact that the categorical analysis shows that benefits anticipated are realized.

The data were further analyzed by looking at the matched percent of the top and top three anticipated benefits that were perceived by the participants to be realized. These benefits are (1) faster decision-making,

TABLE 3 aenef~ by Category

Anticipated Perceived as Category Benefit benefit actual benefit

Productivity Faster decision making 75 68 Increased productivity Enhanced problem solving Solve complex problems Reliability Equipment operation Reduced down time

Knowledge preservation Capture scarce expertise 10 14 Use in remote locations

Quality improvement Increased quality of decisions 29 18 Dealing with uncertainty

Training Educational benefits 15 13 Job enrichment Rexibility 10 15

Integration of knowledge of several experts

r=0.97841 P=0.0007

Benefits from Expert Systems 57

TABLE 4 Significance Results for Differences Between Anticipated

and RmlizKI Benefits

Test tscore Pvalue

Individual benefits 1.05 0.3081 Categories of benefits - 0.444 0.6754

(2) increased productivity and (3) increased quality of decision making. Table 5 contains these results.

The 49 individuals who completed the surveys listed 146 top three anticipated benefits. Of the 146 anticipated benefits, 82 had corresponding matching realized benefits. According to this information, organizations are receiving 57% of anticipated benefits. The top benefit listed was realized 70% of the time.

5. CONCLUSIONS AND RECOMMENDATIONS

Our investigation dealt with benefits obtained from the use of expert system technology and we found that there is convincing evidence that many benefits cited in the literature are realized and, additionally, that ones antici- pated are perceived to be realized. Yet, by further examining the results, several observations are of interest.

First, although the top four anticipated benefits are perceived as received, the aggregated perceived benefits are ordered somewhat differently from what the partici- pants expected. Specifically, and of particular interest to a practitioner, is the fact that faster decision making is more often felt as a result of expert system utilization than an increase in either the quality of a decision or productivity.

Additionally of note is the fact that increases in productivity, the second most cited anticipated benefit, is also the second most cited realized benefit. This individual benefit merged with others that are relevant to productivity provides strong support for the fact that these measures are strong reasons for employing expert systems.

We are not suggesting that all expert systems successfully increase productivity, but we do see a

TABLE 5 Matched Anticipated end Realized Benefits

Top three Top

Total number Number realized Percent of those anticipated

Top benefits: r= 0.89208 P= 0.0001

146 47 83 31 57% 70%

relationship between expert systems that are successfully implemented and common measures of both organiza- tional productivity and quality improvement.

Finally, based upon the survey results, managers should view expert systems as most likely to enhance the following major objectives:

Faster Decision Making. Cash et at. (1994) discuss major strategic objectives which have reSulted from significant and volatile change. Two such changes include shorter product development cycles and higher performance hurdles. The premise that firms can no longer afford the luxury of leisurely product develop- ment life cycles is a fact in today's society. An expert system searches for knowledge in a timely fashion that results in a reduction of critical parts of the development cycle.

Additionally, pressures from customers and com- petitors have resulted in higher performance hurdles with customers expecting products and service faster than ever before. For these reasons, fast response time is critical to even a small organization where decision making must be a part of the rapid cycle of events.

Enhanced Problem Solving and Reliability of Deci- sions. Risks are great when decisions are made by persons without expert knowledge. Cash et at. (1994) point out that there are risks associated with decisions that are made without senior-level review. Yet, the need for timely, accurate decisions is critical. This is a paradox that haunts managers within any medium to large organization and, one that according to our survey participants may well be solved with the introduction of expert systems.

Decision Quality. This benefit, albeit anticipated but not always realized, should be analyzed from the afore- mentioned need to respond to higher performance hurdles. Customers expect service and products that are of the highest quality and expert systems can be supportive of this goal especially where expertise is scarce. For example, many service functions are dele- gated to phone operators who may not have the experience of those who have more experience. Never- theless, with an expert system, experienced decision making may be a "computer away" from the person who is closest to a customer.

We remain convinced that expert systems offer benefits as prescribed by the theorists. Our research prepares managers to assess the probable benefits to their own organization.

Finally, although we advocate expert systems as a way of increasing organizational effectiveness, we also caution those reading this paper that they are not a remedy for all problems. Rather, the proper use of expert systems is suggested only by carefully examining the environment in which it may be developed as well as the

58 B. Martin et aL

capabilities of system developers and experts to partici- pate in such an undertaking.

6. LIMITATIONS AND FUTURE RESEARCH

Because we had no reliable measures upon which to build, our study is restricted by the limited empirical evidence related to ES success. Also, the validity of the results may be suspect due to confounding variables in each organizational environment that might mitigate the effects reported.

We feel, however, that despite the exploratory nature of the research, the importance of examining real environments makes our investigation important. Hope- fully, it provides a foundation for a more rigorous approach to the questions posed. Additionally, our sample was chosen from persons working in organiza- tions that employ Artificial Intelligence. We feel that this type of approach is important if we are to examine potential benefits of expert systems.

Another major criticism may be the sample size. We would have preferred to examine greater numbers of individuals who are working in the field of expert systems but feel that the field, in comparison with other categories of information systems, is relatively small. As others enter the ES market, it is hoped that larger studies will be conducted.

Despite the limitations stated, this study represents an important attempt to empirically explore expert systems. Additionally, because "real world" settings are analyzed, the study provides practical information.

REFERENCES

Ansari, A. & Modarress, B. (1990). Commercial use of expert systems in the U.S. Journal of Systems Management, 42,(12), 10-13.

Ashmore, G. M. (1989). Applying expe~ systems to business strategy. The Journal of Business Strategy, September/October, 6-49.

Beckman, T. J. (1990). An expert system in taxation: The taxpayer service assistant. In J. Liebowitz (Ed.), Expert systems for business management. Englewood Cliffs, NJ: Yourdon Press.

Byrd, A. (1993). Expert systems in production and operation manage- ment: Results of a survey. Interfaces, 23(2), I 18-129.

Coleman, K. (1993). The AI marketplace in the year 2000. Al Expert 8(I), 34-38.

Duchessi, P. & O'Keefe, R. M. (1995). Understanding expert systems success and failure. Expert Systems with Applications, 9(2),

123-134. Edwards, A. & Connell, N. A. D. (1989). Erpertsysterns in accounting.

Herefordshite: Prentien-Hall. F, einstein, J. L., Magnas, H. & Bailey, D. L. (1990). An expert system

at the U.S. encngy information administration for quality assma~e and statistical survey support. In IAebowitz, J. (Ed.), Expert systems for business management. Englewood Cliffs, NJ: Yomdon Press.

Goel, A. (1994, Winter). The reality and future of expert systems, a managers view of AI research issues. Information Systems Manage- rnent, 53-61.

Grandon, G. T. (1995). Early expert systems: where are they now? MIS Quarterly, 19(1), 51-69.

Hauser, R. D. Jr & Herbert, E J. (1992). Managerial issues in expert system implementation. SAM Advanced Management Journal, $7(1), 10-15.

Hellerstein, J. L., Klein, D. A. & Millikon, IC R. (1992). Expert systems in data processing. Reading, MAt Addison-Wesley.

Karlinsky, S. J. & O'Leary, D. E. (1992). Tax-based expert systems: A first principles approach. In O'Leary, D. E. & Watkins, P. R. (Eds), Expert systems in finance. Amsterdam, The Netherlands: Elsevier.

Liang, 1". Y. & Ta, H. P. (1990). Management expert systems for competitive advantage in business. Information and Management, 195-201.

Liebowitz, J. (1990). Introducing expert systems into the firm. In J. IAebowitz (Ed.), Expert systems for business and management. Englewood Cliffs, NJ: Yonrdon Press.

Martin, T. J. & 71ckefonse, M. B. (1992). Expe~ systems in the financial services industry. In D. E. O'Leary & P. R. Watkins (Eds), Expert systems in finanee. Amsterdam, The Netherlands: Elsevier.

Meyer, M., DeTore, A., Siegel, S. & Curley, K. E (1992). The strategic use of expert systems for risk management in the insurance industry. Expert Systems with Applications, 5(I/2), 15-24.

Oisen, P. R. (1989). Safety and risks in the use of financial expert systems. In E. Hollnagel (Ed.), The reliability of expert systems. Chichester, West Sussex, England: Horwood.

Pitt, L. F., Watson, R. T. & Kavan, C. (1995). Service quality: A measure of information systems effectiveness. MIS Quarterly, 19(2), 157-172.

Sharman, R. S. (1991). A socio-technical model for deploying expert systems - - Part l: The general theory. IEEE Transactions on Engineering Management, 38(1), 14-23.

Turban, E. 0988). Decision support and expert systems. New York: MacMillan.

Tyran, C. K. & George, J. E (1993). The implementation of expert systems: A survey of successful implementations. Data Base, 24(4), 5-15.

Van Dijk, J. C. & Williams, P. (1990). Expert systems in auditing. New York: Stockton Press.

Will, R. W., McQuaig, M. K. & Hardaway, D. E. (1994). Identifying long-term success issues of expert systems. Expert Systems with Applications, 7(2). 273-280.

Yoon, Y., Guimaraes, T. & O'Neal, Q. (1995). Exploring the factors associated with expert system success. MIS Quarterly, 19(1), 83-106.