an expert system shell expert system environment vm

6
AN EXPERT SYSTEM SHELL: EXPERT SYSTEM ENVIRONMENT/VM Marianne F. Ostrowsky Ruth C. Swezey / / -7 7 The Expert System EnvironmentNMconsists of a developmentprogram and a consultation program and has a high-level English-like interface. This paper reveals how parts of an expert system shell interact with themselves and with the humans involved, applications of ESE, and limitations and problems aesociated with expert systems. INTRODUCTION When problem solutions require so much knowledge that it is difficult for one person to handle, or when the solutions include too many alternatives for one person to consider, or when quick decisions are needed from multiple sources, an expert system may be the answer. Expert systems is a branch of maificial intelligence that uses software products to solve a problem that normally would require human intelligence. They can be used to store and manipulate human expertise. They can ask questions to draw conclusions and give advice in a way that imitates the human reasoning process. Expert systems are applicable for tasks that require knowledge and reasoning, where the reasoning is repetitive and boring, complex but systematic. [5] Expert systems can be used in many fields: business, military, education, medicine and computers. For example, an insurance salesman has to ask very routine questions to decide if a client is a good candidate to insure and what type of insurance would be best for him. Similar to this would be a banker's problem of determining if a client is worthy of receiving a loan and, if so, the maximum line of credit that this person could borrow. Atthe SUNY Watson School an expert system is being developed to advise students in what courses are available to them based on what they have already taken or had waived. A techno- logical application of an expert system could be one that determines the cause of a car or train engine problem depending on the signs that indicate a problem does exist. An expert system shell is a general purpose building tool used to develop and maintain an expert system, similar to the way database management systems build and use databases. One shell can be used to create many different, unrelated "knowledge bases," which are stored as disk files in the same way databases are stored. The difference between the knowledge base and the database is that the knowledge base is executable, [13] whereas a database is queried or updated. Every expert system shell consists of a way to build and man- age a knowledge base, which is a collection of facts and rules about a particular area of expertise, and an inference engine, which is the generalized reasoning method used. Examples of shells include Essential-MYCIN (EMYCIN) TM, [11] GU- RU TM, [5] OPS5 TM, and IBM's Expert System Environ- ment**. The rest of this paper deals with IBM's Expert System shell: its function, methods of reasoning, and how its knowledge base is built. ISSN0019-0578/89/01/0009/06/$2.50 © ISA 1989 ISA Transactions. Vol. 28, No. 1 9

Upload: aanjpg3649

Post on 12-Nov-2015

223 views

Category:

Documents


2 download

DESCRIPTION

An Expert System Shell Expert System

TRANSCRIPT

  • AN EXPERT SYSTEM SHELL: EXPERT SYSTEM ENVIRONMENT/VM

    Marianne F. Ostrowsky Ruth C. Swezey

    / /

    -7 7

    The Expert System EnvironmentNM consists of a development program and a consultation program and has a high-level English-like interface. This paper reveals how parts of an expert system shell interact with themselves and with the humans involved, applications of ESE, and limitations and problems aesociated with expert systems.

    INTRODUCTION

    When problem solutions require so much knowledge that it is difficult for one person to handle, or when the solutions include too many alternatives for one person to consider, or when quick decisions are needed from multiple sources, an expert system may be the answer. Expert systems is a branch of maificial intelligence that uses software products to solve a problem that normally would require human intelligence. They can be used to store and manipulate human expertise. They can ask questions to draw conclusions and give advice in a way that imitates the human reasoning process. Expert systems are applicable for tasks that require knowledge and reasoning, where the reasoning is repetitive and boring, complex but systematic. [5] Expert systems can be used in many fields: business, military, education, medicine and computers.

    For example, an insurance salesman has to ask very routine questions to decide if a client is a good candidate to insure and what type of insurance would be best for him. Similar to this would be a banker's problem of determining if a client is worthy of receiving a loan and, if so, the maximum line of credit that this person could borrow. Atthe

    SUNY Watson School an expert system is being developed to advise students in what courses are available to them based on what they have already taken or had waived. A techno- logical application of an expert system could be one that determines the cause of a car or train engine problem depending on the signs that indicate a problem does exist.

    An expert system shell is a general purpose building tool used to develop and maintain an expert system, similar to the way database management systems build and use databases. One shell can be used to create many different, unrelated "knowledge bases," which are stored as disk files in the same way databases are stored. The difference between the knowledge base and the database is that the knowledge base is executable, [ 13] whereas a database is queried or updated. Every expert system shell consists of a way to build and man- age a knowledge base, which is a collection of facts and rules about a particular area of expertise, and an inference engine, which is the generalized reasoning method used. Examples of shells include Essential-MYCIN (EMYCIN) TM, [11] GU- RU TM, [5] OPS5 TM, and IBM's Expert System Environ- ment**. The rest of this paper deals with IBM's Expert System shell: its function, methods of reasoning, and how its knowledge base is built.

    ISSN 0019-0578/89/01/0009/06/$2.50 ISA 1989 ISA Transactions. Vol. 28, No. 1 9

  • EXPERT SYSTEM ENVIRONMENT/VM Expert System Environment/VM (hereafter denoted as

    ESE) is a shell written in PASCAL and run on an IBM System~70 TM under a VM/CMS operating system. It origi- nated at the IBM Palo Alto Scientific Center and was announced in the USA and Canada in August, 1986.

    It is supposed to be easy to use with a high level English- like interface so experts can easily build their applications. [4] ESE consists of two programs: ESDE--the develop- ment program, which provides facilities to build a knowl- edge base about a particular domain, and ESCE--the con- sultation program, which conducts a problem solving session with the user. [8]

    Figure 1 represents how the parts of an expert system shell interact with themselves and with humans involved. [7]

    I I I I I Knowledge I . . . . . . . . . . > I Inference I I Base I < . . . . . . . . . . I Mechanism I I I I I

    I v I v

    I I I I Development I I Consultation I Interface I I Interface I I I

    IV IV Builder User

    Figure 1. Interactions

    INFERENCE ENGINES An inference engine is the reasoning technique used by an

    expert system. It is the program that asks the user questions, resolves parameters and rules, and comes to a conclusion. Forward and backward chaining are two such methods ESE provides to resolve parameters by inferring them from rules and the known values of other parameters. Forward chaining is data driven. It works to find new information from known data. The system asks the user for input of all data and then fires all rules referring to that data until there is nothing left to compute---i.e., no more rules apply, the drawback to this method is that the user may be asked to input unnecessary data and extra processing may be done chasing dead end goals. The ESE command to initiate forward chaining is DISCOVER. [1,2,3]

    Morecommonly used is backward chaining, which is goal driven and works backward from a goal to find supporting data. The system is given a goal and then works backward trying to resolve parameters. A sourcing sequence for the parameters is defined, which tells the order of how to obtain parameter values. In order to prove some goal, the system tries to find some rule whose conclusion is the same as the goal and then tries to prove its premise in the same way. If along the way some parameter value is unknown, it is set up

    10 ISA Transactions. Vol. 28, No. 1

    as a subgoal, which is resolved recursively in the same way. The ESE command to start backward chaining is DETER- MINE. This method of inference uses the sourcing sequence defined for each parameter (discussed later).

    Backward chaining is used when the user does not know exactly what information is needed by the expert system or when interactive dialog is desirable. It is practical to use when there are a reasonable number of final answersmthe system tries to prove or disprove each one, gathering the needed information as it goes. Forward chaining is used when the user only needs to supply a small amount of data and knows what data the system needs to solve a problem. It is practical to use when the possible number of right answers seems infinite. [12]

    KNOWLEDGE BASE The knowledge base consists of the variables and rela-

    tionships that the inference engine uses. In order to build an effective knowledge base one must first decide exactly what the system is to do. In ESE the developer uses ESDE to build his knowledge base using parameters, rules, and focus con- trol blocks. Parameters are used to represent facts concern- ing the domain; rules represent relationships between the parameters; and focus control blocks (FCB's) are the struc- tures that control the processing of the rules. Parameters, rules and focus control blocks all have properties the devel- oper defines to control the consultation.

    Parameters [1,2,3] are the variables of ESE. Each pa- rameter has several editable properties. These include constraint, sourcing sequence, prompt, and owning focus control block. The parameter is names and is explicitly or implicitly typed as either a number, bitstring, hexstring, string or Boolean. It may be allowed to have several values such as single-valued, multivalued, or ordered and may have certainties assigned to each value. It may have constraints placed on its value as part of its definition. Constraints include selection from a list, relation to another parameter or specific value, and inclusion or exclusion from a certain range. Some examples of parameter constraints are:

    1. TAKEN FROM ('red', 'green', 'blue') - This is a con- straint on a string parameter allowing it to have only one of these three values.

    2. > FIRST < LAST - This is a constraint on a number parameter allowing it to have a value greater than FIRST value and less than LAST value.

    3. = (1,3,5,7) - This is a constraint on a multivalued number parameter allowing it to be equal to the values in the list.

    4. = TRUE - This is a constraint on a Boolean parameter allowing it to be equal to TRUE.

    5. = HEIGHT + WIDTH - This is a constraint on a num- ber parameter allowing it to be equal to the expression to the sum of HEIGHT and WIDTH values.

    Sourcing sequence is another editable property. This is the sequence of allowable sources and the order in which

  • they should be tried to determine the value of the parameter. If no specific sequence is specified by the developer, then the default sequence of rule, user, default, external program is used. * Rule- ESE tries to acquire a value for the parameter using

    rules only. * User - ESE asks the user to input a value for the parame-

    tel'. * Default - The value of the parameter is obtained from the

    default constraint, which is another editable property. * External - ESE acquires the value of the parameter from

    an external program.

    Another editable property of a parameter is the prompt. This is the actual text of the question that will be asked of the user if 'user' is defined as the parameter sourcing sequence. This property can make the expert system consultation easy to use and understand. Ifa prompt is not specified, ESE will build a prompt based on the name of the parameter, the number of values a parameter can have, and the type of the parameter.

    Examples: 1. For a Boolean parameter if no prompt is specified, the

    system will ask the user, "Is it true that Name?" 2. For multivalued, non-Boolean parameters the system

    prompt would be "What are Name?" If the parameter has a specified constraint, then the prompt would also say, "Choose one or more of the following," and a list of the possible values would follow.

    3. For single-valued non-Boolean parameters the system prompt would be "What is Name?" If the parameter has a specified constraint, then the prompt would also say, 'Choose one of the following," and a list of the possible values would follow.

    Another parameter property is the default constraint, which is used if "default will be taken" is specified as the sourcing sequence method. This value must be a literal or constant since its value must be known immediately. It cannot refer to a parameter and it must be consistent with the pammeter's constraint. There is no default value for this property.

    Owning FCB's is still another parameter property the de- veloper can determine. This is a list of Focus Control Blocks that know about this parameter and can reference it.

    A PASCAL program (ESXEX'ID PASCAL) is associated with ESE; it consists of 99 blank procedures. By interfacing ESE with these external procedures, data already available in the computer as other files can be used and results of ESDE/VM can be sent to other external routines. The developer can fill in one or more of these procedures to perform some function. He can then call one of these procedures by specifying "external data" in the sourcing sequence and then specifying which one of the 99 procedures in the 'procedure name'. Procedure name is the parameter property that tells the name of the external procedure if

    "external data" was chosen as the sourcing sequence method. Once a value has been determined for a parameter, that

    value cannot be changed nor will ESE try to acquire another value for it unless the property "Val can chg fig" has been set to true. It tells ESE that a parameter that already has a value associated with it can be assigned a new value. The major use of this flag is with a counter that needs to be incremented under certain conditions.

    The principal knowledge of a knowledge base is con- tained in the rules. Rules [1,2,3] define relationships be- tween parameters. They are used to express knowledge about the solution of a problem by combining facts that imply other facts or by combining facts to define a sequence of steps to follow in solving the problem. They can direct the flow of control during an expert system consultation. They can deactivate irrelevant parts Of the knowledge base during the consultation.

    As with parameters, each rule has several editable prop- erties. The only required property of a rule is the rule text, which takes the form IF premise, THEN action. A premise is a clause that can be tested. An action is a clause that can be acted on if the premise statement is TRUE. A rule may have multiple premises and/or multiple action.

    An important property is the rule type that tells under which circumstances a rule should be executed. ESE sup- ports two types of rules: * Inference (which is the default value) rules can be proc-

    essed by either forward or backward chaining inference methods (these will be discussed latex). Backward chain- ing tries to determine a value for parameters used in the rule and then fires the rule. Forward chaining will not fire this type of rule unless all the parameters referenced by it already have values.

    * Monitoring rules are ignored by both types ofinferencing engines. They are processed immediately after a parame- ter in the premise is assigned a value. Single-fire moni- tor rules are fired only once. Multifire monitor rules are fired every time a parameter's value changes. Monitoring rules are usually used to display messages to the user during a consultation, increment a counter, and control the consultation. Another editable property of a rule is owning FCB's. This is a list of the focus control blocks that know about the rule and can reference it.

    Examples of rules: * IF Animal_Class IS 'mammal' AND An imal_A~

    IS'tawny' THEN Animal IS 'Tiger'. This rule assigns the value 'tiger' to the parameter. "Animal" when the pa- rameter Animal_Class has the character string 'mammal' and the parameter Animal_Appearance has the character string 'tawny'.

    * IF PACKSMOKE = 0 THEN DONT CONSIDER QUITSMOKE. In this rule if the value of the parameter PACKSMOKE is 0, then the parameter QUITSMOKE is ignored for the rest of the current consultation.

    * If 2 OF THE FOLLOWING (PAGE_NUM=20,

    ISA Transactions. Vol. 28, No. 1 11

  • REFERENCE_NUM--5, DATE=' 12-3') THEN PAPER_DONE=TRLIE. If any two of the premise condi- tions are satisfied, then the parameter PAPER_DONE will be assigned the value of TRUE.

    * IF RESTAURANTS ARE KNOWN THEN SHOW SUGGESTED_ RESTAURANTS. This rule displays the values of the multivalueparameter SUGGESTED_ RES- TAURANTS when the value of the multivalued string parameter RESTAURANTS are known.

    Example of single-f'u monitor rule: IF credit is 're- jected' THEN SHOW 'Contact Sales Dept. Immediately'. Example of Multiple-fire monitor rule: IF occupation=' doctor' THEN doctor count=doctor count+ 1.

    Since most human knowledge is not precise, it is only natural that the rules and parameters of a knowledge base are not totally certain either. A feature of ESE/VM is that the developer is able to provide certainty factors for rules and parameters. A certainty factor is the belief in a value for a parameter or belief in the validity of a rule. Complete belief has a certainty factor of +1 and complete disbelief has a certainty factor of- 1. A certainty factor of 0 corresponds to the unknown case---that of complete uncertainty. A Boolean parameter is represented in the form of certainty: a TRUE value is represented as having certainty of +1.0; FALSE is represented as having a certainty value of -1.0. Certainty factors are not based on probability theory. They do not relate to any scale other than that of the developer. [11]

    In regular IF rules, the certainty from the premise part does not propagate into the action part. In the following the parameter CAR is assigned the value 'Porsche' with a 0.8 certainty: IF COLOR is 'red' THEN there is 0.8 evidence that CAR is 'Porsche'.

    In FIF rules (Fuzzy If) the certainty from the premise does propagate into the action clause. In the following example the certainty of COLOR being red (0.7) is multiplied by the certainty of CAR is Pontiac (0.8), resulting in the parameter CAR being assigned the value of 'Pontiac' with a certainty of 0.56 (0.7 * 0.8).

    FIF COLOR is 'red' THEN there is 0.8 evidence that CAR is 'Porsche'

    In building an expert system architecture the problem- solving knowledge must be organized into logical groups to assist in development and maintenance. The consultation must be controlled so that the questions appear in a logical order to the user. Focus control blocks [1,2,3] are the primary building blocks of ESE that control the consultation. They break the knowledge base into separate modules, each of which represents a single task to be accomplished during the problem-solving session. FCB's select the goals ESE is to try to resolve by defining which parameters are to be asked about first and defining the maximum number of problem repetitions.

    Every knowledge base must have at least one FCB, with the others related in a hierarchical way. If the developer does not define any FCB's, then ESED will generate a minimal

    1 2 ISA Transactions. Vol. 28, No. 1

    working control block, which prompts the user for necessary information such as goals, results, and initial data (editable properties discussed later) and places it in the knowledge base. The consultation can start when this control block is successfully established.

    As with parameters and rules, FCB's have editable prop- erties. The most important property is the Control Text. This consists of a list of commands (control language) that guide the processing by manipulating the knowledge stored in ESE objects. The control language consists of the following commands: * ACQUIRE - obtains parameter values from external data

    routine(s). * ASK - prompts the user to input values for specified

    parameters. * DETERMINE - finds the parameter values by involving

    the BACKWARD chaining inference engine. * DISCOVER - finds the parameter value by invoking the

    FORWARD chaining inference engine. * DISPLAY - shows resulting screens or parameter values

    to user. * PROCESS - passes data and/or control from ESE to an

    external data routine. * ESTABLISH - initiates the processing of another FCB by

    passing control to it.

    Several control strategies can be used to modify state- ments in the control language. The inferencing technique used by the DISCOVER command can be modified by: * USE - to use a certain list or group of rules * STOP ON - stops inferencing when a condition is met * ORDER RULES BY - specifies order in which rules

    should be fired

    Strategies used to alter the sourcing sequence of the DE- TERMINE command include: * DONT ASK - Never ask the user for this value. * DONT ACQUIRE - Never try to get the value from

    external data. * DONT INFER - Never use rules to obtain this value by

    excluding this object from DETERMINE command.

    Slrategies that order groups objects include: ORDER BY - considers rules or parameters in specified order FOCUS ON - focuses on one or more parameters before other parameters.

    Another editable property is Goals. This is an optional list of parameters the FCB is to try to resolve. It is used by the DETERMINE command in the control text.

    Initial Data is an optional list of parameters the user will be prompted for before processing begins. It is used by the ASK command in the control text.

    Externa l Data is an optional list of parameters whose values will be obtained from an external source. It will be

  • used by the PROCESS or ACQUIRE statement in the control text.

    Results is an optional list of parameters whose values will be displayed when processing is completed. It is used by the DISPLAY command in the control text.

    Parameter ownership and rule ownership can be specified by using the Parameters and Rules properties of an FCB.

    Each FCB can be instantiated more than once. The number of times it is executed can be a constant or can be controlled through a rule. Such replication would be useful, for example, to execute a collection of rules for each culture that was taken from a given patient in a biomedical applica- tion.

    ESE under consultation environment (ESCE) requests in- formation from the client specific to the problem through interactive dialog. This information is combined with infor- mation in the knowledge base to provide a solution to the user's problem. An important feature of ESE is that the user is allowed to ask questions such as: * WHY - why is a question being asked * WHAT - to receive more details on the question being

    asked * HOW - how the system determined a parameter value * UNDO - to change answers to previous questions * RERUN - to rerun a previously stored consultation * REVIEW - to review previous questions and answers * STORE - to save consultation answer file

    APPLICATIONS OF ESE ESE/VM has been used to develop a knowledge-based

    system called Contract Support Services Consultant. It is used by field service managers to produce legal contracts for customers by aiding in the process of estimating, bidding and preparing agreements for contract support services. During consultation the system asks the user questions such as the customer's name, address, etc. Then it asks questions such as the type of services, type of machine, quantities and envi- ronmental conditions to determine the specifics of the work to be estimated. The consultant systems uses backward reasoning to estimate the cost and the time required for the service requested. [7] The system then integrates text of a general nature with text based on the particular customer information to produce the final service contract. [8]

    Other areas of application using ESE include computer software and hardware diagnosis, then configuring of com- puters, and financial and banking applications.

    L IMITAT IONS AND PROBLEMS

    Reliability is the major concern with expert systems. In- formation in the expert systems knowledge bases and the process for putting that information in the knowledge bases limits the expert system. If the knowledge is not implicitly stated in the expert system knowledge base, it will notbe able to make any conclusions, or the conclusions may be incon- sistent. This may lead to unreliable results at unexpected times, especially in situations that were not thoroughly

    tested. [10,14] How do you test an expert system in which the conclusion

    for a particular application may be a matter of opinion? This is one of the problems with some expert systems because testing of this type of application is an unreliable way to find errors or, worse, the absence of them. The best these systems can do would be to imitate the human experts as much as possible, who themselves are imperfect. [15] Thebest thing the developer can do would be to raise the confidence level as high as possible for that particular system, but it will never be one hundred percent error-free.

    Since the expert systems cannot be made more reliable than humans, the public may insist that the software pass tests similar for certifying human experts. For example, the Food and Drug Administration is considering that stringent field tests be administered to medical diagnosis systems, like other medical apparatus, before being allowed to be mar- keted to the general public. [14] This may significantly in- crease the time required to bring an expert system to market.

    Another problem is representing the expertise of an expert in a form that can be input to the computer. The computer uses facts, figures, and rules, and experts use "know-how" they have acquired from practice and experience. They do not methodically apply rules in going about their tasks; instead, they recall whole situations from the past and apply them to the present. They use their intuition while they are analytically evaluating what to do. They may have a mature and practiced understanding that has become a part of them of which they are not aware. Other experts may use their emotions or the emotions of the other people involved to draw their conclusions. Therefore it is difficult to instill into an expert system intuition, experience, or emotions, and many applications would not be able to use expert systems. [15]

    Reluctance of experts to share their expertise for fear of losing their jobs is another problem. Also, they may not want to invest the time necessary to build the knowledge base because they are not sure the system will do as good a job as they do. Many users do not have confidence in the conclu- sions drawn by an expert system.

    Another potential problem is when a company relies too heavily on expert systems. Many inexperienced employees may see expertise as a function of the large knowledge bases and inference engines and fail to progress beyond the mini- mal competent level of performance. Eventually, the com- pany may discover that its actual human expertise and wisdom may no longer exist as their employees are relying on the computer to do their work. [15]

    CONCLUSION Expert systems insure consistency and uniformity in the

    application of decision criteria. In tasks that require making judgments or evaluations it is very possible to leave out something that is relevant to the decision. One must also make sure various factors are weighed properly and their interactions accounted for when making assessments. By

    ISA Transactions. Vol. 28, No. 1 13

  • using an expert system you can be sure that all criteria have been considered and weighed properly, resulting in higher quality judgments with greater consistency.

    By using an expert system to make a decision, there is no partiality or "jumping to conclusions". The decision is made on the basis of facts, not human emotions or prejudices. Also, expert systems are reliable and readily available, and they never get sick or have to go on vacation, which results in increased productivity. [5]

    An expert system is never finished, it is evolutionary. It may start out with several rules and grow to hundreds of rules. It may change with the growing complexities and needs of the institution it serves.

    REFERENCES (1) IBM, Expert System Development EnvironmentlVM

    Reference Manual, IBM, SH20-9608, 1985. (2) IBM,ExpertSystemEnvironmentlVMWorkshop, IBM

    Information Services. (3) IBM, Expert System Development Environment/VM

    Users Guide, IBM, SI-I20-9608, 1985. (4) Brad Lemley, "Artificial Expertise: Intelligent Soft-

    ware for Problem Solving," PC, Vol. 4, No. 8, April 16, 1985, pp. 108-112.

    (5) John ttelliv~ell, "Guru: Brave New Expert Systems?" PC, Vol. 5, No. 10, May 27, 1986, pp. 151-163.

    (6) A.J. Symonds, "Introduction to IBM's Knowledge- Systems Products," IBM Systems Journal, Vol. 25, No. 2, 1986, pp. 134-146.

    (7) J.A. Voelker and G. B. Ratica, "The Genesis of a Knowledge-Based Expert Syslem," IBM Systems Jour- nal, Vol. 25, No. 2, 1986, pp. 181-189.

    (8) P. Hirsch, et al., "Interfaces for Knowledge-Base Builders' Control Knowledge and Application-Spe- cific Procedures," IBM Journal of Research and De- velopment, Vol. 30, No. 1, January, 1986.

    (9) Dave Robbins, "Knowledge Base Construction," Ex- pertise: The Expert Systems Journal, Vol. 1, No. 1, January, 1986, Part 1.

    (10) Vance Giboney, "Conventional Programming and Expert Systems," Computer Language, Vol. 3, No. 8, August, 1986, pp. 53-60.

    (11) Koenraad Lecot and D. Stott Parker, "Control Over Inexact Reasoning, A/Expert, Premier, 1986, pp. 32- 43.

    (12) Dennis Merritt, "Forward Chaining in Prolog," A/ Expert, November, 1986, pp. 30-34.

    (13) "Interview: PeterHartTalksAboutExpertSystems," IEEE Expert, Vol. 1, No. 1, Spring, 1986, pp. 96-99.

    (14) Peter J. Denning, "Towards a Science of Expert Systems," IEEE Expert, Vol. 1, No. 2, Summer, 1986, pp. 80-83.

    (15) Hubert and Smart Dreyfus, "Why Expert Systems Do Not Exhibit Expertise," IEEE Expert, Vol. 1, No. 2, Summer, 1986, pp. 86-90.

    14 ISA Transactions. Vol. 28, No. 1