author woo, chong w. apr 92 to small type.document resume ed 366 293 ir 016 487 author woo, chong w....
TRANSCRIPT
DOCUMENT RESUME
ED 366 293 IR 016 487
AUTHOR Woo, Chong W.TITLE A Multi-Level Dynamic Instructional Planner for an
Intelligent Physiology Tutoring System.INSTITUTION Illinois Inst. of Tech., Chicago.SPONS AGENCY Office of Naval Research, Arlington, VA. Cognitive
and Neural Science* Div.PUB DATE Apr 92CONTRACT N00014-89-J-1952; NR4422554NOTE 37p.; Distribution list will not reproduce well due
to small type.PUB TYPE Reports - Descriptive (141)
EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS *Computer Assisted Instruction; Higher Educatiun;
*Instructional Development; Medical Students;*physiology; Problem Solving; *Programmed Tutoring
IDENTIFIERS *Intelligent Tutoring Systems
ABSTRACT
This paper describes the design and development of aninstructional planner for an intelligent tutoring system forcardiovascular physiology that assists medical students to learn thecausal relationships between the parameters of the circulatorysystem, to understand how a negative feedback system works, and tosolve problems involving disturbances to the system. Theinstructional planner is responsible for deciding what to do at eachpoint during a tutoring session. It integrates opportunistic controlwith sophisticated instructional planning, combining lesson planningwith discourse planning. The lesson planning is organized in threelevels: goal generation, determination of planning strategies, andchoice of tactics to refine the goal into subgoals. Themixed-initiative discourse planning is implemented using a two-levelapproach: pedagogical decision making at the upper level and tacticaldiscourse state-based planning at the lower level in its discoursemanagement network. It generates plans dynamically based on thestudent model, monitors their execution, repairs plans whennecessary, and rrIplans when the student asks a question or makes acomment. (Contains 49 references.) (Author)
***********************************************************************Reproductions supplied by EDRS are the best that can be made
from the original document.***********************************************************************
A MULTI-LEVEL DYNAMICINSTRUCTIONAL PLANNER
FOR AN INTELLIGENT PHYSIOLOGYTUTORING SYSTEM
CHONG W. WOO
U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and improvement
EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)
CI This document has Peen reproduced asreceived from the person or organizationOriginating il
0 Minor changes have Oton made to improvereproduction Quality
Points ot view Or opinions stated in this document do not necessarily represent officialOERI position or policy
GENERATION OF COMPLEX TUTORIAL DIALOGUESMARTHA W. EVENS, PRINCIPAL INVESTIGATOR
COMPUTER SCIENCE DEPARTMENTILLINOIS INSTITUTE OF TECHNOLOGY
CHICA130, ILLINOIS 60616312-567-5153, [email protected]
This work was supported by the Cognitive Science Program, Office of NavalResearch under Grant No. N00014-89-J-1952, Grant Authority Identifica-tion Number NR4422554, to Illinois Institute of Technology. The contentdoes not reflect the position or policy of the government and no officialendorsement should be inferred.
Reproduction in whole or in part is permittedfor any purpose of the United States Government.
Approved for Public Release; Distribution Unlimited
2
BEST COPY AVAILABLE
REPORT DOCUMENTATION PAGE 0Form Approved0/148 No. 0704-0188
Pudic reporting burden for this koilection of information is estimated to average I hour per response. including the time for reviewing instructions, searching existing dataSOUrCeS.gathering and maintaining the data needed, and completing arid reviewing the collection of information Send comments regarding this burden estimate o any other aspe t of thisiollection of information, incliading suggestions tor reducing this burden to Nashington HeadouarterS Services, Directorate for Information Operations and Reports, 1215 JeffersonDavis Highway. Suite 1204. Arlington, VA 222024302. and to the Office of Management and Budget. Paperwork Reduction Project (0704.0188), Washington, DC 20503
1. AGENCY USE ONLY (Leave blank) 1 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED
1 April, 1992 I Interim4. TITLE AND SUBTITLE 5. FUNDING NUMBERS
A Multi-Level Dynamic Instructional Plannerfor an Intelligent Physiology Tutoring System G
N00014-897..7-19526. AUTHOR(S) R&T 4422554
Chong W. Woo
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
Dr. Martha Evens REPORT NUMBER
Computer Science DepartmentIllinois Institute of Technology10 West 31st StreetChicago, IL 60616
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING / MONITORING
Dr. Susan Chipman AGENCY REPORT NUMBER
Cognitive Science ProgramOffice of Naval Research800 North Quincy StreetArlington VA 22217-5000
11. SUPPLEMENTARY NOTES
12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE
Approved for public release;distribution unlimited
Alsvagdyatmliaxivnig the design and development of an instructionallanner for an intelligent tutoring system for cardiovascular physiologyhat assists medical students to learn the causal relationships betweenhe parameters of the circulatory system, to understand how a negativeeedback system works, and to solve problems involving disturbances tohe system. The instructional planner is responsible for deciding whato do at'each point during a tutoring session. It integratespportunistic control with sophisticated instructional planning,ombining lesson planning with discourse planning. The lesson plannings organized in three levels: goal generation, determination of planningtrategies, and choice of tactics to refine the goal into subgoals. Theixed-initiative discourse planning is implemented using a two levelpproach: pedagogical decision making at the upper level and tacticaliscourse state-based planning at the lower level in its discourseanagement network. It generates plans dynamically based on the studentodel, monitors their execution, repairs plans when necessary, andeplans when the student asks a uuest'on o m.k- mm14. SUBJECT TERMS 15. NUMBER OF PAGES
Planning, Discourse planning, Tutorial dialoguesText generation 16. PRICE CODE
17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF ABSTRACTOF REPORT OF THIS PAGE OF ABSTRACT
Unclassified Unclassified Unclassified UL
NSN 7540-01-280-5500
3Standard Form 298 (Rev. 2-89)Prescrit ed by ANSI SNI 139-18290.10
1
A MULTI LEVEL DYNAMIC INSTRUCTIONAL PLANNER FOR ANINTELLIGENT PHYSIOLOGY TUTORING SYSTEM
Chong WooDepartment of Computer ScienceIllinois Institute of Technology
Chicago, Illinois 60616Phone: (312) 567-5153
E-Mail: [email protected]
J. INTROWCTIONAn instructional planner in an intelligent tutoring system (ITS) is
responsible for deciding what to do next at each step during the tutoringsession. The planner has to decide what subject matter to focus on, how topresent it to the student and when to interrupt the student's problem-solvingactivity [Dede, 1986; Kearsley, 1987]. This pedagogical decision making isvery complex and there is no one correct choice due to the dynamic changes inthe student's learning state. Hence, the deci3ion must be based on manydifferent knowledge sources, such as knowledge about the domain, knowledgeabout the student, and pedagogical knwledge about tutoring.
Recent approaches to designing tutoring aystems view the decision makingprocess as a planning problem [Peachey and McCalla, 1986; Macmillan et al.,1988; Brecht et al., 1989; Murray, 1990]. Adaptive planning techniques in thetutoring domain enable the generation of customized plans for individualizedinstruction. Among the recent research systems, MENO-TUTOR [Woolf, 1984]represents an important attempt at planning the discourse strategies observedin human tutors, but it lacks global lesson goals [Murray, 1988]. IDE-INTERPRETER [Russell, 1988] is another attempt at planning the lesson goals atvarious levels of abstraction, but this system lacks power at the localdiagnostic level. Thus, there is a need to build an instructional planner thatcombines globally coherent lesson goals with flexible local discourse plans.
In this research, I am building a planner that integrates opportunisticcontrol with a sophisticated instructional planning methods; combiningcapabilities of lesson planning with discourse planning. This planner is adynamic instructional planner that supports customized, globally coherentinstruction, carries out a mixed initiative strategy. It monitors currentplans in progress, repairs those plans, or replans as needed. This hasrequired the invention of multi-level instructional planning.
The goal of this research is to develop an ITS, CIRCSIM-TUTOR, thatassists first year medical students to learn the behavior of thecardiovascular reflex system that stabilizes blood pressure. Since thestudents have already attended lectures about the domain, CIRCSIM-TUTORassumes prerequisite knowledge and assists them tc; correct theirmisconceptions in a problem solving environment. This system is beingdeveloped as a joint projecL of Rush Medical College and Illinois Institute ofTechnology.
1.1 prolutioa sat Computer-Based Instructiou At 112
Computer Aided Instruction (CAI) in the cardiovascular domain at RushMedical College has evolved from HEARTSIM [Rovick and Brenner, 1983], toCIRCSIM [Rovick and Michael, 1986], to the CIRCSIM-TUTOR prototype [Kim etal., 1989] and finally to CIRCSIM-TUTOR over the last ten years.
HEARTSIM was a Plato program and CIRCSIM is a stand-alone Basic program.The CIRCSIM-TUTOR prototype is a Prolog prototype of our ITS designed andimplemented by Kim [1989]. Its design is based on major ITS architecture.However, the prototype system still does not possess all of the capabilitiesneeded for an ITS. It lacks natural language capabilities, it does not analyzethe student's misconceptions, and the instructional planner is very primitive;
4
2
a discourse planner could not be implemented since complete discoursestrategies for all the primitive actions had not been developed, planningknowledge is not explicitly represented as a separate module, and there was noreplanning capability so that the system could not respond to studentinitiatives. CIRCSIM-TUTOR uses the same architecture as Kim's prototype butincludes complete student modelling, instructional planning, and naturallanguage understanding and generation facilities.
1,2 OrganizatiouSection 2 describes the environmcnt in which the system runs. The
subject area of CIRCSIM-TUTOR is cardiivascular physiology and the systemassists students to understand the behavior of the complex negative feedbacksystem. Section 3 begins with a brief introduction to ITS: the generalstructure and the issues involved in each module of the ITS. Then eachcomponent of CIRCSIM-TUTOR will be briefly introduced. Section 4 presentsdesign issues for building the planner: levels of planning and tutoringstrategies. A short tutoring excerpt iz displayed, from a transcript of humantutor and student interaction. And then a short scenario ',Lows how the systemworks. It concludes with a discussion of the overall organization of theplanner: lesson planning, discourse planning, and plan monitoring. Section 5explains the generation of the content of lesson plans in detail. It firstdiscusses the main features of the planner: goal generation and plangeneration. And then it describes its own lesson planning rules: goalgeneration rules and plan expansion rules. Section 6 discusses the discourseplanner. The structure of the planner is a two level discourse managementnetwork, which consists of a set of states that represent tutorial actions.The control mechanism is separated into default and meta-rule transitions. Thepaper concludes in section 7 with a discussion of the significance of theplanner, describes some of its limitations, and gives suggestions for future
research.
Z. 111X 21100.114171112Qualitative reasoning or simulation [deKleer and Brown, 1984; Forbus,
1984; Kuipers, 1984] is an approach to problem solving that reasons about thecausal relationships that structure our world. Anderson [1988] argues thatqualitative reasoning is the most demanding approach and essential to producea high performance tutoring system. He states that qualitative modelling canmaximize the pedagogical effectiveness since it is human-like reasoning,although the implementation effort is much larger than that required for thetraditional black box models or glass box models. CIRCSIM-TUTOR is an approachto qualitative simulation in cardiovascular physiology [Michael et al., 1990].
2.1 Subject AreaThe cardiovascular system consists of many mutually interacting
components, and the student must understand the cause and effect relationshipsfor each individual component of the system. Figure 1 shows a causal model ofCIRCSIM-TUTOR, called the "Concept Map," designed by Michael and Rovick [Kimet al., 1989]. Each box in the map represents a physiological variable, suchas SV for Stroke Volume and RAI for Right Atrial Pressure. An arrow with aor "-" sign between two boxes tells the direction of the causal effects andwhether the causal relationship between the connected variables is direct orinverse. For example, a qualitative change in one component of the system, adecrease in RAP, directly causes a decrease in SV. This qualitative changepropagates to other adjacent components of the system according to thepropagation rule.
There are three stages in the human body's response to a perturbation inthe system that controls blood pressure. The first stage is the DirectResponse (DR) in which a perturbation in the system will physically affect
5
3
many other parameters. The second stage is the Reflex Response (RR), in whichother parameters are affected by the negative feedback mechanism to stabilizethe blood pressure. The final stage is the steady State (SS), which isachieved as a balance between the changes directly caused by the initialperturbation and the further changes induced by negative feedback.
Figure 1. The Concept Map
arsanizatianCIRCSIM-TUTOR begins with a brief introductory message and then asks the
student to choose any procedure from the curriculum list. The curriculum(Figure 2) is stored as a set of four different experimental proceduresdesigned by our expert human tutors (JAM and AAR).
List of Available Procedures
. Hemorrhage: Remove 1.0 Liter of Blood.
. Decrease Cardiac Contractility (CC) to 50% of Normal.
3. Increase Venous Resistance (RV) to 200% of Normal.
4. Increase Intrathoracic Pressure (PIT) to 2 mmHg.
. Quit.
Figure 2. List of Available Procedures
4
Each procedure begins by describing a perturbation of the cardiovascularsystem, and asking the student to predict how the system variables willrespond to the perturbation by making qualitative entries in the PredictionTable (see Figure 3); using a "+" sign to represent an increase, a "-" for adecrease, and "0" to indicate no change. The first column of the table is usedto predict the Direct Response (DR) of each variable to the perturbation, thesecond is used for the Reflex Responses (RR), and the third for the SteadyState (SS).
ParametersDR HR SS
Cardiac Contractility0
Right Atrial Pressure -
Stroke Volume -
Heart Rate 0
Cardiac Output
Total Peripheral Resistance 0
Mean Arterial Pressure
Figure 3. The Prediction Table
When the student finishes predicting all four parameters in one columnof the table, for example the DR stage, the student's answers are comparedwith the correct answers. If the student has made any errors, a naturallanguage tutoring session will begin, based on the result of this evaluationin order to correct the student's misconceptions.
2,1 2y3tem ConstraintsThere are some system variables that need to be described; the procedure
variable is the variable changed by the perturbation; the primary variable isthe first variable in the Prediction Table affected by the procedure variable,(in some cases the procedure variable is the primary variable); the neuralvariables are the variables directly under nervous system control. The rest ofthe variables we call physical variables. The students are not allowed topredict the variables in any arbitrary order, since there are some constraintsthat they must follow. For example, the constraints for DR are fairly complex:
constraint DR1:
Constraint DR2:
Constraint DR3:
The student must predict the primaryvariable first, and the value must becorrect.The student must predict the physicalvariables in the correct causal sequence.The student may predict the neuralvariables at any time and in any order.
7
5
The student receives a canned error message, when either of the firsttwo constraints is violated, and is told what to do next. The purpose offorcing the student into the correct sequence is to make sure the causalbehavior of the system is followed correctly. Neural variables can be enteredat any time since neural variables do not change during the DR period exceptwhen one is a primary variable. The constraints for the RR stage are designedto teach the students about the effect of the baroreceptor reflex:
Constraint RR1:
Constraint RR2:
Constraint RR3:
The student must predict either theneural variables or MAP first.The student must finish predicting allthe neural variables before predictingother physical variables.The student must predict the physicalvariables in the correct causal sequence.
Finally, when predicting the SS stage, the student is allowed to enterpredictions in any arbitrary order since there are no specific constraints forthis stage.
Nultinle Vinultaneous InputsIn a mixed-initiative type of ITS, tutor begins by posing a question and
the student either responds to the question or takes the initiative. Sometimesthis style of tutoring leaves students confused and frustrated if they do nothave enough background in the domain knowledge. Rather than blindly walkingthrough the domain, it would be much more effective if the tutor provides asimulated problem situation in the domain for the student before the actualinteractive tutoring begins.
CIRCSIM-TUTOR begins with a Prediction Table, in which the student isasked to make qualitative predictions about the behavior of the system given aparticular perturbation. After the student finishes all the predictions, thetutor analyzes the student's answers and shows what errors were made if any.Based on a careful analysis of these errors, the tutor can generate a globallesson plan, and interactive tutoring begins by using a mixed-initiativeSocratic strategy in natural language. Thus, the Prediction Table provides aqualitative simulation environment for the student by requiring multiplesimultaneous inputs (multiple responses to different aspects of a problemprovided by the student in a single uninterrupted turn) before interactivetutoring begins.
There re several benefits of adapting this kind of design strategy.First, the tutor receives enough initial knowledge about the student so thatit can narrow the focus for tutoring. It can also detect some common studentm:_sconceptions [Michael et al., 1991] or bugs. Second, the students can see asimple mental model of the entire domain at the start, which prevents thestudents from getting too far off the track [Reiser, 1989]. Elsom-Cook [1988]argues that using multiple pedagogic strategies can provide a very powerfullearning environment. CIRCSIM-TUTOR begins with a coach-like environmentduring the Prediction Table entry, and then moves to Socratic tutoring for theinteractive tutoring session. This flexibility in adapting to the student'sneeds at different stages provides another benefit.
2. QRGANIZATIO Q CIRCSIM-TUTORMost of ICAI systems have been separated into four major components
[Carr and Goldstein, 1977; Sleeman and Brown, 1982; Barr and Feigenbaum,1982]: the domain knowledge base, a collection of instructional strategies andan algorithm for applying them, a student modeler, and an interface. Since amajor goal of CIRCSIM-TUTOR is to carry on a natural language dialogue, wehave divided the interface module into three pieces, an input understander, a
6
text generator, and a screen manager. As a result, CIRCSIM-TUTOR has sevensubmodules: a domain knowledge base, a problem solver, a student modeler, aninstructional planner, an input understander, a text generator, and a screenmanager. Figure 5 shows the overall architecture of CIRCSIM-TUTOR.
U. Domain expertiseDomain Knowledge Base. Anderson [1988] describes three different
categories of knowledge encoding: the black box model, the glass box model,and the cognitive model. The cognitive model is the approach that CIRCSIM-TUTOR is attempting tc implement. The domain knowledge is decomposed intomeaningful, human-like components and a causal reasoning mechanism is appliedto it, so that the system can teach the student to solve problems in a human-like manner. For a detailed discussion of this problem see Wielinga andBreuker [1990].
Domain knowledge can be divided into three different types of knowledgeto be tutored: declarative knowledge, procedural knowledge, and knowledge oftutoring heuristics. Declarative knowledge includes domain concepts and causalrelationships between them. Procedural knowledge involves the rules for usingthe concepts in solving problems. For example, in CIRCSIM-TUTOR, a rule thatfigures out the actual determinant of SV is if the primary variable is RAP,then RAP is the actual determinant of SV. Knowledge of tutoring heuristicsmust be extracted from the experience of domain experts; it involves ways ofteaching the student about the particularly difficult points in the domain.
We have built a small domain knowledge base encoded as a network offrames (see Figure 4). Each frame represents domain concepts and how theyrelate to each other causally. There are three conceptual levels in the domainknowledge; level 0 consists of the definitions and static facts, level 1
consists of the cause-effect relationships between the parameters of thecardiovascular system, and level 2 contains a deeper knowledge of underlyingphysiology. The level 2 knowledge is used when the tutor needs to give a hintto the student. Currently, the level 2 knowledge is under refinement anddevelopment. Hence, in the present program the domain knowledge base isconstructed as a set of components that is used for both problem solving andcausal explanation. This is the most important and the basic knowledge thatconstitutes the domain expertise.
(frame SV
(frame-typevar-typeframe-nameclassinstance-ofnamedefinition
part-ofanatomy ventriclecausal-relation-in causal-RAP-SV causal-CC-SVcausal-relation-out causal-SV-CO))
variablephysically-affectedSVinstancevariableStroke Volumevolume of blood ejected eachheart beatheart
Figure 4. A Frame from the Domain Knowledge Base
Problem Solver. The intelligence of an ITS comes from its ability tosolve the problems [Clancey, 1987]. The problem solver solves the problems
7
presented to the student or asked by the student. If the problem solver solvesthe problems but can not explain how it solves them, it may just as wellretrieve stored answers. The ability to solve the problem, using the expert'sproblem solving behavior, can be used to identify the student'smisconceptions, to give an explanation, and to provide a basis for tutoringstrategies.
Problem solving in CIRCSIM-TUTOR is carried out by two problem solvers:the main problem solver and the subproblein solver. The main problem solversolves the problem, generates correct answers, and produces the same problemsolving path as an expert in the domain. This solution path can be used tomonitor the student's problem solving behavior while the student is makingentries in the predictions table. The subproblem solver solves currentproblems generated by the planner, such as determinant of X, relationshipbetween X and 7, and also problems coming from the student questions. Theother modules of the system may consult these problem solvers to get anyinformation they need.
1,2 .gtudent ModelerThe student modeler is responsible for representing the student's
understanding of the subject by building a student model [VanLehn, 1988]. Thestudent model is a data structure that represents the student's current stateof knowledge; what the student knows, what the student does not know, and whatmisconceptions he or she may have. Based on this information, the tutor cangive individualized instruction to the student. There are two major approachesfor student modeling. One approach, the overlay model [Carr and Goldstein,1977], is designed to represent the student's knowledge state as a subset ofan expert's knowledge state. Another approach, the buggy model [Brown andBurton 1978], represents the student's misconceptions not as subsets of theexpert's knowledge, but as variants of the expert's knowledge. In CIRCSIM-TUTOR, the student modeler integrates overlay and buggy strategies into one[Shim et al., 1991].
The student modeler begins analyzing the student's entries in thePrediction Table. Based on this analysis, the planner generates a lesson plan.During the tutoring session, the planner sends the student's answer to themodeler and the modeler analyzes it and returns the result. Based on thisinformation, the planner can decide the next instruction. Currently only theoverlay information is used for choosing the next tutoring strategy.
SnRut DndamitanduzThe input understander is responsible for understanding the student's
natural language input. It handles not only well-formed but also ill-formedstudent inputs [Lee et al., 1990; Lee, 1990]. The student input may be eitheran answer to the tutor's question, or a question from the student. If thestudent's answer is The actual determinant of SV is RAP, then the planner willpass the sentence to the understander along with the current lesson topic inlogical form, (actual-determinant SV). Then the input understander parses thesentence, checks its coherence with the current topic, and returns the logicform, (answer (actual-determinant Sy (RAP)). Then the planner extracts thestudent answer, RAP, and passes it to the student modeler to diagnose thestudent answer.
The input understander must also understand student initiatives whetherthe student is asking for an explanation, or referring to the previous remarksof the tutor, or w.:nts to stop the session. For example, if the studentinitiative is I don't understand about SV, then the input understander returnsthe logical form (question (explain sV)). This process needs to be studied indetail and we are currently investigating the student initiatives by analyzingtranscripts.
1 0
8
ILA Text GeneratorThe text generator is responsible for turning the tutor's output into a
natural language sentence. It receives necessary information as a logical form
from the planner and generates a natural language sentence or sequence ofsentences phang and Evens, 1990]. This information includes the current topicand text styles: question, hint, answer, etc. For example, the text generator
is given a logic form from the planner, (question (affected-by SV 2)), then itproduces the English sentence, "What are the determinants of SW" The textgenerator can handle this kind of simple question, explanation, or
acknowledgement. The current version of the text generator only receives thenecessary information from the planner, not from all the other modules, sothat its behavior is somewhat passive.
creen ManagerThe screen manager takes care of the interaction between the student and
the system. First, the screen manager displays system messages through theintroductory windows. Then it displays the list of procedures that the studentcan select. When the student selects the problem, it displays the predictiontable with instructions about how to use che mouse and how to make entriesinto the table. Then it receives qualitative answers, (+, 0), from theprediction table one by one from the clicking of the mouse and passes them tothe planner. It also handles two other windows, the student window and thetutor window. From the student window, it receives the student's naturallanguage input in English sentences. In the tutor window, it displays naturallanguage sentences created and passed to it from the text generator.
Instructional 21annuxThe instructional planner is responsible for determining what to do next
at each point during a tutoring session. It interacts with the inputunderstander, the text generator, the student modeler, and the screen manager,in order to carry out tutorial activities. Although the design of the plannermay vary depending on the purpose of the ITS, several researchers haverecently proposed combining opportunistic control with a plan-based approach[Derry et al., 1988; Murray, 1990; Macmillan and Sleeman. 1987]. For instance,Murray [1990] suggests that the way to provide opportunistic control withglobal lesson plans is to implement a dynamic instructional planner. ForCIRCS/M-TUTOR, the planner needs to generate the global lesson plan and takecare of the discourse control as well (Woo et al., 1991a, 1991b].
A. PLANNINQ agTRUCTIONThe instructional planner is the central component of the ITS; it is
responsible for selecting or generating instructional goals, deciding how toteach the selected goals, monitoring and critiquing the student's behavior,and determining what to do next at each point during a tutoring session. Thatis, the planner makes two different types of important decisions during thetutoring session, decisions about the content of the lesson and decisionsabout its presentation strategy. Although the early ITSs largely focused onthe delivery strategy of the planner, some recent planning research shows theintegration of both aspects in building the planner [Macmillan et al., 1988;Derry et al., 1988; Murray, 1990].
The planning component of CIRCSIM-TUTOR must carry out both functions,since it needs to provide a global lesson plan, and it needs to carry on anatural language exchange with the student. This section eiscusses generaldesign issues of the planner with the goal of providing the most effective5nstruction possible to the student, a sampL dialogue extracted from thetranscript of an actual human tutor-student A.Ateraction and a scenarioimplementing that dialogue, and a description cf the overall organization ofthe planner.
11
9
STUDENT MODELLER
if
Student'sAnswers verlay
Student'sCurrent State
Expert's BehaviorAi PROBLEM SOLVER
Correct AnswersA A
KNOWLEDGE BASE
urriculum
INSTRUCTIONAL PLANNER
A.Student'sInput
INPUT UNDERSTANDERMIIMOOi.wn.1O.MVIeexla1IrwmawmwmlilMmwnIxsomowal1
Dialogue
V
TEXT GENERATOR
SCREEN MANAGER
STUDENT
Figure 5. The Structure of CIRCSIM-TUTOR
9A.1 22.1i= .I.1.21111A
Capabilitica of the Planner. Most machine planning systems, like STRIPS(Fikes and Nilsima, 1971), HACKER [Sussman, 1975), and NOAH (Sacerdoti, 1977),deal with the observable physical world, whereas instructional planningsystems deal with unpredictable dynamic changes in the student's knowledge.The student's current learning status can never be observed directly. It canonly be guessed; the results of this guesswork are stored in the form of thestudent model (VanLehn, 1980). Thus, the planner must possess uniquecapabilities for handling unpredictable situations an an expert human tutordoes.
The planner must plan at different levels of the hierarchy; ahierarchical planning technique can reduce the complexity of the planningprocess. The plan is first developed at a higher level and the details aredeveloped later; this technique prevents development of unnecessary plans in
1 0
advance. The planner must plan at a global level; when the planner generates
the next instruction, it must consider the past plan and the student'sresponses to provide continuity of instruction. The planner must replan whenthe current plan fails or a request is made by the student. The planner mustbe able to monitor the plan to identify the need for replanning. The plannerof CIRCSIM-TUTOR provides all these capabilities.
Levels of Planning. Research by Leinhardt and Greeno (1986, cited in
Derry et al., 1988] has shown that experienced teachers employ levels of
planning in acccmplishing their goals; planning instructional goals occurs at
the most global level, planning actions and decision making occur at a lessglobal level. Inspired by this research, Derry et al. [1988] designed theirTAPS system with three levels of instructional activity: curriculum planning(the agenda), lesson planning (instructional actions), and on-line tutorialintervention. Murray (1988] also distinguished three levels of instructionalplanning; curriculum planning (planning a sequence of lessons), lessonplanning (determining the subject matter in a single lescon), and discourseplanning (planning communicative actions between the tutor and the student).He argues that at least two levels of planning, lesson planning and discourseplanning, must exist in an ITS to deliver more effective and flexible
instruction.CIRCSIM-TUToR is capable of both lesson planning and discourse planning.
It can be set up so that the student can select a problem from a list of fourexperimental procedures or it can do complex curriculum planning. The numberand types of procedures will be extended further in future versions of the
system.
4.2.2 ScenarioA Sample Tutoring Session. We have recorded a number of tutoring
sessions with our experts, Joel Michael and Allen Rovick, who are Professorsof Physiology at Rush Medical College, and some of their first year medicalstudents. After careful studies of the recorded transcripts, we extracted somepossible tutr-rial strategies and tactics that provided us with the frameworkfor building 'Ale instructional planner and the overall system. It is assumedthat students have already learned much of the domain knowledge, hence thesystem will mainly assist the students to correct their misconceptions and to
solve problems. Our current system can handle dialogues like the following.Example 1:
Tutor> What are the determinants of SV?Student> SV is determined by RAP and CO.Tutor> RAP is correct, but CO is not a determinant of
SV. Remember. SV is the amount of blood pumpedper beat. What is the other determinant of SV?
One important point about the above tutor-student interaction is thecontent of the questions posed by the tutor. For example, on the first line ofthe excerpt, the tutor is asking the student about the determdnants of strokevolume. Asking a question about determinants is the first part of the planthat the tutor is using to teach the student about the causal relationshipsbetween two variables, RAP and SV. Thus the content of the question has to begenerated by the lesson planner before the tutoring begins. Another importantaspect is how to present the selected topic. From the above short excerpt, we
can see four different kinds of delivery modes: a direct question (line 1),positive and negative acknowledgements (line 3), hints (line 4), and follow upquestions after hints (line 5). Thus, the planner (discourse planner) needs toplan how to present the selected content to the student effectively.
Example 2:Tutor> By what mechanism is TPR controlled?Student> Nervous System.Tutor> Correct, TPR is controlled by the nervous system.
13
11
Then what is the correct prediction of TPR?Student> No change.
Example 2 is an another tutoring situation that focusses on one of theneurally controlled variables, TPR. The tutor first asks the student about itscontrol mechanism in line 1. This control mechanism is the first strategy toteach the student about the neurally controlled variables. Since the studentanswered correctly, the tutor gives a positive acknowledgement and then usesits second strategy, asking for a prediction, in line 4. We have extractedthis kind of tutoring strategy from the transcripts and designed explicitlesson planning rules.
From the above examples of tutor-student interaction, we candistinguish between the subject matter and its presentation formats. Ohlsson[1986, p. 2171 argues that an effective ITS should be able to generatedifferent presentations of each piece of subject matter in order to provideadaptive instruction to the student. The content of the questions posed by thetutor and its delivery modes lead to the development of two different kinds ofinstructional planning, lesson planning and discourse planning, because thesubject to be taught has to be generated adaptively, and also its presentationform can vary according to the situation.
Implementation of the Scenario. Assume that the current lesson goal isto tutor the causal relationships between two parameters, RAP and SV. Thisgoal gets refined into a set of hierarchical subgoals by using strategic andtactical rules. The subgoals generated at the tactical level, such asdeterminants, actual determinant, relation, and value, are kept in a stack,which is used by the discourse planner to pick the next topic.
The following scenario describes what each component of the system does,what kind of information it needs, and what is the result after each step. Thesteps are numbered to show the execution sequence. This tutorial interactionbegins after the lesson planning is done. So that the discourse planner beginswith the first topic in the stack, the determinants, and when that topic iscompleted, continues with the next topic, the actual determinant, and so on.
1. Planner: Picks the current topic from the stack,selects the discourse tactic, and passes it to thetext generator as an internal logical form.
current topic:discourse tactic:call Text Generator:
(determinant SV),question.(question (determinant SV))
2. Text Generator: Generates a sentence,"What are the determinants of SV?"
3. Screen Manager: Displays the sentence in the window.
4. STUDENT: "SV is determined by RAP and CO."
5. Planner: Passes the student's input with the currentlesson topic to the input understander.
(question (determinant SV)(SV is determined by RAP and CO))
6. Input Understander: Parses the student's answer, checksits coherence with the dialog history, and returnsthe answer to the planner in logic form.
call planner: (answer. ((determinant SV)(RAP CO)))
1 4
1 2
7. Planner: Passes the current topic and student answerto the the student modeler in logic form.
current topic: (determinant sV),student answer: (RAP, CO),
call Student Modeler: ((determinant SV) (RAP, CO))
8. Student Modeler: Calls the problem solver, gets thecorrect answer: (RAP, CC), compares the correctanswer with the student answer, and updates thestudent model.
In step I, the disco=se planner picks the topic, determinant, from the
subgoal stack, selects the discourse tactic, question, binds these twotogether with the current variable, SV, into a logical form, (question(determdnant SV)), which is passed to the text generator to generate a naturallanguage sentence. After receiving the logical form from the planner, the textgenerator generates a sentence like the one in step 2. In step 3, the screen
manager displays the sentence on the student window, and the student respondswith the answer in step 4. So the current dialogue is:
Tutor> What are the determinants of SV?Student> RAP and CO.
In step 5, the planner passes the student's input along with the current
topic. The input understander has to recognize the student's answer; parse the
answer, check its coherence with the dialogue history, and return the answerto the planner in its logical form. Then the planner sends the current topicwith the student's answer to the student modeler in step 7. Finally, thestudent modeler analyzes the student's answer, and records the result in thestudent model. The next step will start with the planner checking tha studentmodel, and then deciding what to do. Since one of the student's answers iswrong, the planner consults its tutoring rules and decides to give someacknowledgement first:
Tutor> RAP is correct, but CO is not a determinant of SV.
At this point the tutor has two choices, either give a hint or just give
an answer and continue with the next topic. Since this is the first trial, thetutor decides to "give a hint" and then ask a question to complete theprevious answer. So a possible response would be:
Tutor> Remember. SV is the amount of blood pumped per beat.What is the other determinant of SV?
A different tutoring rule will be applied if the student again makes anerror after receiving a hint; the student will be given a direct answsr forthe second question. Our current tutoring rules vary according to the topicand the student's responses (i.e., the tutor gives different responses indifferent situations). The question may be about neural variables or causalrelationships; in each case the tutoring rules are different. Also we havedifferent rules for each stage, DR, RR and SS.
Orcanizatioa gd. the Instructional glanugzThe instructional planner of CIRCSIM-TUTOR consists of two parts (see
Figure 6); the lesson planner and the discourse planner. The plan controllermonitors the execution of the current plan. The planner can be thought of as asmall expert system, which consists of two main parts: a knowledge base and an
15.
1 3
inference engine [Harmon, 1987]. Thus, the lesson planner consists of threesets of lesson planning rules, and an inference mechanism. The discourseplanner consists of four sets of discourse planning rules and an inferencemechanism. This section introduces the organization and the main features ofthe instructional planner briefly.
LESSON00- PLANNER:
P LAN
CONTROLLER
INSTRUCTIONAL PLANNER
0. DISCOURSEPLANNER:
Level 1: Goal Generation
Goal
Plan Generation
Level 2: Strategy
Level 3: Tactic
Lesson Plan
DISCOURSE NETWORK
Level 4: DiscoursePedagogic State
Level 5: DiscourseTactical State
VSTUDENT
Figure 6. Instructional Planner
Lesson Planning. Lesson planning determines the content and sequence ofthe subject matter to be taught in a single lesson [Murray, 1988; Brecht etal., 1989; Russell, 1988]. The lesson planning in CIRCSIM-TUTOR consists oftwo phases: goal generation and plan generation. The generation of the lesson
16
14
goals is guided by a set of explicit domain-dependent heuristics (goal
generation rules), and the lesson plans are determined by applying two set of
rules, rules for selecting strategies and rules for selecting tactics. As aresult the lesson planner does hierarchical lesson planning with its threesets of rules; at the topmost level it generates lesson goals, and then itexpands one of the goals into a sat of subgoals (a plan) at the next level.The generated goals will be saved in the goal stack and the subgoals in thesubgoal stack. The lesson planner must update the goals dynamically as the
student model changes.Discourse Planning. Discourse planning is a mechanism for planning
communicative actions between the tutor and the student within a lesson[Woolf, 1984; Winkels et al., 19883. CIRCSIM-TUTOR communicates with thestudent in natural language. Thus, the discourse planner must interact withthe student modeler, the screen manager, the input underetander, and the text
generator using a flexible control mechanism. This cont.:ol mechanism resides
in its discourse network.The network consists of two levels; the top level of the network
specifies pedagogic decisions and the lower level consists of a set ofdiscourse tactical states, the executica of which causes text generation,student model updates, and moves to the other states. It represents thediscourse planning rules ahd the control mechanism in explicit form. The rulesinclude all the necessary information to carry out the discourse with thestudent, and the control mechanism is also specified within the rules; twosets of default rules manage the fixed control flow, and two sets of metarules handle dynamic control flow.
Plan Monitoring. AI research on planning emphasizes tha_t execution of aplan requires some monitoring [Charniak and McDermott, 1986]. In the recentrobot planning systems [Wilkins, 1988; Swartout, 1988, the plan monitoring isdone by inserting monitoring steps in the plan, which behaves like a studentmodel in instructional planning. In an ITS, since the student's learningstatus is unpredictable, the planner also needs to monitor the execution ofthe plan and revise the plan if necessary. As a result, plan monitoring shouldoccur whenever there is a change in the student model. Plan revision may occurwhen the current-plan is completed or when the student takes the initiative.
For the current version of cIRCSIM-TUTOR, the planner monitors thestudent problem solving in two different places. First, when the studententers predictions in the prediction table, the planner monitors the student'sentries in the table and interrupts with some warning messages if the studentviolates the system constraints. The messages are designed by the experts, tohelp the students in their problem solving. The system gives differentmessages depending on the procedure, the variables, and the stages. second,the planner monitors the student answer at each step during the tutoringsession, by referring to the student model, and then decides what to say next;give a hint, give an answer, or continue with the next topic, etc. when thestudent takes control by asking a question during the tutoring session, theplanner suspends the current plan, carries out the student's request and thenresumes the suspended plan.
5. TIE LEZ1411 ELM=The lesson planner decides on the contents of a lesson, based on the
student's current knowledge about the domain. The planner has to generatelesson goals, sequence the goals, and select the appropriate planningstrategies to create a plan for the current lesson goal. Figure 7 shows thearchitecture of the lesson planner including the necessary planning steps,student model, and lesson planning rules. The result of the lesson planning isa set of subgoals (a plan), each of which will be the topic for a dialoguewith the student.
17
1 5
The lesson planning mechanism is an essential component of theinstructional planner, since the system must generate globally coherent andconsisten instruction for the student (Macmillan et al., 1988; Murray, 1990],in such a way that the topics are logically connected wit each other, andsequenced and presented in a manner sensitive to the tutorial goals and thestudent's needs. This section describes the lesson planner: lesson planningrules, an architecture, and its two main mechanisms (i.e., goal generation andplan generation).
LESSON PLANNER
Goal Generation
Goal GenerationRules
Student Model
Goal
Plan Generation
Strategy 4Student Model
Ii11...Strategic Rules
Tactics 4Tactical Rules
Lesson Plan
Figure 7. Structure of the Lesson Planner
J. Ligag2a plannina /Rules
The contents of the tutoring strategies are extracted from thetranscripts of the human tutor and student interaction, and we need to encodethem explicitly in the program as rules. I designed this part as a productionsystem, which consists of a rule interpreter and a set of rules. This is themost common approach used in expert systems [Hasemer and Dominque, 1989]. Thissubsection, describes the design of the rules and the implementation of therule interpreter to parse the rules.
The Rule Interpreter. The rule interpreter consists of three main parts:its main loop, its working memory, and its pattern matcher. The working memoryis crucial to the operation of the rule interpreter, because the workingmemory holds an initial representation of the problem that the system istrying to solve. Each time around the loop, the contents of the working memorywill be compared to the antecedent of the rules, and then will fire only onerule if it matches. If an antecedent matches with the working memory, theconsequent will be executed, and the content of the working memory will bechanged for the next inference. The matching cycle will continue until norules match. At this point the interpreter halts, and the content of theworking memory is the desired result for the given problem.
The interpreter is built using LISP macro functions, which understandand interpret the rules for the system. The rule format consists of threeparts: the name part of the rule, the antecedent part, and the consequentpart. For example, (Rule_name: (antecedent) => (consequent)). This approachmakes the system efficient in representing the rules explicitly.
How to Encode the Lesson Planning Rules. The lessou planner uses threesets of lesson planning rules (goal generation rules, strategy rules, andtactical rules). The general form in which the rules are written is if X thenY. Here X is the antecedent or left-hand side of the rule and Y the consequentpart or right-hand side of the rule. Both the antecedent and the consequentmay contain one or more terms.
For example, assume that the student made an error in predicting thevariable TPR. One of the goal generation rules applies; if the student doesnot know TPR, then build the lesson goal, tutor TPR about the neural control.
18
1 6
This rule can be expressed as (G Rulel: ((do-not-know TPR) => (neural-control
TPR))). If the current lesson goal is teach the causal relationship between
RAP and SV, and the student does not know the direction, then this rule can be
written as (S Rulel: ((causal-relation)(do-not-know direction)) => (tutor-
causality))). This is the strategy rule for dealing with non-neural variables.
If the strategy rule is tutor-causality, then the corresponding tactical rule
is to teach determinants, actual-determinant, relation, and value. This rule
can be written as (T Rulel: ((tutor-causality) => (determinants) (actual-
determinant) (relation) (value))).Currently, there are about 50 goal generation rules, 20 strategy rules,
and 20 tactical rules that handle DR, RR and SS phases, and for procedures 4,
6 and 7. The rules may need to be extended to handle the other procedures.
Lesson PlanninaInstructional planning centers around instructional goals. The lesson
planning generates the lesson goals, the knowledge that the system intends the
student to acquire through the tutoring session. This subsection describes how
to generate the lesson goals, and how to develop a lesson plan for the each of
the goals. The two main mechanisms of the lesson planning process, goal
generation and plan generation, are explained below.Goal Generation. CIRCSIM-TUTOR generates instructional goals based on
the student's knowledge demonstrated as entries in the Prediction Table. Thegeneration of the goals is guided by a set of explicit goal generation rules
designed by our experts (Joel Michael and Allen Rovick), which ensures that
the most serious misconception is selected and tutored first.
Goal Generation Rules
1. IF Current Primary Variable is CC and
Student Answer is not NOCHANGE for TPR
Then Build Lesson Goal (NEURAL-CONTROL (TPR))
2. IF Current Primary Variable is RAP and
Student does not know the CAUSAL-RELATIONSHIP
between RAP and SV
Then Build Lesson Goal (CAUSAL-RELATION (RAP, SV))
3. IF Current Primary Variable is RAP and
Student does not know the CAUSAL-RELATIONSHIP
between SV and CO
Then Build Lesson Goal (CAUSAL-RELATION (SV, CO))
Figure 8. Goal Generation Rules
For example, suppose the student made wrong predictions in the table for
the variables, TPR and SV. The student modeler has determined, from itsanalysis, that the student is confused about the mechanism controlling TPR and
19
1 7
the causal relationships between RAP and SV and SV and CO. So the lessonplanner retrieves the information from the student model, applies the goalgeneration rules (see Figure 8), and generates the lesson goals dynamically.The result is a set of lesson goals in the goal stack (see Figure 9).
Order Lesson Goals
1. NEURAL-CONTROL (TPR)
2. CAUSAL-RELATION (RAP,SV)
3. CAUSAL-RELATION (SV, CO)
Figure 9. Generated Lesson Goals in the Goalstack
The goal generation is significant in many ways; the goals are generateddynamically and adaptively; the goals are sequenced in the order that theexpert tutors this material; the goals provide a global context that remainscoherent and consistent throughout the tutoring session, unless the goals arerevised. New goals can also be generated, which tutor the student about acommon misconception (a bug), if the student modeler detects such amisconception. The goals remain in force until they are changed by the plannerdynamically.
Strategic Rule
1. If the Goal = CAUSAL-RELATION and
Student does not know and
direction is incorrect
Then Strategy = TUTOR-CAUSALITY
2. If the Goal = CAUSAL-RELATION and
Student does not know
direction is correct
Then Strategy = REMIND-RELATION
and
3. If the Goal = NEURAL-CONTROL and
this is the first procedure
Then Strategy = DEFINE-TUTOR-NEURAL
Figure 10. The Strategy Rules
Plan Generation. The second stage of the lesson planning is the plangeneration mechanism, which creates the instructional plan by applying twosets of rules, rules for selecting tutorial strategies to achieve the selectedgoal and rules for selecting pedagogic tactics to execute those strategies.
20
1 8
strategy rules (Figure 10) describe the tutorial approach from a domain-
independent point of view. These include tutoring prerequisites before the
material they underlie, reminding the student about relations between two
parameters, explaining the definition before tutoring about it, and so on.
Tactical rules (Figure 11) also represent a domain-independent tutorialapproach; they involve asking about concepts and relations between the
concepts.
Tactical Rule
If Strategy
Then Tactic
If Strategy
Then Tactic
If Strategy
Then Tactic
= TUTOR-CAUSALITY
= DETERMINANTS,ACTUAL-DETERMINANT,RELATIONSHIP,VALUE
= TUTOR-NEURAL-CONTROL
= MECHANISM,VALUE
= TUTOR-SS-PHYSICAL-VARIABLE
= VALUE-DR,VALUE-RR,VALUE-SS
Figure 11. The Tactical Rules'
For instance, if the goal is teach the causal relationship between thetwo parameters, then the fired strategy rule is tutor the causality, and thisthen fires the pedagogic tactical rule: ask about: determinants, actualdetermdnant, relationship, and correct value. The result is a hierarchical
goal tree (Figure 12).
Goal: CAUSAL-RELATION (RAP, SV)
Strategy: Tutor Causality
Tactic:
1
DeterminantsV
Actual Relationship ValueDeterminant
Figure 12. Generated Plan for "causal_relation(RAP,SV)"
21
1 9
Thus the current goal is ultimately refined into four subgoals by two-step goal transformations. In order to solve the current goal, all thesubgoals must be solved. This is the well-known AI problem-reductiontechnique, which transforms a goal into a set of immediate subproblems by asequence of transformations [Barr and Feigenbaum, 1982]. The four subgoalsgenerated at the tactical level are the current plan for the goal. These arekept in a subgoal stack (Figure 13), which is used by the discourse planner topick the next topic.
Order Subgoals
1. Determinants
2. Actual-determinant
3. Relation
4. Value
Figure 13. The Subgoal Stack
5...1 An ExamalaFigure 14 shows an example of the lesson planning process for the
causal-relationship between RAP and SV. From the top of the Figure, the goalgeneration step is described with its other information: student model, rulesused, goal stack, and current goal. Then the plan generation step is describedin two steps, the strategic and the tactical steps. The lesson planner waitsfor the discourse planner to complete the current lesson plan, and when theplan controller sends a wake-up signal, then the planner gets reactivated andcontinues with the next goal in the goal stack.
THE DISCOURSE RUH=The discourse planner is responsible for controlling interactions
between the tutor and the student. It needs to decide how the tutor shouldrespond to a student with a given problem [Woolf, 1984; Winkels et al., 1988].This discourse strategy must be planned explicitly by the discourse planner,so that the system can enter into flexible and coherent interactions. InCIRCSIM-TUTOR, the discourse planner is combined with the lesson planner, sothat the discourse planner receives a global lesson plan from the lessonplanner. The plan controller monitors the execution of the plan and forces thediscourse planner to suspend or resume the current plan when the student takescontrol. The planner consists of sets of discourse planning rules and a twolevel discourse network.
J. Architecture 21 Iha Discourse PlannerFlow Chart Approach. Meta knowledge is knowledge about knowledge [Davis
and Buchanan, 1987]; what you know and don't know (operational metaknowledge), and how you do things (control meta knowledge). The operationalmeta knowledge is needed to recognize a question outside the limits of thesystem. It can be ignored in the discourse planner, since the inputunderstander receives such a question or answer and responds with I don'tunderstand, please rephrase. The control meta knowledge determines how thesystem interacts with the student; it is based on our observations about howthe human expert tutors the student. The integration of this knowledge intothe system ensures that it appears to ask questions in a logical order.
22
2 0
Goal Generation
Rules Used:
Student Model: do-not-know (SV) DR_G_Rule8
Goal Stack: Causal-relation (RAP, SV)Causal-relation (SV, CO)
Current Goal: Causal-relation (RAP, SV)
Plan GenerationRules Used:
Strategy: Tutor-causality DR S Rulel_ _
Tactics: (determinants) DR_T_Rule6(actual-determinant)(relation) (value)
Subgoal Stack: (determinants)(actual-determinant)
(Plan) (relation) (value)
Discourse Planner
executes "determinants of SV"
Plan Monitoring: Waits for the student response
Figure 14. An Example of Lesson Planning
The basic representation of the control meta knowledge in CIRCSIM-TUTORis the flow chart. This is a model of what the expert does and when he doesit. For our system, Allen Rovick designed several flow charts (see Figure 15),each of which is used for tutoring the student in a different situation. Weneed different tutoring strategiea for handling different variables anddifferent phases (DR/RR/sS).
23
2 1
Tutoring DR-Non-Neural Variables
(1 determinant)
What are thedeterminants
=NIMININ --ill. Give hint from ley 1 2
2nd N -40. Give answer
(2 det rminants)
Yes
1Y, 1N answer*Give hint fromlevel 2
2N answers---0Give answer
What isrelationshi
Yes I 2nd N.----0.Give relationship
If no equation,--Wfrom level 2
give hint
N-------4If equation, state in words
Predict againN-----4Give entire level
Yes
Still Npsend to textbook
Next error
Figure 15. The Flow Chart for Tutoring Non-Neural Variablesin DR
Figure 15 is the one that tutors the non-neural variables in DR. The contentof the questions is determined by the lesson planner and passed on to thediscourse planner, which must then decide how to express this content,determining whether to ask a question, give an answer, and so on. After thechart was created, I encoded this information as discourse planning rules. The
2 2
next step is to create a sophisticated inference mechanism that can utilize
these rules.Discourse Network. The network is the main knowledge structure of the
discourse planner. It consists of states, links, and arcs (see Figure 16). Thestates represent tutorial actions, the arca imply state transitions, and thelinks indicate hierarchical dependencies; a state at the tactical levelrepresents the refinements of the level above. Three important mechanisms needto be discussed: levels of planning, representation of the tutorial states,and control structures.
Begin
IINTRODUCE
..1.11110.1.1...
Finish
TUTO1 rOMPLETI
emind
trlanation
Live_Hin
pave Ans
"omplete_Topil
k_Questionive_Correct. Ack
eguestion
lEvaluate_Inpu
Give Incorrect Ac
pave Half Correct_Ack
Figure 16. The Discourse Network
A. Levels of Planning. The discourse planner is divided into twoplanning levels: pedagogical and tactical. The pedagogical level makesdecisions about the style of tutoring; introduces a topic, remediates thestudent's misconceptions, and completes a topic. The discourse action beginswith the pedagogical level, introduce state, and then it traverses the networkand finishes one topic as it reaches the complete state. The tactical levelchooses an expository style to implement the pedagogy; question the student,give acknowledgement, or give an answer. The states at this level arerefinements of the states at the pedagogic level.
B. Representation of Discourse Strategies. The second importantmechanism is the representation of the tutorial strategies in the form ofstates. The discourse strategies were than extracted from the flow chart andexpressed as discourse rules. The rules are written as a frame-like structureusing Lisp macro functions, which represent the states in the network. Thestates are divided into default states and meta states, and each state isfurther divided into pedagogic and tactical states. Each state consists of astate name and slots. The slots in the default state contain information about
25
23
tutoring strategy, text style, and explicit control. The meta states mainlyinclude explicit control mechanism and preconditions. In Figure 19, theexecution of the Ask Question state will cause the text generator to generatea question, and then move on to the next default state, Eval_Input. The slotsalso contain a register to keep track of the completion of the topic, and aflag to update the student model.
C. Control Structure. The discourse control in the network can bedivided into a default control structure and a meta control structure. Thedefault control is specified in the default states, no that thu tutor movesfrom one state to another according to a pre-determined path. The meta controlabandons the default path and moves to the state that is specified in themeta-rule. The system checks the meta-rules first and if none of the meta-rules fire, then the control flow will follow the default path. This controlpath is hidden in Figure 16, because the exceptional behavior by the meta-rules can not be predicted in advance. For example, the Eval_Input state willbe selected right after the Ask Question state as a default path, but the nextstate is unpredictable, since the student answer could be correct, wrong, orpartially correct. This mechanism enables the dynamic behavior of thediscourse planner.
Pedagogic Default Rule
(Pedagogic_default(subgoalupdatenext-state
(Pedagogic_default(subgoalupdatenext-state
*introduce*current-tasktopic-completed*tutor*))
*tutor*current-tasktopic-completed*complete*))
Figure 17. The Pedagogic Default Rule
Pedagogic Meta Rule
(Pedagogic_meta(preconditionprior-statenext-state
(Pedagogic_meta(preconditionprior-statenext-state
*m_tutor*topic-completed*tutor**introduce*))
*m_complete*no-more-topics(*introduce* *complete*)*stop*))
Figure 18. Some Pedagogical Meta Rules
The main disadvantagu of earlier discourse management networks [Woolf,1984; Clancey, 1982] is that they needed to be coupled with some other controlmechanism, such as an agenda and an external memory to provide a topic. In
26
24
CIRCSIM-TUTOR, since the lesson planner provides a globally coherent lessonplan, the network itself can function solely for delivery purposes whilekeeping all the advantages of the discourse management network, such as
flexible discourse control and explicit representation of discourse
strategies.
Tactical Default Rule
(Tactical_default(text-stylecontentupdatenext-state
(Tactical_default(text-stylecontentupdatenext-state
*ask_question*questioncurrent-tasknil*eval-input*))
*give_answer*give-answer(current-task correct-answer)student-model*complete-topic*))
Figure 19. Tactical Default Rules
Tactical Meta Rule
(Tactical_meta(preconditionprior-statenext-state
(Tactical_meta(preconditionprior-statenext-state
*m_correct*correct-response*eval-input**correct-ack*))
*m_incorrect*incorrect-response
*eval-input**incorrect-ack*))
Figure 20. Tactical Meta Rules
122 Discourse PlanningDiscourse planning in C/RCSIM-TUTOR is managed by a simple algorithm. It
iterates through the states until a topic becomes complete. Either the studentresponds with a correct answer or the tutor gives the answer. This sectiondescribes important features of the discourse planning.
The Discourse Goal. The discourse planner needs a goal to tutor thestudent. This goal can be found in the subgoal stack, which the lesson plannerhas produced. In Figure 13, the subgoals are sequenced by number, so that thediscourse planner can carry them out in that order. When the planner finishescarrying out one of the subgoals, it will be removed from the stack, and theplanner picks the next one. This cycle continues until the stack is empty, or
27
2 5
is suspended by the plan controller in favor of a student initiative [Woo,1991c].
Generating Natural Language Sentence. The tactical default states haveslots containing information for the text generator. when the plannerprocesses the states, the text-style and content slots will be extracted fromthe current state. For example, assume that the planner is processing the*ask_question* state (Figure 19), while the text-style slot contains questionand the content slot contains the current-task, such as determinant (SV).Binding these two slot values provides us with a logic form, (question(determinant (SV))), which will be passed to the text generator, whichgenerates the sentence, What are the determinants of SV? Then the screenmanager will display the sentence in the tutor window.
The logic form may need to be extended to generate richer sentences,since thie k!nd of the logic form only contains information about a particulartask or the solution of a problem. The text generator may need to collect moreinformation from many other sources, the domain knowledge base, the studentmodel, the dialog history, and so on.
How to Recognize a Student Initiative. cmCsIM-TUTOR allows studentinitiatives during the tutoring session. So the planner must understandwhether the student response is a question or an answer by checking the inputlogic form, which is being nassed from the input understander. For example, ifthe input understander passes a logic form, (answer (determinant SV)(RAP CO)),the first item of the list, answer, indicates that this is an answer. Thesecond item of the list, (determinant SV), is the current topic, and the thirditem, (RAP CO), is the student answer. Let's assume that the tutor asks thequestion, What are the determinants of SV? and the student responds withdon't know about SV. Then the input understander recognizes this as animplicit question and returns a logic form, (question (do-not-know) (SV)). Theplanner receives the logic form and recognizes that this is a studentinitiative, so it suspends the current plan and carries out the studentrequest; asks the problem solver to get the definition of SV from theknowledge base, and then asks the screen manager to display it.
A,2 Trace ni Discourse TransitionFigure 21 shows a short trace of a sequence of discourse transitions.
The short arrows represent the pedagogic level transitions; the long arrowsrepresent the tactical level transitions; and the double arrows represent themeta level transitions. The left side of the figure shows the processing ofstates, and the right side of the figure shows the discourse actions resultingfrom visiting the states.
The tutor begins by asking a question, then it moves to the evaluatestate by the default control rule. At this time, the student responds with ahalf correct answer, which is recognized by the meta tactical rule3, whichforces a move to the half-correct state. This state produces anacknowledgement and then another meta rule fires, which recognizes that thisis the first try. So the meta rule forces a move to the give-hint state, whichproduces a hint. Since there is no default and a meta rule applies, thecontrol pops up to the upper level and checks whether the topic has beencompleted. If not, then control goes back to the introduce state again, andmoves down to the tactical level. This time the requestion state is selected,since this is the second try on the same topic.
7. CONCLUSION
7.1 Significant FeaturesThis paper describes the design and development of an instructional
planner for a Physiology ITS, CIRCSIM-TUTOR. The planner has severalsignificant features.
23
2 6
Current Topic: Determinants of SV
- >,=>: Pedagogic Level
-->, ==>: Tactical Level ( -> Default , => Meta )
Discourse States Discourse Action
- > INTRODUCE
--> Ask-question Tutor: What are the determinantsof SV?
--> Eval-input Student: RAP and CO
==> Meta-tactic3(Incorrect-one)
- -> Half-Correct Tutor: RAP is correct, but CO isnot a determinant of SV.
==> Meta-tactic6(First-try)
- -> Give-hint Tutor: Remember. SV is the amountof blood pumped per beat.
-> TUTOR
=> Meta-pedagogic(Not-completed)
- > INTRODUCE
- -> Requestion Tutor: What is the otherdeterminant of SV?
Figure 21. Trace of the Discourse Transition Process
First, the planner combines two different instructional planningapproaches: lesson planning and discourse planning. Lesson planning producesglobal lesson plans, which will be carried out during the discourse planningstage. Second, the planner plans dynamically based on the inferred studentmodel; it generates plans, monitors the execution of the plans, and replanswhen the student interrupts with a question during the tutoring session.Third, the pedagogical kncmledge is extracted from the experts and representedexplicitly as rules, lesson planning ruler- and discourse planning rules. Therules are used to generate lesson plans alai to control discourse strategies.The system interprets the rules and builds the lesson plans or returns anappropriate discourse action. Fourth, the planner plans at different levels ofthe hierarchy; the higher level is a simplification or abstraction of the plan(lesson goals) and the lower level is a detailed plan (subgoals), sufficient
2 7
to solve the problem. Fifth, the planner allows minimal student initiativesduring the tutoring session. If the student asks a question the plannersuspends the current plan, carries out the student request, and then resumesthe suspended plan.
Since one of the main goals of CIRCSIM-TUTOR is to provide a naturallanguage interface, the discourse planner is designed not only to providesophisticated discourse control, but also to create the internal logic formsfor the text generator to generate the sentence. A short tutoring scenario isintroduced, which came from a transcript of human tutor and studentinteraction, to explain the internal process of the system.
7.2 Future ResearchThe current version of the student model is limited to the overlay
strategy, so the planner can support tutoring on the overlay errors only, notthe bugs. The tutoring strategy for the bug library has not been developedyet, so the system cannot tutor the student about bugs at the moment.
Another important tutoring strategy is giving a more detailed level hintduring the tutoring session. Also it needs more anticipation from othermodules; the domain knowledge base needs much more detailed knowledge, theinput understander and the text generator need to expand their lexicon andlogic forms to contain all the variables at the detailed level, the problemsolver needs to be able to access the knowledge base and extract a hint, andthe planner needs to have a general strategy for deciding the content of thehint for every situation during the tutoring session.
CIRCSIM-TUTOR supports four pre-determined problems as a curriculum, sothat it does not really require curriculum planning. Our expert tutors aredeveloping many more procedures for the system, which may requiresophisticated curriculum planning in future versions of the system.
I.Q. BIBLIQQBAZZX
Anderson, J.R. 1988. The Expert Module. In: Polson, M. C. and Richardson,J.J., Eds., Foundations of Intelligent Tutoring Systems, LawrenceErlbaum Associates Publishers, Hillsdale, New Jersey, 21-53.
Barr, A. and Feigenbaum, E.A. 1982. The Handbook of Artificial Intelligence.William Kaufmann, Inc., Los Altos, California, 2:225-294.
Brecht, B., McCalla, G., Greer, J., and Jones, M. 1989. Planning the Contentof Instruction. Proceedings of the 4th International Conference on ATand Education, Amsterdam, Netherlands, 32-41.
Brown, J.S. and Burton, R.R. 1978. Diagnostic Models for Procedural Bugs inBasic Mathematical Skills. Cognitive Science, 2:155-191.
Carr, B. and Goldstein, I.P. 1977. Overlays: a Theory of Modelling forComputer Aided Instruction. AI Memo 406, AI Laboratory, MassachusettsInstitute of Technology.
Charniak, E. and McDermott, D. 1986. Introduction to Artificial Intelligence,Addison-Wesley, Reading, Massachusetts.
Clancey, W.J. 1982. Tutoring Rules for Guiding a Case Methods Dialogue. In:Sleeman, D.H. and Brown, J.S., Eds., Intelligent Tutoring Systems,Academic Press, New York, 201-225.
Clancey, W.J. 1987. Intelligent Tutoring Systems: A Tutorial Survey. ReportNo. STAN-CS-87-1174.
Davis, R. and Buchanan, B. G. 1985. Meta-Level Knowledge. In: Buchanan, B. G.and Shortliffe, E. H., Eds., Rule-Based Expert Systems. Addison-Wesley,Reading, Massachusetts, 507-530.
deKleer, J. and Brown, J.S. 1984. A Physics Based on Confluences. ArtificialIntelliaence, 24:7-83.
30
2 8
Dede, C.J. 1986. A Review and Synthesis of Recent Research in IntelligentComputer-Assisted Instruction. International Journal of Man-Machine
Studies, 24:329-353.Derry, S.J., Hawkes, L.W. and Ziegler,U. 1988. A Plan-Based Opportunistic
Architecture for Intelliyent Tutoring. Proceedings of IntelligentTutoring Systems. ITS-88, Montreal, Quebec, Canada, 116-123.
Elsom-Cook, M. 1988. Using Multiple Teaching Strategies in an ITS. Proceedingsof Intelligent Tutoring Systems: ITS-BB, Montreal, Quebec, Canada, 286-
290.Pikes, R. and Nilsson, N.J. 1971. STRIPS: A New Approach to the Application of
Theorem Proving to Problem Solving. Artificial Intelligence, 2:189-208.
Forbus, K.D. 1984. Qualitative Process Theory. Artificial Intelligence, 24:85-
168Harmon, P. 1987. Intelligent Job Aids: How AI will Change Training in the Next
Five Years. In: Kearsley, G.P., Ed., Artificial Intelligence andInstruction, Addison-Wesley, Reading, Massachusetts, 165-190.
Hasemer, T. and Domingue, J. 1989. Common Lisp Programming for ArtificialIntelligence, Addison-Wesley, Reading, Massachusetts.
Kearsley, G. 1987. Artificial Intelligence and Instruction: Application andMethods, Addison-Wesley, Reading, Massachusetts.
Kim, N., Evens, M., Michael, J.A., and Rovick, A.A. 1989. CIRCSIM-TUTOR: AnIntelligent Tutoring System for Circulatory Physiology. In: Mauer, H.,Ed., Computer Assisted Learning, Proceedings of the InternationalConference on Computer Assisted Learning, Dallas, Texas, Springer-Verlag, Berlin:254-266.
Kim, N. 1989. CIRCSIM-TUTOR: An Intelligent Tutoring System for CirculatoryPhysiology. PhD. Dissertation, Department of Computer Science, Illinois
Institute of Technology, Chicago, Illinois.Kuipers, B. 1984. Commonsense Rearzning about Causality: Deriving Behavior
from Structure. Artificial Intelligence,_24:169-203.Lee, Y., Evens, M., Michael, J.A., and Rovick, A.A. 1990.IFIHS: Ill-Formed
Input Handling System. Proceedings of the Second Midwest ArtificialIntelligence and Cognitive Science Conference. Carbondale, IL, 93-97.
Lee, Y. 1990. Handling Ill-Formed Netural Language Input for an IntelligentTutoring System. Ph.D. Dissertation, Computer Science Department,Illinois Institute of Technology, Chicago, Illinois.
Leinhardt, G. and Greeno, J.G. 1986. The Cognitive Skill of Teaching. journal
of Educational Psychology, 78:75-95.Macmillan, S.A. and Sleeman, D.H. 1987. An Architecture for a Self-Improving
Instructional Planner for Intelligent Tutoring Systems. Computational
Intelligence, 3:17-27.Macmillan, S.A., Emme, D. and Berkowitz, M. 1988. Instructional
Planners:Lessons Learned, In: Psotka, J., Massey, L.D., and Mutter,S.A., Eds., Intelligent Tutoring Systems: Lessons Learned, LawrenceErlbaum Publishers, Hillsdale, New Jersey, 229-256.
Michael, J. A., Rovick, A. A., Evens, M., and Kim, N. 1990. A Smart TutorBased on a Qualitative Causal Model. Proceedings of the AAAI SpringSymposium on Knowledge-Based Environments for Learning and Teaching,
Stanford, March 27-29, 112-117.Michael, J.A., Rovick, A.A., Evens, M., Shim, L., Woo, C., and Kim, N. 1991.
The Uses of Multiple Student Inputs in Modeling and Lesson Planning inCAI and ICAI Programs. Submitted to the International Conference onComputer Assisted Learning, Nova Scotia, CANADA.
Murray, W.R. 1988. Control for Intelligent Tutoring Systems: A Comparison ofBlackboard Architecture and Discourse Management Networks. ResearchReport R-6267, FMC Corporation, Santa Clara, California.
Murray, W.R. 1990. A Blackboard-based Dynamic Instructional Planner. ResearchReport No. R-6376, FMC Corporation, Santa Clara, California.
31
2 9
Ohlsson, S. 1986. Some Principles of Intelligent tutoring. In: Lawler, R. andYazdani, M., Eds., Artificial Intelligence and Education: LearningEnvironment and Tutoring Systems,_Ablex Publishing, Norwood, New Jersey,204-237.
Peachey, D.R. and McCalla, G.I. 1986. Using Planning Techniques in IntelligentTutoring Systems. International journal of Man-Machine Studies, 24:77-88.
Rovick, A. A. and Brenner, L. 1983. HEARTSIM: A Cardiovascular Simulation withDidactic Feedback. The Physiologist, 26:236-239.
Rovick, A.A. and Michael, J.A. 1986. CIRCSIM: An IBM PC Computer TeachingExercise on Blood Pressure Regulation. XXX IUPS Congress, Vancouver,Canada.
Russell, D.M. 1988. IDE: The Interpreter. In: Psotka, J., Massey, L.D., andMutter, S.A., Eds., Intelligent Tutoring Systems: Lessons Learned,Lawrence Erlbaum Publishers, Hillsdale, New Jersey, 323-349.
Sacerdoti, E.D., 1977. A Structure for Plans and Behavior. Elsevier-NorthHolland, Amsterdam, the Netherlands.
Shim, L., Evens, M., Rovick, A.A., and Michael, J.A. 1991. Effective CognitiveModeling in an Intelligent Tutoring System for CardiovascularPhysiology. Proceedings of Fourth IEEE Symposium on Computer-BasedMedical Systems Baltimore, Maryland, 338-345.
Sleeman, D.H. and Brown, J.S. 1982. Intelligent Tutoring Systems. AcademicPress, New York.
Sussman, G.A. 1975. A Computational Model of Skill Acquisition, Elsevier-NorthHolland, New York.
Swartout, W. 1988. DARPA Santa Cruz Workshop on Planning. AI Magazine, Summer,115-131.
VanLehn, K. 1988. Student Modelling. In: Polson, M. C. and Richardson, J.J.,Eds., Foundations of Intelligent Tutoring Systems, Lawrence ErlbaumAssociates Publishers, Hillsdale, New Jersey: 55-78.
Wielinga, B.J. and Breuker, J.A. 1990. Models of Expertise. InternationalJournal of Intelligent Systems, 5(5):497-509.
Winkels, R., Breuker, J, and Sandberg, J. 1988. Didactic Discourse inIntelligent Help Systems. Proceedings of Intelligent Tutoring Systems:ITS-88, Montreal, Quebec, Canada, 279-285.
Wilkins, D. E. 1988. Practical Planning. Morgan Kaufmann, Los Altos,California.
Woo, C., Evens, M., Michael, J.A. and Rovick, A.A. 1991a. DynamicInstructional Planning for an Intelligent Physiology Tutoring System.Proceedings of Fourth IEEE Synposium on Computer-Based Medical Systems,Baltimore, MD:226-233.
Woo, C., Evens, M., Michael, J.A., and Rovick, A.A. 1991b. InstructionalPlanning for an Intelligent Medical Tutoring System. Proceedings of theThird Midwest Artificial Intelligence and Cognitive Science Conference.Carbondale, IL, 31-35.
Woo, C., Evens, M., Michael, J.A. and Rovick, A.A. 1991c. Planning in anIntelligent Tutoring System. Postar Session, International Conference onthe Learning Sciences, Evanston, Illinois, U.S.A.
Woolf, B. 1984. Context Dependent Planning in a Machine Tutor.Ph.D.Dissertation, University of Massachusetts, Amherst, Massachusetts.
Zhang, Y. and Evens, M., 1990. Extending a Knowledge Base to SupportExplanations. Proceedings of Third Annual IEEE Symposium on Computer-Based Medical Systems, 259-266.
ILVIALTC1. 4 MAN 934/10/4 ALL.AittA. 11147.771Y. NAT _LANG
Dr. nollip L. Ar AmmoUmwsroy 11444449444
D499449444 el 244474.4.0
75 E4+4 ItAn Rae414214 IOWA HA
7.6990441414 MN 33131
Dr. VW MaraDeper04944 1.4494orIlmens 1110~/Gods% 00102
De. DWI AMowe. C944 151No, 21949.491 PAD CrowUrn Deep. CA 92152.400
Dr. Tony MordCa49 1142MO 111.4 54 $944 114.499414
SOO N. Owen SA
Ar14414444 VA name")
Dr. Tow H. MimesCaesor Us As Lady el 144444.5174 C441444Ve R.44.444 Craw31 47409
C34.4441., IL SIM
Dr. SNI49444 J. AMA.* C144.4.Celleped 191onseme StudiesDrawl UsA9NaryPt4140190944 PA 11404
Dr. Cow, AsOoE lowieed Tobas 544.449249949/444 NJ 011541
Dr. 141s04.1 MawrNYNIXAl LaterearySOO Warsaw/ AwareMire N.M. NY 10101
Dr. POWs YapsSAW ideolor410 L UsioreOr. RA IMIDUsreario el MieriiseMe Moe. MI 161104720
Dr. Marl 3. SW*Noy Peroosol RAD CMNYSo Dime, CA 12122400
poet MA. Irmo 0. IroiUsMm..dI
64.109.444 NvisIsleUsirswiia WarsDX72 law Os P. Moro 11ITALY
floariAI DM ds. thAermeaPeseheb 1640
D.4100 SWUM L WEST GERMANYMOM AMU WC OP DORMANT
MOW,. D. lorrMow el 1144 494BrOneereorl Saws
MPOIR/141., NI, 114 410Af11, DC X0324148
Dr. Throes 0. PeroDoesowir *I brawler,Urivraity el PaausertKiwi &WeeRoMerso NV 14627
Dr. New NowDowers. el Gower Uwe
$440, &Wm I14444.144114 U14944444
114.6911141, TN 17221
Dr. Jobe 419.14
Theboos Caw,
Diorama. Um
Colosble Llerwmity373 Wail 1306 SowsX.. York, NV 10027
Dr. Melt Illiss0414DNA 44 NA CAM.,Iwo thereftioWiwi', UM. Roo asPorMaess. RI MI:
Dr.l. C. 11.04.weMosher* lisrosore. IAAHowew WOW elfordo& ard TeeMelogr
Orlareebors MD WM
Dr. Mow IwoCade 212
Noel TOW. 5 roomOriodo PL 33361124
Dr. Aro aweOodeerr SOW el BMW*Uslwrier of CaiilsodeIMST.4313 TAW 14.11brelaMy, CA NM
Lk. Abe T. tomJew 3. MeDeweil FrodoleeSelo IMO1034 isol Mowed MO34. LoW MO 0117
Amber* MA 00003
ed Noel adowes*DOWN (11.3)
NM rfo. P1. 32301
Dr. Peel CAWPurism UrinereirrDiseerlesW. 'Arrow. IN 47007
Oar ri NoweI RowebCede 1142Kr/ N. 04O4f Ssw1AlVaoso. VA W174030
Meow'NOW 3row DepswestCsdeNor Poiewel RAD C.a...Sir Mph CA 121124600
'Weer, Cede DIN. Porseerel RAD Cowire Deers, CA 103323000
NsW Todakag Swear C44441Awe Dr. Rebers Noe, Core 24217130 Rowed PretwerOriels* II. 32126.3224
RAD Csordrisser, Awe Jra HatOdke Of doe MHO, mrr, '24.11K1drama* el IN NA% A4443417%Aimee% DC 7027>X00
Coessa494 01544Noval 11.44994.14 labonooryCods 4427
WNW." DC 207143003De. Jorge Cow
Alressore Sews. NW De. /Me it Comm*%Wyss, DC X001 Derenwor el brebelery
10 PryAkhogy PornoDs. PO Carlow Wow UeiverairCrewpir.31441.0 Urivrokr New °Was, LA 70115Dormosei M royeeropPloWors, PA ISW Dr. IOWA Cow
Cede 142De. kowder Csoollre N. Poressel RAD CesarIdeoriers1 Tway Senior 36s Mops, CA 9215241100WWI* RodNew" NI 01141 De. Iwo& IL Coo
Aar,. Senwon,De. Rob W. CNA" P.O. ea 110MSC, Wakes Ha um. lietten. CA 13102Cow., MOW WOW./P1.Sorg0, PA 11113 CTM4.I.4434 LAnny
no Owls. RcMPae10. Cu* Morow, CA 1001133311
IMplems Progrw Roes 136/WW1 Wow Parolotles Dr. Poo Celiemer%Alvah DC 1315s Dowser, C41141 ter CorOwe
SamosDr. Pool R. CUM*/ 101 011. LAWN 3MPowiresies 11611 Toole Poi Mew1111 Nenh IPL Myer Dr. Celeorsw OH 41210.1102kW WAMO." VA 722111 Dr. COoler I. DM.
11.1wrireel Toler UAWD o kw drew Mod 34., 72.TC.M. *awl Avow Prlararo /41 01341011ei et Noel Row**No* Odor k Marlow Di% 'WNWASirmao VA 72/374400 Applied Slew We:we
P.O. lea 1172Do Rowed I. G.M. Isier, PA WMUM LAMP /Woo AiMmeAL/IIRMIL Dr. /Wee DowIrrehs M/A TX WM mow sm. Usipmmity
DqeM..I54 PrWeleryTellehows, fl. 37106
0.o1re ClassTe164 1444
Doprisese FINNAlop1.1.1.94444
Marseebews
33
BEST COPY MAILABLE
Dr. I. Soot Demhot" M IldeemiseUMM.lpe4 BIWA Cilorrisnts 1.161. mVasorrsorr, IC CANADA VrT I ZA
Deka. Thboisalleforwiee Caw
1711CODA.2 Dr. J. D. IIrseer Oldest OX1X30Cesare Woos SII4 3 Immo We Dame Asap.. G11111213 KINGDOMAkahrkft, VA 22114 MO N. 114ewraerd Si.44 Ceps.) Alweedes. VA 2/311 Di. Sass R. 04444sir
Nat* Calleis, See 45Dr. New [Apse Dr. Usk ISseer Vas4441414 GeirewayOrpeirmws isr Bwereie CassairAlrlles Wire*" 14444414. TN 37303
Cresermies rod Dereleriews Depeemosi el &Orb2, no Asay.P.4441 bake" PA 13313 Me. Iimold 04461444710141 PARIS Gaiweerre 44 DCFRANCS Dr. Cad H. Produelaws Drpedwom OM Ilsekeseieg
Deft 44 Sawsdeser Piweelsw 1134. Q. Re= 112Dr. UAW Nese M40411Uelerser OM Cessiessw AVOW. NM.Greasier Sabeel *I Easmiss 3701 aureole 111114 W.110141. DC 20[011.14.sied7 44 C.4441.ess 0g10414 GoebelSwim Swam% CA 10101 CANADA RSA IT2 Dr. Sloth Ore
A114141./MOWDr. Jobs Mks Dr. N. Predwiliew Iamb All. TX nz0.1401N. Prswasa RA 0 Crew ISMsaiessi Tales Sorrow0444 If (16.R) Dr. T. Delrodsre1Sre Lisp, CA 121524404 Pewedee, NJ 0134 Osoile whom a
TweeelesyDr. lefteg Eldela De. AVM N. Ileeild Same* 44 lalearielGawenely a ca4..... Armin. swil, 410 ser Slower Pagiosing
S. Dote Seale. AFS. DC =33244411 Aires GA 301324305Drrerswer el Lapsieei. C-CalLa Jells. CA NOM De. ALM. Friedele CA. Streirre K. Oweisi
Derailment 44 Prpoolop 0454. 41 retweesel RADFAIC FeraityAsierseres Deiveredy rlAbbrem ISX1 11 Se, NW, Re*. 44421301 Piweta Deny. 5M4* ))0 1544.44414e. *lbws 0113w el Penee441 MwastArreRedwdle. MD 20130.4301 CANADA TiO ISM Wmariese. DC 20410
De. Moak DomDors M Ceerese SinewMOOS 111600161 ei Tedweirgr10 Wes 31s SsedMays U. gals
Ds WWI IL GimeDI..44 Copses SomeDereneree PITA*Mgh, Row 312Usiweekre M MowToros AZ UM
Dr. Lamar D. 44y44 Oar. Deponmea elUS 0664 44 Pswee44 Mawiereet Csersew Sawa001.44 Perrerwl Rows* NM Omer MIMS throveity
Derdereirm Cepaies bake, VA190R M. NWWerbisoes, DC 30411 Di. 11444 0414s
trawl Ramat LeS. CM4 3334Dr. hee-423eses Sasso* 431$ Cerweere Awe., SW.Irmo Reneese Om is Wirlisews. DC 7W73-10))Maessiseel 11414444 Sarsers
114.44wer C.4144ne Dr. Plialp 04144Iris" CA 14717 AR1444 Orsiee
ATT10 PERLICDDr. 1444.4e J. Parr Pert 04014e. OA WASAra* 114w44 Looms71111.10 Di. /WSW OisibergIOW Siireberrer Meese Res M4A2ear4/44. VA WU Tarbes CeWye
GUAM@ WismarDr. Menbell 1. ha 511 Wei 4334 SaraFinliedd Ca. 3140 Y4014 HY WV2320 14.M Versos SinaiMr*" VA WV Di. Drwr Dimmer
Sieediwel Todiei SeniorDr. PA. name, Peewees NJ 0041C444 31NPRDCSea Despe. CA 023 0314011 XS De* Awsw
Lasisra. VA 22773De. Cleidisie FelleeesCopies. Slime LAWeewerParser Ihnereas211 Nauss SweetPrimers, NJI 01344
Dr. 114.4441 L. /mu.eAmmar Casio T.P.O. See 1M
041, IA Ina
Dr. Rabin GIeweLemma Raw*
A DerwMpseel CoraLIelveieley Nide*3100 0114re $ emiMew^ PA WM
Gc. Menai D. OM*101 HMISMONI TWINS
Dr. VIseer PirMs Wee, NY 14131Dowiriseer FreeSeergeMiesesaser Gael* Ds. 4.. OisEabergReawkle, 140 30130 D.u.w,44 Prraole&
IWess. lleirwerreDr. Miebsel Plesiegsw PrIswares NJ St 141.10111Code IdNPRDC . PreL Amapa GorseSw Dies% CA 021324000 PRO, Heiv. Ordwd
II Kaele Rmd
34
D.. AAA," C. GreewirDeperuwm PorAologyWs*. Sum LlswereryMirspria. TN 31152
Dr. Worse OmGriarem 346444 44 114.44044Fwdbes Oriverwry113 Wew AAA StlletNM Tads HY 10023
Dr. how O. GrosseUMW 13aermweSusleed UivernftryRims 311Sesawa, CA 04303
De. Meier' OrwerrrrgCame he Merin. Spiro.Room 244111 0444.4.004 Nom"sere GeireeserDears MA 02213
Dr. 1404.44 NOBDORSI= OMSHP.O. Sus 1420D-7901 lidearodasfre IWES OERMANY
Dr. Hew? M. HMGKai 11swereesMIS 33r4 P.M. NorthMap" VA 32)37
bkard HsIu.. lik4404Srliftes Laser,11444e044 Taste Servs.Careww. wa Roslae R44414Priewers, 441 0041
Mt. H. Hasher/srDeyomem 44 Campos SneersGeier Mews GeSsnarPeirm. VA DM
Dr. %Ore J. HawsLairmiss Kass**
Aersidise Reiser*Siam. Rowe* Crew733 Cellos PAM SowPriersiers, NJ 013411
Stews HorsedPAW*, The lkiwiers1 441
II lin 1110 Ji IIti fli flhili ?fill fit
I1 1 g ill ! I t I 11.11/1 ill
; )1 3 ; lid 1 11 iiihr 111 igh 114 Pp 36A iv3 iv8 g Jiv I I 1 pi"I lit I 1111 1/111 Nil 11011 Jill I 'JP/ JOIN 11111 :111%1Ags cl; zi A 21 A A .EJ dd it. s
`1100382
Yenned CAOIDOMOWN1 Dgaaa la el PenneyFrown Lean. NFILDC Let benne ei C4444.44
MO la alai len Cols 041 Peeler, CO 1103044144Unman n Metope Soo Mop, CA W3124400Aso Arno, 141 01100 Dr. Hen. Non
Wrens ATM MSCDr. Mend B. lOyer Noe Cann Set 4044 Xenon Anvi iroore 1sonntDemesnes el Pennine Is Millen Ineerne 3444 Ftnehewer Aa.Unworn ell Canner Nreel Palatal Lebeanwp Alonnon VA 7:2333.3400Sono 11444ers. CA MOO Ca& 3110
Weenen, DC Y13734000 Dr. Senna PeterDr. Konen MaKeeen MOILCambia Unreney 011eo a Nen Ilernee, 101 SW Men Sete 300Deponent ed Canner Senn Cele 114X3 Perna& 001 4171114451 Caernor Seaga leanin ea N. Oala katNew Yen, XY 10077 AelkiterA VA 1331744410 Dr. 1. Wren Palen
(4 Capin) A714il1D1Dr. Hone Mtlenno Men An TX 70273Nen Pennon Peneni Spain MeieI he Peemen
old Denleneno Cense Wortopern Dr. Ben PennCede 14 Cain el Noon Ponesed (11313-0U7) CamilinerleMPIPteb ILPlee Dien. CA 11121324 10) Oneonta el he Non 21314ineSnet. 1S9r) "anle !"-Pnti.-Woreleopes, DC 111311440 PreonleinionnyDv. Menet Menem 114".1111-41$42 17Vanton . TT rmo1DEIDL ALACFH1 Dr. heili Orme'111.110 24/1 WY Son 739.1 Dr. Lawn RenickWriet.Porontes AIL OH 4102 NASA Ann Planed GNAW Leonel% A It 0 Cater
Wen NW, CA 94013 Usienny of PinnenoDr. Joel A. Mena N011 Mtn SeenDrperinna of Plysiene 14r. hem Oriel Pienne. PA 13213Mg PrennenesSe Lens Novo Tenn Sewn
Seise Conte Omer (One 212) Dr. Chinn Rien4Pen Moen Cake 12110 Renton rertarry WI Nap 501.14anon, IL 40612 Orloode FL 31436.3224 Oman Anna Swan
144414pser, NY 11714Dr..144ros D. MOON DIL ha* 011ola,LP.DC Intern fee Deere Anent Mr. W. A. NanUnman et benne 1101 M. lknoreped So. Mei% Haan been Dine..1104 ov. Sweet Anweine VA 3:D11 Neon Trento Syne. CCM.Pieteorge PA 13310 Cede A
Dr. Mann Port 17330 Prawn PanneDr. Alin Moore Arey Alreaten Intiont Oren% FL 321134.3224Ilennern Teener P6111.2
Looneerno USC 10111 lIaaaborer Anna Dr. Line 0. Newelno Pl. Hann Dr., 14444 3011 Alossidels, VA 77133 Sam" Innen& anItedeno leen. CA WTI Yrosopenten Prepsn
Dr. Ma P. Otleo ol Totetelny OneontaDioritoime Seinen Dinette ILNiono ke ne Canna n doe Untr4 SuluPenn el Men Poneebol Luning Seinen Wooleene DC 20510PER3.4710 Nnokreente Uneven,Woolnen. DC 3111711 INS Mono Anon Dr. Wenn I Mtn
anonen 11. 40101 Senn Treanor. InAnemia Dee. It Mown Nun 47221 honorer Comm CiftleNMI Innen Meng Ceetatal D. Rey S. Pens See XOGee N.42 AM (F1311.11) Romesor, GA 30002HAS Plenoblo (73) X04 lintkoreer AnneMininen TN 30014 Aimee* VA 24)331 Dr. Nene Wm
Men Pala. Devisee (GO 262)Dupery Dinetor Moneoree, Dr. Men N. rem 17110 Mown Parkway
Peremrei an Moen Der. 14.444111eseneo sel 'Donal Mon Moine Sower Cent.Men Set Snores Cerren Penne *fon Afro, Celine FL 32424.3224ATM: Celt OIMP SU 0e1re47Weinen, DC am kaolin 303
Powneen FL 3130114004
Dr. Moor SennTee latrine Mr ter
CDR Mel C. Petbo Lenin SnowMod Ponosiont Nonienewo Uoivertery
Mr. A Mean Seen ION Men, AnneTonne Uneven Cede 041.11 inenee U. 40201Fer. /1011. lletepons Ooneentoon Mnesvp, CA MUP. 0. lert 217 Dv. Mark Wein7100 All Ronk& Dept. 4 Airelahotwvo Senna 331 WienoonenThe MITHERLANDS Con 34 313 lerneown1 An.
Med Patornono Senn Ore 11.01Deese% 'Ma Nen on Plootton. CA 1130414014 Wee Pert CA 114023
Fen hoe Sp% Dopl.Mee See. Yoe Pros Dm Sinn MAW O&M*, 544044er
MIL Ila Ma* (MOTHS%) Donoenet e4 Onto 1.4m414. RAD CrierInernen FL MOP oe1 Gonne Solemn l 1 aver* 44 Piens*
Mein 1930 Olen SinnMinn SOT liteberolo, PA 1)140Trines Senn Deponent Gonne% MA OSNPROC (Coe 14) Dr. Mon $ennaSoo Diego, CA 10152410* Do. Prete FINN 4140 Porten
Seen of Inerens Centbn, CA MOHDimon Unneny el GagneTrento Themeleg Doparonna Ikeloolop, CA PITH Merle MonteNFRDC (Cede 13) Dep. Penn& YenSem One, CA 1121524005 DK Mores Peen thie. lenolor
36
BEST COPY AVAILABLE
11 fill WV ilifi III
iii; fig
1111
IP; 111
cq
&f
&.
Ff iffr
at:
-sk
<if-
I*
if
.YA
q=4
qui !Ito
t
liti !
ff iff
f
rt gli
plo it,11
lif IIFI