PROBLEMS OF COMPUTER SIMULATION1

Nico H. Frijda

Psychology Department, University of Amsterdam

Computer simulation of psychological processes presents a number of problems, whichare too infrequently discussed explicitly. The most important of those are the relationbetween a program and the theory it embodies—it is often difficult to distinguish betweenthe theoreticallyrelevant aspects and those of a merely technical nature—and validationof program output. Some illustrations of the activity of detailed process simulation andsome suggestions concerning validation are presented.

c+.i

Computer simulation of psychologicalprocesseshas proved itself an important

tool in the development of psychologicaltheory. A number of successful programsparallel human problem-solving, pattern-recognition, or other behavior. Yet severalmethodological issues make it difficult toappreciate the contribution of these pro-grams to psychology. In this paper some ofthese issues are discussed.

It may be useful to repeat the differentfunctions in which computer simulation canbe of value. First, computer programs canserve as unambiguous formulations of atheory. The program language is precise:the meaning of a given process is fully de-fined by what it does. Moreover, program-ming principles such as looping, nesting, andrecursion enable one to express clearly com-plicated processes. Such programming al-ready operatesat the level of flow charting.

Second, computer simulation is a meansto demonstrate and test the consistency andsufficiency of a theory. If thebehavioral datawhich the theorywants to explain are in factreproduced by running the program, thetheory has beenproved capable of explainingthese facts. Moreover, running the programunder a variety of conditions may generateconsequences of the theory which can betested against new evidence. These conse-

1 This paper is based on a contribution to theRound Table Discussion on Computers in Psy-chology at the 15th International Conference ofApplied Psychology. The research is supported bythe Netherlands Organization for Pure ResearchZ.W.O.

quences may be unforeseen and they may bequite important—such as the discovery ofperformance fluctuations without the intro-duction of stochastic elements (Feigenbaum,1959). Extensive experimentation is possibleby running different versions of the program;decreasing or increasing fit with behavioraldata can indicate the role of various com-ponents and parameters. An example isGelernter's geometryprogramwhich was runwith and without the diagram-based in-ference heuristic, with resulting difference incapability (Gelernter, Hansen, and Loveland,1960).

Third, computer simulation mayserve as aheuristic in the search for models. The effortof getting a computer to perform a giventask may lead to illuminating psychologicalhypotheses, even if no behavioral evidencehas been taken into account. Moreover, aprogram which solves problems is by thatsole virtue a candidate for a model and de-serves the psychologists' attention. After all,proving theorems or recognizing patternswas until recently uniquely human or ani-mal.

Computer programming principles of non-psychological origins may as such suggestpsychological models. The TOTE is a clearexample (Miller, Galanter, and Pribram,1960) ; Yntema's (1962) model of immediatememory is another. To regard problem solv-ing as the functioning of sets of programsowes its existence, or at least a renewedimpetus, to the way computers have beenmade to work.

->\)

Behavioral Science, Volume 12, 1967

60 Computers in Behavioral Science

Another source of psychological hypothe-ses resides in the necessity, evident duringprogramming, to introduce specific condi-tions or auxiliary mechanisms. It may ap-pear, for instance, that a given process canbe realized only if the program is able to re-tain its prior efforts or to perform a givengeneralization. A theorymay sound fine andplausible, but usually it is difficult to seewhat is neededto make it work. In the neuro-psychological field the simulation of Hebb'scell assemblies has demonstrated this quiteclearly (Rochester, Holland, Haibt, andDuda, 1956). As a matter of fact, one of themost important functions of computer simu-lation resides in uncovering the implicationsof a theory. By implications we mean here:necessary assumptions or conditions withoutwhich the process could notbe realized in theway desired.

In order to appreciate thecontribution of agiven simulation effort to psychologicalknowledge, there should be clarity in tworespects. On the one hand, the relation ofprogram to theory should be clear; otherwiseit is not possible to make conclusions abouttheory adequacy on the basis of programsuccess or failure. On the other hand, thecriteria for correspondencebetween programoutput and observable behavior should beexplicit; otherwise no evaluation of simula-tion success, and thereby of theprogram, canbe made. I shall discuss the methodologicalissues in both problems.

THE RELATION BETWEEN PROGRAMAND THEORY

According to some formulations in the lit-erature, aprogram is a theory. This seems anincorrect way of putting things. Rather, aprogram represents a theory. It does this withthe help of a number of mechanisms whichare irrelevant to the theory or which thetheory might explicitly disclaim. The lowerorder subroutines and a number of technicalnecessities are determined by the particu-larities of the programming language, themode of operation of theparticular computer,and the special limitations inherent in seriallyoperatingdigital machines. Manyoperations,too, are just shortcuts for convenience or

results of ignorance about the psychologicalmechanisms involved. Random selection bymeans of random number generators, scan-ning of serial lists to find a certain alterna-tive, use of counters to check on capacitiessuch as permissible depth or effort, or settingof afixed number of working storagecells areexamples.

Much in the program, then, is not theory.This has methodological consequences. Al-though parts of the program do not belongto the psychological theory, nothing in theprogram indicates when this is the case, andwhen it is not. Nothing in theprogram indi-cates from what level onward it is meant tobe theory, and up to what level disagreementwith psychological data or psychologicalplausibility is of no concern.

If the program does not show what istheory, the program's author has to. We aredependent upon the author's explicit state-ment of which routines embody his theoryand how and under what conditions theyfunction. Serious problems of communica-tion arise in this connection. Descriptions ofprograms are usually presented in a dis-cursive manner. Processes are described inmore or less informal language and in arather global way. Presentation is apt to beabout as vague as in purely verbal theories.There is full loss of the programclarity, andit seems that one of the main advantagesofcomputer simulation—unambiguous theoryformulation—disappears at the moment itshould manifest itself. One must take theauthor's word that the loosely indicatedprocesses do what they are supposed to doand that what they do is what the theoryprescribed. One must, moreover, take hisword that the processes do what they aresupposed to do in the wayandunder the condi-tions stated. By this proviso I mean withoutadditional mechanisms not mentioned in thedescription. I will return to this point. Exam-ples may be found in the most unsuspectedquarters. One, for instance, is present inSimon and Kotovsky's otherwise beautifulpaper on their letter series completion pro-gram (1963). "The pattern generator seeksperiodicity in the sequence by looking for arelation that repeats at regular intervals"

Behavioral

Science,

Volume

12,

1967

Computers in Behavioral Science

(p. 540), and "If this kind of periodicity isnot found, the pattern generator looks for arelation that is interrupted at regular inter-vals" (p. 540) are the only descriptive state-ments concerning the main process of theprogram. It is true that the program isavailable for closer and convincing inspec-tion. Few, however, understand the languageand the programs are noteasily obtained.

Finding a solution to this communicationproblem is not a simple matter. Global, in-formal description usually negates the bene-fits of computer simulation. Publishing theprogram itself, of course, is neither feasiblenoruseful. Thebest road seems to lie midwaybetween the two. It will in generalbe possibleto describe the relevant processes unambig-uously by naming the subroutines concerned,by stating their precise input and outputconditions, the conditions of their activation,and the transformations they achieve. It willin general also be possible to do this com-pletely—in the sense of mentioning all tests,parameters, and so forth upon which theseoutputs depend—without leaving the level oftheoretically relevant subroutines and with-out descending into technical detail. Theoriginal report on EPAM (Feigenbaum,1959) and the latest report on the GeneralProblem Solver (Newell, 1963) contain pres-entations of this kind.

This recommendation hides two require-ments. The first is that the program struc-ture reflects the structure of the theory. Adescription as meant here will be possibleonly if theprogram hasbeen written in termsof independent, theory-relevant routines;only under this condition, that is, can onemake sure that the description is faithful.The theory-relevant routines must, so tospeak, be isolated from theoretically irrele-vant auxiliary operations and must be asindependent of technical realizations aspossible. The goal is to specifyfully thefunc-tions of each routine even without relatinganything of the mode of operation of thelower order subroutines. Simulation pro-grams are usually organized in this way—orreorganized if the original plan was jumbled(Baker, 1964)—but published work tends tofollow this organization only incompletely.

The secondrequirement is quite important,though few published reports meet it. It isthat the statement of the theoretically im-portant program segmentsshould contain alldetails about auxiliary operations and otherconditions (such as memory storageregistersor counters) which influence or codeterminethe operation of the main routines.

Let us take an example. The success ofGPS with logic tasks is highly dependentupon the ordering of the difference betweenpresent and desired state; or, Simon andKotovsky's program (1963) needs, for reallysuccessful operation, some fairly sophisti-cated routines to check on the correctness ofits hypotheses. Since these features collab-orate in determining the program's successor the success of the simulation effort, andsince they do this at the same level as themain subroutines, they belong to the theory.Therefore, they should not only be men-tioned; they should be stated explicitly asparts of the theory, if not in the specific formthey have in the program, then at least ingeneral functional terms.

Not to do so is not only misleading, it isalso unwise. As indicated before, program-ming points to the implications of a givenmodel. This can be one of theprincipal gainsof computer simulation, which should not belost or devaluated. For instance, naturallanguage understanding by means of predic-tive analysis necessitates a series of specificmemory cells for holding unfinished preposi-tional phrases such as adverbs in search of averb (Lindsay, 1961). Here are implicationsas to memory load, then. As another example,recursive problem-solving routines, such asin the older version of GPS, presupposeelaborate retention of intermediate results inorder to enable retracing of steps in case offailure (Newell, 1963), a presuppositionwhose psychological realism is doubtful. Oragain, learning in a communication networkproves possible only when behavior changesare based upon general strategies called"impatience" and "persistence," and notwhen based upon simple reinforcement ofuseful behaviors (McWhinney, 1964). Thismajor result of simulation effort is men-tioned by its author only in passing; but

Behavioral Science, Volume 12, 1967

Computers in Behavioral Science62

such implications should serve as startingpoints for further research and tests of thevalidity of the model.

Details of this sort—the influence of aux-iliary operations or eventhe specific mode ofoperation of the major routines—are ingeneral hardly discussed. Simon andKotovsky (1963), for instance, do not de-scribe the nature of the differences whichdistinguish the several versions of theirpro-gram. These differencescould have been oneof the most interesting points of their study.Lack of necessary detail in program presen-tations is, I think, a shortcoming of verywide and general occurrence. In the case ofsimulation programs, it seriously detractsfrom the value of the work; performance assuch is not so important here as convincingthe reader that the reasons for this per-formance are plausible. Lack of informationimpairs the effectwhich computer simulationcould have on psychology as a whole.

THE RELATION BETWEEN OUTPUTAND BEHAVIOR

Evaluation of program performance, thesecond major problem area, involves twoactivities: evaluation of the program's con-tribution to psychology; and evaluation ofthefit between program output andbehavior.These two, value of the theory and predic-tive accuracy, are separate in this case.

First, the value of a program as the;embodiment of a theory depends largelyupon the generality of the processes in-volved, upon the number of ad hoc assump-tions and preprogrammed niceties. Thetheory of problem solving incorporated inGPS gains in appeal to the extent that theprogram was successful in several differentproblem areas. The program remains un-convincing qua theory to the extent that itssuccess may depend upon similarities in taskstructure among a still quite limited set ofmathematical tasks.

As for the fit between output and be-havior, value, or fruitfulness for thatmatter, is of course highly dependent uponit. It is a function not only of degree of fitbut also of number of correspondencies be-tween output and behavior. There can be

correspondence with respect to problem solu-tion only, or to quantitative performancemeasures also—time, errors, orders of diffi-culty—or even to details of the process bywhich solution is sought or reached—intro-spections or intermediate results, such asused in Feldman's (1962) binary choiceprogram and in GPS (Newell and Simon1959).

The kind of correspondence one tries toachieve dependsupon the natureof the taskssimulated and upon the intent of theinvestigator. With the more automatic proc-esses such as recognition (Uhr, Vossler, andUleman, 1962), immediate memory (Bower,1963), or serial reproduction (Feigenbaumand Simon, 1962), performance measures andsolution achievement are the only humanoutput data available. For some purposessolution achievement is a sufficient criterioneven with more complex tasks andevenwhensome data process would conflict with somefeatures of the machine process. Severalquestion-answering programs, for example,are only simulations in so far as theyprovethat correct answers to questions in naturallanguage can be produced by means ofspecific mechanisms. These mechanisms in-clude the use of implication-rich data modelsin the case of SAD SAM (Lindsay, 1961), ofsearch schemata in baseball (Green, 1963),or of single relational representations andtransitivity rules in SIR (Raphael, 1964).They are helpful for giving this proof, andfor demonstrating the limitations and impli-cations of the respective models.

DETAILED PROCESS SIMULATION

Sometimes one would want to reproducenot only the workings of one or two mecha-nisms but also the full stream of behaviorduring a given type of mental activity. Onetries to achievereproduction of the details ofthe process. Computer simulation seemsideallysuited to so complex a task.

Efforts for mirroring this stream of be-havior exist, with Newell, Shaw, and Simon'sGeneral Problem Solver (Newell and Simon,1961) as the major example. The machineprotocol was placed side by side with humanprotocols, and two or threecomparisons were

Behavioral

Science.

Volume 12, 1967

Computers in Behavioral Science 63

published. Laughery and Gregg's paper(1962) is one of the few clear instances ofprogram construction on the basis of protocolanalysis. So is Feldman's work on binarychoice. Besides these studies, very little at-tempt has been made torelate protocol dataand programs systematically. Programs areusually constructed by drawing ideas fromgross behavioral evidence. Analyzing intro-spective reports in systematic fashion hasseldom been practised, at least to judgefrompublished reports. This is a pity. Introspec-tion gathered in a systematic way, thisstandby of the psychology of old, might playits role anew, precisely because programs canbe fitted step by step from the protocol andreproduced.

To illustrate this approach, let me refer tothe work which is in progress at the psy-chology laboratory of Amsterdam Uni-versity. It concerns simulation of humaninformation storage andretrieval.

We plunge, somewhere, into the mass ofphenomena. We give our subjects some taskand ask for introspection, in the traditionalway of the Wurzburgers. We ask, for in-stance, informal definition of concepts suchas "What does 'lucidity' mean?" or "What isa cigarette?" We obtain protocols such as:"Eh, what is a cigarette . . . that is a long. . . thin object with white paper around it,and with tobacco in it, used to ... besmokedfor pleasure. Boy, what acomplicatedstatement. I just see it, a vague kind ofimage. After saying that about white paper,I thought about cigarettes with yellow maizepaper in France after the war, and then Ithought of marihuana cigarettes. I had tosearch for the expression "for pleasure;" Itried to classify things which you have justfor fun. Also after describing the shape, Ithought, nowI should classify thefunction".

We try to transform this output into aprogram-oriented process description first byconverting it into a series of short, simplestatements containing the solution efforts,and second, by constructing a program-likedescription. In constructing the description,we try as much as possible to account forprocess comments such as "I just see it,""What a complicated statement," "I had tosearch for the expression." Comments such

as thesepoint, howevervaguely, to the proc-esses going on: verbal control processes,immediate nonverbal actualization of knowl-edge, and the like. Table 1 presents theresulting hypothesized sequence of subrou-tines. Defining a known concept is conceivedas evocation of a mass of denotations, serialscanning of prominent elements in that mass,and testing the relevance of each in turn;naming relevant ones; checking the ade-quacy of the result by confronting thedenotation of the definition words with theoriginal denotation.

As a following step, other protocols arescanned to test the adequacy of the hypoth-esis. This can be done systematically byscoring the simple statements referred toabove in terms of the subroutines whose out-put they are supposed to be. Deficienciesin terms of unclassifiable items, nonfittingitems, or sequence violations rapidly mani-fest themselves. In the example given,several things became clear at once: testphase 4 did not always take the form givento it (an aspect maybe rejected because it isunessential), and it did not always occur atthe predicted place; the relevance test fre-quently followed naming. Program changeswere made, which in this case had to bequite extensive. Modification of the contentof the test phase necessitated flexible accessto several relevance criteria dependent uponthe type of solution proposal just made bythe subject. The mobility of the test phasenecessitated a much more hierarchical proc-ess structure than that given in Table 1,involving separation into executive andworking routines as in GPS. The cycle ofprogram correction is repeated until a suffi-cient fit seems to obtain, after which theroutine denotedby "scanning prominent ele-ments," "confronting denotations," and soforth, are to be spelled out and actual pro-gramming can begin.

It is clear that a very large part of thework is performed prior to writing the actualcomputer program. This is particularly truesince process description at a fairly high levelis what is really psychologically relevant.The yieldof the work seemsrather high, pro-vided that all program changes which weredictated by the date are retained and re-

Behavioral

Science,

Volume 12, 1967

TABLE 1Defining word meanings

input stimulus word

exit.

64

Behavioral Science, Volume 12, 1967

Computers in Behavioral Science 65

ported. That a problem-solving executivehad to be introduced, not because of pro-gramming convenience but because of thedemands of the behavior data, is aninteresting finding. Equally important is theemergence of arguments for flexible controlprocedures. It should be stressed that, al-though work up to this stage is entirely in-formal, results are nevertheless fruits ofcomputer simulation. Only the flow chartstructure of the hypotheses permits scanningof the protocol data in search of insufficien-cies, and only the program model permitsphrasing in terms of recurrent cycles andsuccessive operations. The program func-tions clearly as a tool for analysis and search.

Although most of the psychological workoccurs at this informal level, formal pro-gramming (in this case, ALGOL 60) is stillnecessary. There are two reasons for this.The first is that actual programming is theonly way to make sure that everythingreallyworks; it is also the only way to clear upmysterious notions such as "confronting de-notations." The second reason is that onlyby formal programming can the necessarysubstructure be worked out—which must bedone to discover the implications of themodel. By "substructure" we mean the basicstructure of datarepresentation and of lowerorder search and retrieval processes. Thesubstructure in the present case has to con-form to a theory of human memory which isnot very directly suggestedby behavior databut which is in many ways constrained bythem. A number of different memory prob-lem-solving tasks (such as similarities, re-quests for supraordinate concepts, and soon) will be analyzed and programmed;considerations of parsimony will here be ofprimary concern. We hope the constraintsmentioned will become more and more speci-fic. The system is to absorb information fromsimple sentences and to produce, whenquestioned, protocols of retrieval activitiessimilar to the one given as an example.

MATCHING HUMAN PROTOCOLS ANDMACHINE PROTOCOLS

How is one to estimate the degree of fitbetween machine output and human output?

There is hardly any methodology existinghere. As much ingenuity as has been investedin the making of programs, as little has beenspent on the assessment of their value. Nextto high precision there always seem to bespots of rough approximations which under-cut this veryprecision. We are left largely toour subjective impressions of what we con-sider good or bad correspondence.

When judging correspondence betweenmachine output and human output we haveto distinguish between those programs re-sulting only in quantitative performancedata and those giving qualitative processdetails. With those giving only numericalresults, one could be tempted to employ theusual significance tests. Their appropriate-ness, however, seems doubtful. The degreesof freedom of a computer program are ex-tremely great, The construction of alterna-tive models seems the only control, and anecessary one at that, Clarkson (1961) is theonly author I know of who actually testedhis model (of portfolio selection) in thismanner.

Detailed process simulation does notusually lend itself to significance tests. Com-mon sense impression of similarity seems theonly basis for judgement. There is nothing-wrong with this use of common sense. Theparallels between what subjects do and whatthe program does are often so striking (aswith GPS or the Simon and Kotovsky pro-gram) as to render coincidence unlikely andto confer plausibility upon the machinery.This rough form of what has come to becalled "Turing's test" (Newell and Simon,1959) is a useful basis for evaluation.

In evaluating the degree of fit betweenmachine protocol and human protocol, theinvestigator is confronted with problems ofselection. On the one hand, he wants asfaithfully as possible to simulate (and thus tounderstand) the human behavior. On theother hand, it is mostly trivial and unen-lightening to aim at reproduction of all ir-relevant human limitations or idiosyncrasies.Distractions, certain limitations of workingmemory, viscissitudes of information gather-ing, peculiarities of verbal expression, oftencontribute very little to understanding the

Behavioral

Science,

Volume 12, 1967

Computers in Behavioral Science66

processes in which we are interested. It iscertainly admissible, andeven wise, to selectwhich data to simulate and which to discardeven though this involves theoretical pre-conceptions and may entail the risk of dis-carding features which later may turn out tobe consequences of the major process itself.

Problems of selection are most serious andessential in connection with generalizingsimulation results. One has to make decisionswhere similarity is required and where data-specific differences can be neglected.

When a program is constructed on thebasis of a given set of data, how can we makesure that it applies to other sets? With ab-stract tasks there is not too much difficulty.One varies the inputs for both subjects andprogram. With tasks yielding more qualita-tive data (such as memory tasks) checkinggets complicated. How can a program con-structed on the basis of definitions of acigarette be checked on protocols of defini-tions of a cigar? The scoring systemsuggested above may offer a solution. Scor-ing the simple statements in terms of thesubroutines whose outputs they are supposedto be permits construction of a quantitativeindex of fit. This is the case even when thecontent Of the human protocols is differentfrom what the machine would produce. Sub-routine sequences should correspond to therequirements of the program, and deviationscan be counted and weighed. Such an indexpermits the application of reliability tests ofthe scoring systemused.

Of course, evaluation of programs andtheories is not finished when output andhuman protocol are compared and are es-sentially alike. Any theory is useful onlyinsofar as it gives rise to newinvestigation orleads to new integration of data. The sameholds in connection with simulation. Fromtheprogram written on the basis of protocols,we hope and expect cues for direct experi-ment, Our memory model, for instance, de-mands that the number of ways of access tothe memory store be limited to sensory cuesand names. This at once leads toexperimentsin which cue complexes are varied in con-creteness and, for instance, reaction timesare tobe measured. A more consequential ex-

ample of integration of computer simulationwith other methods of research is given byanother project, undertaken by Jan Elshoutat the Amsterdam University psychologylaboratory. Here, protocol analysis of prob-lem-solving tasks will be related to factoranalytic investigation of tests containingsimilar tasks (a Guilford type battery), aswell as to performance on a concept forma-tion criterion task. Parallels between theresults of factor analysis and program com-ponents similar over different tasks will belooked for. The computer simulation hereseems to be essential to cany process anal-ysis beyond a purely verbal stage: it mayshow another of its aspects as a helpful tech-nique in psychological research.

REFERENCES

Baker, F. B. An IPL-V program for conceptattainment.Educ. psychol. Measmt., 1964, 24,119-127.

Bower, G. A modelof immediatememory. Workingpaper at Summer Institute in HeuristicProgramming, Santa Monica, 1963.

Clarkson,

G. P. E.A simulation of trust investment.Englewood

Cliffs,

N.J.: Prentice-Hall, 1961.Feigenbaum, E. A. EPAM: An elementary per-

ceiver and memoriser. RAND Report, 1959,1817.

Feigenbaum,E. A., &

Simon,

H. A. A theory of theserial position effect. Brit. J.Psychol., 1962,53, 307-320.

Feldman, J. Computer simulation of cognitiveprocesses. In H. Borko (Ed.) Computer ap-plications in the behavioral sciences. Engle-wood

Cliffs,

N.J.: Prentice-Hall, 1962, Pp.336-356.

Gelernter,

IL, Hansen, J. R., & Loveland, D. W.Empirical exploration of the geometrytheorem machine. Proc. Western Joint Com-puter

Conference,

1960, 17, 143-147.

Green,

B. F. Digital computers in research. NewYork:

McGraw-Hill,

1963.Laughery, K. R., & Gregg, L. W. Simulation of

human problem-solving behavior. Psycho-melrika, 1962, 27, 265-282.

Lindsay, R. L. The reading machine problem. Mim-eographedreport, University of Texas, 1961.

McWhinney, W. H. Simulating the communicationnetwork experiments. Behav.

Sci.,

1964, 9,80-84.

Miller, G. A.,

Galanter,

E., & Pribram, K. H.Plans and the structure of behavior. NewYork: Holt, Rinehart, Winston, 1960.

Newell, A. A guide to GPS-2-2. RAND ReportRM-3337-PR, 1963.

Newell, A., &

Simon,

FI. A. The simulation of

Behavioral Science, Volume 12, 1967

Computers in Behavioral Science 67

human thought. RAND Report P 1734, 1959.Newell, A., &

Simon,

H. A.

GPS,

a program thatsimulates human thought. In H. Billing(Ed.) Lernende Automaten. Munich: Olden-bourg, 1961.

Raphael,B. SIR: A computer programfor semanticinformation retrieval. Unpublished doctoraldissertation, Massachusetts Institute ofTechnology, 1964.

Rochester, N., Holland, J. H., Haibt, L. H., &Duda, W. L. Test on a cell assembly theoryof the action of the brain, using a large digi-tal computer. IRE Trans. Infor. Theory,1956,

PGIT-2,3,80-93.

Simon,

H. A., & Kotovsky, K. Human acquisitionof concepts forsequential patterns. Psychol.Rev., 1963,70, 534-546.

Uhr, L., Vossler,

C,

& Uleman,J. Pattern recogni-tions over distortions by human subjectsand a computer model of human formperception. /. exp. Psychol., 1962, 63, 227--234.

Yntema, D. B. Recall as a form of data-process-ing. Paper read at psychology section ofthe AAAS meetings, Philadelphia, 1962.

(Manuscript received March 28, 1966)

tr+3

To know the truth partially is to distort the Universe. ... Anunflinching determination to take the whole evidence into accountis the only method of preservation against the fluctuating extremesof fashionable opinion.

Alfred North Whitehead

Behavioral Science, Volume 12. 1967

CONCEPT FREQUENCY IN POLITICAL TEXT: AN APPLICATION OF ATOTAL INDEXING METHOD OF AUTOMATED CONTENT ANALYSIS1

Arthur R. CarlsonSystem Development Corporation, and Department of Psychology, the University of Illinois

A computer routine, originally designed for informationretrieval in natural language, isused to analyze the content of a data base ofpolitical text. In the latter sections of the paperthere is a rather informal discussion of some substantive relations generated by the appli-cation of the method to the particular data base used. The uniqueness of the method lies inthe fact that the data base is totally indexed. Some pessimism is expressedconcerning thepossibility of devising a deterministic method of content analysis.

C*J>

IN recent years there has been much in-terest in the development of computer-

assisted content analysis methods. Inparticular, research utilizing the GeneralInquirer (Stone, Bales, Namenwirth, andOgilvie, 1962), a subsystem of COMIT(North, Holsti, Zaninovich, and Zinnes,1963), has been undertaken (for exampleby Holsti, 1963). The research reported inthis paper differs markedly in techniquefrom previous attempts to harness the speedand accuracy of the computer to the taskof content analysis.

Protosynthex is the operating question-answering system developed by the Syn-thex Project at System Development Cor-poration (Simmons, 1964). This system,originally designed to answer questionsposed in English, was used to analyse thecontent of a set of political statements byLuis Corvalan, a doctrinaire Chilean com-munist, Fidel Castro, John Kennedy, andRichard Nixon.

THE TECHNOLOGY

The routines which comprise the Proto-synthex system are written in JTS, JOVIALfor time sharing. The system operates online in the time-sharing mode. This meansthat the user can sit at his input device,

1 The author would like to thank Dr. WilliamBlanchard and Mr. Howard Manelowitz, both ofSystem Development Corporation, for helpfulsuggestions both substantive and methodological,and Zoe Ann Matlock and Aryline Strobel of SDCfor assistance.

usually a teletypemachine, asking questionsand receiving immediate responses. Ofcourse, there is no reason why the usershould restrict himself to questions. Re-trieval is accomplished by searching thetext for passages containing the same words(or synonyms) appearing in the query.Using the standard measure of information,the inverse of the frequency of occurrence,the retrieval program extracts the passagein the text containing maximum inter-section of question words weighted bytheir information content. Thus, it producesthe statement most similar to the questionstatementin terms of the words used.

The system functions quite well as aquestion answerer when its data base is acompendium of verbal information; theoriginal data base was the Golden BookEncyclopedia. However, it is obvious thatit can be put to other uses as well for, in-sofar as it yields the closest match to astandard, it can be used to measure thesimilarity of texts (see Baum, 1965, p. 59)or the frequency of occurrence of concepts.The latter task is accomplished notby askingthat a question be answered or a statementmatched, but by sending the machine aset of synonyms or, perhaps more exactly,words which are in the same semanticballpark. Indeed, this last procedure wasthe one employed by the present author.The reason for choosing this method has todo with the nature of the retrieval process.As pointed out above, the retriever extracts

(iS

Behavioral

Science,

Volume 12, 1967