A study of how individuals solve complex andill-structured problems

RONALD FERNANDES1 & HERBERTA. SIMON2

1H. John Heinz III School of Public Policy and Management, 2Department of Psychology, CarnegieMellon University, U.S.A.E-mail: ronald@andrew.cmu.edu

Abstract. A number of factors cause individuals to use diverse strategies to solve problems. Thispaper presents a methodology for examining these di¡erences in strategy. Verbal protocols areelicited to collect data on the cognitive processes occurring during problem solving. These data,codi¢ed into propositional representations, and non-parametric statistical comparisons are thenused to evaluate the signi¢cance of strategy di¡erences. These strategies are then mapped withdynamical graphs, with which we examine the task-independent and the task-speci¢c cognitiverepresentations the participants used. As an illustrative example we apply this methodology tostudy the in£uence of two contributing factors, professional training and national culture, on thestrategies adopted by professionals to solve a complex and ill-structured problem (hunger in acountry). The problem-solving strategies of professionals from di¡erent countries and trained inarchitecture, engineering, law or medicine are analyzed to show some intriguing di¡erences in thegeneral strategies adopted by individuals belonging to di¡erent professions, and the outcomes fromusing these strategies.

1. Introduction

There has been a resurgence of interest in understanding how individuals solvereal-life policy problems. Such problems are usually complex and ill structured,and research has been hindered in the past by inadequate empirical and ana-lytical tools for isolating strategies used during the problem-solving process.New developments in software now permit a ¢ner-grained analysis of dataacross a larger number of participants, including the use of non-parametricstatistical methods. We propose in this research note to describe brie£y themethodology and results of our research. A more detailed report on our ¢nd-ings is available from the authors on request.

Policy problems have most of the characteristics of complex and ill-struc-tured problems. Funke (1991) has de¢ned complex problems as having thefollowing features:

1. Intransparency: only knowledge about the symptoms is available, onlysome variables lend themselves to direct observation, or, the large num-ber of variables requires selection by the problem solver of a few relevantones.

2. Polytely: multiple goals may be present that could interfere with each other.

225Policy Sciences 32: 225^245, 1999.ß 1999Kluwer Academic Publishers. Printed in the Netherlands.

3. Situational complexity: there are complex connectivity patterns betweenvariables, and

4. Time-delayed e¡ects: not every action shows immediate consequences.

Problems also vary along a continuum between the ill structured and the wellstructured. The degree of de¢niteness depends upon the power of the problem-solving techniques available.Well-structured problems are those having:

1. A de¢nite criterion for recognizing solutions and a mechanizable processfor applying that criterion.

2. At least one problem space in which the successive problem states may berepresented.

3. A structure wherein attainable state changes (legal moves) and consider-able moves can be represented in a problem space as transitions.

4. A representation for any knowledge a problem solver can acquire, in oneor more problem spaces.

5. A re£ection in the state changes, of the laws that govern the externalworld and the e¡ects upon a state of applying any operator, and

6. Basic processes that require only practicable amounts of computing andonly information that is available with practicable amounts of search(Simon, 1973).

The di¡erences in problem-solving strategies among individuals or groups canbe explained by a phenomenon called identi¢cation (Simon, 1997). A personidenti¢es with a group when, in making a decision, he evaluates the severalalternatives of choice in terms of their consequences for the speci¢ed group'.This leads to decision-makers acquiring a representation of a problem thatfocuses attention on operative goals, and interpreting them in terms of thepartial information attended to. When presented with a complex stimulus aperson perceives in it what he or she is ready to perceive; the more complex orambiguous the stimulus, the more the perception is determined by what isalready `in' the individual and less by what is `in' the stimulus. Thus identi¢ca-tion can explain the adoption of available general problem-solving strategies byindividuals to solve complex problems, and the preference by individuals forspeci¢c cognitive strategies over others.

Identi¢cation based on professional, ethnic or other characteristics cancause individuals to apply problem-solving strategies that match the goals ornorms of the group identi¢ed with. These strategies can also be due to identify-ing a problem as relevant to a group's expertise. Professional identi¢cationcould contribute to the formation of general problem-solving strategies due tothe inculcation of `best practices' during the course of professional educa-tion. The limited focus of attention of professionals could contribute to thesestrategies. These strategies could also occur due to a self-selection of indi-viduals into professions they could identify with i.e. that match their cognitivepreferences.

226

Thus Altmeyer (1966) has shown that the cognitive abilities of under-graduate arts and science students change over the course of their professionaleducation. Engineering majors enhanced their analytical and logical reasoningabilities, but showed a decline in some of their imaginative abilities; while artsmajors enhanced their imaginative abilities but exhibited a decline in some oftheir analytical and logical reasoning abilities. Specialization of study leads todi¡erentiation of style, and mature, capable and motivated students displayvery distinct thinking habits that are correlated with their choice of major ¢eld(Doktor, 1969). Winter et al. (1981) document abundant evidence of cognitivechanges in individuals during the course of training in the liberal arts at theundergraduate level.

Voss et al. (1983) investigated how experts and novices solved a problem oflow crop productivity in Soviet agriculture, an ill-structured social scienceproblem. One salient di¡erence between experts and novices was in the timetaken to develop problem representations, with experts spending more time todo so. In the solution process, experts also typically proposed relatively fewerbut more abstract solutions and spent considerable time in developing argu-ments related to the solutions. Novices on the other hand, proposed moreand simpler solutions with very little argument development. Voss and Post(1988) then compared strategies used by these experts with those used bymagistrates and physicians as they solved problems in their ¢eld. They founddistinct similarities in the problem representational process across all cases.There were marked similarities in the use of general problem-solving strategiesby experts, the most frequently employed method being decomposition. How-ever, during the solution process, Soviet experts departed substantially frommagistrates and physicians. While for magistrates and physicians the solutionwas presumed to be stated when the representation was established, Sovietexperts typically began the solution process by advancing a relatively abstractsolution, then a solution for each of the component sub-problems and ¢nallyan integration of these solutions into a solution covering all aspects of theproblem.

Amsel et al. (1991) examined whether lawyers displayed a style of reasoningdistinguishable from the style that other professionals such as psychologistsexhibited. They showed that while lawyers relied primarily upon past-orientedand diagnostic causal inference rules as represented by counterfactual rules,psychologists preferred statistical evidence, the generation of hypotheses andthe evaluation of internal and external validity of an experimental design tomake causal inferences. Amsel et al. also suggested that the processes oftraining in law and psychology induced professional di¡erences in causalreasoning.

Of the various methods that have been used to study individual problem-solving in complex and ill-structured situations, the approach used byVoss andothers (1983) was the most similar to ours in spirit. The data collected fromconcurrent verbal protocols was segmented and classi¢ed into argument cate-gories ¢rst proposed by Toulmin (datum, claim, warrant, backing, quali¢er

227

and rebuttal). This approach gives us a rich insight into the problem-speci¢cstrategies used during problem solving, but tends to obscure general strategiesused to represent or solve the problem. It also does not lend itself easily tostatistical evaluation of the di¡erences. Our methodology, described in the nextsection, can be applied to examining individual and/or group di¡erences intask-independent and task-speci¢c strategies used for solving complex and ill-structured problems.

2. Data collection, coding and veri¢cation

We begin by describing a complex and ill-structured situation. Each participantis instructed to read the problem and then `think aloud' while solving it withina limited time. The response is audiotaped without interaction between theexperimenter and the participant during the experiment. The audiotaped pro-tocols are transcribed verbatim, segmented into cognitive chunks or actionsidenti¢able as separate units of thought, and encoded as one of nine basicactions or nine meta-actions.

The nine basic actions were:

1. Recall: To recall facts speci¢cally mentioned in the problem.2. Read: To read words, phrases or statements from the problem.3. Assume: To take for granted or suppose some fact or perceived fact not

mentioned in the problem and to use it.4. Know: To be certain of some knowledge and to use it.5. Infer: To conclude from evidence or premises or to adopt as a logical

consequence.6. Evaluate: To ¢x the value of; or to examine and judge, information

speci¢ed in the problem.7. Calculate: To calculate the numerical value of information.8. Query: To utter a question, inquiry or a doubt, and9. Recommend: To counsel or advise that something be done to solve the

problem.

Meta-actions describe or refer to any of the basic cognitive actions. They areusually statements of the participant when he is thinking about the problem-solving process but not about speci¢c problem details. Eighteen distinct actions(9 actions and 9 meta-actions) are used to encode, at a propositional level, thethoughts expressed by each participant during the problem representation andsolving process. Independent coding can be carried out and the inter-coderreliability veri¢ed.

228

3. Data analysis

The preliminary analysis of the protocols can be carried out in three stages:

1. Analysis of task-independent cognitive processes to determine generalproblem-solving strategies.

2. Task speci¢c analysis for insights into how individuals solve a speci¢cproblem, and

3. Synthesis of the analyses to examine how general problem-solving strat-egies a¡ect problem representation and solution outcomes.

3.1. Task independent analysis

Each protocol can be encoded using software to facilitate the process. ProtocolAnalyst's Workbench (PAW; Fisher, 1991) is one such software tool that isindependent of the problem domain. The interactive feature of PAW facilitatesencoding that helps preserves the structure and content of the original tran-script. PAW creates a matrix for each individual showing the frequency of useof each action in the protocol, and a table of percentages of each action usedwhile solving the problem. Analysis of these percentages indicates the relativeimportance of di¡erent actions. PAW also gives the frequency of transitionaland re£exive ties. Transitional ties occur when one action precedes another,re£ecting the relations between the cognitive actions used to solve the problem.Thus if the action `infer' is followed by the action `recommend' it is recorded asa transition from the one to the other action. Re£exive ties occur when oneaction is followed by the same action, an example of which would be one `infer'statement followed by another `infer' statement where both statements aresubstantially di¡erent in content. Transitional and re£exive ties enable us toassess an action's weight in a £ow model of the problem representation andsolution process. They can also be used to set a quantitative threshold forconsidering a tie to be part of a cognitive process, to eliminate noise.

The weights or percentages of these ties for each individual's protocolcan then be entered in matrix form in any matrix manipulation software suchas UCINET (Borgatti, Everett and Freeman, 1992) to create an individualmatrix showing the percentage of transitional and re£exive ties on actions foreach individual. Each matrix so obtained can be tested for similarity with thematrices of other individuals, using the correlation between an individual'smatrix, and all the other individual matrices. This gives a data matrix showingthe correlation between each individual and every other individual. This datamatrix measures individual similarity or di¡erences in transitional and re£exiveties.

In order to test whether the results obtained are consistent with our hypothe-sized di¡erences in characteristics between individuals, the data matrix canthen be compared for similarity with a characteristic structure matrix. This

229

matrix has cells with 1's if the pair of individuals share a similar characteristicand 0's if they do not. In making statistical comparisons, structural auto-correlation may occur in the data matrix, due to the dyadic nature of relation-ships between individuals and non-independence of observations, leading tobiased results. However Quadratic Assignment Procedure or QAP1,2, a non-parametric method of statistical analysis available in UCINET has been shownto give relatively unbiased results in such situations (Krackhardt, 1988).

Other characteristics might also coincidentally in£uence di¡erences amongindividuals in their problem solving strategies as represented by the data ma-trix. In such a situation QAP can be used to compare3 the similarity betweenthe data matrix and the new characteristic structure matrix. The magnitude ofthe correlation coe¤cient between the two matrices and the level of signi¢-cance would then indicate mutual in£uence. On the other hand the factors wehypothesize as in£uencing problem solving strategies may be correlated. QAPcan then be used to determine the collinearity between the two characteristicstructure matrices. If this collinearity is severe, we can separate out the relativee¡ects of both characteristics on the observed di¡erences in problem-solvingstrategies between individuals. QAP multiple regression analysis4 can be per-formed using the data matrix as the dependent matrix and the two charac-teristic structure matrices as independent matrices.

Having determined strategies used by individuals with speci¢c character-istics or belonging to certain groups we can ask: Can these speci¢c strategiesbe identi¢ed? We found helpful a graphical method for visualizing some detailsof the problem-solving strategies adopted by professionals. To make graphicalcomparisons, KrackPlot (Krackhardt et al., 1994) can display transitional andre£exive ties and the centrality and proportion of these ties for each action. A£ow diagram displays the sequence of actions each individual follows whilesolving the problem.We found it appropriate5 to draw individual £ow diagramsthat showed only those sequential pairs of successive actions with transitionaland re£exive ties that were common to individuals within a speci¢ed group.

Our next step is to understand how these general problem-solving strategiescould in£uence how individuals actually go about solving a complex and ill-structured problem. The next two sub-sections indicate how general problem-solving strategies might in£uence the task speci¢c outcomes and recommenda-tions given by individuals.

3.2. Task speci¢c process analysis

Task speci¢c process analysis requires an understanding of the domain knowl-edge that professionals use in problem solving. It asks whether professionalsdi¡er in their use of information to solve complex ill-structured problems.Problem behavior graphs (Newell and Simon, 1972) display the sequence ofactions within a protocol and changes in representation of problem states in theindividual's mind. A careful examination of the content of these graphs reveals

230

similarities and di¡erences in how individuals belonging to similar and di¡er-ent groups solve problems.

3.3. Synthesis of results

The most critical section of the analysis is the synthesis of what we know aboutthe general problem-solving strategies used by individuals, and the speci¢coutcomes of the problem-solving process. If done appropriately this synthesiscan determine whether general problem-solving strategies play any part indetermining problem representation and solution process and the nature of thesolutions o¡ered. A successful synthesis relates what we know about individualor group characteristics to speci¢c strategies used by that individual or group.We will demonstrate the use of this method in the next section, using anexample of the analysis of strategies in a complex and ill-structured problem.

4. Illustrative example: Policy problems as complex and ill-structuredproblems

The problem of hunger in a country was chosen as an example. The problem ofhunger meets the criteria of complexity and ill structuredness in many ways,so that individuals could use both domain-speci¢c knowledge and generalproblem-solving strategies to solve the problem. There could be a wide varia-tion in the knowledge each individual had or applied to the problem, derivedboth from information provided in the problem statement, and from externalknowledge.

4.1. Policy problem

The policy problem given to the participants consisted of the role to be playedby the problem-solver followed by a brief description of the problem:

As a senior policy maker in your country you are called upon to advise aninternational organization about what is needed to solve a speci¢c policyproblem in a country called Hungeria. Please explain what other informationyou might need to solve this problem and then what your recommendationswould be?

Brief description of the problem:

Hungeria, with a population of 26 million, has nearly four million people whoexist below the poverty line and around one million people who are under-nourished and hungry because they do not have su¤cient food to eat. In

231

estimating the numbers of people going hungry the Food and AgricultureOrganization of the United Nations (FAO) uses as its criterion the energyintake level at which a person can barely survive which is a daily calorie intakebelow 1.2 basal metabolic rate (around 2100 Calories). Hungeria has a GrossDomestic Product (GDP) of $638 billion and is a net food exporting country.The birth rate in Hungeria is 15 per thousand and the death rate is 8 perthousand.

4.2. Specifying the complexity and ill-structuredness of the problem

The problem of hunger frequently arises from complex linkages between theproduction, distribution and consumption of food within the context of politi-cal, social and economic institutions (Timmer, 1983). Lack of income and long-term employment can lead to hunger even in relatively developed countries. Thefood policy of a country is also constrained by the international economy,through its impact on balance of payments, foreign exchange rates and theimport and export of food commodities. Instability in international food pricescan seriously a¡ect domestic food policies.

Some of the actions that may help address the problem of hunger are:

1. Redistribution of assets: Highly unequal ownership of assets frequentlya¡ects food production and consumption. Land reform may achieve aredistribution of assets, as may taxation of the rich and subsidies to thepoor.

2. Growth in income for the poor: Growth strategies may provide an increasein real incomes for the poor by providing new employment opportunities.

3. Reduced discrimination against certain social or demographic groups: Jobopportunities may be improved for speci¢c social or demographic groupsby reducing discrimination against them.

4. Increase in the production of food: An increase in the production of foodby a country may be achieved through technical change.

The problem of hunger in a country therefore meets the criteria of complexityspeci¢ed earlier. In order to ensure that it was also ill structured, the problemwas designed with some `contradictions'. This would induce multiple problemrepresentations both in terms of problem structure and domain knowledge.

The ¢rst `contradiction' was in the facts that Hungeria had a Gross Domes-tic Product (GDP) of $638 billion and a population of 26 million. This worksout to a per capita GDP of around $24,000, among the highest in the world.Another `contradiction' was the statement that Hungeria was a net food export-ing country, implying that food production was not a serious problem. A third`contradiction' was in the rather moderate birth rate and death rate, which wererepresentative of a highly developed country. Fourth, no information was givenabout the occurrence of any natural disaster that could have caused the prob-

232

lem. These `contradictions' eliminated various potential causes of hunger in thecountry. A good solution would require that professionals either question thatinformation or make suitable assumptions to assure consistency before o¡eringrecommendations. The information provided in the problem was su¤cient tosupport recommendations consistent with the `contradictions', along two direc-tions:

1. Targeted food subsidies could provide immediate relief to the hungry (e.g.food stamp programs, monetary support programs and food-for-workprograms). Given the high per capita GDP, this was an appropriatesolution of the problem in the short run.

2. Long-term recommendations might include redistribution of incomethrough tax policies or through productive jobs for people below thepoverty line.

4.3. Data collection, coding and veri¢cation

The professions chosen were architecture, medicine, law and engineering. Eachof the participants6 held undergraduate degrees in these four ¢elds. Each par-ticipant1 was given 30^45 minutes to solve the problem while thinking aloud.The coding was carried out according to the methodology described earlier(9 basic and 9 meta-actions). To test reliability of this coding, independentcoding was carried out on a random sample (20%) of the encoded actions.Intercoder reliability for this sample was 96%. Finally Protocol AnalystsWork-bench (PAW) was used to encode each protocol7 systematically.

4.4. Task independent analysis

PAW was used to generate the percentages of actions used by each professional.The results are shown in Table 1. The percentages in Table 1 indicate thatprofessionals used substantially more basic actions than meta-actions. However,architects were an exception among the professional groups in their ratherextensive use of meta-actions. PAW was then used to analyze the relativeproportions of transitional and re£exive ties.With a total of 18 types of actions(9 basic, 9 meta) for each professional we obtained a 18 ¾ 18 square matrix likethat in Table 2.

Each matrix was then tested for similarity with the matrices of all the otherprofessionals using UCINET. This produced an 8 ¾ 8 data matrix of the corre-lation coe¤cients as displayed in Table 3. Table 3 shows that individualsbelonging to the same profession have higher correlation coe¤cients in generalthan individuals belonging to dissimilar professions, with two exceptions.Engineer 2 (E2) shows a higher correlation with Lawyer 1 (L1) and Lawyer 2(L2) than he does with Engineer 1 while Physician 2 (P2) shows a higher

233

correlation with Architect 1 (A1) and Architect 2 (A2) than with Physician 1(P1). In order to test whether the results were consistent with di¡erences inprofessional education among the participants, the data matrix was comparedfor similarity with a profession structure matrix (Table 4), which had entries of1 if a pair of participants belonged to the same profession and 0 if they belongedto di¡erent professions.

QAP was used to calculate the correlation coe¤cient between the datamatrix and the profession structure matrix. Table 5 shows that the correlationwas 0.37. The probability of a correlation as large or larger than 0.37 with arandom matrix was 1.1% indicating that the correlation was signi¢cant at the0.05 level.

As our sample included professionals who had obtained their professionaleducation outside the U.S., these di¡erences between professions might be dueto other cultural di¡erences. In order to examine the signi¢cance of such di¡er-ences, a country structure matrix was created (Table 6) that had 1's if the pair ofparticipants were educated in the same country and 0's if they were educatedin di¡erent countries. QAP was used to compare the similarity between thedata matrix and the country structure matrix. Table 7 shows that the observedcorrelation between the two matrices was 0.26. The percentage of random

Table 1. Relative importance of actions used by professionals (in %).

Actions Participants

Archi-tect 1

Archi-tect 2

Physi-cian 1

Physi-cian 2

Engi-neer 1

Engi-neer 2

Lawyer1

Lawyer2

Assume 2.9 6.6 3.3 3.2 12.2 7.8 2.5 14.9Calculate 1.5 3.9 3.3 1.6 0.0 3.1 1.9 0.0Evaluate 13.2 14.5 0.0 11.3 9.8 4.7 15.4 2.1Infer 10.3 2.6 13.3 4.8 17.1 20.3 16.7 6.4Know 16.2 6.6 6.7 8.1 4.9 14.1 19.8 36.2Query 19.1 17.1 13.3 25.8 22.0 6.3 0.6 5.3Read 4.4 1.3 21.7 12.9 0.0 10.9 3.1 0.0Recall 5.9 2.6 18.3 9.7 2.4 0.0 6.8 1.1Recommend 5.9 13.2 8.3 4.8 19.5 23.4 20.4 28.7M-Assume 0.0 3.9 0.0 0.0 0.0 0.0 0.0 0.0M-Calculate 4.4 5.3 0.0 0.0 0.0 4.7 0.0 1.1M-Evaluate 4.4 5.3 1.7 0.0 4.9 3.1 3.7 3.2M-Infer 7.4 11.8 5.0 12.9 2.4 1.6 3.7 1.1M-Know 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0M-Query 0.0 3.9 3.3 1.6 2.4 0.0 1.2 0.0M-Read 0.0 1.3 0.0 0.0 0.0 0.0 0.0 0.0M-Recall 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0M-Recommend 4.4 0.0 1.7 3.2 2.4 0.0 3.7 0.0

Total meta-actions 20.6 31.6 11.7 17.7 12.2 9.4 13.0 5.3

Total basic actions 79.4 68.4 88.3 82.3 87.8 90.6 87.0 94.7

Total 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0

234

Table2.

Tran

sitio

nala

ndre£exive

ties

matrixforArchitect1.

Assu

Calc

Eval

Infer

Kno

wM-A

ssM-C

alM-Eva

M-Inf

M-K

now

M-Q

ueM-R

eaM-R

eca

M-R

eco

Que

ryRead

Recal

Recom

Assum

^^

^^

^^

^^

1^

^^

^^

^^

^1

2Calcu

^^

1^

^^

^^

^^

^^

^^

^^

^^

1Evalu

^^

^2

2^

^^

^^

^^

^^

21

2^

9Infer

^^

31

2^

^^

^^

^^

^1

^^

^^

7Kno

w^

^2

14

^^

2^

^^

^^

11

^^

^11

M-A

ss^

^^

^^

^^

^^

^^

^^

^^

^^

^0

M-C

al^

^^

^^

^1

1^

^^

^^

^^

^1

^3

M-Eva

^^

^^

^^

^^

^^

^^

^1

2^

^^

3M-Inf

1^

^^

^^

1^

2^

^^

^^

1^

^^

5M-K

no^

^^

^^

^^

^^

^^

^^

^^

^^

^0

M-Q

ue^

^^

^^

^^

^^

^^

^^

^^

^^

^0

M-R

ea^

^^

^^

^^

^^

^^

^^

^^

^^

^0

M-R

ecal

^^

^^

^^

^^

^^

^^

^^

^^

^^

0M-R

ecom

^^

^^

1^

^^

^^

^^

^^

^1

^1

3Query

^^

21

1^

1^

1^

^^

^^

6^

1^

13Read

^^

^2

^^

^^

^^

^^

^^

1^

^^

3Recal

11

1^

^^

^^

^^

^^

^^

1^

^^

4Recom

^^

^^

1^

^^

^^

^^

^^

^1

^2

4

21

97

110

33

40

00

03

143

44

68

235

Table 3. Data matrix.

1 2 3 4 5 6 7 8A1 A2 P1 P2 E1 E2 L1 L2

1 A1 1.00 0.55 0.15 0.44 0.43 0.38 0.38 0.362 A2 1.00 0.07 0.38 0.45 0.30 0.23 0.273 P1 1.00 0.30 0.14 0.25 0.17 0.194 P2 1.00 0.30 0.13 0.14 0.095 E1 1.00 0.42 0.36 0.246 E2 1.00 0.68 0.637 L1 1.00 0.628 L2 1.00

Bold ^ signi¢cant at 0.01 level. Bold italics ^ signi¢cant at 0.05 level. Italics ^ signi¢cant at 0.10 level.

Table 4. Profession structure matrix.

1 2 3 4 5 6 7 8A1 A2 P1 P2 E1 E2 L1 L2

1 A1 1 1 0 0 0 0 0 02 A2 1 1 0 0 0 0 0 03 P1 1 1 0 0 0 04 P2 1 1 0 0 0 05 E1 1 1 0 06 E2 1 1 0 07 L1 1 18 L2 1 1

Table 5. QAP correlation between data matrix and profession structure matrix.

Correlation Matches

Observed value 0.372 0.000Average 0.003 0.000Standard deviation 0.178 0.000Proportion as large 0.011 1.000Proportion as small 1.000 1.000

Table 6. Country structure matrix.

1 2 3 4 5 6 7 8A1 A2 P1 P2 E1 E2 L1 L2

1 A1 1 0 0 0 0 0 0 02 A2 0 1 0 0 1 0 0 03 P1 0 0 1 1 0 0 0 04 P2 0 0 1 1 0 0 0 05 E1 0 1 0 0 1 0 0 06 E2 0 0 0 0 0 1 0 07 L1 0 0 0 0 0 0 1 18 L2 0 0 0 0 0 0 1 1

236

correlation's as large or larger than 0.26 was 10.2% indicating a correlationbetween the data and the country structure matrix lower than that betweenthe data and profession structure matrix, and not statistically signi¢cant at the0.10 level.

The profession and country structure matrices had a fairly high level ofcollinearity. The QAP correlation coe¤cient between the two matrices was0.52, which was signi¢cant at the 0. 10 level as shown in Table 8. In order toseparate out the relative signi¢cance of professional education and countrye¡ects on the general problem-solving strategies used by professionals, QAPmultiple regression analysis was performed using the data matrix as thedependent matrix and the profession structure and country structure matricesas independent matrices. Table 9 shows the results of the QAP regression.Profession appeared signi¢cant at the 0.10 level while country is not signi¢cant.Therefore there is evidence of profession-speci¢c general problem-solving

Table 7. QAP correlation between data matrix and country structure matrix.

Correlation Matches

Observed value 0.258 0.000Average 0.002 0.000Standard deviation 0.175 0.000Proportion as large 0.102 1.000Proportion as small 0.899 1.000

Table 8. QAP correlation between profession structure matrix and country structure matrix.

Correlation Matches

Observed value 0.519 0.893Average 0.001 0.781Standard deviation 0.210 0.045Proportion as large 0.070 0.070Proportion as small 0.995 0.995

Table 9. QAP regression of data matrix on profession structure matrix and country structure matrix.

Independent Regression coe¤cients

Un-stdizedcoe¤cient

Proportionas large

Proportionas small

Intercept 0.28 0.994 0.006Profession 0.14 0.082 0.918Country 0.05 0.293 0.707

R-square: 0.144. Probability: 0.077.

237

strategies, re£ected in the transitional and re£exive ties between actions, thatare signi¢cantly similar for professionals belonging to the same profession anddi¡er across individuals belonging to di¡erent professions.

We then used the graphical method recommended earlier to view and com-pare these general problem solving strategies. Shown below are the £ow dia-grams created using Krackplot (Figures 1 and 2) for two professional groups:engineers and lawyers in our sample, along with our interpretation.

The common actions for engineers (shown in Figure 1) consisted of 5 out ofthe 8 actions and no meta-actions. These 5 actions account for 81% of allactions for Engineer 1 and 66% for Engineer 2. The actions Recommend , Infer,Assume and Query also show re£exive ties indicating the repeated use of theseactions in solving the problem. No action appears to be truly central to theprocess for either engineer.

The common actions for lawyers (shown in Figure 2) consisted of 6 out ofthe 8 actions and no meta-actions. These 6 actions account for 77% of allactions for Lawyer 1 and 94% for Lawyer 2. The actions Recommend andAssume also show re£exive ties indicating the repeated use of these actions insolving the problem. On the whole the actionKnow appears to be central to theprocess with the most cyclic ties to other actions.

InterpretationThe lawyers and engineers tend to use a large proportion of Recommendactions as compared to other professionals. However the process throughwhich they arrive at a Recommend action di¡ers substantially. Lawyers showa transition from actions Know and Infer to Recommend , while engineersshowed transitions from the actions Assume, Query and Evaluate to Recom-mend. Another di¡erence is that transitions between Assume and Infer, appearfor lawyers but not for engineers, although engineers use both these actionsmore than lawyers do.

4.5. Task speci¢c process analysis

Task speci¢c process analysis relies upon the domain knowledge that profes-sionals use. Problem behavior graphs were built to display the sequence ofactions and changes in representation of problem states in the protocols forboth physicians (extracts shown below), followed by our interpretation of thegraphs.

238

Fig. 1.

239

Fig. 2.

240

Extracts from problem behavior graphs: Physicians

Population control, which has medical implications, was considered by bothphysicians during the problem-solving process, with Physician 1 recommendingpopulation control as a solution. Physician 2 however critically considered theinformation given in the problem about the birth and death rates in the countryand correctly inferred that population control was not the problem. They werealso similar in their attempt to focus on diagnosis of the problem with aninability to diagnose due to the `contradictions' (especially Physician 2, state-ments 29 and 61).

4.6. Synthesis of results

Do general problem solving strategies adopted by professionals play any part indetermining the problem representation and solving process and the nature ofsolutions o¡ered? Given below are some examples of use of these strategies bytwo professional groups: architects and lawyers.

ArchitectsThe professional education architects receive trains them in architectural de-sign (Akin, 1986: pp. 176^177). The design process shares with policy problemsa complex and ill structured nature. Both architects sought a solution usingstrategies of architectural design. One of the dominant strategies was theQueryÅÆ Evaluate strategy shown below.

241

QueryÅÆ Evaluate strategy

Thus Architect 1 used this strategy to determine that the problem was a wealthdistribution problem for which he needed more information. He queried landuse facts and food production facts that were not apt in the context of thisproblem, as the country had net food exports. He however heeded this pointlater on, which led him to realize the complexity of the problem (statement 55)and to o¡er a meta-recommendation (statement 58) rather than a recommen-dation. Similarly Architect 2 used this strategy to make a short-term recom-mendation of feeding the undernourished (statement 29). However his longterm recommendations for creating jobs were restricted by insu¤cient infor-mation (statements 49 to 53), and his consequent recommendations for creat-ing jobs were based upon assumptions (statements 61 and 63) he made aboutthe skills of the people.

LawyersLawyers are trained to choose between alternative views, while focussing onfacts that support their view (Wrightsman, 1987). The nature of their trainingpredisposes them to focus on con£ict resolution rather than on discovering thetruth. Some elements of this professional training are re£ected in the use ofthe actions Know and Recommend in their problem-solving strategies. Theirprimary problem-solving strategy used was theKnowÅÆRecommend strategywith re£exive ties on the actions Know andRecommend as shown below:

242

KnowÅÆRecommend strategy

The strong Know ÅÆ Recommend transitional tie for both lawyers appears toassist them to make recommendations. Thus Lawyer I uses his knowledge ofthe funding norms of theWorld Bank (statement 64) to foresee a problem for ashort-term program of auto-consumption (statement 66). Similarly Lawyer 2uses his knowledge of how hungry people behave (statements 24 through 27)to recommend education in practical skills as a way of solving the problem(statement 28).

5. Conclusions

Our examination of the protocols suggests that the use of general strategiesappears to be sometimes helpful and sometimes harmful in enabling profes-sionals to solve complex ill-structured problems. For example the use of theKnow ÅÆ Recommend strategy by the two lawyers in our sample hinderedtheir ability to use information provided in the form of `contradictions'. Theessential issue here seems to be determining when some general cognitivestrategies facilitate the problem solving process and when they hinder it.

A question that has interested researchers for some time has been: Are theresingular di¡erences in the general problem-solving strategies of professionalssolving complex and ill-structured problems and can these di¡erences be iden-ti¢ed and measured? The methodology and analysis of this research noteappears to identify such di¡erences and may be applied to other real-worldpublic policy problems. Using verbal protocol analysis, the sophisticatedsoftware currently available to analyze large and complex protocols, and thestatistical and graphical tools described in this research note, valuable insightcould be gained into the decisions taken by professionals.

243

Acknowledgements

The authors are grateful for the comments of Linda Babcock, Danny Fernandes,David Krackhardt, Denise Rousseau and the comments of two anonymousreferees on earlier drafts of this paper. Stephanie Mathews and Zhaoli Rongprovided invaluable research assistance. The usual disclaimers apply.

Notes

1. The algorithm testing the similarity of matrices proceeds in two steps. In the ¢rst step itcomputes Pearson's correlation coe¤cient (as well as the simple Matching coe¤cient) betweencorresponding cells of the two matrices. In the second step, it randomly permutes rows andcolumns of one matrix and recomputes the correlation. The second step is carried out hundredsof times in order to compute the proportion of times that the correlation based on randompermutations is equal to or larger than the observed correlation calculated in step 1. If thecorrelation is positive, a low percentage (50.05) suggests that the observed similarity (asindicated by the observed correlation) between the matrices is unlikely to have occurred bychance (cf. UCINET IV Version 1.0 ^ Network Analysis Software: User's Guide by Borgatti,Everett and Freeman, 1992: pp. 127^128).

2. The comparisons ignore the diagonals of both matrices. This avoids an arti¢cial similarity dueto both matrices having all 1's in their diagonals.

3. See note 2.4. QAP regression procedure performs a standard multiple regression across corresponding cells

of the dependent and independent matrices in the ¢rst step. In the second step, it randomlypermutes rows and columns (together) of the dependent matrix repeatedly and recomputes theregression, as described in Note 1.

5. Two factors motivated this decision. First, that this would act as an approximation to eliminat-ing any noise or false sequences between or within actions, from the diagram. Second, we wouldhave established that individuals show higher levels of similarity within a group.

6. All the participants were male. There was some variation in the nature of the professionaleducation each had received subsequent to completing his undergraduate degree and theircurrent occupation, although the basic categories were restricted to professionals who hadundergraduate and graduate degrees in those speci¢c ¢elds. The exceptions were one lawyerwho was presently in a graduate program in public policy (Lawyer 1) and one engineer,currently in a graduate program in industrial administration (Engineer 2). Both physicianswere practicing at a local hospital while the remaining participants were in graduate schoolrelated to their ¢eld of undergraduate study. The professionals di¡ered in the country where theydid their undergraduate studies. Both physicians (India), both lawyers (Mexico), one engineer(France) and one architect (Switzerland) had non-U.S. undergraduate degrees.

7. Encoded protocols available on request.

References

Akin, O. (1986). Psychology of Architectural Design. London: Pion Limited, pp. 176^177.Altemeyer, B. (1966). `Education in the arts and siences: Divergent paths,' doctoral dissertation,

Carnegie Mellon University.Amsel, E., R. Langer and L. Loutzenhiser (1991). `Do lawyers reason di¡erently from psycholo-

gists? A comparative design for studying expertise,' in R. Sternberg and P. Frensch, eds.,ComplexProblem Solving-Principles and Mechanisms, Hillsdale, New Jersey: Lawrence Erlbaum Asso-ciates, pp. 223^250.

244

Borgatti, E. and Freeman (1992). `UCINET IV Version 1.0 ^ Network Analysis Software: User'sguide'.

Doktor, R. (1969). `The development and mapping of certain cognitive styles of problem solving,'doctoral dissertation, Stanford University.

Ericsson, A.K. and H.A. Simon (1983). Protocol Analysis: Verbal Reports as Data. Cambridge,MA: The MIT Press.

Fisher, C. (1991). `Protocol analyst's workbench: Design and evaluation of computer-aided protocolanalysis,' doctoral dissertation, Carnegie Mellon University.

Funke, J. (1991). `Solving complex problems: Exploration and control of complex systems,' in R.Sternberg and P. Frensch, eds., Complex Problem Solving ^ Principles and Mechanisms, Hill-sdale, New Jersey: Lawrence Erlbaum Associates, pp. 185^222.

Krackhardt, D., J. Blythe and K. McGrath (1994). `KrackPlot 3.0: An improved network drawingprogram,'Connections 17 (2): 53^55.

Krackhardt, D. (1988). `Predicting with networks: Nonparametric multiple regression analysis ofdyadic data,'Social Networks 10, pp. 359^381.

Newell, A. and H.A. Simon (1972).Human Problem Solving. Englewood Cli¡s, N.J.: Prentice-Hall.Schwartz, S. and T. Gri¡en (1986). Medical Thinking ^ The Psychology of Medical Judgment and

Decision-Making, NewYork: Springer Verlag.Simon, Herbert A. (1997).Administrative Behavior, 4th edition, NewYork: The Free Press.Simon, H.A. (1973). `The structure of ill-structured problems,'Arti¢cial Intelligence 4, pp.181^201.Timmer, P., Walter F. and Scott Pearson (1983). Food Policy Analysis, Baltimore: The Johns

Hopkins University Press.Voss, J., S.W. Tyler and L.A. Yengo (1983). `Individual di¡erences in the solving of social science

problems,' in R. F. Dillon, R.R. Schmeck, eds., Individual Di¡erences in Cognition, New York:Academic Press.

Voss, J. F. and T.A. Post (1988). `On the solving of ill-structured problems,' in M.T.H. Chi, R.Glaser, and M. J. Farr, eds., The Nature of Expertis , Hillsdale, N.J.: Lawrence Erlbaurn Asso-ciates, pp. 261^285

Winter, D.G., D.C. McClelland and A. J. Stewart (1981). A New Case for the Liberal Arts, SanFrancisco, California: Jossey Bass Publishers.

Wrightsman, L. S. (1987). Psychology and the Legal System, Monterey, California: Brooks/ColePublishing Company.

245