mentor modeling: the internalization of modeled professional thinking in an epistemic game

Mentor modeling: the internalization ofmodeled professional thinking in anepistemic gamejcal_385 173..189

Padraig Nash & David Williamson ShafferDepartment of Education Psychology, University of Wisconsin-Madison, Madison, WI, USA

Abstract Players of epistemic games – computer games that simulate professional practica – have beenshown to develop epistemic frames: a profession’s particular way of seeing and solving prob-lems. This study examined the interactions between players and mentors in one epistemicgame, Urban Science. Using a new method called epistemic network analysis, we explored howplayers develop epistemic frames through playing the game. Our results show that playersimitate and internalize the professional way of thinking that the mentors model, suggesting thatmentors can effectively model epistemic frames, and that epistemic network analysis is a usefulway to chart the development of learning through mentoring relationships.

Keywords educational games, mentoring, assessment.

Introduction

There is a broad consensus that mentoring is an impor-tant part of young people’s intellectual, social, andemotional development (Freedman 1999). In this paper,we look at one powerful form of mentoring – the men-toring that takes place in professional training – andexamine how one feature of professional mentoring –modeling – was used to help players in a computergame learn real-world skills, knowledge, values, andways of thinking.

The context for this study is the epistemic gameUrban Science. Epistemic games are computer-basedrole-playing games that simulate professional training.Epistemic games recreate in game form professionalpractica such as design studios, capstone courses, andother occasions where professionals-in-training workin a supervised setting to prepare for entry into the

workforce. Urban Science, for example, is modeled ona practicum course for graduate urban planning stu-dents (Shaffer 2006b). In the planning practicum,students are hired by organizations to complete a plan-ning project: with the help of a mentor, they visit the sitein question, meet with stakeholders, use geographicinformation system (GIS) models to weigh tradeoffs,create preference surveys and final plans, and presenttheir findings (Shaffer 2006b). In Urban Science,players as young as 12 years old do these same things:they log into an online portal where they receiveinstructions from their non-player-character supervi-sors, go on virtual site visits, use a GIS model, writepreference surveys and receive feedback, and presenttheir final plans, all with the help of mentors who canlook over their shoulders as they work.

Acritical element of professional practica is the inter-action between young professionals and their mentors(Schön 1983, 1987). In the urban planning practicum,students meet regularly with planning consultants:experienced professionals with whom the students talkabout their work. These reflective conversations play acritical role in developing a professional way of think-ing. In Urban Science, mentors – who are also called

Accepted: 14 May 2010Correspondence: Padraig Nash, 1025 West Johnson Street,Madison, WI 53715. Email: [email protected] are no conflict(s) of interest.All co-authors had complete access to data supporting the manuscript.Study had complete approval from the University of Wisconsin-MadisonIRB.

doi: 10.1111/j.1365-2729.2010.00385.x

Original article

© 2010 Blackwell Publishing Ltd Journal of Computer Assisted Learning (2011), 27, 173–189 173

planning consultants – do the same for the players.As inthe planning practicum, planning consultants in UrbanScience help novices with the intricacies of the work,provide feedback, and inspire them when they are stuck.

A central role of professional mentors, however, is tomodel the way of thinking that is unique to their profes-sion: what Shaffer (2006a) calls the epistemic frame.This paper looks at whether this same form of mentor-ing takes place in one epistemic game, and if so,whether it has the same effect as mentoring in the pro-fessional context. We ask to what extent players in anepistemic game imitate the epistemic frame thatin-game mentors model, and whether this imitationleads to the internalization of a professional way ofthinking. While the results are specific to this study andto this epistemic game, the type of mentoring and learn-ing that happens in epistemic games could be usefulmodels for designing new types of educational settings,in schools or elsewhere.

Theory

There is extensive research on the role that mentors canplay in promoting positive behaviour and habits inyouth (DuBois et al. 2002; DuBois & Silverthorn2005). Vygotsky’s (1978) work on cognitive develop-ment describes the role of mediating agents in thechild’s social learning environment: namely that thedevelopment of the child’s higher cognitive processesdepends on their presence. While Vygotsky himself‘emphasized symbolic tools-mediators appropriated bychildren in the context of particular sociocultural activi-ties, the most important of which he considered formaleducation’ (Kozulin 2003, p. 17), scholars haveextended the discussion of those mediating agents toemphasize mediation through another human being.Much of the literature on social mediation is concernedwith how adults can facilitate the child’s problem-solving process by arranging and structuring theproblem for them, and by participating in the problemsolving itself. Joint problem solving (Wertsch 1978)and other situations where a child’s current capabilitiesare extended through the support of an adult, are oftenframed as scaffolded learning (Wood 1999) or as cogni-tive apprenticeship (Rogoff 1990). Mediating strategieslike scaffolding, while perhaps too context-dependentand numerous to be simply classified (Kozulin 2003),have been well described (Schaffer 1996), but how par-

ticular aspects of social mediation contribute to a child’scognitive development is unclear. How social learningprocesses are converted into internal developmentalprocesses, Vygotsky leaves murky, suggesting thatshowing ‘how external knowledge and abilities in chil-dren become internalized’ (Vygotsky 1978, p. 91) is animportant agenda.

In particular, Vygotsky (1978, p. 88) describes howchildren imitate problem-solving techniques; as he putsit, ‘using imitation, children are capable of doing muchmore in collective activity or under the guidance ofadults.’ Children can imitate adults or more advancedpeers to handle problems that would otherwise bebeyond them.As Valsiner and Van der Veer (1999) pointout, Vygotsky’s use of the concept of imitation is moresophisticated than simple copying; it is part of a learningand developmental process. When Vygotsky argues thatchildren ‘can imitate only that which is within . . .[their] developmental level’ (1978, p. 88), he is describ-ing what he calls the zone of proximal development.Problems that are in the learners’ zone of proximaldevelopment are not just those problems that theycannot yet do alone, they are problems that they have thepotential to one day solve by themselves. Thus, imita-tion leads to internalization.

While Vygotsky used this social learning process todescribe young children who were learning to do simplemath problems, the scope of the process is larger. Themovement from imitation to internalization is theprocess by which, as Vygotsky puts it, learners ‘growinto the intellectual life of those around them’ (1978, p.88). Researchers have described how knowledge is situ-ated in the activity, context, and culture in which it isused (Collins et al. 1991; Lave & Wenger 1991). Hutch-ins (1995), for example, examined the role that moreexperienced naval navigators play in novice crewmem-bers’ development of essential navigational skills andknowledge. The quartermaster monitors the actions ofthe novice watch standers as they attempt their duties,and is ready to help or take over if they are unable tocomplete the task to the ship’s requirements. More gen-erally, participating in the practices of a professionalcommunity, under the supervision and guidance ofmentors, gives individuals access to that profession’srepertoire of ways of seeing and solving problems (Lave& Wenger 1991). Mentoring practices in professionalcommunities are situated in the activities that the learneris attempting to master.

174 P. Nash & D.W. Shaffer

© 2010 Blackwell Publishing Ltd

Many professions have institutionalized this mentor–mentee relationship in the form of a professional practi-cum. Schön (1983, 1987) describes how novicesparticipate in practices they wish to learn in ‘simulated,partial, or protected form’under the guidance of a seniorpractitioner. This way of learning is suitable for learningthe mores of a professional community because profes-sionals, more than knowing basic facts and using basicskills, have a particular way of thinking (Goodwin1994). They make decisions based on a set of profes-sional values, and defend those choices with profession-specific modes of argumentation and standards ofevidence – that is, with a particular professional episte-mology (Schön 1983). To learn to participate in thecommunity of practice that Lave and Wenger describe,professionals-in-training need to learn to assume anidentity and values consonant with the fundamental pur-poses of their profession (Sullivan 1995).

Shaffer (2006b), building on professional repertoire,communities of practice, the role of practica in profes-sional preparation, and Vygotsky’s socio-culturalexplanation of cognitive development, introducesepistemic frame theory. ‘The work of creative profes-sionals is organized around epistemic frames,’ Shaffer(2006b, p. 12) argues: the ‘skills, knowledge, identities,values and epistemology that professionals use to thinkin innovative ways’. Professional thinking requires themastery of knowledge and skills, to be sure, but in thecontext of actual work, professionals rely on profes-sional values to direct their skills and knowledge. Simi-larly, professionals assume certain identity markersthat help them identify themselves as members of thegroup. And finally, they agree on accepted ways of jus-tifying their decisions, an epistemology. Yet epistemicframes are not merely collections of these unrelatedelements. As Shaffer (2006b, p. 160) argues, ‘theepistemic frame of a profession is the combination –linked and interrelated – of values, knowledge, skills,epistemology, and identity’ that professionals use to seeand solve problems. Professionals use their epistemicframe in the context of professional action. Practica, insimulating the world of professional practice, providelearners with the occasion to practice professionalaction – in essence, to begin to develop the epistemicframe of that profession.

In professional practica, where learners are facedwith problems that are often beyond their capabilities,mentors are there to guide them. Throughout the practi-

cum, mentors monitor learners’ performance, interven-ing at critical moments of confusion and struggle(Schön 1983, 1987). As Schön describes, while in thesecases, the mentors may sometimes instruct in the con-ventional sense, they mainly function as coaches whoseconversations with the learners’ highlight how to navi-gate the obstacles of the profession. As the novicesengage in the activities of the profession, the mentorsreflect, and invite the learners to reflect, on the work. Inthis facilitation of how to accomplish professionalwork, mentors reveal to apprentices ways to go aboutsolving problems. In helping the apprentice, they showthe way it is done: they model (Collins, et al. 1991).Through these reflective conversations, mentors modela way of working that requires the complex of skills,knowledge, values, identity, and ways of thinking asso-ciated with the community of practice. In short, theymodel the epistemic frame. In modeling the epistemicframe, mentors offer learners a professional vision thatthey can imitate and eventually internalize.

Assessing mentor modeling in epistemic games

There is a literature on educational games (Gee 2003;Squire & Jenkins 2004), and in particular literature thatdescribes the theory behind epistemic games or othergames that are based on rules and roles (Salen & Zim-merman 2004; Shaffer 2006a, 2006b, 2010). Shaffer(2006a) describes epistemic games as simulations ofprofessional training in which players play the role ofnovice professionals. Shaffer and others (Shaffer2006a, b; Svarovsky & Shaffer 2006) have shown thatthose who play epistemic games develop epistemicframes; in addition, these developed frames can persistmonths after the game is over (Bagley & Shaffer 2009).As in practica, epistemic games feature mentors wholead learners to the right way of working. In UrbanScience, for example, planning consultants guide theplayers through a series of activities drawn directlyfrom ethnographic study of an undergraduate planningpracticum. Urban Science provides us a case study ofmentor–learner interactions that can show the formationof an epistemic frame. Conversations between playersand mentors in Urban Science can show the extent towhich a player not only uses elements of the epistemicframe of a practice, but the extent to which the playeruses elements of the frame the way a more experiencedpractitioner does.

Mentor modeling 175


In order to accurately capture players’ demonstrationof epistemic frames while playing epistemic games, wehave developed a method to measure how game partici-pants, both mentors and players, link the elements of theepistemic frame during gameplay (Shaffer et al. 2009).Called Epistemic Network Analysis (ENA), this methodallows us to look at when and how often these frame ele-ments are linked, and the relationship between trends inthe players’ demonstration of the epistemic frame anddifferent features of the game. This study examines howplayers of Urban Science develop the epistemic frame ofurban planning through one particular feature of thegame, namely the mentors’ modeling of the epistemicframe. Specifically, we look at players’ pre and postinterviews and the reflective conversations betweenmentors and players to see whether the players imitateand internalize the epistemic frame that the mentorsmodel.

Research questions

This study asks three questions. First, did the players ofUrban Science develop planning epistemic frames?Second, during the game, did the players imitate theepistemic frame that the in-game mentors modeled?And finally, did the epistemic frames that players dem-onstrated during the game with the mentors persistwhen the mentors were not present after the game?

Methods

Study design

BackgroundIn the epistemic game Urban Science, students play therole of urban planners charged with redesigning neigh-bourhoods in their own city. Game activities weremodeled on an ethnographic study of a graduate-levelplanning practicum, Urban and Regional Planning 912,at the University of Wisconsin, Madison. The planningpracticum’s series of activities included:

• background research, where novice planners readabout the history of the planning site, including pastplanning decisions that have impacted it.

• a site visit, where novice planners learned about thefeatures of the planning challenge from first-handobservations and meetings with stakeholders.

• the creation of preference surveys, where noviceplanners prepared alternative plans using GIS soft-ware to elicit feedback from stakeholders about fea-tures of the neighbourhood they wanted preserved.

• staff meetings, where teams of novice planners dis-cussed information gathered and proposed planningsolutions.

• drafting of a final plan, where teams decided on andconstructed a proposed plan using GIS software.

• proposal preparation, where teams developed a pre-sentation that explained and justified their proposedplan.

• final proposal, where teams presented their proposalsto relevant stakeholders.

Gameplay in this version of Urban Science adaptedthese activities to be played by middle-school agestudents.

ParticipantsOffered as part of a summer programme called Collegefor Kids at the University of Wisconsin, Madison, 14middle-school age students played Urban Science 4 h aday during weekdays for 4 weeks during the summer of2007. Players had no prior experience with urbanplanning.

The four planning consultants in the game, whoguided the players in team meetings and as they worked,were played by graduate students who underwent a1-day training that covered the urban planning profes-sion, the game’s activities, and preferred mentoringstrategies. In addition, the mentors met as a group beforeeach session to plan for the day’s activities,and after each session to reflect on how the session went.

Game structureIn the game, players redesigned three neighbourhoodsites in Madison: State Street, Schenk-Atwood, and theNorthside. Each of the first 3 weeks focused on the rede-sign of a single neighbourhood site; in the fourth week,the players created a comprehensive plan that incorpo-rated redesigns of all three sites. This study focuses onthe first 3 weeks of the game.

The sequence of the activities was the same in each ofthe first 3 weeks. The game began with the players arriv-ing at a computer lab at the University of Wisconsin,Madison; this lab served as the office of the fictionalurban planning firm called Urban Design Associates.



The players signed into the game website, an intranetportal, where they found many of the tools that theywould use to play the game (Fig 1).

Once signed in, the players checked their inboxes andreceived emails from non-player characters (NPCs),including the head planner and community facilitator,who explained the planning process and tools used inthe game, provided background materials, and collectedand assessed work products. Players were directed toread the City Council’s Request for Proposals (RFP),conduct background research on their site assignment,and write a background report summarizing the site’simportant characteristics and especially any historicallysignificant planning decisions. For example, the RFPfor State Street, a popular pedestrian thoroughfare inMadison, Wisconsin, states that:

. . . a new plan for State Street will provide a uniqueopportunity for Madison to develop a long-term vision to

address unmet wishes of stakeholders within the city andcontinue Madison’s tradition of balance between urbanand ecological needs. The anticipated planning shallfocus on maximizing civic utility by addressing eco-nomic development and housing concerns for the area.Proposals must incorporate approaches for analyzingimpact on trash, sales revenue, jobs, greenspace, parking,housing, crime, and cultural index to be considered.

For each neighbourhood site, players acted as plan-ning advocates for one of four stakeholder groups. Forexample, the stakeholder groups concerned with thedevelopment of State Street were:

• Business Council• People for Greenspace• Equal Opportunities for All• Cultural Preservation Organization

Equipped with digital cameras, players traveled tothe site to conduct a site visit. On the site visit, players

Fig 1 The Urban Design Associates homepage, the game site for Urban Science.

Mentor modeling 177


took photographs features of the site that they thoughtshould be preserved or changed. Upon returning to theiroffice, they filed a neighbourhood assessment com-prised of their photographs, descriptions of the featuresthey photographed, and justifications for why theythought the features should be changed or preserved.

Next, working in teams, players conducted a virtualsite visit, in which they interviewed NPCs representingstakeholders in the community to find out what types ofissues they cared about. For example, one stakeholderfrom the State Street site, Ed, says:

Hi! My name is Ed and I helped form Equal Opportuni-ties for All because there are a lot of good people in thisworld that can’t normally afford high quality housing.Our organization works with people like you to findplaces to develop housing properties to help first-timerenters and homeowners establish themselves, particu-larly for individuals that have respectable jobs but can’tnormally afford a home. We’ve noticed that State Streetcurrently only has housing available to groups that canafford to pay incredibly high rents, which we think isinappropriate. We recommend finding a way to includemore housing on State Street so that more hard-working,blue-collar individuals can afford to live in a great placelike State Street.

The players recorded the stakeholders’ opinions intheir planning notebooks, and then began to translatetheir findings into proposed land use changes. Topropose land use changes, they used iPlan, an interac-tive GIS model of the planning site that let them assessthe ramifications of those changes. In iPlan, playerscould change zoning designations for the parcels, unitsof land held by a single owner. Zoning codes were rep-resented on the map in a unique colour (Fig 2):

The iPlan model also included graphs representingsocial and economic indicators important to each neigh-bourhood site. For example, the issues that wereimpacted by zoning changes to the State Street site were:

• Crime• Cultural index• Greenspace• Housing• Jobs• Parking• Trash• Total sales

As players made changes in the zoning of parcels, thegraphs dynamically updated, showing the projected

impact of the zoning changes on the social and eco-nomic conditions of the neighbourhood. For example, ifplayers chose to rezone a large arts complex as a surfacelot to increase parking, not only would the culturalindex decrease, but jobs and total sales would alsosuffer. Any single change to the physical representationof a site resulted in changes to its eight indicator values.

Using iPlan, in other words, players saw a physicalrepresentation of the site, the land use allocations for thestreet, and the consequences of their zoning changes(Fig 3).

Using iPlan, players worked in their stakeholderteams to construct preference surveys. As in the plan-ning practicum, preference surveys in Urban Sciencewere a set of possible planning alternatives designed toelicit information about the desires and hopes that stake-holders had for their neighbourhood. Specifically,players in Urban Science developed and used prefer-ence surveys to try to determine the minimum or‘threshold’ values that would lead stakeholders tosupport (or reject) a proposal. For example, playersmight have used a preference survey to determine howmany additional housing units were needed in a plan togain the support of the Equal Opportunities for Allstakeholder group – or how many additional square feetof parks were needed for the support of the People forGreenspace.

Once completed, players submitted their preferencesurveys to their stakeholder group. The virtual stake-holders responded to the preference surveys throughshort dialogue based on the specific indicator levels,delivered in the form of a printed report from a focusgroup. For example, one player working with the EqualOpportunities for All received the following feedbackfrom the stakeholder, Ed:

I’ve looked at your plan and there’s really no way that it’sgoing to work for us. There just isn’t enough housing onthe street! With so few places to live, landlords will beable to raise rents as much as they want, and there will beeven less affordable housing. I’m sorry, but this isunacceptable.

Next, players held a staff meeting with their planningteams to summarize the feedback they received. Eachplanning team presented their findings to the group as awhole, and new planning teams (with one player fromeach stakeholder planning team) were formed to draft afinal plan.



Each team used iPlan to create a final plan that couldincorporate the needs of all of the stakeholder groups.When plans were complete, each team prepared a pre-sentation of their findings and recommendations, whichwas delivered to a local planner acting as a representa-tive of the city council.

Data collection

Individual pre/post interviewsData were collected through individual interviews witheach player before and after the game. The interviewswere composed of questions about science, technology,

and urban planning practices. Pre- and post-gameinterviews from the game were recorded andtranscribed.

The questions asked were:

• What do think urban planning is?• Do you think urban planning is important?• What do you think it means to be a planner?• How would you say urban planners get information

for the plans they propose?• Do planners ever work with other people?• Do you think environmental issues are important to

cities?

Fig 2 Zoning codes used by players tochange land use designations of StateStreet’s parcels.

Mentor modeling 179


In-game interactionsData were also collected through recorded and tran-scribed interactions between the players and mentorsduring the game. There were three types of interactionsbetween players and mentors.

The first type consisted of one-on-one conver-sations where the mentor approached a player at

work and asked a prepared set of questions,including:

• What are you working on?• How is it going?• If not going well: What have you tried? Why?• If going well: Why is it going well?

Fig 3 An image of the iPlan interface. Zoning changes were dynamically reflected in graphs representing indicators such as crime, culturalindex, greenspace, housing, jobs, parking, trash, and total sales.



The second type consisted of team meetings whereplayer teams met to reflect on the work that they hadbeen doing. The mentor facilitated the meeting, askingquestions that included:

• What were you working on today/before?• How did it go?• What did you try/How did you do it? Why?• How would you do it differently next time? Why?

The third type of interaction consisted of spontane-ous conversations between players and mentors duringgameplay. These interactions were initiated either by aplayer or group of players in need of some help, or by amentor who saw that some guidance was needed. Thementor usually asked the same questions as detailed inthe first type of interaction, but depending on the natureof the situation, the conversations varied.

As detailed in Game Structure, once the playersbegan working on their final plans, the teams shifted sothat every final plan team was comprised of playerswho had conducted a preference survey with differentstakeholder groups. With each new week, and each newneighbourhood site, the player personnel in the stake-holder teams varied. The mentors, assigned to differentteams throughout the game, worked with all of theplayers, but the amount any one mentor worked withany one player varied week to week.

Data analysis

ENA is a method designed to assess learner perfor-mance based on the theory of epistemic frames. Thismethod measures whether epistemic gameplayersdevelop a particular epistemic frame over the course ofgameplay (Shaffer et al. 2009). Epistemic games arebased on a theory of learning that looks not at isolatedskills and knowledge, but at the way skills and knowl-edge are systematically linked to each other, and linkedto a set of values, epistemology, and identity markers.To assess a way of thinking about a professional domainmeans to measure a learner’s formation of connectionsbetween frame elements, because it is this constructionof a network of skills, knowledge, values, identity, andepistemology that allows the learner to see and solveproblems as a professional does (Shaffer 2006b; Shafferet al. 2009). ENA is an appropriate assessment because

it measures the extent to which epistemic frame ele-ments become linked.

ENA measures the links between frame elementsthrough the co-occurance of those elements in dis-course. Social network analysis uses analytical tools forrepresenting networks of relationships between people,which are ‘similar to . . . the complex and dynamic rela-tionships of the kind that characterize epistemic frames’(Shaffer et al. 2009). Social network analysis might usedata on the number of times people interact in a socialsituation to create maps to determine the connectionsbetween those people. ENA uses maps of the connec-tions between frame elements the way that socialnetwork analysis might map connections betweenpeople. ENA makes the assumption that the more timesframe elements occur together in discourse, the moreclosely they are related.

CodingTranscriptions from individual interviews were seg-mented into units representing one complete answerto a question, and included any follow-up questions orclarifications between the player and the interviewer.Transcriptions from in-game interactions betweenmentors and players were segmented into unitsrepresenting one complete interaction between aplayer, or group of players, and mentor. A single ratercoded all excerpts for elements of an urban planningepistemic frame: the interrelated set of skills, knowl-edge, values, identity, and epistemology of the profes-sion. The in-game interactions were coded both forthe players’ and mentors’ epistemic frames. Table 1describes the analytic codes used in our qualitativedata analysis of the in-game discourse. The analysis ofthe pre and post interviews used the same analyticcodes.

For example, if during a conversation with a plan-ning consultant a player mentions the value of servingstakeholders and the skill of zoning particular parcelsto create a site plan that will satisfy those stakeholders,that player’s excerpt is coded for values and skills, andthose elements are considered linked at that moment.Similarly, if during a conversation a mentor asks aplayer to justify a particular zoning choice in light of aparticular stakeholder’s needs, that mentor’s excerpt iscoded for values and epistemology, and those elementsare considered linked in that moment.

Mentor modeling 181


Tab

le1.

Ana

lytic

code

sus

edin

qual

itativ

eda

taan

alys

isof

in-g

ame

disc

ours

e.

Co

de

Des

crip

tio

nPl

ayer

exam

ple

Men

tor

exam

ple

Skill

sab

iliti

esn

eed

edto

bec

om

ean

urb

anp

lan

ner

...c

rim

eIm

anag

edto

keep

low

by

just

no

tad

din

gto

om

uch

hig

hd

ensi

tyh

ou

sin

g.

Are

ther

eo

ther

thin

gs

you

can

chan

ge

into

ho

usi

ng

rath

erth

anch

ang

ing

exis

tin

gh

ou

sin

g?

An

dw

ou

ldth

atm

ake

anim

pac

t?K

no

wle

dg

eas

pec

tso

fu

rban

pla

nn

ing

do

mai

nkn

ow

led

ge

...o

ne

of

the

stak

eho

lder

sw

ants

alo

to

fh

ou

sin

g..

..an

dI’m

also

mak

ing

sure

that

the

bu

sin

ess,

tras

h,a

nd

crim

ed

on

’tg

ou

pto

oh

igh

.

An

yth

ing

else

inte

rms

of

zon

ing

that

you

thin

kyo

um

igh

th

ave

tob

alan

ce?

Iden

tity

Feel

ing

so

fb

elo

ng

ing

toa

urb

anp

lan

nin

gco

mm

un

ity

or

of

bei

ng

ap

rofe

ssio

nal

Men

tor:

Wh

atd

idth

eysa

y?Pl

ayer

1:Th

ato

ur

pla

nw

asso

ph

isti

cate

d.

...

Men

tor:

Did

you

feel

sop

his

tica

ted

,pre

sen

tin

g?

Play

er1:

Kin

do

fPl

ayer

2:Y

eah

Play

er1:

Mo

reso

ph

isti

cate

dth

anlik

e,sc

ho

ol

Bu

tis

ther

ea

way

we

can

sort

of

loo

kit

asp

lan

ner

s?

Val

ues

thin

gs

that

are

imp

ort

ant

tou

rban

pla

nn

ing

pra

ctic

eIt

was

pre

tty

easy

top

leas

em

yst

akeh

old

ers,

bu

tth

isp

lan

pro

bab

lyw

ou

ldn

’tw

ork

very

wel

lfo

ran

yon

eel

se.

They

[sta

keh

old

ers]

are

allc

on

cern

edab

ou

tth

esa

me

thin

go

rd

iffe

ren

tth

ing

s?

Epis

tem

olo

gy

way

so

fth

inki

ng

abo

ut

or

just

ifyi

ng

acti

vity

wit

hin

the

urb

anp

lan

nin

gco

mm

un

ity

...w

hat

’sa

just

ifica

tio

nw

eca

nh

ave

for

crim

e,th

atw

hen

we

add

edth

en

eed

edh

ou

sin

gfo

rp

eop

lew

ho

wo

rkin

the

Sch

enk-

Atw

oo

dn

eig

hb

ou

rho

od

,an

dn

eed

toliv

eth

ere,

ino

rder

tod

oth

at,c

rim

ew

ent

up

by

on

ein

cid

ent

per

year

.

Alr

igh

t,so

then

wh

atw

ou

ldb

ea

just

ifica

tio

nfo

rth

est

akeh

old

erw

ho

wan

tsit

way

low

er,w

hat

wo

uld

you

say

toth

atst

akeh

old

er?



Dynamic epistemic network graphs

Computing epistemic adjacency matricesThe epistemic frame of a given profession, P, has ele-ments f1. . .n, where each fi is some element of theepistemic frame of P. Further, the epistemic game basedon P can be described as a series of activities aboutwhich we collect data, D.

For any participant p, we can look at Dpt, containing

the evidence that at time t that player p is using one ormore of the elements of the epistemic frame of the pro-fession P. To construct an epistemic network from datasuch as this, we create an adjacency matrix, Ap,t, forplayer p at time t, recording the links between elementsof the frame for which there is evidence in Dp

t:

A f f Dp ti j i j

pt

,, = 1 if and are both in (1)

By representing the epistemic frame in use during astrip of activity as an adjacency matrix, we can use thetools of network analysis to examine the cumulativeimpact of strips of activity on a developing epistemicframe. We construct the cumulative adjacency matrixfor player p, Fp, by summing the adjacency matrices Ap,t

from time t = 0 to the end of the game at time t = e as:

F Ap p n

n

e

==∑ ,

0

(2)

We use the same equation to compute Fp,t theepistemic frame for player p at any point of time t in thegame. That is, we sum the adjacency matrices for all ofthe strips up to time t by:

F Ap t p n

n

t, ,=

=∑

0

(2a)

If Ap,t represents the epistemic frame in use forplayer p during strip at time t, then Fp,t represents thecumulative epistemic frame for player p at time slice t.We thus represent the development of player p’sepistemic frame through a series of cumulative adja-cency matrices Fp,0 . . . Fp,e where e represents the endof the game.

We computed a series of epistemic adjacencymatrices:

• one for each player’s pre interview.• one for each player’s post interview.• one for each player’s in-game interactions, weeks

1–3.

• one for each player’s in-game interactions in weekthree only.

• one for the players’ collective in-game interactions,weeks 1–3.

• one for the mentors’ collective in-game interactions,weeks 1–3.

Derived network characteristics

Weighted density and adjusted weighted densityWeighted density provides a measure of a frame’s com-plexity: the overall strength of association of the frame(Shaffer, et al. 2009). We compute W(M), the weighteddensity of an epistemic network M (either for a strip oftime or a slice of frame development – that is, either forAp,t or Fp,t) by computing:

W MMi j

i j

( ) =( )

∑ ,

,

2

2(3)

We choose this measure rather than the more tradi-tional network density function because networkdensity measures only the number of links in thenetwork as a percentage of the total links. Such ameasure therefore does not take into account the weightof the associations in the network – which is a moreuseful measure of the strength of an epistemic frame.Computing weighted density from the squares of theassociations weights the core of the network, represent-ing networks with a small number of strong associationsas stronger than networks with a large number of weakassociations. We divide by 2 because the symmetricalnature of the adjacency matrix counts each linkagetwice. We compute the root of the sum of squares to uselinks as the units of weighted density.

In order to compare weighted densities when therewere a variable number of strips of time, we normalizedthe data by calculating an adjusted weighted density. Wecalculated the adjusted weighted density by dividing theweighted density by the number of its constituent stripsof time.

Relative centralityRelative centrality is a measure of the relative weight ofan epistemic network’s constituent frame elements.First, we compute the sum-of-squares-centrality C(fi) orweight of an individual node fi in matrix M as:

Mentor modeling 183


C f Mi i jj

( ) = ( )∑ ,2 (4)

Finally, we compute the relative centrality R(fi) of anindividual node fi in matrix M as:

R fC f

C Mi

i( ) =( )( )max

(5)

Where Cmax(M) is maximum node weight of any nodein M.

Statistical tests

T-testTo compare the complexity of the players’ epistemicframes between pre and post interviews, we calculatedthe weighted density of their frames based on theiranswers to pre and post interview questions. We thenused a paired t-test to determine whether or notthe weighted density of the players’ frames significantlyincreased between the pre- and post-game interviews.

CorrelationWe correlated adjusted weighted density of the players’3rd week frames with the weighted density of their post-interview frames.1

Results

Data in this section support three claims about the expe-rience of players in the epistemic game Urban Science.First, players developed epistemic frames by playingUrban Science. Second, the players imitated theepistemic frame modeled by the in-game mentors.Third, the players’ frames, as developed in-game, per-sisted even when the mentors were not present, after thegame.

Developing an epistemic frame

The matched pair questions that the players answered inthe post-interviews contained more co-occurring frameelements than those answered in the pre-interview. As aresult, the weighted density of the players’ post-interview frames was significantly greater than that oftheir pre-interviews (mean pre = 0.1, mean post = 4.4;P < 0.01). Figure 4 shows this change in weighteddensity:

These changes in weighted density corresponded to aqualitative difference in players’ development of theepistemic frame. For example, one player, when asked,‘What do you think it means to be a planner’ in the pre-interview, replied:

You sort of sketch out and you sort of visualize what willgo where and how that will work out.

Fig 4 Players’ change in weighteddensity from pre to post interviews.



In the post interview, the player answers the samequestion with considerably more detail:

I think it means collecting as much information as you canand it means listening to peoples’ opinion and takingthem into consideration. It also takes humor becauseyou’re not going to plan a place by yourself, you’re goingto have to collaborate with a lot of people and it takes a lotof compromising and coming up with justifications. Themain goal is trying to plan and design a city and trying toimprove it and making it the best you possibly can to fitthe people’s needs and what they want and trying to comeup with a solution for all the different opinions and pointof views.

Before the game, the player’s answer was vague andgeneral, without any mention of the elaborate process oflearning about stakeholders’ needs, or working with col-leagues to try and meet the needs of the stakeholders andthe community as a whole. After the game, in a more el-aborate response, the player refers to important elementsof the epistemic frame of urban planning: research, col-laboration, compromise, and justification all in theservice of improving a community for its constituents.

Imitating the modeled epistemic frame

In their conversations with players, mentors modeledthe planning epistemic frame. For example, in week 1, amentor reinforced a planning value by prompting a teamof players to remember that their job is to represent theneeds of stakeholders:

Mentor:Are you arguing your opinion or your stakehold-er’s opinion?Player 1: My stakeholders.Player 2: My stakeholders’ opinion. I don’t agree with it,but I’m arguing for them.

The mentor’s question makes clear what the correctanswer is. The players’ responses adopt the mentor’suse of the professional term ‘stakeholder’ and the plan-ning value of advocating for the stakeholders’ opinionsas opposed to one’s own whims.

The mentor then follows up with another questiondesigned to remind the planners that they are collabora-tors, not adversaries:

Mentor: My other question is: whose team are you on?Players 1 & 2: My stakeholders . . .Player 1: Oh, our team. Our team!

By bringing to the players’ attention that they are on ateam together, the mentor gets Player 1 to shift her per-spective from that of a simple advocate for one stake-

holder group to that of a colleague whose job it is towork with a team to serve the greater public good by sat-isfying all of the stakeholder groups.

In week 2, player 1, now on a different team with adifferent mentor, and working with a different stake-holder group, now thinks of her teammates’ stakeholdergroups when considering zoning changes:

Mentor 2: . . . . do you feel like there are some decisionsyou would adjust based on what you heard here andgoing forward to a final plan?Player 1: Well yeah cause we gotta compromise, I can’tjust like only want these people’s views to push throughall the problems.Mentor 2: Mmhm, so what are some of the ones you feelyou might have to adjust? The taxes was one . . .Player 1: Yeah taxes, um, I don’t know we have a lot ofthings in common kind of, well, Robert’s group doesn’treally care about greenspace, but um Cheryl’s group doeswant more greenspace and so does mine so that’s onething we have in common and also everyone hereincreased housing in their plan, um and I think everyonewants low crime, I mean, I don’t think anyone wants highcrime . . .

The player is now prepared to compromise becauseshe knows that planners do not only want to serve onegroup of stakeholders. She is ready to proceed directlyto figuring out how to serve the stakeholders by findingwhat they want ‘in common’.

In other words, players emphasized the same frameelements that the mentors emphasized, both in immedi-ate conversations and in conversations as the game pro-gressed. In terms of their relative centrality, the players’frame elements followed the same sequence as those ofthe mentors (Fig 5):

Together, knowledge and skills were the most centralfor both players and mentors; as the most central frameelements, their relative centrality was always at or near100. For both the mentors and players, the values frameelement was the third most central element, the episte-mology frame element the fourth, and the identity frameelement was the least central element. In the conversa-tions between mentors and players, mentors modeled aparticular planning epistemic frame, and the playersemulated that frame.

Internalizing the modeled epistemic frame

Comparing the players’ frames as adopted in the gamewith their frames as demonstrated in post-interviewsshows a significant relationship between them. Theadjusted weighted density of the players’ frames in the

Mentor modeling 185


third week of the game correlates to the weighted densityof the players’ post interview frames (R = 0.6535737,P = 0.021). Figure 6 shows the correlation betweenplayers’ in-game and post-game epistemic frames:

In summary, these results suggest that (a) playersbegan to develop an urban planning epistemic frame by

playing Urban Science; (b) during the game, playersadopted the version of the urban planning frame that wasmodeled by in-game mentors; and (c) the planningframe that the mentors modeled and players imitatedpersisted after the game when the mentors were notpresent.

Fig 5 Collective relative centrality of players’ and mentors’ epistemic frames throughout Urban Science.

Fig 6 Correlation between players’ in-game and post-game epistemic frames.



Discussion

Mentoring relationships are widely thought to positivelyinfluence young people’s development. This studyexamined whether a feature of one particular form ofmentoring – the modeling of professional thinking thattakes place in a professional practicum – was repro-duced in the epistemic game Urban Science. Specifi-cally, we looked at whether mentors’ modeling ofprofessional thinking in the game contributed to players’development of the epistemic frame of urban planningthrough gameplay.

We addressed this issue in three parts. First, we usedENA to examine the weighted density of players’epistemic frames in pre- and post-interviews. Theplayers’ post-interview frames were significantly moredense. This change suggests that in answering the post-interview questions, players saw more connectionsbetween the skills, knowledge, identity, values, and epis-temology of urban planning than they had in answeringmatched questions in the pre-interview. That is, theplayers of Urban Science began to develop a morecomplex epistemic frame in the domain of urban plan-ning. If, as Shaffer (2006b) suggests, professional think-ing is characterized by ‘the combination – linked andinterrelated – of values, knowledge, skills, epistemol-ogy, and identity,’ players of Urban Science appear tohave developed a more professional way of thinkingthrough playing Urban Science.

Next, we used ENA to examine the relative centralityof individual frame elements of both the players’ andmentors’ frames, as enacted in their conversationstogether throughout the game. In those conversations,the players emphasized the same frame elements that thementors did. As might be expected of more experiencedpractitioners, the mentors connected more frame ele-ments more often, suggesting a more mature epistemicframe. But the ordinal position of the players’ epistemicframe elements, and thus the shape of their epistemicframe, was the same as that of their mentors’. In otherwords, the players of Urban Science were able to imitatethe mentors’ professional way of talking about urbanplanning work. If, as Vygotsky (1978, p. 88) argues, chil-dren ‘can imitate only that which is within . . . [their]developmental level,’ the players’ successful emulationof what the mentors were modeling suggests that thegame activities of Urban Science were within theplayers’ zone of proximal development.

The zone of proximal development only describes thepotential for internalization. In order to determinewhether the mentors’ modeled frame was adopted, notmerely parroted, by the players, we compared theweighted density of the players’ frames in the third weekof the game with that of their post-interview frames. Theweighted densities of these frames were correlated. Thiscorrelation suggests that the epistemic frame that theplayers imitated during the game persisted after thegame. That the players were able to reproduce the imi-tated frame after the game is evidence that the players ofUrban Science internalized professional thinking to theextent that they no longer needed the mentors’ scaffold.The professional thinking used by urban planners wentfrom being in the players’ zone of proximal develop-ment during the game Urban Science, to being withintheir actual development level by the time the game wasfinished. Vygotsky’s assertion, that ‘what a child can[do] with assistance today she will be able to do byherself tomorrow’(1978, p. 87), was confirmed. While itis unclear exactly when the transformation took place,there is evidence that the players of Urban Sciencebegan to achieve some autonomy in their ability to thinkas professionals, and that their autonomy was derivedfrom their interactions with mentors.

Vygotsky’s hypothesis presupposes that learning pro-cesses, such as the imitation of modeled behaviour, areconverted into internal developmental processes. Whilethis study does not claim to completely demonstrate theprocess by which children internalize external knowl-edge and abilities, the results presented here do suggestthat the imitation of modeled behaviour is one importantstep in the process of internalization. We have shownhow mentors modeling professional thinking contrib-uted to the Urban Science players’ development ofepistemic frames. These epistemic frames were not justassociated with the mentors’ guidance in that moment:they were internalized by the players. The players’ imi-tation of the mentors’ modeling is part of the process bywhich they learned to think like professional adults.What was modeled mattered, as the participants ofUrban Science readily internalized the mentors’ way ofworking. Future work requires us to collect more datato conduct similar analysis of other aspects ofprofessional-style mentoring in epistemic games.

A second finding of this study is that ENA was shownto be a useful way to measure the development ofepistemic frames, as well as the relationship between the

Mentor modeling 187


players’ and mentors’ frames. Network analysis isuseful for measuring the relationships between things,and the complexity of those relationships. ENA mea-sures the interrelated dimensions of expertise. Onecould imagine using ENA to study any game or otherlearning environment where success is contingent on thedevelopment of multiple interrelated dimensions. Whatthose dimensions are in any given learning environmentdepend on the markers of expertise. Computer games,and in particular computer games grounded in valuedreal-world cultures like Urban Science, both providereal-life challenges for players and dynamically assessproblem-solving; as Gee and Shaffer (2010) argue, com-puter games are ideal for 21st century assessment. Infuture studies, we hope to further examine the role ofprofessional mentoring in epistemic games, and in par-ticular, to use ENA to investigate the process by whichplayers of epistemic games internalize epistemicframes.

Limitations

The results presented here have several limitations.First, this preliminary study only describes what a smallnumber of students did while participating in 80 hours ofthe Urban Science epistemic game.As a result, this workprovides insufficient grounds for making causal claims,nor does it by itself warrant immediately implementingepistemic games at a large scale. None of these resultsare more than suggestive, in the sense that this experi-ment is not a controlled study, conducted at large scale,and was not designed to distinguish individual differ-ences in performance. On the contrary, this was a small-scale experiment, fundamentally qualitative in nature,designed to see whether this approach to education hadpromise for further development, and if so, to under-stand the mechanisms at work that would be tested morerigorously in further experiments. Follow-up work thatlooks at a larger volume of data is already underway, andwe look forward to establishing more broad claims infuture papers.

Urban Science is a simulation of one planning practi-cum and one type of planning. Not all urban plannersfocus on zoning, and not all planners follow the samesteps that participants of Urban Science did. We feel,however, that Urban Science is an appropriately rigor-ous simulation of a particularly important aspect ofurban planning.

During this iteration of Urban Science, all of theplanning consultants were game researchers withlimited urban planning experience. The planning con-sultants were minimally trained and were in the sameplace as the players at all times. The planning consult-ant to player ratio (1:3 in this case) is not a ratio thatcan be duplicated in most traditional school settings.Follow-up studies in which planning consultants inter-act with players remotely, and in which the ratio ofplayer-to-mentor is much larger, are being conductedcurrently.

When looking at the relationships between thementor and player frames, this study treated all mentorsand players collectively. Further, the nature of how theplanning consultants mentored the players was notobserved. Future studies will include a closer analysisof the relationship between what the mentors are doingand what the players are learning. For example, specificmentoring strategies, such as asking certain types ofquestions or referring players to other resources, mighttend to work differently for different players, or differ-ently during different activities, or at different stages ofthe game.

Finally, ENA is a new method for understanding thedevelopment of an epistemic frame. As such, we expectto develop it in ways that allow us to better test signifi-cant events in player frame development. Spikes in therelative centrality of specific frame elements for playersduring certain activities will be more closely analysed.Also, frame density is a proxy measure in this study; wewould like to link it to practical demonstrations of think-ing in the domain of urban planning. In addition, theframe elements coded in this study are coarse-grained.Future studies will seek to measure frame elementsmore specific than the basic skills, knowledge, identity,values, and epistemology.

Despite these limitations, however, the results heresuggest several implications for the larger communityof people interested in mentoring, and in particular, theways players of games receive guidance from mentors.This study shows that mentors can model the sophisti-cated way that urban planners think, and that players ofthe game not only imitate the mentors, but also developthe ability to think as urban planners themselves. Thepersistence of the modeled epistemic frame suggeststhat the frame, once developed, constitutes a profes-sional vision that can be applied in a range of situationsin the world.



Acknowledgements

This work was funded in part by the Macarthur Founda-tion and the National Science Foundation throughgrants REC-0347000, DUE-091934, DRL-0918409,and DRL-0946372. The opinions, findings, and conclu-sions do not reflect the views of the funding agencies,cooperating institutions, or other individuals.

Note

1The data for one player’s third week consisted of only one interaction, so that

player was removed from the test.

References

Bagley E.A.S. & Shaffer D.W. (2009) When people get in theway: promoting civic thinking through epistemic game-play. International Journal of Gaming and Computer-Mediated Simulations 1, 36–52.

Collins A., Brown J.S. & Holum A. (1991) Cognitive appren-ticeship: making thinking visible. American Educator 15,6–11.

DuBois D.L. & Silverthorn N. (2005) Natural mentoringrelationships and adolescent health: evidence from anational study. American Journal of Public Health 95,518–524.

DuBois D.L., Holloway B.E., Valentine J.C. & Cooper H.(2002) Effectiveness of mentoring programs for youth: ameta-analytic review. American Journal of CommunityPsychology 30, 157–197.

Freedman M. (1999) The Kindness of Strangers: AdultMentors, Urban Youth, and the New Voluntarism. Cam-bridge Univ Press, Cambridge.

Gee J.P. (2003) What Video Games have to Teach Us aboutLearning and Literacy. Palgrave/Macmillan, NewYork.

Gee J.P. & Shaffer D.W. (2010) Looking where the light isbad: video games and the future of assessment. EpistemicGames Group Working Paper No. 2010-02.

Goodwin C. (1994) Professional vision. American Anthro-pologist 96, 606–633.

Hutchins E. (1995) Cognition in the Wild. MIT Press, Cam-bridge, MA.

Kozulin A. (2003) Psychological tools and mediated learning:Vygotsky’s educational theory in cultural context. InVygotsky’s Educational Theory in Cultural Context (eds A.Kozulin, B. Gindis, V.S. Ageyev & S.M. Miller), pp. 15–38.Cambridge University Press, Cambridge.

Lave J. & Wenger E. (1991) Situated Learning: LegitimatePeripheral Participation. Cambridge University Press,Cambridge.

Rogoff B. (1990) Apprenticeship in Thinking: CognitiveDevelopment in Social Context. Oxford University Press,NewYork.

Salen K. & Zimmerman E. (2004) Rules of Play. MIT Press,Cambridge, MA.

Schaffer R. (1996) Joint involvement episodes as context fordevelopment. In An Introduction to Vygotsky (ed. H.Daniels), pp. 251–280. Routledge, London.

Schön D.A. (1983) The Reflective Practitioner: How Profes-sionals Think in Action. Basic Books, London.

Schön D.A. (1987) Educating the Reflective Practitioner:Toward a New Design for Teaching and Learning in theProfessions. Jossey-Bass, San Francisco, CA.

Shaffer D.W. (2006a) Epistemic frames for epistemic games.Computers & Education 46, 223–234.

Shaffer D.W. (2006b) How Computer Games Help ChildrenLearn. Palgrave Macmillan, NewYork.

Shaffer D.W. (2010) The Bicycle Helmets of ‘Amsterdam’:Computer Games and the Problem of Transfer. Universityof Wisconsin-Madison, Madison, WI.

Shaffer D.W., Hatfield D.L., Svarovsky G.N., Nash P., NultyA., Bagley E.A.S., Franke K., Rupp A.A. & Misley R.(2009) Epistemic network analysis: a prototype for 21stcentury assessment of learning. The International Journalof Learning and Media 1, 33–53.

Squire K.D. & Jenkins H. (2004) Harnessing the power ofgames in education. Insight 3, 5–33.

Sullivan W.M. (1995) Work and Integrity: The Crisis andPromise of Professionalism in America. Harpercollins,NewYork.

Svarovsky G.N. & Shaffer D.W. (2006) Engineering GirlsGone Wild: Developing an Engineering Identity in DigitalZoo. Poster at the International Conference of the LearningScience (ICLS), Bloomington, IN.

Valsiner J. & Van der Veer R. (1999) The encoding of distance:the concept of the ‘zone of proximal development’ and itsinterpretations. In Lev Vygotsky: Critical Assessments,Vol. 3 (eds P. Lloyd & C. Fernyhough), pp. 3–31. Rout-ledge, London.

Vygotsky L.S. (1978) Mind in Society: The Development ofHigher Mental Processes. Harvard University Press, Cam-bridge, MA.

Wertsch J.V. (1978) Adult-child interaction and the roots ofmetacognition. Quarterly Newsletter of the Institute forComparative Human Development 2, 15–18.

Wood D. (1999) Teaching the young child: some relationshipsbetween social interaction, language, and thought. In LevVygotsky: Critical Assessments, Vol. 3 (eds P. Lloyd & C.Fernyhough), pp. 259–275. Routledge, London.

Mentor modeling 189


mentor modeling: the internalization of modeled professional thinking in an epistemic game

Documents

epistemic game urban

epistemic game jcal

model epistemic frames

professional way

professional training

epistemic network analysis

urban planning practicum

mentors model