1

Scientific visualization tools render quantitative data into visual representations that can be analyzed and manipulated by scientists. More recently, K-12 students are also being exposed to such tools, enabling them to conduct authentic inquiry in classroom contexts. But certain problems require more qualitative, observational data that cannot be easily represented with current visualization software. This paper describes an approach to using photographs and video as a primary data source for observational inquiry. We describe a framework for students to collaborate around photographs and video, collaboration that leads to inquiry and the development of explanatory models. We also describe two of our learning environments to illustrate how students can begin to develop predictive theories from image data.

Introduction

Visualization tools have changed the ways that scien-tists model and explain data about quantitative phe-nomena. In recent years, these tools have beenadopted and modified by educational researchers foruse in K-12 classrooms1, 2. By representing quantita-tive data visually, students can begin to grasp complexideas that were once foreign to classroom activities.For many problem domains, the visual representationsprovided by scientific visualizations are simpler tomaster and manipulate than symbolic and numericalones, allowing students to more easily detect patternsand make sense of complex data. Instead of simplymemorizing and applying formulas and algorithms,students can use static and dynamic visualizations togain a deeper understanding of domain phenomena.

Most scientific visualization tools map quantitativedata into visual representations. For instance, temper-

ature may be displayed as a color spectrum, with redbeing the hottest points, blue being the coolest points.Such mappings work when numerical data can be col-lected, but many problems require more qualitative,observational data in order to be explained. Forinstance, census values can tell us a great deal aboutthe population of migrating animals, but observingtheir behaviors may provide insight into why theirnumbers are increasing or decreasing. In other words,quantitative data can often help us see that there areissues to be explored, but qualitative analyses ofbehaviors and processes are required to understandwhy such issues arise.

In this paper, we discuss ways to use qualitative datain visualization and modeling activities. Rather thanmapping numerical quantities into graphical forms,we use ordinary photographs and video clips as pri-mary sources of data. Imagery can provide evidencefor claims about observable processes and behaviors,and our research provides students with tools to ana-lyze and generalize explanations from still and mov-ing images.

We describe a class of activities where students usephotographs and video to investigate and explaincomplex phenomena. By removing the narratives andcaptions that generally accompany images and film,we create situations where students must discovervisual patterns for themselves. Once patterns are dis-covered, students develop qualitative models to gener-alize their findings and make predictions aboutadditional, related visual data. In this way, we shiftfrom the use of imagery as an information source toimagery as data to facilitate model building andexplanation.

Explaining Imagery

Brian K. SmithErik Blankinship

2

We describe a framework for developing models fromimage data and two learning environments for highschool classrooms that embody the framework. Thefirst, Animal Landlord, presents digitized nature filmsto students exploring issues in behavioral ecology.These students annotate and compare film clips toproduce qualitative models of predator-prey behav-iors. The second application, Image Maps, presentsstudents with historical images of their communitiesthat are used to develop understandings of urban plan-ning and cultural change. In both cases, students col-laborate around image data to construct qualitativemodels of observable processes (e.g., predationbehaviors, community change). The remainder of thepaper describes how students use images in inquirytasks.

Why Imagery?

Visual events are rich with opportunities for studentsto pose questions and reflect on behaviors andprocesses3-5. Teachers can encourage students to thinkabout events and anomalies present in the imagery,including issues not explicitly mentioned in a film'snarration or a photograph's caption. Being able toview moments in time with photographs and videocan also facilitate learning — watching a lion chaseits prey or seeing the styles of dress in the 1940's isdramatically different than simply reading about ani-mal behavior or history. And because imagery main-tains a historical record, students can return to it,reexamine it, and continue to learn from it. As a result,imagery establishes a context for problem solving, forgeneralizing explanations from pictorial evidence.

Imagery as data. However, photographs and videoare not typically thought of as artifacts for problemsolving and inquiry; they are more likely to be treatedas “visual aids” to accompany text or audio informa-tion. For video, we can draw distinctions betweenvideo as information and video as data6, 7. The formertreats video as an additional sensory channel accom-panied by text and/or audio information. In educa-tional films, narratives provide most of the content,creating self-contained “lectures” complemented withvisuals8. Similar comparisons can be made for photo-graphs. Where many have studied how illustrationsaccompanied by text present information, other havelooked at the types of observation and interpretationskills that must be applied to understand photographs.

While there can be useful content in imagery accom-panied by explanations, getting students to generatetheir own questions and hypotheses around photo-graphs and video can also be instructive. Video as dataassumes that viewers will analyze and interpret visualinformation for themselves. More so, video as dataprovides opportunities for collaboration betweenteachers and students, as problem solving becomesthe central task. For instance, there are learning envi-ronments that allow students to analyze properties ofmotion9-11 and kinesiology12 with digital video. Withthese, students measure physical phenomena directlyfrom video clips to develop quantitative stories aboutdistance, rate, and time relationships or how musclemovements relate to human actions. Educational tele-vision formats can also present contexts for usingimagery as data. For instance, The Adventures of Jas-per Woodbury provide problems to be solved andclues to their solutions13, rather than giving awaysolutions as many educational films do14.

Similarly, Goldman-Segall's Learning Constellations15 allows students and teachers to become “multime-dia ethnographers” as they study video of their class-room practices. Nardi et al.6 use video to facilitatereal-time collaboration during neurosurgery. Medicalstudents use the same video clips after surgery tounderstand the demands of operating room practices.And collaboration around video also plays a role inMedia Fusion, a system integrating digitized videowith quantitative data tools (e.g., spreadsheets) forstudents to back imagery with numerical evidence forproblems16, 17.

Investigating and modeling with image data. In allof these applications, students use video as data toconduct authentic inquiry around a problem. We buildupon this previous work by allowing students to cre-ate explanatory models and narratives for collectionsof photographs and video. To do this, they examineimagery seeking answers to a particular question (e.g.,“Why are lions a ‘bad’ hunters?”, “Why was the traf-fic circle in Harvard Square eliminated?”). Ratherthan simply receiving answers to such questions, thestudents are responsible for examining image data andassembling them into models that explain how andwhy particular events occur. Gradually, they movefrom raw image data to working with evidence in theform of significant photographs or video frames.These images are used to construct more complexgeneralizations of the processes being observed.

3

Our goal is to change the use of imagery in class-rooms by shifting students from recipients of contentto producers of media artifacts. Textbooks and tradi-tional school curricula can bias students to think thatlearning is a process of memorizing factual informa-tion without argument. Documentary narratives andphotographic captions often do the same, presentingcarefully crafted stories suggesting a “right” way toview a complex phenomenon. In a sense, students per-ceive the development of scientific knowledge asinformation simply waiting to be discovered18-20. Wewould like to help them understand that it is experi-mentation, argumentation, and iterative refinement ofideas that lead to the “truths” found in these imagesources. Instead of simply understand facts, we wouldlike them to understand the reasoning strategies thatunderlie scientific endeavors.

We are trying to have students perform tasks as expertpractitioners do, using imagery to develop models andtheories about phenomena. For instance, behavioralecologists might use videos of animals to study andanalyze behaviors and patterns. Urban planners oftenuse historical images to make decisions about futurezoning and construction issues. Students can alsoengage in this use of imagery as data to develop mod-els and predictions of visual events. Just as studentscan learn by analyzing and modeling quantitative data21, 22, we suspect that much can be learned throughsimilar activities with image data.

Students need support to become active observers andinvestigators of visual data. In particular, if we wantthem to develop causal explanations and models fromvisual data, they will need task structures to facilitatethe inquiry process. Students can learn by generatingquestions and hypotheses for themselves, but theyalso need to understand what makes a good question,a reasonable hypothesis. More so, they need to under-stand how to analyze photographs and video to createexplanatory models. The average student, accustomedto seeing imagery accompanied by narrations andcaptions, will likely have difficulties performing criti-cal observation and interpretation of images todevelop their own explanatory models. Our applica-tions are designed to help them make the leap toinquiry with image data by explicitly modelingexpert, investigation strategies23, articulating strategicknowledge needed to explain complex events and pro-cesses with image data.

Strategic activity. Assembling a causal story about

complex behavior means organizing observationaldata into coherent structures or models for explana-tion. It means thinking about the types of actions orevents involved in the process and understanding howthey influence final outcomes. Through consultingwith experts in several domains, we developed astructure for constructing explanatory models fromimage data. That structure defines four steps thatassist in building models from casual observations ofstill and moving images:

1) Decompose. Complex processes consist of manyconstituent, related actions. Interactions betweenpredators and prey, for instance, must progressthrough stages of detection, stalking, chasing, andfinally capturing. The changes in a city's major modesof transport may progress from horses to trains toautomobiles. Identifying these components providesthe building blocks for the remaining strategic steps.

2) Compare. It is not enough to analyze a single filmor photograph of a complex process. Our studentsinvestigate libraries of video and images and comparethem to look for similar events. By looking for varia-tions in a routine or across time, students can identifypatterns that may prove critical to explaining the pro-cess.

3) Identify factors. Once variations are detectedthrough comparison, students need to perform addi-tional analyses to determine the factors influencingthe variance. For example, one might observe thattrees are disappearing over time in a collection ofurban photographs. To explain why this is the case, itis necessary to look deeper at the images, to identifyadditional factors that may account for the disappear-ance (e.g., the number of electrical poles are increas-ing).

4) Model. With variations and influencing factorsidentified, students can generalize causal models thatexplain the phenomenon under investigation. Thesemay take the form of decision trees explaining theflow of an event or causal chains describing changesover time. Regardless of the form, the modeling stepcreates an explanatory framework that can be used topredict and design future configurations of the prob-lem space.

This investigation model provides structure for ana-lyzing complex, observable processes, whether thatmeans field observations or observations of imagery.While students are accustomed to looking at photo-

4

graphs and films, they are not necessarily accustomedto making fine-grained observations and explanationswith imagery. The investigation model helps themmove from raw image data to predictive theoriesabout observable phenomena. We also providedomain-specific heuristics to help students understandthe types of questions to ask during their investiga-tions. For instance, in behavioral ecology, askingabout costs and benefits of particular behaviors is agood strategy when trying to explain how and why anaction has evolved. In urban planning, one may wantto look for variations in land use patterns to under-stand how neighborhoods arise.

Explaining Animal Behavior with Video

To understand how the investigation model is instanti-ated in curricular materials, we provide two examplesof learning environments that we have developed. Thefirst is concerned with animal behavior. Nature docu-mentary films are commonly used in biology class-rooms to introduce concepts in animal behavior, butthey tend to provide descriptive overviews of behav-ior, neglecting many interesting causal processes infavor of straightforward outcomes. For instance, afilm might mention that a creature performs a particu-lar behavior without explaining the complexities ofhow and why it does so. Narration is the primarysource of knowledge in such educational films8, 14, yetthe video itself contains implicit information that stu-dents can observe and explain.

We developed a video environment, called AnimalLandlord, for high school students to investigate thehunting behaviors of the Serengeti lion24, 25. Only 15-30% of all hunts attempted by lions result in success-ful capture26, 27 and understanding the reasons for thisrequires investigating the causal interactions betweenthe lion, its prey, and the environment. Students usedigitized nature films to become field researchers,working to understanding how and why lions andtheir prey interact during the hunt. In conducting theirinvestigations, they explore topics in behavioral ecol-ogy such as social organization, resource competition,variation between individuals and species, and envi-ronmental pressures.

Animal Landlord provides support for the investiga-tion structure described earlier. Students begin withthe task of decomposing hunting sequences intosmaller behaviors. To do this, they label movie frameswith action titles (e.g., “Predator stalks prey,” “Prey

escapes from predator”). The actions that studentscollect for each film are assembled into an annotationwindow (Figure 1) where additional information isprovided. Besides identifying the action, studentsrecord the observations that led them to decide thatthe action is important to the hunt outcome (e.g.,“What do we observe as ‘predator stalks prey’? It fol-lows at the rear and crouches down low.”). They alsorecord interpretations or inferences that can be drawnfrom the action (e.g., “What can we interpret or askabout ‘predator picks target’? The lionesses probablychose the fat one because it would provide the mostmeat.”). The collection of actions, observations, andinterpretations form the basis for later discussion andgeneralization of behavioral models.

Figure 1 Student annotations for a hunting video.

Once a number of films are annotated, students loadthem into a comparison tool to detect similarities anddifferences across filmed events (Figure 2). Studentsuse the comparison tool to identify reasons for varia-tion in the films. For instance, by lining up all actionsmarked “Predator stalks prey”, students can visuallyinspect all films to see how stalking actions differacross multiple films (e.g., the type of prey, theamount of ground cover). The interface also allowsstudents to inspect actions before and after a selected

5

event to see how different interactions can lead to orresult from similar behaviors. By considering the dif-ferent paths to outcomes and identifying selectivepressures from the video, students can begin toexplain behavior in terms of evolutionary theory.

Figure 2 Comparing annotations with Animal Landlord. Each column is a film annotated by students. The window is aligned on the action “Predator stalks prey”. The faded columns are films that do not contain this action.

Students create models of the possible predator-preyinteractions that can occur during a hunting episode.They currently do this by taking their video annota-tions and creating decision trees on posters. Thesetrees represent the space of all hunting decisions madeby predator and prey during the observed videos. Theposter-sized decision trees are displayed around theclassroom, and teachers lead whole-class discussionsto help students think about the evolutionary reasonsfor the paths through the tree (Figure 3). For instance,a teacher might focus on a node marked “Predatorignores prey” to get students discussing the energycosts related to predation. Such prompting might alsolead to discussions of variance between male andfemale lions, why their energy costs might differ,hence their different hunting behaviors. In otherwords, the decision trees allow students (and teachers)to question why certain behaviors seem to reoccur

during hunting and to examine behavioral transitionsin light of optimization and evolutionary adaptation.

Figure 3 A student presenting a decision tree during a whole-class discussion. These trees are created from the video clips and model all possible actions that predator and prey can take during the hunt.

The decision trees are also used when viewing naturefilms after the computer intervention. That is, when-ever additional hunting films are shown in class, stu-dents use their decision trees to make predictionsabout the behaviors of the animals and to refine theirmodels if needed. For example, a film on chimpanzeesmight violate the students’ models, for chimps —unlike lions — hunt better when there is less vegeta-tion in the area. This helps students generalize theiroriginal models to include creatures like chimpanzees.In this way, we tried to make all classroom nature filmexercises incorporate model testing and refinementafter the Animal Landlord intervention.

Explaining Communities with Photographs

Our second example explores urban planning andcommunity change with historical photographs. Theseemingly ordinary landscapes that we live in containa great deal of history that goes unnoticed by mostresidents of a community. When students are taught toexplore their outdoor surroundings, they becomemore aware of the intricacies of man-madeenvironments28. Not only can they begin to appreciatearchitectural patterns, they may begin questioning andposing hypotheses about historical and social aspectsof their communities. For instance, the high rent dis-trict of Cambridge, Massachusetts still holds evidenceof its industrial past, and observant students may

6

begin to wonder when the area shifted to high-tech-nology. A key to answering such a question lies in thehistorical images of Cambridge. By making theseimages accessible to students, we hope to developnew ways for them to collaborate and investigate howand why local communities have evolved over time.

Camera historica. Careful observation of the presentcan yield interesting questions, but we also need toprovide students with a glimpse of a community'spast. The photographs found in a local, historicalarchive are well suited for developing a story of acommunity's past, but these are typically difficult forthe general public to access. To gain access to theseimage collections, we send students into their commu-nities to become urban researchers armed with a digi-tal camera augmented with a global positioningsystem (GPS) and a digital compass (Figure 4). Whenstudents take photographs, the position and orienta-tion of the camera are recorded directly into the digi-tal image file. By integrating geographicalinformation systems (GIS) with multimedia29, 30 , wecan record a “geo-referenced” trail of students' photo-graphs that can be used for inquiry. And, unlike Ani-mal Landlord, students are now responsible forcollecting field data as well as producing models fromimagery.

Students go outside their schools to take photographsof buildings in their neighborhoods. When they returnto their classrooms and download their images into anapplication we have developed, they can peer into thepast. Our program parses each image, extracts theposition metadata, and performs a search31 of a Cam-bridge GIS map to return the name of the photo-graphed building. By identifying the current building,we can retrieve and display historical images of thephotographed location (Figure 5). In this way, ourcamera provides a window into the past: Studentsphotograph the present and receive historical imagesfor their investigations of community change.

Figure 4 Image Maps hardware. A Kodak DC260 camera is attached to a Trimble GPS and Precision Navigation digital compass to provide position and orientation metadata.

Figure 5 Viewing the past with images of the present. Thumbnails on the right are images taken by students. Choosing one of these displays its larger image and an array of historical thumbnails across the top. The left image is the historical photo chosen from the retrieved collection.

Once historical images are retrieved and displayed,students can begin annotating and comparing them.Photographs are annotated with features that appear tochange over decades (Figure 6). For instance, a trail ofCambridge photographs shows the evolution of trans-portation from horses to railways to automobiles. Stu-dents can mark photos with appropriate labels (e.g.,“automobiles”, “trains”) and search on these tags toretrieve similar photos from different eras. The pur-pose of this activity is to help them notice how similar

7

features may vary across time.

Figure 6 Annotating images. Students develop ontologies to characterize interesting features of images. Objects in the photographs are labeled with these features and used to develop explanations of community change.

More importantly, students can begin to build modelsof how and why their local communities havechanged over time. The models that they construct arebased on the architectural patterns described by Chris-topher Alexander and his colleagues32. A problem/theme is chosen (e.g., “Crosswalks for people”), thecontext for the problem is described (e.g., pedestriantraffic is conflicting with transportation), and evidenceis provided in the form of historical images. In thecrosswalk case, students would construct a causalchain illustrating the progression from unmarkedpavement to marked crosswalks. After constructing anumber of chains, they can return to the field to seehow well their generalizations hold up in unexploredparts of the city. That is, the exercise does not con-clude with a single community outing; we expect stu-dents to iterate on their hypotheses. For instance, ifthey think that Harvard Square was rearranged to min-imize traffic flow, they may need to return to the loca-tion to discover how traffic was rerouted. Additionalphotographs in the present lead to historical picturesthat may help them discover the answer to traffic rout-ing issues.

As with Animal Landlord, students use image data tocreate models of “behavior”; in this case, the behav-iors are changes in a community over time. As well,students will collaborate and argue around these datato develop hypotheses about change. For instance, wemay divide a class into groups where each one studies

a sector of the city. As a class, they can assemble amore complete model of community change than asingle group could on its own. We also imagine thatmuch discussion and debate will revolve around thecausal chains that students produce. Teachers will beresponsible for helping students make use of investi-gation strategies as they go into the world to collecttheir data and to moderate arguments around theirhypotheses.

The Culture of Imagery as Data

It is important to stress that computer software cannotchange classroom learning without additional support.When imagery is used as data rather than information,student attitudes and practices must change from theirnorms. Teachers must also change their practices, asthey can no longer rely on narrative explanations toprovide the “right” answer for a problem. Rather theynow have to prepare for student-directed questionsthat they may not have answers for. In fact, for thedomains we have chosen, experts often lack answersfor student questions (for instance, no one reallyknows why lions, unlike most felines, live in groups).Thus, for teachers and students, investigating imagedata can lead to many unknowns.

In four deployments of Animal Landlord to Chicago-area high schools, we made many iterations of ouroriginal design in order to “fit” into the culture of theclassroom. We also worked with teachers to help themunderstand how to guide student inquiry, thus helpingthem to change their expectations about the use ofimagery as curricular materials. In a sense, the soft-ware tools and video database act as “conversationalprops”33, digital artifacts that people can refer to dur-ing learning conversations. Collaborative inquiry canbe mediated by such props34, but teachers must alsoguide students to search through the image data toseek multiple explanations for phenomena being stud-ied.

One goal for our classroom interventions was to fosterdifferent attitudes about the use of video in class-rooms and to understand what types of learning wouldoccur as a result. Ordinarily, nature films are viewedquietly by students, and they may be quizzed at theend to assess their recall of the content. We were try-ing to create an environment where argument anddebate occur during viewing, where students generatetheir own hypotheses and explanations about filmedevents, and where teachers probe and guide studentexplanations to become increasingly sophisticated. In

8

this section, we discuss some of our findings from adeployment of Animal Landlord, focusing on howclassroom practice and the software tools help supportthe use of imagery as data.

Our observations are based on work with 44 highschool freshman in two Chicago-area biology class-rooms, serving mostly upper- to middle-class socio-economic communities. In this particular school, 54%of the students are language minorities (i.e., English isnot their first language), and the classrooms weworked with reflected this diversity. The majority ofthe students were fourteen years old, and all of themwere enrolled in their first high school science course.

Classroom discussion. Discussions before, during,and after work with Animal Landlord were crucial tothe learning experience. As the investigations ofbehavior progress, teachers talk with students insmall-group or whole-class discussions, directingtheir activities and encouraging argumentation aroundtheir findings. These discussions tend to be student-centered; the teacher’s primary role is to respond totheir queries and to suggest directions for investiga-tion. Students do their own share of independent dis-cussion as they argue around the films to constructtheir annotations and decision trees, and these argu-ments spill over into classroom discussions. Ulti-mately, learning seems to emerge from student-initiated discussions fueled by the observations madeon the computer. This is very different than traditionalclassroom activities where discussions are initiated byteachers.

Being sneaky. Teachers encourage students to developcausal explanations by constantly prompting them toelaborate their hypotheses. Some of these elaborationprompts are generic — “why,” “what else,” “tell memore.” Others are more domain specific, drawing onevidence from the video clips and biological theories.A discussion from one of our classrooms where stu-dents were arguing that a lion in one of the video clipswas “being sneaky” appears below.

1. Teacher: What is the lion doing there [points to video on screen]?2. Student A: It’s being sneaky.3. Teacher: Sneaky … I’m not sure what you mean. What do you mean

by sneaky?4. Student A: Sneaky, you know, it sneaks around, it’s being clever.5. Student B: Yeah, but that seems different than the other things.

Shouldn’t it be stalking?6. Student A: Whatever … it’s still being sneaky.7. Teacher: How do you measure sneaky?8. Student A: What do you mean?9. Teacher: How do you describe it?10. Student B: You mean how can you tell it’s being sneaky? Like what’s it

doing?

11. Teacher: Yes. 12. Student A: It’s creeping along in the grass. It’s trying not to be seen.

It’s being sneaky!13. Student B: Yeah, but that’s stalking. Sneaky is more like an interpreta-

tion …14. Student A: Sneaky, stalking … it’s the same thing.15. Student B: It’s not cause sneaky doesn’t say how the lion acts.16. Student A: It’s acting sneaky!17. Student B: But what is it doing? It’s crouching and going slow in the

grass. So it’s stalking.

Lines 1 and 3 show the teacher asking for clarificationabout the students’ work. In line 7, she gives a specificsuggestion to consider how “sneaky” should be mea-sured; in a sense, she is asking to them to think morescientifically about stalking behaviors. “Sneaky” sug-gests that lions acts as humans might, intentionallyplanning to quietly approach their prey. The teacher ispushing the students to describe “sneaky” in terms ofmeasurable attributes. For instance, she later tellsthem to think about the amount of ground vegetationin the stalking area, as this can hide the lion’sapproach. Ultimately, one of the students begins tounderstand the point of the teacher’s questioning, andshe begins to argue with her partner (lines 12-17).

Being hairy. Another example of classroom discourseoccurred when students suggested that female lionshunt more than their male counterparts. This is evi-dent from simply comparing the number of femalesand males involved in the video corpus. But, as westated at the beginning of the paper, quantitative dataoften fail to explain why a particular event is happen-ing. In this case, the teacher tried to get students to usethe video to explain why females might be huntingmore than males.

Using the video as evidence, the class claimed thatmales are significantly larger than females, makingthem unable to hide themselves in the Serengeti grass-lands. Along with their physical size (which can be 2-3 times that of the female), they also have large manesthat also increase their chance of being spotted bypotential prey. This answer may seem adequate, butthe teacher prompted them with a biological strategy,thinking about the costs and benefits of a particularbehavior or feature.

18. Teacher: Having a big mane then is a cost to the lion. So there must be a reason for it. What’s the benefit?

This is a much harder question to answer, and stu-dents began developing hypotheses, many based onsexual selection (e.g., “If you have a big mane, you’rethe king of the pride,” “Bigger manes attract mates.”).

9

Such hypotheses could be valid, but the teacherpushed them back to the video to see if there was any-thing that might suggest why natural selection wouldfavor males to have manes. In this case, the studentsbenefited from the teacher’s guidance, for they haddifficulties making the leap from hunting behaviors towhat appears to be a cosmetic feature.

19. Teacher: How do they kill, lions? You watched the videos.20. Student 1: Fangs and bites to the upper neck.21. Teacher: The upper where?22. Student 1: To the jugular vein…23. Teacher: (interrupting) Found where?24. Student 1: Huh?25. Student 2: Where’s the jugular found?26. Student 1: On the neck.27. Teacher: Oh, so where would that be on the lion?28. Student 3: Underneath his mane?29. Class: Oh!30. Teacher: Oh really … so anyone have another theory?31. Student 4: Oh, so it’s like it bites the mane and misses it.32. Teacher: Yeah, the bigger the mane…33. Class: The harder it is to grab the neck.

In other words, males may have manes to defendthemselves from attack. The teachers has studentsarticulate an alternative theory for the presence of themane; she pushes them to associate a morphologicalfeature — the location of the jugular vein underneaththe mane — with an adaptive trait — manes are hardto bite through. More importantly, she prompts stu-dents to recall the video data they worked with,encouraging later justification of theories with evi-dence.

In both examples, student work is driving the contentof the discussions. Our teachers tried not to enter theclassroom with prepared lectures or topics; rather,they responded to student investigations, choosingparticular aspects of their unfolding explanations tocritique and further elaborate. In traditional hands-on,classroom experiments, students rarely have a chanceto investigate questions of their own, develop methodsfor testing hypotheses, or connect data toconclusions35-37. There are also few opportunities forstudents to engage in theory articulation, applyingtheoretical knowledge to actual problem solving38.The experience of using video as data works becauseteachers allow and coach students to develop theirown observations, interpretations, and questions fromnature films. Discussions based on student findingshelp them to create models from data, hopefully pro-viding them with a stronger understanding of the pro-cess of doing and explaining science.

Artifacts as conversational props. The discussionsabove suggest that students can engage in scientificdiscourse around video data when provided with

guidance. But without going through the exercise ofannotating, comparing, and modeling video as data, itis unlikely that such discussions could have occurred.Students require more than opportunities to “talk sci-ence”; they also need opportunities to “do science”.We claim that the activities that students perform attheir computers before and during these discussionsallow them to respond to teacher prompts and to suc-cessfully collaborate to produce explanations ofbehavior. More so, it is possible that the software andinvestigation model influence teacher goals andexpectations, shaping the strategies they use toencourage student inquiry.

Each strategy in the investigation model (see p. 3) isreflected as an artifact in software or on paper (Table1). For instance, students decompose behaviors withannotation tools designed to help them see that a com-plex process, like hunting, can be split into multiple,important actions. Decision trees became useful forillustrating multiple paths to the two outcomes, killingor not killing one’s prey. The representations providedby each artifact seem to help students focus on impor-tant issues and guide them through the process ofusing video as data.

Being vigilant. One example of how the artifacts helpstudents during inquiry came when students discov-ered a behavior known as vigilance. Although it israrely mentioned in high school biology textbooks ornature films, vigilant or “scanning” behavior, the fre-quency that a prey animal alternates between feedingand observing its environment to detect potentialpredators, have been the study of much behavioralecology research39-41. Our students detected thisbehavior while using Animal Landlord’s comparisontool. It happens that some films show prey animals

Table 1: The relationship between investigation strategies and student-created artifacts in Animal

Landlord.

Investigation strategy Artifact

Observation vs. inference Annotation notes

Behavior decomposition Annotation notes

Comparison Comparison tool

Identifying variation Comparison tool & decision trees

Modeling Decision trees

10

cycling between scanning and feeding. In a singlefilm, students may annotate these actions (“Prey looksaround”, “Prey eats grass”) and not notice that there isan interesting pattern. But when multiple films werecompared, students noticed these recurring events andbegan forming generalizations about the behavior.

At least one group in each classroom we observeddetected vigilant behavior using the comparison tool.Once it was detected, teachers could prompt studentsto explain what they were seeing. For example, oneteacher asked the students if they could detect varia-tions in scanning patterns across different video clips.That is, does a zebra check its surroundings as oftenas a buffalo? For that matter, does scan length andtime vary when the number of prey animals increase?

Being articulate. A second example of artifacts play-ing a role in learning considers the annotation tool.We administered pre- and post tests to students on thefirst and final days of their work with Animal Land-lord. The tests consisted of open-ended essay ques-tions drawn from university, ecology examinations.We were curious to see if student performance wouldvary as a result of investigating animal behavior withvideo. While the data are covered more extensivelyelsewhere42, we want to discuss how some of theseresults can be tied to use of the software artifacts.

The questions that we asked students could never becompletely answered with a single response. Forexample, the question, “What limits the amount ofprey consumed by a predator,” raises many potentialissues (e.g., the effort required to capture prey, thepercentage of unsuccessful captures, and so on). Onthe pretest, many students gave a single response tothe question, answers such as, “If they’re not hungry,they won’t eat,” and, “They know they have to savefood for times when prey are scarce.” Our first step inanalyzing the responses was to note the number ofissues raised for each essay question. Table 2 showstypical examples of student responses and the number

of points raised in each. An increase in the number of

points between the pre- and post tests indicates thatstudents understand the need to articulate multiplereasons for the execution of a behavior.

Similarly, each point raised may contain a justificationor explanation. Raising an issue such as, “a cost ofpredation is being out in the open,” is useful, but itsays nothing about why it is important to the creature.Justifying each point raised goes beyond stating whatoccurred in the video data, moving from descriptive tocausal explanations of behavior. Example justifica-tions are shown in Table 3.

Table 2: Sample student responses to pre/post questions and the number of issues coded for each

(issues are in italics).

Student response # of issues

1) If a predator cannot catch the prey, then that would limit it’s food consumption.

2) If a predator has offspring, it may have to watch the offspring instead of find food.

2

His physical characteristics such as its teeth, claws. The speed that he has. Ability to see close and far. His diet. Knowing what looks pleasing and healthy.

4

1) If the predator is hunting with a group it may have to save food for the others.

2) If another predator comes along the 1st predator may not eat all the prey and will save some for the other predator. Example — cheetah and lions meet.

3) They may not be hungry because they already ate.

4) Predator needs only enough to survive. Not to eat a lot in case of some-thing dangerous comes (another predator).

4

Table 3: Sample responses from the pre/post questions with justifications (in italics).

Student response

If it is at night. This is important because at night I think it would be hard to catch prey.

Takes a lot of energy to make the catch so by the time it catches it, it is too tired to eat it. So it wastes energy and gets nothing out of it, no energy put back in.

11

Both the number of points and justifications increasefrom pretest to post test (Figure 7). The mean numberof points raised for each question increases from 2.43to 3.93, (F(1, 42) = 28.63, p < .001), and the meannumber of justifications for each question alsoincreased from 1.25 to 2.41, (F(1, 42) = 14.14, p <.001). These increases suggest that students are refin-ing their initial conceptions of behavior and/or howbehavior should be explained to include more knowl-edge and rationales for this knowledge.

Figure 7 Mean number of points raised by the students in pre/post tests. The shading within each bar shows the number of points with and without a rationale or justification.

What accounts for these increases? We hypothesizethat exploring the nature films as data and discussingfindings in groups helps students discover and articu-late more behavioral issues. More important, we sus-pect that the annotation exercise plays a large role inthese results. The annotation tools provide a structurethat reinforces students to 1) be explicit about allactions leading to the success or failure of a hunt, and2) justify these actions with observations and interpre-tations. When forced to be explicit about the interme-diate actions in the hunt, students gain anunderstanding of their importance to the overall out-come; this is reflected in the increased number ofpoints. Discussions around the comparison tool mayalso contribute to our results. With that tool, studentsargue with each other about issues that were omittedfrom the annotations and question each others’assumptions about their rationales for including par-ticular events.

We lack data to support the claim that the structure ofour curricular artifacts contributes to learning. Weonly know of one study that examines differencesbetween multiple representations and their impact oncollaborative inquiry43, but those results suggest that

the expressive qualities of a representation can impactthe ways that students discuss and make sense of dataand evidence. In future work with Animal Landlordand Image Maps, we hope to work with multiple rep-resentations for the same task to better understandhow they effect investigations of image data.

Conclusion

We have been developing a class of visualization andmodeling applications that use imagery as a primarysource for learning through inquiry. Unlike most sci-entific visualization tools where quantitative data isrepresented visually, our students work directly withphotographs and video, constructing qualitative mod-els to predict future outcomes and events. Becausestudents often lack an understanding for the impor-tance of modeling37, 44, we imagine that the immedi-acy and concrete qualities of imagery may be anappropriate way to scaffold students into additionalmodeling tasks. Rather than simply looking at photo-graphs or watching videos, we want students arguingand debating over differences in image data.

While the software environments give teachers andstudents tools to begin doing investigations, usingimagery as data also means also means learning totalk about evidence in new ways. Students initiate dis-cussions through questions, observations, and infer-ences about patterns and behaviors that they discoverin the image data sets. Teachers lose some of theircontrol over the classroom agenda, but they compen-sate for this by guiding discussion, argument, andpublic criticism of student hypotheses. While we havefocused on student learning in this paper, it is also evi-dent that teachers are learning with their students,changing their practices and expectations away fromproduct (do you know X?) to process (can you do X?).

300+ students in twelve Chicago-area classroomshave used Animal Landlord, and a new set of studentsbegan using it in 1999 with video content tailored forstudies of conservation biology. As with the originalversion, we hope to see students developing causaljustifications of behaviors and their importance formaking conservation decisions. The Image Mapsproject is being readied for a deployment with stu-dents in spring 2000. As we continue to work withstudents and teachers, we hope to discover more aboutthe types of representations and strategies that canassist inquiry with qualitative, image data.

With Animal Landlord and Image Maps, students use

0

1

2

3

4

Pretest Posttest

Mean Number of

Points

JustifiedUnjustified

12

imagery as data to construct explanatory models ofcomplex processes — the interactions of predatorsand their prey and the changes in a community'sarchitecture. The applications also share the sameinvestigation model, the process of annotating, com-paring, identifying factors, and creating predictivemodels to explain the image data. Together, they rep-resent a first step towards reusing existing photo-graphs and video for inquiry learning and modelconstruction.

References

1. Gordin, D.N. & Pea, R.D. (1995). Prospects for scientific visual-ization as an educational technology. Journal of the LearningSciences, 4(3), 249-279.

2. Edelson, D.C., Gordin, D.N., & Pea, R.D. (1997, March). Creat-ing science learning tools from experts' investigation tools: Adesign framework. In Annual Meeting of the National Associa-tion for Research in Science Teaching. Oak Brook, IL.

3. Wetzel, C.D., Radtke, P.H., & Stern, H.W. (1994). InstructionalEffectiveness of Video Media. Hillsdale, NJ: Lawrence ErlbaumAssociates.

4. Bransford, J.D., Franks, J.J., Vye, N.J., & Sherwood, R.D.(1989). New approaches to instruction: Because wisdom can’tbe told. In S. Vosniadou & A. Ortony (Eds.), Similarity andAnalogical Reasoning (pp. 470-497). New York: CambridgeUniversity Press.

5. Collins, A. & Brown, J.S. (1988). The computer as a tool forlearning through reflection. In H. Mandl & A. Lesgold (Eds.),Learning Issues for Intelligent Tutoring Systems (pp. 1-18).New York: Springer-Verlag.

6. Nardi, B.A., Kuchinsky, A., Whittaker, S., Leichner, R., &Schwarz, H. (1996). Video-as-data: Technical and socialaspects of a collaborative multimedia application. ComputerSupported Collaborative Work, 4, 73-100.

7. Whittaker, S. & O'Conaill, B. (1997). The role of vision in face-to-face and mediated communication. In K.E. Finn, A.J. Sellen,& S.B. Wilbur (Eds.), Video-Mediated Communication (pp. 23-49). Hillsdale, NJ: Lawrence Erlbaum Associates.

8. Erdman, B. (1990). The closely guided viewer: Form, style, andteaching in the educational film. In E. Ellsworth & M.H. What-ley (Eds.), The Ideology of Images in Educational Media: Hid-den Curriculums in the Classroom (pp. 27-42). New York:Teachers College Press.

9. Cappo, M. & Darling, K. (1996). Measurement in Motion. Com-munications of the ACM, 39(8), 91-93.

10. Rubin, A. & Win, D. (1994). Studying motion with KidVid: Adata collection and analysis tool for digitized video . In Confer-ence Companion to CHI '94 (pp. 13-14), New York: ACMPress.

11. Rubin, A., Bresnahan, S., & Ducas, T. (1996). Cartwheelingthrough CamMotion. Communications of the ACM, 39(8), 84-85.

12. Gross, M.M. (1998). Analysis of human movement using digitalvideo. Journal of Educational Multimedia and Hypermedia,7(4), 375-395.

13. Barron, B.J.S., Schwartz, D.L., Vye, N.J., Moore, A., Petrosino,A., Zech, L., Bransford, J.D., & Cognition and TechnologyGroup at Vanderbilt. (1998). Doing with understanding: Les-sons from research on problem- and project-based learning.The Journal of the Learning Sciences, 7(3&4), 271-311.

14. Ellsworth, E. (1990). Educational films against critical peda-gogy. In E. Ellsworth & M.H. Whatley (Eds.), The Ideology ofImages in Educational Media: Hidden Curriculums in theClassroom (pp. 10-26). New York: Teachers College Press.

15. Goldman-Segall, R. (1997). Points of Viewing Children's Think-ing: A Digital Ethnographer's Journey. Mahwah, NJ: LawrenceErlbaum Associates.

16. Bellamy, R.K.E. (1996). Designing educational technology:Computer-mediated change. In B.A. Nardi (Ed.), Context andConsciousness: Activity Theory and Human-Computer Interac-tion (pp. 123-146). Cambridge, MA: The MIT Press.

17. Borovoy, R.D., Cooper, E.B.W., & Bellamy, R.K.E. (1994).Media Fusion: An application of model-based communication.In Conference Companion to CHI '94 (pp. 17-18), New York:ACM Press.

18. Carey, S. & Smith, C. (1995). On understanding the nature ofscientific knowledge. In D.N. Perkins, J.L. Schwartz, M.M.West, & M.S. Wiske (Eds.), Software Goes to School: Teachingfor Understanding with New Technologies (pp. 39-55). NewYork: Oxford University Press.

19. Lederman, N.G. (1992). Students' and teachers' conceptions ofthe nature of science: A review of the research. Journal ofResearch in Science Teaching, 29, 331-359.

20. Songer, N.B. & Linn, M.C. (1991). How do students’ views ofscience influence knowledge integration? Journal of Researchin Science Teaching, 28, 761-784.

21. Jackson, S.L., Stratford, S.J., Krajcik, J., & Soloway, E. (1995).Making systems dynamics modeling accessible to pre-collegescience students . In National Association on Research on Sci-ence Teaching (pp. , San Francisco, CA: .

22. Mandinach, E.B. & Cline, H.F. (1994). Classroom Dynamics:Implementing a Technology-Based Learning Environment.Maywah, NJ: Lawrence Erlbaum Associates.

23. Tabak, I., Smith, B.K., Sandoval, W.A., & Reiser, B.J. (1996).Combining general and domain-specific strategic support forbiological inquiry. In Proceedings of the Third InternationalConference on Intelligent Tutoring Systems (pp. 288-296), Lon-don: Springer Verlag.

24. Smith, B.K. & Reiser, B.J. (1997). What should a wildebeestsay? Interactive nature films for high school classrooms. InACM Multimedia 97 Proceedings (pp. 193-201), New York:ACM Press.

25. Smith, B.K. & Reiser, B.J. (1998). National Geographicunplugged: Classroom-centered design of interactive naturefilms. In Proceedings of the CHI 98 Conference on HumanFactors in Computing Systems (pp. 424-431), New York: ACMPress.

26. Bertram, B.C.R. (1979). Serengeti predators and their socialsystems. In A.R.E. Sinclair & M. Norton-Griffiths (Eds.),Serengeti: Dynamics of an Ecosystem (pp. 223-248). Chicago:University of Chicago Press.

27. Schaller, G.B. (1972). The Serengeti Lion: A Study of Predator-Prey Relations. Chicago, IL: University of Chicago Press.

28. Stilgoe, J.R. (1998). Outside Lies Magic: Regaining Historyand Awareness in Everyday Places. New York: Walker andCompany.

29. Kraak, M.-J. (1996). Integrating multimedia in geographicalinformation systems. IEEE Multimedia, 3(2), 59-65.

30. Spohrer, J. (1998). Worldboard: What comes after the WWW?Available: http://www.worldboard.org/pub/spohrer/wbconcept/default.html.

31. Tsui, C. (1998). Multimedia Data Integration and Retrieval inPlanning Support Systems. Unpublished M.S. thesis, Massa-chusetts Institute of Technology, Cambridge, MA.

32. Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A Pattern

13

Language: Towns, Buildings, Construction. Oxford: OxfordUniversity Press.

33. Brinck, T. & Gomez, L.M. (1992). A collaborative medium forthe support of conversational props. In Conference Proceedingson Computer-Supported Collaborative Work (pp. 171-178),New York: ACM Press.

34. Roschelle, J. (1992). Learning by collaboration: Convergentconceptual change. The Journal of the Learning Sciences, 2,235-276.

35. Germann, P.J., Haskins, S., & Auls, S. (1996). Analysis of highschool biology laboratory manuals: Promoting scientificinquiry. Journal of Research in Science Teaching, 33(5), 475-500.

36. Pizzini, E.L., Shepardson, D.P., & Abell, S.K. (1991). Theinquiry level of junior high activities: Implications to scienceteaching. Journal of Research in Science Teaching, 28(2), 111-121.

37. Schauble, L., Glaser, R., Duschl, R.A., Schulze, S., & John, J.(1995). Students' understanding of the objectives and proce-dures of experimentation in the science classroom. The Journalof the Learning Sciences, 4(2), 131-166.

38. Ohlsson, S. (1992). The cognitive skill of theory articulation: Aneglected aspect of science education? Science & Education, 1,181-192.

39. Scheel, D. (1993). Waiting for lions in the grass: The usefulnessof scanning and its effects during hunts. Animal Behaviour, 46,695-704.

40. Lima, S.L. & Dill, L.M. (1990). Behavioral decisions madeunder the risk of predation: A review and prospectus. CanadianJournal of Zoology, 68, 619-640.

41. Lima, S.L. (1987). Vigilance while feeding and its relation tothe risk of predation. Journal of Theoretical Biology, 124, 303-316.

42. Smith, B.K. (1998). Classroom Cinema: Explaining Behaviorwith Video. Unpublished PhD dissertation, School of Education& Social Policy, Northwestern University, Evanston, IL.

43. Suthers, D.D. (1999). Effects of alternate representations of evi-dential relations on collaborative learning discourse. In Pro-ceedings of the Third Conference on Computer SupportedCollaborative Learning. Palo Alto, CA.

44. Grosslight, L., Unger, C., Jay, E., & Smith, C.L. (1991). Under-standing models and their use in science: Conceptions of mid-dle and high school students and experts. Journal of Researchin Science Teaching, 28(9), 799-822.