making student learning visible: using concept map analysis as an assessment tool

Course and Assignment

Course: How People Learn is the gateway course for the eLearning and

Instructional Design M.Ed. and an elective for two other concentrations. It

introduces students to learning sciences research, including the cognitive

processes of learning and high impact practices that improve learning.

Assignment: In the first week, students create a visualization of how they

conceptualize “learning,” using the free online tool bubbl.us, and share it with

the class. In Week 10 they create a second map that depicts their revised

and refined understanding of “learning.” The images below illustrate one

student’s pre/post map set. This activity is both a learning strategy and an

opportunity for course assessment. This poster presents insights we gained

when we analyzed and compared pre/post maps.

Methodology

Sample:

We analyzed pairs of maps from 37 students in 2 different classes that occurred

during consecutive semesters. We omitted all maps created by students who

completed one, but not two maps.

Data Dimensions:

To create a data collection instrument, we selected half a dozen maps that we felt

demonstrated a range of sophistication and tried to identify salient characteristics.

Prominent features included content types, content relationships, shape, and design.

Norming:

We selected 12 maps that we both analyzed to compare our results, and used

discrepancies in our ratings to clarify our data form and our understandings of the

categories. When we were confident that our interpretations were reliably similar, we

divided the final maps and each analyzed a group.

Data Collection and Analysis:

Using a Google form for data collection enabled us to download results into an Excel

spreadsheet and sort to achieve multiple views. The images below illustrate our

analysis categories, data collection form, and graphs of corresponding results.

Making Student Learning Visible:

Using Concept Map Analysis as an Assessment Tool Gail Matthews-DeNatale, Ph.D.

Assistant Teaching Professor, Graduate School of Education

College of Professional Studies

Laurie Poklop, Ed.D.

Associate Director

Center for Advancing Teaching and Learning Through Research

Kaleena Seeley, B.A.

Research and Program Manager

Center for Advancing Teaching and Learning Through Research

Research Questions

• Do the pre/post maps exhibit progression from novice to expert

knowledge? Assuming there is evidence of progression from novice to

expert, what are the markers and patterns of development?

• Are some areas of development more prominent than others? Are there

gaps or areas in which they continue, as a group, to show little progress?

FIN

DIN

GS

Surprises and Reflections

Prior Knowledge: We assumed the baseline maps would feature affective

dimensions (e.g., learning as “fun”) and post maps would emphasize cognitive

processes (e.g., memory, transfer). Few baseline maps referenced affective

terms, and most pre/post maps included process-oriented terms.

Indicators of Development: If most maps focus on process,

what are the markers of novice, intermediate, and advanced

maps? We need a more sophisticated framework for analysis.

Redesign Decisions

Provide Context: We will add a required reading that orients students to concept

maps as an instructional strategy, providing a rationale for the assignment. The

revised assignment directions will also prompt students to include connecting terms,

as depicted in the reading’s illustrations.

Guide the Revision: If every term is afforded equal weight, then nothing is

important. Students need strategies for approaching the tasks of assigning value and

articulating relationships. These are important steps in conceptualizing and

visualizing a model. The revised Week 10 assignment directions will prompt students

to sort, categorize, look for relationships, and consider process flow between

conceptual domains.

Streamline and Focus: Students will submit the revised map with a written narrative

that explicates the revisions and provides a rationale for their conceptual model. This

will be in lieu of a synthesis paper that was not explicitly connected to the mapping

exercise.

Align Tools with Task: We will revise the bubbl.us tutorial to provide step-by-step

directions for adding terms to the connecting lines.

TOTAL 1 2 3 4 5

PRE 8 10 17 2 0

POST 0 1 9 22 5

0

5

10

15

20

25

Content Areas Referenced

Student 30:

Post-course map

OU

R S

TU

DY

IMP

LIC

AT

ION

S

FreeAssociation

Tiered/Hierarchical

Conceptual

PRE 24 11 2

POST 7 19 11

0

5

10

15

20

25

30

Relationship Between Terms

Constellation Branching Synaptic

PRE 22 14 1

POST 5 22 10

0

5

10

15

20

25

Map Shape

Fallacy of Complexity as Indicator: Post maps show an increase in all

content areas, terms, and the percentage categorized as branching or

synaptic. But more is not necessarily better. Authors who distilled their

ideas into a conceptual model created simplified maps. Perhaps complex

maps indicate intermediate as opposed to advanced development?

Conceptual Relationships: Some authors apparently

placed terms at random. Other maps indicate intentional

arrangement. Several students put

connecting words between terms. This made it

easier to follow the author’s thinking process.

Student 30:

Pre-course map