researching e-portfolios: the current state of play

Researching e-portfolios: the current state of play

Darren Cambridge, Barbara Cambridge, and Kathleen Blake Yancey

Inter/National Coalition for Electronic Portfolio Research

February 6, 2014

Overview

Our research model: Philosophy and process (Darren)

Some results: Scaling out (Barbara)

Current questions: The four propositions(Kathi)

Implementation Research

The Importance of Having ProblemsIn scholarship and research, having a "problem" is at the heart of the investigative process; it is the compound of the generative questions around which all creative and productive activity revolves. But in one’s teaching, a "problem" is something you don’t want to have, and if you have one, you probably want to fix it. … How might we make the problematization of teaching a matter of regular communal discourse? How might we think of teaching practice, and the evidence of student learning, as problems to be investigated, analyzed, represented, and debated? —Randy Bass

Three curricula

Kathleen Yancey, Reflection in the Writing Classroom

Research into the Swamp

“There is a high, hard ground where practitioners can make effective use of [traditional] research-based theory and techniques, and there is a swampy lowland where situations are confusing ‘messes’ incapable of technical solution. The difficulty is that the problems of the high ground, however great their technical interest, are often relatively unimportant to clients or to the larger society, while in the swamp are the problems of greatest human concern.” Donald Schön

Transactional Research• Practitioners generate research questions• Goal is to influence practice• Methodologies chosen based on knowledge

about learning, not exclusively current disciplinarily-accepted methodologies

• Agency for answering the questions resides in multiple constituents– practitioner researchers– learners– peer practitioner researchers

• Diversity provides robustness

Coalition structure


Coalition Exigency

• Rapid growth in use of electronic portfolios in the United States (and beyond)

• Wide diversity of models • Considerable potential to impact learning and

engagement• Evidence uneven and fragmented• Implementation unconnected to research


Coalition Structure

• Inter/National Coalition for Electronic Portfolio Research established in 2003

• Led by Barbara Cambridge (AAHE/NCTE), Kathleen Yancey (Clemson/FSU), Darren Cambridge (EDUCAUSE/GMU/AIR)

• Seven cohorts of about ten campuses that work together for three years


Coalition Activities

• Individual questions and collaborative themes• Two meetings a year• Intervening online communication• Interaction between cohorts • Consultations with Coalition leadership• Coordinated dissemination


Intra-organizational Practices

• Diverse team • Space for forming• Narrow but open question• Balance between intellectual and pragmatic

purposes


Diverse Team

• Both people who have research in their job title and those who don’t

• Reflective of the range of people involved in portfolio practice on the campus– Include administrators– Include students

• Portland State: Administrators, students, faculty from multiple disciplines


Space for Forming

• Need sufficient time and space to develop– Shared expectations– Shared conceptual framework– Personal relationships within team


Narrow but Open Question

• Well-focused research question• Openness to the data taking you elsewhere


Intellectual and Pragmatic Purposes

• Clear sense of audiences and purposes of research • Practitioner research doesn’t have to be just

evaluation • Balance between what you need to justify your work

and what’s intellectually meaningful• Practice as inquiry


Diversity and Balance

• Who might you ask to join your team you’ve not previously considered?

• What aspects of your project can you expand or emphasize to balance intellectual and pragmatic value?


Inter-organizational Practices

• Senior leadership sponsorship• Triangulation rather than replication • Collaborative exploration of

methodologies• Regular conversations with neutral

experts • Multiple genres of reporting out


Senior Leadership Sponsorship

• Three-year commitment of travel funding from institutional budget– Confirmation of commitment to portfolio practice

• Regular updates and notes of thanks• Ideally, member of the team


Triangulation

• Triangulation rather than replication • Enough structure to focus and connect, but not

restrict– No one strict definition of “research” – Shared themes but not a mandated research question

• Cohorts One and Two: Catalog and taxonomy of reflective artifacts

• Critical friends


Collaborative Exploration of Methodology

• Guided exploration of research methodologies and methods

• Both a way to plan the project and a way to develop shared understanding of research

• Breaking out of received notions of research through conversations– Across disciplines / practice areas – Across campuses / organizations


Conversations with Experts

• Quarterly conference calls with a Coalition leader

• Periodic occasions for reviewing and asking questions

• The questioning is probably more important than the advice


Multiple Reporting Genres

• Variety of forms of reporting– One-pagers– Blue Skies questions– Thick descriptions of artifacts– Presentations of evidence – Chats

• Helps to stimulate creativity and accommodate multiple styles

Scaling out

A local site implements a practice, does research on its effects, and shares the findings

with another site that adopts or adapts the inquiry question.

The process continues.

Over 60 campus research teams are linked through the Coalition in a network in which some teams have scaled out from others.

Scaling out can be around category of inquiry question, the inquiry question itself, methodology, purpose, audience, or emergent findings.

Today we’ll look at category of inquiry question as an illustration of the value of scaled out research having been done or being done in the Coalition.

1. Institutional mission

Goshen College, a small college in mid-America, strives to provide an education for the whole person to create leaders for the church and for the world. To that end, it chooses to focus on experiential learning and connections between academic affairs and student life. Specifically, each student spends a semester-long study abroad with classes and service.

Content analysis of reflective statements in eportfolios that address each of the college’s student

learning outcomesCoding of fifty reflective

statements about ways students had changed and about ways students had been strengthened in relation to learning outcomes:– Knowledge– Skills– Responsibilities– Integration

Six themes emerged, the strongest being- Affecting identity (values,

beliefs, and assumptions)- Increased self-awareness

Finding: Eportfolios enable students to turn experience into evidence of learning through reflection and through multiple representations of experience.

Implication for Goshen College:“We have a way to assess if we are acting out our

mission as a college through the structure of learning for students.”

Institutional mission

Loyola University, a medium-sized private university, in the middle-part of the US. It emphasizes the Jesuit value of social justice. It plays that out through experiential learning. The university strives to engage students in their own learning as they work to embody the university’s mission.

Inquiry into the Effectiveness of Introductions to Engaged Learning Assignments

In Fall 2013 a group of faculty gave a reflection assignment that generated an artefact for the students’ portfolio. Some faculty within the group used a common prompt; others did not.

Using activity theory, constructivism, and flow theory, another group of faculty and administrators (some in the group who gave the assignment and others not) catalogued the artefacts, used a rubric to judge the artefacts (norming), will review a random sample of the artefacts, and will continue the process during Spring 2014. Comparison and contrast will be made between the quality of the artefacts generated from the common prompt versus other prompts.

Relation to Mission

If the common prompt helped yield more effective reflective artefacts, ones that according to the rubric demonstrate engagement in experiential learning related to social justice, the university will consider use of that prompt across more classes in the university.

2. Professional identity

Clemson University-Cohort IIUniversity of Michigan-Cohort VITeams from two large universities in different parts of

the United States, one private and one public, chose to investigate professional identity, one of majors in psychology and one of minors in writing.

Clemson: How do eportfolios contribute to development of identity as a psychologist?

Psychological Assessment Survey, 169-item survey based on American Psychological Association’s learning goals and outcomes.

Five psychological factors: Five general factors;Knowledge base Information and technical literacyResearch methods Communication skillsCritical thinking skills Sociocultural/international awarenessApplication Personal developmentValues in psychology Career planning/development

Methodology and Finding

PAS at beginning of the semester; eportfolio-based PAS at end of the semester where each item tapped how their eportfolio affected the learning outcomes (n=95)

Finding: Both set of factors improved significantly over the semester between 0.4 and 0.8 on a 6-point Likert scale. Psychological factors improved at a greater rate than general competencies.

Instructor, self-rated, and peer-rated PAS scores were similar, confirming reliability of the instrument revealed by the initial confirmatory factor analysis.

Michigan: How do affordances in eportfolios of multiple connections, spaces for reflection, linkages, and navigations contribute to the ways

students develop and come to think about themselves as writers?

Four cohorts of writing minors (N=66) and comparison group (N=113)Mixed methods:Writing minors: Statistical analyses on entry• Instructional and demographic data and exit surveys• Entry surveys in semester students entered the program Coded for emergent themes• Exit surveys from semester they graduated related to writerly self-interest• Entry interviews during second semester of study and frameworks for thinking• Exit interviews as they graduate about writing• Application letters to Minor in Writing• Archived gateway and capstone eportfoliosComparison group: Rhetorical move analysis and• Writing samples from each semester in the study corpus linguistics• Directed self-placement essays• Writing samples from first-year writing classes

Five major findings

• Minors express stronger sense of themselves as writers than in the comparison group.

• Minors demonstrate greater genre awareness than students in the comparison group.

• Minors demonstrate a more developed sense of rhetorical situation than students in the comparison group.

• Minors demonstrate a more developed practice of reflection on writing than students in the comparison group.

• Eportfolios play an important role in helping students in the minor group understand their development as writers.

• According to the research of Clemson University and the University of Michigan, eportfolios contribute to the professional identity development of students in the respective professions of psychology and writing.

3. Use of eportfolios within a system

Lamar University and DePaul University have investigated the use of eportfolios across systems. Lamar, in an on-line graduate program, asked questions about effects of eportfolio practice in the university on teachers use of eportfolios in earlier academic levels. DePaul is comparing and contrasting an eportfolio element across disciplines and learning situations.

Lamar research question: Has participation as a master’s candidate using the eportfolio process contributed to

transference of eportfolio practices with PreK-12 students?

Method: Web-based survey to 271 program graduates in Fall 2011 and to 202 in Spring 2013 with 49% response rate. Based on Crosswell method of explanatory sequential design using collection of qualitative date to explain quantitative data results.

Findings about conditions for implementation in PreK-12:Support factors: administrative support, grading, parental support, internet

safety, eportfolio retention timeChallenges: assessment, eportfolios as official records, eportfolio reflection

timeNext inquiry: Case study of entire school district to investigate eportfolio implementation,

including contributions of teachers who have and have not studied through eportfolios themselves

DePaul research: The affordances of eportfolios enable more kinds and opportunities for reflection for all students. Reflections

are different across schools (professional, adult learners, first-year students). DePaul is studying how and why the differences.

• Current work includes coding eportfolios. Much discussion about methodology, but they have developed what they call a “mash-up methodology” on which they have shared agreement. They affirm the great benefit of having researchers from different backgrounds and settings working together. The acceptance of research results and commitment to application to practice are greater with this approach.

• Scaling out across an institution will be more possible after this initial work that signals the possibility of application of eportfolios in multiple educational contexts.

Other sites for further specific information about Cohort teams’ research

Final reports of all campus teams are posted on the Inter/National Coalition for Electronic Portfolio Research website (2008-2013)

Electronic Portfolios 2.0: Emergent Research on Implementation and Impact (Cambridge, Cambridge, and Yancey. Stylus Publishing, 2009)

Eportfolios for Lifelong Learning and Assessment (Cambridge, D., Jossey-Bass, 2010)

Scaling out can lead to core practices.

• Scaling out is important, but so is the evolution of core assumptions and practices as more and more local entities embody eportfolio practice and research. The next section of our webinar will focus on core propositions regarding eportfolio learning.

Cohorts 6 and 7

The Non-Negotiable Core of ePortfolio and the Four Propositions

COHORT SIX

Curtin University of Technology (Australia)

Goshen CollegeIndiana University Purdue University IndianapolisLamar UniversityNortheastern UniversityPortland State UniversityUniversity of GeorgiaUniversity of MichiganUniversity of MississippiVirginia Military InstituteWestminster College .

COHORT SEVEN

Deakin University (AUS)

ClemsonDePaulLamar UniversityLoyola of ChicagoSanta Clara UniversityUniversity of Cincinnati (two campuses)

The Non-Negotiable Core of Electronic Portfolio

--Confusion among definitions--Confusion among practices

--Interest in identifying what is distinctive about ePortfolios

When you complete a "Critical Task" assignment in a course you will upload this assignment to your Assessment Portfolio (AP). Chalk and Wire will notify your instructor that the assignment is ready to be graded. The instructor will grade your Critical Task assignment using a rubric. All Critical Task assignments must be submitted and graded through Chalk and Wire. At the end of your program you will have a complete portfolio of Critical Task assignments that document your professional growth.

Where is reflection?

FOUR PROPOSITIONS

1. Interaction of Pieces of Evidence2. Relationship between Evidence and Reflection3. Material Practices4. Meaningful Comparison without Standardization

PROPOSITION ONEInteraction of Pieces of Evidence

For meaningful assessment, interaction of pieces of evidence within an

eportfolio is more important than single pieces of evidence

PROPOSITION TWORelationship between Evidence and Reflection

Reflection on pieces of evidence within an eportfolio and on the eportfolio as a

whole provides information for assessment that isn’t available by other

means

IN TEAMS

PROPOSITION THREEMaterial Practices

The material practice of eportfolio composition generates distinctive

knowledge about learning

Reading portfolios as a practice. Confusion of port reading linked to media: print as linear harder to process; reflection providing role of narrator; what kinds of tools/commentary/ navigations do we need for the design and the readings of portfolios?

New students have limited experiences with various media; and their thinking is patterned and routinized in a way that’s not keyed to insight: they may need new experiences with all kinds of texts. Also complicated by proficiency with technologies and faculty facility with them.

PROPOSITION FOURMeaningful Comparison without

Standardization

Eportfolios enable meaningful comparison of student learning across

institutions (and other contexts) without standardization

What is validity?

How do different perspectives impact the way we assess an ePortfolio?

What is the process of assessment, and how does it impact judgments?

CONSIDER: The decision-making process should be an interpretive conversation guided by norms.

What criteria and evidence can be used in making a decision? When, how, and by whom?

And how do you document that decision-making process so others understand how you got there?

Some Tentative Results

1. Students areoften asked to collect artifacts, but not to consider what they learn from seeing how they interact with each other. Considering the interaction of artifacts=an opportunity for learning that is often missed.

2. Cohort members agree that reflection makes a distinctive contribution to learning, but we—faculty, staff, and students—are frequently vague about what we want that distinctive contribution to be. Strong reflection seems to tap at least two domains of learning (e.g., sociolinguistic; psychological; epistemic).

3. There is some resistance to the idea that we can have comparability without standardization. It’s not that cohort members want standardization, but they don’t see how they can manage assessment without it. It requires a paradigmatic shift in thinking—much like outcomes have.

Learn More

• Europortfolio: europortfolio.org • Inter/National Coalition for Electronic Portfolio

Research: ncepr.org• Eportfolios 2.0: Emergent Findings about

Implementation and Impact (Stylus, 2009)• Darren: [email protected], @dcambrid• Barbara: [email protected]• Kathi: [email protected]

researching e-portfolios: the current state of play

Education

critical task assignments

team inter national coalition

inter national coalition

inquiry question

minors demonstrate

experiential learning

negotiable core

united states