design-based implementation research to study and drive an undergraduate stem education improvement...

Design-Based Implementation Research to Study and Drive an Undergraduate

STEM Education Improvement Project at Oregon State

June 3, 2015SMTI 2015 National Conference

Jana Bouwma-Gearhart, Kathy Quardokus Fisher, Ann Sitomer, David Little, John Ivanovitch, Christina Smith, Milo Koretsky

1

Theory of action: Catalyze broad institutional change with respect to the use of Evidence-Based Instructional Practices (EBIPs) in large enrollment STEM classrooms through communities of practice (Wenger, 2009).

Research goal: Study changes to STEM instructional practices and pedagogical knowledge via

Micro-level educators’ and students’ practicesMeso-level units’ routines, structures and

cultureMacro-level institutional routines, structures,

and culture

DBIR (Penuel et al., 2011)Multiple stakeholdersPersistent problems of practiceIterative and collaborative design

Design-Based Implementation Research (DBIR)

2

Events Analysis

Theory of

Action

Enhancing STEM Education at Oregon State University (ESTEME@OSU)

Within unitsCommunities of Practice (CoP)Conference participationInstruction-related data

consultation

Across unitsCommunity of Practice (CoP)Collaboration with:

Center for Teaching and Learning (CTL)

STEM education research center

Other STEM change initiatives

Events

Tenure track and non-tenure track facultyGraduate teaching assistants (GTA) and undergraduate learning

assistants (LA)Chairs, deans, and vice provosts

Participants

3

Identifying a Model for Organizational Change Research Needs: Interpret a complicated system

Individual practices (cognition and action) The social/cultural context Aspects of their dynamic interaction

View of practitioners in light of change initiatives meant to foster more effective decision-making and other professional practices (Fumasoli & Stensaker, 2013)

“Building of [interventions] from practice” (Honig, 2003)

Past studies of organizational change Focus on policy adoption with fidelity (Fumasoli & Stensaker,

2013) Focus on mostly the macro scale or micro scale (Trowler et al.,

2005; Trowler & Cooper, 2002)

4

Theoretical Frameworks

Organizational LearningCreating, retaining, or transferring

knowledge in an organization

Cultural-Historical Activity TheoryAn organization understood as a

collective, artifact-mediated, and object-oriented activity system

Cultural ModelsShared information internalized through

socialization, and used for guiding behavior in a complicated environment

5

Events

Macro - Institution

Meso - Department

Micro - Individual

6

Center for Teaching and

Learning EventsAcross Units CoP

STEM Center Events

Project Synthesis

CoP on Specific Projects

Conference Participation

Instruction-related Data Consultation


Organizational Structure

7

Data SourcesEvents

Macro - Institution

Meso - Department

Micro - Individual


Learning Events

Across Units CoP

STEM Center Events

Project Synthesis



Organizational Structure



Administrative Interviews

Policy Documents

Project Team Interviews

Event and CoP Observations

Project Team

Interviews

CoP Observatio

nsAdministrative and Educator

Interviews GTA Practices Observation Educato

rs Survey

Educator Interviews

Educators Survey

Classroom Observations

Student Data

Events

8


Learning Events

Across Units CoP

STEM Center Events

Project Synthesis





Data Sources


Policy DocumentsProject Team Interviews


Project Team

Interviews

CoP Observatio

ns

Administrative and Educator

Interviews

GTA Practices Observation

Educators Survey

Educator Interviews

Educators Survey


Student Data

Macro - Institution

Meso - Department

Micro - Individual

EventsCollect Data

based upon Theoretical Frameworks

Analysis based upon


Info

rm

Theory of

ActionDesign

Design-Based Implementation ResearchOrganizational Structure

Macro - Institution

Meso – Unit

Organizational Learning (Argote & Miron-Spektor, 2011)

9

Learning process: As an organization accomplishes tasks (experience) it interacts with its context and “learns” by creating, retaining, or transferring knowledge.

Context is structure, culture, technology, identity, memory, goals, incentives, strategies, etc.

Active: members and toolsLatent: structure and culture

Knowledge Presented as: individuals’ schemas, routines, and the range of potential behaviorDistributed routines and cognition: Data sources and analysis

Educator and administrator interviews – qualitative analysisEducator surveys – social network analysis

Analyzed at the macro and/or meso level and informed by the micro level

Macro - Institution

Meso – Unit

Cultural-Historical Activity Theory (CHAT) and Expansive Learning (Engestrom, 2001)

Learning Process: Expansive learning is driven by contradictions and accomplished when the object and motive are radically re-conceptualized (zone of proximal development of the activity)

Context: An activity system is collective, artifact-mediated, and object-oriented. It contains subjects, tools and signs, objects, rules, community, division of labor.

Activity system history is needed to understand a system’s problems and potential.Historical context of division and labor: Data sources and analysis

Educator and administrator interviews – qualitative analysisPolicy documents – content analysis

Analyzed at the macro and/or meso level and informed by the micro level

10

Macro - Institution

Meso – Unit

Affordances of CHAT/Expansive Learning and Organizational Learning

11

Topic Organizational Learning

CHAT/Expansive Learning

Sources of change and development

Context interacts with experience

Contradictions

Types of Change Creating, retaining or transferring knowledge

Mini-learning or expansive learning

Context Active: Members, tools and tasks

Subjects, tools and signs, objects

Latent: which individuals are members, what tools they have and which tasks they perform

Rules, community, division of labor

History Lens by which changes may be identified

Lens by which changes may be understood

Micro/Meso/Macro Incorporates what is happening at the micro level, but focusing more at meso and macro level

Could be at any level, but tends to be based around an object at the meso level

Micro - Individual

Cultural Models (Ferrare & Hora, 2014)

Shared information internalized through socialization within and between groups.

Norms and practices are adopted, adapted and enacted by individuals to function in a complex environment Cultural models of teaching and learning: Data sources

and analysis Educator and administrator interviews – qualitative analysis Classroom observations - incidence/repertoires of practice

Analyzed at the micro level based upon individual’s judgments, perceptions and explanations of specific situations.

12

Events

13


Learning Events

Across Units CoP

STEM Center Events

Project Synthesis





Data Sources




Project Team

Interviews

CoP Observatio

ns


Interviews


Educators Survey

Educator Interviews

Educators Survey


Student Data

Macro - Institution

Meso - Department

Micro - Individual

EventsCollect Data

based upon Theoretical Frameworks

Analysis based upon


Info

rm

Theory of

ActionDesign

Design-Based Implementation Research

Events

14


Learning Events

Across Units CoP

STEM Center Events

Project Synthesis




Data Sources





Project Team

Interviews

CoP Observatio

ns


Interviews

Educators SurveyEducator

Interviews


Student Data

Macro - Institution

Meso - Department

Micro - Individual

Analysis Examples


Educators Survey

Potential for Organizational Learning in Networks Research Question: What is the potential for organizational

learning at one research university regarding issues of teaching and learning/curriculum and instruction?

15

Organizational learning via transfer

may be difficult because B_12 is not a

participant

CHAT question: How can the activity system of the unit make sense of the disconnect between

B_04 and B_12

Legend:Non participant

Participant

Cultural models question: To what

extent do individuals with many social connections have

shared cultural models?

DBIR: Given these analyses, what theory of action may be used to design events to improve the potential for

organizational learning?

Faculty Teaching Climate Survey Data

Ten items piloted from Knoreck (2012)

Two items omitted due to lack of fit in Confirmatory Factor Analysis

Analysis of eight climate itemsN = 140 respondentsReliability: α = .915Unidimensional factor accounted for

68.6% of the variance

Scale scores produced using item response theory

Equal variances assumed between climate scores of units,

Significant differences between means of climate scores of units,

Differences between units are of “large” effect size, η2

16

Item Content (Knorek, 2012)

Share teaching resources about how to improve their teaching with colleagues

Discuss the challenges they face in the classroom with colleagues

Consult with each other on teaching related issuesAre encouraged by department administrators (e.g. department chair) to communicate with colleagues about their teaching

Regularly discuss teaching related issues with department leadershipValue faculty teaching development services available on campus as a way to enhance their teachingBelieve that engaging in teaching improvement opportunities is part of their jobProvides new faculty with teaching development opportunities and resources

DBIR Theory of Action with Low Climate Score

17

Legend:Non participant

Participant

Focus on the core group that has participated

and discusses teaching regularly despite the

low climate score

Strengths and Limitations of our Project and Meld of Theoretical Frameworks

18

Strengths Limitations

Our projects capacity 2 faculty members, 3 postdocs, 2 graduate students

Are there too many details, as a result is the larger picture lost?

Multi-perspective of various modelsIndividuals, context, and interactions between the two

When do we stop considering new frameworks?

Potential for triangulation Will other change agents be able to use our model with less capacity for analysis?

Discussion and Questions

• How are you documenting and researching change in response to STEM education improvement initiatives in your institution or related organizations?

• What methodologies are you employing?

• What theoretical constructs concerning organizations and change guide your study?

• What is our model over-privileging/under-privileging?

• Per this discussion, what are some takeaways that can inform your plans for studying organizational change?

• What in our model may be most applicable, meaningful or replicable to your plans to study and document change?

19

DEPARTMENT CLIMATEIRT Item Parameters

Item ContentItem a b1 b2 b3 b4 b5

ShareRes 2.94 -2.16 -1.53 -1.07 -0.01 1.07 Share teaching resources about how to improve their teaching with colleagues

Challenges

2.33 -2.93 -1.84 -1.49 -0.43 1.16 Discuss the challenges they face in the classroom with colleagues

Consult 2.51 -2.57 -1.89 -1.22 -0.24 1.1 Consult with each other on teaching related issues

Encouraged

2.63 -2.11 -1.27 -0.74 0.04 1.05Are encouraged by department administrators (e.g. department chair) to communicate with colleagues about their teaching

Discuss 1.93 -2.85 -1.06 -0.52 0.35 1.77 Regularly discuss teaching related issues with department leadership

Value 1.71 -1.81 -0.92 -0.21 1.04 1.94 Value faculty teaching development services available on campus as a way to enhance their teaching

Believe 1.69 -2.3 -1.36 -0.51 0.4 2.04Believe that engaging in teaching improvement opportunities is part of their job

NewFaculty

1.44 -2.21 -1.57 -0.77 0.59 2.01 Provides new faculty with teaching development opportunities and resources

21

CLIMATE SCORES

22

ReferencesArgote, L., & Miron-spektor, E. (2011). Organizational Learning : From Experience to Knowledge.

Organization Science, 22(5), 1123–1137.

Engeström, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of education and work, 14(1), 133-156.

Ferrare, J. J., & Hora, M. T. (2014). Cultural Models of Teaching and learning in Math and science: exploring the intersections of Culture, Cognition, and Pedagogical situations. The Journal of Higher Education, 85(6), 792-825.

Fumasoli, T., & Stensaker, B. (2013). Organizational studies in higher education: A reflection on historical themes and prospective trends. Higher Education Policy, 26(4), 479-496.

Honig, M. I. (2003). Building policy from practice: District central office administrators' roles and capacity for implementing collaborative education policy. Educational Administration Quarterly, 39(3), 292-338.

Knorek, J. K. (2012). Faculty teaching climate: Scale construction and initial validation. Unpublished PhD dissertation, University of Illinois, Urbana, IL

Penuel, W. R., Fishman, B. J., Haugan Cheng, B., & Sabelli, N. (2011). Organizing Research and Development at the Intersection of Learning, Implementation, and Design. Educational Researcher, 40(7), 331–337. doi:10.3102/0013189X11421826

Trowler, P. & Cooper, A. (2002) Teaching and learning regimes: implicit theories and recurrent practices in the enhancement of teaching and learning through educational development programmes, Higher Education Research and Development, 21(3), 221–240.

Trowler, P et al. (2005). Freeing the chi of change: the Higher Education Academy and enhancing teaching and learning in higher education. Studies In Higher Education, 30(4), 427-444.

Wenger, E. (2009). Communities of practice. Communities, 22, 57. 23

Institutional culture models, structures, routines, synergy across projects, artifacts of

organizational history, leadership and other

resourcesExternal organizational

influences also accounted for

Community activities and networks, distributed

cognition/routines/felt affordances regarding

pedagogy and improvements

Community activities and networks, distributed

cognition/routines/felt affordances regarding pedagogy

and improvementsDepartmental culture models, decision-making, training and

other resources regarding pedagogy

Educators’ schemas, sensemaking, routines, felt

affordances and felt self-determination regarding

pedagogical practices/knowledge/commitmen

ts/ change activities/individual improvements

MESO

Interdisciplinary communities of practice concerning projects’ targeted evidence-based

instructional practices and community-identified problems; Informal socials

Project Activities

Data Sources (and analytics)

Intradisciplinary communities of practice concerning community-identified problems;

Disciplinary conference participation

Synergy with projects/stakeholders/suborganizations

to revise structures to better reward/support pedagogical

innovation and interdisciplinary work

Data of educators’ practices and student impacts/factors provided for reflection;

Support for individual pedagogical innovation (leadership)

Administrator interviews (qualitative inductive analysis)

Policy analysis (content analysis)Project(s) leadership interviews (qualitative

inductive analysis)

Community of practice observations (distributed routines/cognition analysis)Educator and administrator interviews

(distributed routines/cognition analysis)Project leadership interviews (qualitative

inductive analysis)

GTA practices and development video (discourse analysis)

Educator and administrator interviews (qualitative inductive analysis)

Educator surveys (department climate analysis, social network analysis,

distributed routines/cognition analysis)

Educator interviews (qualitative inductive analysis)Educator surveys (knowledge/incidences self-report of

practices, scaled metrics for determination, social network analysis)

Class observations (incidence/repertoires of practice, characterization of students’ cognitive demand)

Student data (qualitative and quantitative formative analysis)

Course artifacts (document and content analysis)

MACRO

MICRO

Possibilities for and evidence of organizational status and change centered in Cultural M

odels Theory, Organizational Learning, and Cultural H

istorical Activity Theory Fram

eworks

Design-Based Implementation Research To Study and Drive An Undergraduate STEM Education Improvement Project at Oregon State

24

design-based implementation research to study and drive an undergraduate stem education improvement...

Documents

learning events

stem instructional practices

enhancing stem education

practice honig

research goal

instructional practices

light of change initiatives

communities of practice