Experimental Designs: A Framework to

Evaluate High Impact Practices

Rajeeb Das28 October 2020

Outline

• High Impact Practices Summary

• The Problem: Lack of Experimental Designs

• Literature Review

• Analyses and Results

• Recommendations

• Solutions and Collaborations

– Community of Practice group

Kuh, G. D. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Association of American Colleges & Universities.

11 High Impact Practices (HIPs)

• First-Year Seminars and Experiences• Undergraduate Research• Diversity/Study Away/Global Learning• Learning Communities• Writing and Inquiry Intensive Courses• Service Learning, Community-Based Learning• Internships and Field Experiences• Capstone Courses and Projects• ePortfolios• Collaborative Assignments and Projects• Common Intellectual Experiences

https://www.aacu.org/node/4084

https://www.aacu.org/node/4084

• Research tends to focus on

– Profiles and descriptive statistics

– Practice, implementation, modification

– HIP on First Year Seminars is an exception

Lack of Experimental and Quasi-Experimental Designs

Concerns About HIPs

• there are “consistent problems in the research related to student outcomes … the evidence for the outcomes described in this review is moderate at best, and very often weak”– Swaner, L. E., and J. E. Brownell. 2010. Five high-impact

practices: Research on learning outcomes, completion and quality. Washington, DC: Association of American Colleges and Universities.

• First Year Seminars/Experiences

– 10,000+ practice articles

– 50 case-control articles identified

• Study Abroad (SA) Literature

• 17,000+ practice articles

• Almost no case-control studies reported

Select HIPs Literature

details) for study abroad group.https://frontiersjournal.org/wp-content/uploads/2015/11/TARRANT-RUBIN-STONER-FrontiersXXVI-TheEffectsofStudyingAbroadandStudyingSustainabilityonStudentsGlobalPerspectives.pdf

Not open access

Terzuolo 2018 Intercultural development Quasi Experimental DesignN=108 experimental group students completed IDI before/after studying abroad for a semester. N=65 in control group.SA students had significant pre- to post-test increases in intercultural mindset on IDI Developmental Orientation (DO) score.Characteristics were associated with statistically significant increases in IDI scores for those who studied abroad: female gender, self-declared multicultural identity, and having a grandparent born and raised abroad.https://doi.org/10.1016/j.ijintrel.2018.05.001

Open access

Young 2008 Persistence at University, SA Participants vs No SA Quasi ExperimentTreatment = participation in study abroad program sophomore year. Control group students qualified to participate in study abroad, but did not. Dependent variable was number of fall and spring semesters enrolled post-treatment through spring 2003.79% of treatment group graduated in 4 years. 51% of controls graduated in 4 yearsParticipation in study abroad explained 4.2% of variation in number of semesters enrolled post-treatment. This var contributed more to R2 than did all of the other nine variables collectivelyhttps://eric.ed.gov/?id=ED491021

SA Literature with Good Control Groups

1. Heinzmann 2015 ANCOVA• Found difference in willingness to engage with people from other

cultural backgrounds

2. Leutwyler 2016• No difference on pre/post survey and regression analysis for study

abroad cases and controls

3. Mule 2018 Short-Term, Faculty-Led Study Abroad Quasi-Experimental• Short-term study abroad did not show differences in antecedents,

identification, and pro-social values of global citizenship between study abroad cases and controls

4. Terzuolo 2018 Intercultural development• SA students had significant pre- to post-test increases in intercultural

mindset on IDI Developmental Orientation (DO) score

5. Hadis 2005. Differences in survey responses.6. Young 2008 79% of treatment group graduated in 4 years. 51% of

controls graduated in 4 years7. Petzold 2017 SA survey respondents reported higher scores.

• N=136 x 3 = 408, University of Florida First Time in College

• Records matched on 9 strata: gender, race/ethnicity, honors flag, degree college, class level, residency flag, high school GPA, first gen flag, county of residence

Study Design

Studied Abroad Applied to Study Abroad

Never Abroad

• No differences in academic outcomes

Study Abroad Analysis

Average Time to Degree

(Years)GPA

Degree4 Yr Grad

Yes6 Yr Grad

Yes

Study Abroad Group 3.90 3.59 81% 97%

SA Withdrew Group 3.83 3.59 81% 97%

Never Abroad Group 3.85 3.48 81% 96%

• N=136x3 and University of Florida First Time in College

• Records matched per previous slide

Study Abroad Climate Survey

2013 Timeline 2015

PreClimate Survey

PostClimate Survey

Participant GroupsStudy Abroad

WithdrewNever Abroad

• Difference in mean percentage values 2013 to 2015

• χ2 statistical difference between rows on most items

Study Abroad Climate Survey Results

Superscripts

1Agree + Strongly

agree

2Very good +

excellent 1I feel that I belong at this institution

2When you started here-Ability to appreciate and understand racial and ethnic diversity

2Current ability level-Ability to appreciate and understand racial and ethnic diversity

2When you started here-Ability to understand international perspectives (economic, political, social, cultural)

2Current ability level-Ability to understand international perspectives (economic, political, social, cultural)

Study Abroad Group -9% -16% -2% -11% -3%SA Withdrew Group 1% -28% -12% -18% 2%Never Abroad Group 14% -37% -17% -2% 0%

• Results offer a more nuanced understanding of the impact

• Steps I took were not a burden

• Analysis wasn’t that difficult

Features of This Methodology

• What are the mechanisms by which the Study Abroad HIP functions?

• Addressed by existing literature

• Add post-graduate intent (graduation survey)

• Analyze subpopulations

• First generation in college

• Low income students

Next Steps

• Policymakers/researchers lament limited impact of education research – Lagemann, 2002

• To have impact, the quality of education research must first improve – Penuel 2020

• Experimental methods are strongly encouraged– Education Sciences Research Act

– Every Student Succeeds Act

– NRC 2002, IES 2013, NSF 2013, AERA 2009

Research and Practice Concerns

Adapted from Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J.

C. (2006, July). What matters to student success: A review of the literature.

Washington, DC: National Postsecondary Education Cooperative.

Framework: Let’s Work Together

Community of Practice

Focused Literature

Review

Advocacy &

Research

Share Findings

• Discuss issues related to study designs, data access, IRB

• Gather a robust body of (quasi) experimental design literature

• Report progress on your efforts

– Advocacy, upcoming meetings, potential partnerships

Invitation to Collaborate

• Virtual meeting second week of November

– Poll will be sent out to set date/time

• Introductions

• Describe areas of interest/concern

• Share your work, struggles, and solutions

• Identify concerns and next steps

• Set date for spring virtual meeting

CoP Agenda

• Share concerns about HIPs, namely lack of robust evidence

• Advocate for study designs with comparison groups, especially quasi-experimental designs

• Identify ways to collaborate or contribute

– Community of Practice

– Use my literature to focus your own work

Summary

Session Evaluation and Request Form to Participate in a HIP

Community of Practice in Chat Link

Presented by Rajeeb Das, Senior Data Scientist, r.dasn.@tamu.edu