First Two Years Project

Cathy Crosby-Currie

Christine Zimmerman

Bringing Theory to Practice

March 2007

Modeling the Multiple Influences on Civic

Development and Well-Being

ENVIRONMENT

INPUT OUTPUT

B impact

C Impact

= Who is learning(pre-test attributes: demographics, abilities, views, etc.)

= What are students learning (goals and objectives; post-test)

CognitiveCritical thinkingAcademic abilityVoting behaviorMental Health

AffectiveValuesInterestsSatisfactionAttitudes&Beliefs

= How is the output influenced/do students learn(forces behind something; program/intervention)

Academic or co-curr. programEngaged Learning PedagogyCommunity Service Program

Adopted from Astin, A. (1993). Assessment For Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Educaiton. Phonenix: ORYX Press, 18

Astin’s Theory of Involvement: I-E-O Model

Terenzini’s General Conceptual Model of College Influence on

Student Learning

Terenzini, P., Springer, L., Pascarella, E., Nora, A.(1995). Influences Affecting the Development of Students’ Critical Thinking Skills. Research in Higher Education, 36, 23-39.

Methodological Considerations

Experimental v. Quasi-Experimental Design Key difference between experimental

and quasi-experimental Researcher’s control over the “input”

variable Experimental: YES! -> cause/effect conclusions Quasi-Experimental: NO! -> examine

relationships Control v. comparison groups

Experimental v. Quasi-Experimental Design Quasi-experimental power comes

from: Ability to detect change through design

e.g., interrupted time series design Equivalence of comparison group to

experimental group

Longitudinal v. Cross-Sectional Designs

Cross-sectional:Comparing groups of different ages at one point in time

Convenient but lacks statistical and conceptual power

Longitudinal:Comparing individuals to themselves across time

Multiple cohorts is ideal

St. Lawrence’s Quasi-Experimental, Longitudinal Design

Participants:Two Cohorts – Selected Students from Classes ‘09 and ’10

Experimental Group – students in Brown College

Second year added a second experimental group

Comparison Group – non-equivalent group matched on key variables of interest

Comparison Group Sample II.xls

St. Lawrence’s Quasi-Experimental, Longitudinal Design

Data CollectionPretest (9/05 & 9/06)Posttest (2/06 & 2/07)Follow-up (4/07 & 4/08)

Challenges of Quasi-Experimental and/or Longitudinal Designs

Creating comparison group(s)Participant attrition

Communication incl. letter from president

Personalized letters & email Contacting students multiple

times/multiple ways Accommodate students’ schedules

Institutional Review Board Approval Reframe as a positive contribution to

your research not a hurdle to overcome

Challenges of Measurement

• Valid and Reliable Measures

• Direct - Indirect Measures

• Quantitative - Qualitative Data

• Process - Outcomes Measures

Reliability

Random error (noise)

Systematic error (bias)

… is the consistency or repeatability of responses

Reliability (cont.)Ways to increase data reliability:

Clear directions Clear questions Consistent order of questions Clear survey layout Trained proctors/interviewers Consistent data entry and scoring

Reliability (cont.)

Pilot-test your study

Test-retest your survey

Focus-group survey or interview questions

Include similar questions in same questionnaire

How to assess the reliability of your instrument:

Validity… the extent to which the instrument truthfully measures what we want to measure

How well does the instrument content match what we want to measure?

Do respondents interpret the questions correctly?

Do respondents’ answers reflect what they think?

Are the inferences we make from this study accurate? Can they be generalized?

Validity (cont.)

Use multiple measures and multiple methods

Derive measures from literature review & existing research / participate in national survey instruments and tests

Expert review

Pilot-test your own survey

How to establish validity:

Direct – Indirect Measures

Direct Measures Indirect Measures Portfolio

Essay/reflection

Performance task/test

Actual student behavior

Self-reported behavior, attitudes, gains

Grades

Participation rates

Time spent at task

Direct: tangible, actual evidence

Indirect: proxy for what we try to measure

Qualitative – Quantitative Measures

Qualitative Measures Quantitative Measures Focus groups Structured interviews Self-reflections/diaries Open-ended survey questions

Surveys with closed questions (Likert scale, check list, etc.) Grades Actuary data such as participation rates, attendance, etc.

Qualitative: unit of data = words

Quantitative: unit of data = numbers

Process – Outcomes MeasuresProcess Measures

What did we do?(=data to demonstrate the implementation of an activity/program)

Outcomes Measures What are the results?

(= data used to measure the achievement of an objective/goal)

Initial Intermediate Long term

Administrative ChallengesAnd Best Practices

Buy-in and Support Form campus partnerships early on Build on existing data collections

Institutional survey cycles and survey timing

Copyrights of survey instruments

Liability for use of certain measures

Survey recruitment & retention

Select Survey Instruments

and Literature

Sampling of Survey Instruments Entering Student Survey

CIRP Freshman Survey (HERI, UCLA) College Students Expectations Questionnaire

CSXQ (Indiana)

Enrolled Undergraduate Students/Alumni As a continuation of CIRP: Your First College

Year/College Senior Survey As a continuation of CSXQ: College Student

Experience Questionnaire (CSEQ) National Survey of Student Engagement (NSSE) Consortia Senior and Alumni Surveys (e.g. HEDS.

COFHE)

Sampling of Survey Instruments Depression/Mental Health Measures

Beck’s Depression Inventory (BDI II) Brief Symptoms Inventory (BSI)

Optimism/Pessimism/Happiness Scales Mehrabian Optimism/Pessimism Scale http://www.authentichappiness.sas.upenn.edu/

Alcohol/Drugs/General Wellness CORE Alcohol And Other Drugs Survey ACHA-NCHA

Sampling of Survey Instruments Civic Development (from Lynn Swaner)

CASA TELEPHONE SURVEY INSTRUMENT

Other National Surveys HERI Faculty Survey

Other In-House Institutional Surveys Course evaluations Program evaluations Satisfaction studies

Select Literature The National Center on Addiction and Substance Abuse at

Columbia University (2003): Depression, Substance Abuse, and College Student Engagement: A Review of the Literature. Report to The Charles Engelhard Foundation and The Bringing Theory to Practice Planning Group.http://www.aacu.org/bringing_theory/research.cfm

The National Center on Addiction and Substance Abuse at Columbia University (2005): Substance Abuse, Mental Health and Engaged Learning: Summary of Findings from CASA’s Focus Groups and National Survey. Report to Sally Engelhard Pingree and The Charles Engelhard Foundation for the Bringing Theory to Practice Project, in partnership with the Association of American Colleges and Universities.http://www.aacu.org/bringing_theory/research.cfm

Swaner, L.E. (2005). Linking Engaged Learning, Student Mental Health and Well-being, and Civic Development: A Review of the Literature. Prepared for BTtoPhttp://www.aacu.org/bringing_theory/research.cfm

Select Literature Pascarella, E., Terenzini, P.(1991). How College

Affects Students: Findings and Insights from Twenty Years of Research. San Francisco: Jossey-Bass.

Bringle, R. G., Phillips, M.A., Hudson, M. (2004). The measure of service learning: Research scales to assess student experiences. Washington, D.C. American Psychological Association

Suskie, L. (1996). Questionnaire Survey Research: What works. Tallahassee: Association for Institutional Research