does orientation promote learning and development? an assessment plan unearths the answers. megan...

Does Orientation Promote Learning and Development?

An Assessment Plan Unearths the Answers.

Megan France, Steve Grande, andSara Finney

James Madison University

An overview Welcome The Orientation Program at JMU Our Assessment Plan Data Collection Results Use of results to make informed decisions about

program changes Q & A

Orientation

As a result of your Orientation program, how should students change?

James Madison University Public; four-year Shenandoah Valley, Virginia 3,800 first year students Primarily residential Center for Assessment and Research Studies

JMU Orientation One-day summer program: Summer

Springboard Part of a three-phase experience

1. One Book

2. Summer Springboard

3. August Orientation

Components for students and family members

Orientation Overview University welcome

~45 minutes; University President Focus on high expectations, student support, sense of community

Small group discussions led by peer mentors ~35 min; Personalizing university welcome message

Information on majors and unique academic programs ~60 min; Meetings between students and faculty

Advising and course registration ~2 hrs; Small group and 1-on-1 with advisor

Take care of university “business” and experience campus ~2.5 hrs; Staff and OPAs guide student through pre-enrollment

activities and campus

Activities1. Interaction with peers

2. Receive orientation materials

3. Experience the university culture

4. Inspiring speakers

5. Meet and interact with faculty and advisers

6. Interact with current JMU student mentors

Assessment Plan Development One year to develop goals and objectives

Labor intensive Be realistic Focus on a few goals

Finding or developing appropriate outcome measures

Data collection plan On-going process

As a result of Summer Springboard... Orientation Goal 1: Increase perceived cohesion to

JMU Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale

Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy

Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation

Learning Goals?

Perceived Cohesion Measures the extent to which an individual

experiences and reports a sense of belonging and feelings of morale towards a group (JMU) Two subscales:

3 items measure Sense of Belonging “I see myself as part of the JMU community.”

3 items measure Feelings of Morale “I am enthusiastic about JMU.”

Scale of 1 (strongly disagree) to 9 (strongly agree)

Bollen and Hoyle (1990)

Goals and Objectives: Perceived Cohesion

ObjectiveObjective ActivityActivity MeasuresMeasures

Sense of Belonging

University Welcome and university

business

Perceived Cohesion Scale

(PCS)

Feelings of Morale

Peer mentor meetings


(PCS)

As a result of Summer Springboard... Orientation Goal 1: Increase perceived cohesion to JMU

Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale


Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation

Learning Goals?

College Self-Efficacy Inventory Measures self-efficacy or the confidence a student has

with respect to a range of college-related behaviors Three subscales:

4 items measure Roommate Efficacy “Divide chores with others you live with.”

7 items measure Course Efficacy “Take good class notes.”

9 items measure Social/Social Integration “Ask a professor a question.”

Scale of 1 (no confidence at all) to 10 (complete confidence)

Solberg, O'Brien, Villareal, Kennel, and Davis (1993)

Goals and Objectives: College Self-Efficacy


College Self-Efficacy

Faculty, adviser, and student staff

interactions

College Self-Efficacy Inventory

(CSEI)

As a result of Summer Springboard... Orientation Goal 1: Increase perceived cohesion to JMU

Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale


Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse

orientation

Learning Goals?

Miville-Guzman UniversalityDiversity Scale – Short Form Measures Universal-Diverse Orientation, which refers to an

awareness and acceptance of both similarities and differences among people Three subscales:

5 items measure Diversity of Contact “I am interested in learning about the many cultures that have

existed in this world.” 5 items measure Relativistic Appreciation

“Persons with disabilities can teach me things that I could not learn elsewhere.”

4 items measure Sense of Connection “Getting to know someone of another race is generally an

uncomfortable experience for me.” (R) Scale of 1 (strongly disagree) to 6 (strongly agree)

Miville (1992)

Goals and Objectives: Universal-Diverse Orientation*


Universal-Diverse Orientation

Role modeling and interaction

Miville-Guzman Universality-

Diverse Scale

(MGUDS)

Goals and Objectives

ObjectiveObjective ActivityActivity MeasuresMeasuresSense of Belonging University Welcome

and university business


Feelings of Morale Peer mentor meetings


College Self-Efficacy Faculty, adviser, and student staff interactions

College Self-Efficacy Inventory

Universal-Diverse Orientation

Role modeling and interaction

Miville-Guzman Universality-Diverse

Scale

Data Collection Who?

All first-year incoming students (not including transfer students)

What? First Year Student Survey Summer Springboard Survey

Both surveys included all three measures (PCS, CSEI, MGUDS)

Where? Surveys are on-line (i.e.,websurveyor)

Data Collection

When? First Year Student Survey is distributed early May and

needs to be completed prior to Summer Springboard Summer Springboard Survey is distributed via e-mail

after the students attend Orientation programming

Why this Design? To assess growth: The First Year Student Survey

serves as a pre-test to Summer Springboard (post-test)

There is no control population

Data Collection Issues Missing data

Assessment was not mandatory, but is expected 3,748 first year students 3,259 students completed the First Year Student Survey 1,909 completed both the First Year Student Survey and the Summer

Springboard Survey Students disregard of directions

249 students completed the post-test prior to the pre-test and their data could not be used (sample reduced to 1,660 students)

Invalid data 38 students from 1,660 were removed due to obvious response sets

Total sample was 1,622 out of 3,748 43.3%

Results: Explanation of Analyses Significance tests

Paired samples t-test used for analyses Compare mean of scores on pre-test with mean of scores on

post-test using the same students

Effect sizes Measure of practical significance

Important to investigate due to large sample

Cohen’s d Examined change in the metric of the variable Examined if students were more likely to reach

practical “cutoff” score after Summer Springboard.

Analyses: Perceived Cohesion Scale

Freshman Survey

(Pre)

Summer Springboard

(Post)

Mean SD Mean SDMean

Difference p d

Sense of Belonging 21.30 4.88 22.47 4.39 1.17 <.001 .25

Feelings of Morale 22.55 4.95 23.43 4.34 0.89 <.001 .18

Note. N = 1622. Scores range from 3 to 27 for both subscales.

Results: Perceived Cohesion

Students practically and significantly increased in sense of belonging and feelings of morale for JMU after attending Summer Springboard, but small effect size

What level of perceived cohesion would we like to see for all students after Summer Springboard?

Overall level of Sense of Belonging

Three items, 1 (strongly disagree) to 9 (strongly agree); range of total = 3 to 27 Wanted the majority of students to score above the

midpoint of scale at end of Springboard 18 or higher

Sense of Belonging at pre-test 76.7% (1,244 students out of 1,622)

Sense of Belonging at post-test 85% (1,379 students out of 1,622)

Overall level of Feelings of Morale

Three items, 1 (strongly disagree) to 9 (strongly agree); range of total = 3 to 27 Wanted the majority of students to score above the

midpoint of scale at end of Springboard 18 or higher

Feelings of Morale at pre-test 87.5% (1,419 students out of 1,622)

Feelings of Morale at post-test 90.3% (1,465 students out of 1,622)

Analyses: College Self-Efficacy Inventory

Freshman Survey

Summer Springboard

Mean SD Mean SDMean

Difference p d

Course Efficacy a 54.15 9.61 53.72 9.60 -0.43 .02 -.06

Roommate Efficacy b 33.90 4.83 33.58 4.95 -0.32 <.001 -.08

Social Efficacy c 67.43 13.38 68.60 12.90 1.16 <.001 .13

Note. N = 1619. a = scores range from 7 to 70, b = scores range from 4 to 40,c = scores range from 9 to 90

Results: College Self-Efficacy

Students practically and significantly increased only for social efficacy items, but again small effect size

What level of efficacy would we like all students to score after attending Summer Springboard with respect to interacting with faculty/staff?

Overall levels of social self-efficacy on CSEI

Five items; 1 (no confidence) to 10 (complete confidence); range of total = 5 to 50 Wanted the majority of students to score above the

midpoint of scale at end of Springboard 30 or above

Social efficacy at pre-test 78.9% (1,277 students out of 1,619)

Social efficacy at post-test 82.3% (1,333 students out of 1,619)

Analyses: Miville-Guzman Universality-Diversity Scale

Freshman Survey

Summer Springboard

Mean SD Mean SDMean

Difference p d

Diversity of Contact a 17.58 5.24 17.79 5.38 .21 .01 .06

Relativistic Appreciation a 22.55 4.13 22.46 4.34 -.08 .20 -.02

Sense of Connection b 20.92 3.92 20.73 4.04 -.19 .02 -.05

Note. N = 1,620. a = scores range from 5 to 30. b = scores range from 4 to 24.

Results: MGUDS

Students did not significantly increase in Diversity of Contact, Relativistic Appreciation, or Sense of Connection after attending Summer Springboard

What does this mean for Orientation at JMU? Goal 1: Increase perceived cohesion to JMU

Objective 1: Supported Objective 2: Supported

Goal 2: Assist in social integration to JMU Objective 1: Partially supported

Goal 3: Learn the value of a diverse campus Objective 1: Not supported

Use of Results: Perceived Cohesion Issues to consider for future assessment

Ceiling effect Students are scoring very high overall on this measure

Solution Change directions Find a more sensitive measure for perceived cohesion

2006: The following 6 items concern how you feel about JMU. Please read each item carefully and indicate the extent to which you agree with the statements…

2007: The following 6 items are asking how you feel about JMU. Not all students feel the same way or are expected to feel the same way. Please read each item carefully and respond according to how you feel at this current moment, not how you hope to feel as a first year student…

Use of Results: College Self-Efficacy

Issues to consider for future assessment Why didn’t students increase in roommate or course

efficacy? There is no programming (treatment); No need to assess

constructs

Minor change in social self-efficacy Investigating new outcome measures focused on confidence

to interact with faculty and staff Kicking around the idea of assessing help-seeking

What about the M-GUDS? As a result of our findings we realize that

Openness to Diversity is not substantially addressed during Summer Springboard, nor will it be.

Although increased Universal-Diverse Orientation was our goal, we realized that we first needed to help students get connected to JMU and meet faculty, mentors and peers before we could fully address openness to diversity.

Future Assessment for 2007

Clearer focus regarding goals of program Collecting PCS and CSEI No longer assess MGUDS during Summer

Springboard

Investigate ways to increase student participation in assessment endeavors

Revisiting goals and objectives

Thank you Questions?

Megan France ([email protected]) Steve Grande ([email protected]) Sara Finney ([email protected])

Thanks to Andrew Jones for helping with data analysis and shaping this proposal.

Perceived Cohesion1. I feel a sense of belonging to JMU.

2. I feel that I am a member of the JMU community.

3. I see myself as part of the JMU community.

4. I am enthusiastic about JMU.

5. I am happy to be at JMU.

6. JMU is one of the best schools in the nation.

CSEI1. Make new friends at college.

2. Divide chores with others you live with.

3. Talk to university staff.

4. Manage time effectively.

5. Ask a question in class.

6. Participate in class discussions.

7. Get a date when you want one.

8. Research a term paper.

9. Do well on your exams.

10. Join a student organization.

11. Talk to your professors.

12. Join an intramural sports team.

13. Ask a professor a question.

14. Take good class notes.

15. Get along with others you live with.

16. Divide space in your residence.

17. Understand your textbooks.

18. Keep up to date with your schoolwork.

19. Write course papers.

20. Socialize with others you live with.

MGUDS It is very important that a friend

agrees with me on most issues. I would like to join an organization

that emphasizes getting to know people from different countries.

In getting to know someone, I like knowing both how he/she differs from me and is similar to me.

Persons with disabilities can teach me things that I could not learn elsewhere.

I would like to go to dances that feature music from other countries.

Knowing about the different experiences of other people helps me understand my own problems better.

I often listen to the music of other cultures.

Getting to know someone of another race is generally an uncomfortable experience for me.

It's really hard for me to feel close to a person from another race.

I am interested in learning about the many cultures that have existed in this world.

I can best understand someone after I get to know how he/she is both similar and different from me.

I attend events where I might get to know people from different racial backgrounds.

I am only at ease with people of my own race.

Knowing how a person differs from me greatly enhances our friendship.

I often feel irritated by persons of a different race.

References Bollen, K.A. & Hoyle, R.H. (1990). Perceived cohesion: A

conceptual and empirical examination. Social Forces, 69(2), 479-504.

Solberg, V. S., O’Brien, K., Villareal, P., Kennel, R., & Davis, B. (1993). Self-efficacy and Hispanic college students: validation of the College Self-Efficacy Instrument. Hispanic Journal of Behavioral Sciences, 15, 80-95.

Miville, M.L., Gelso, C.J., Pannu, R., Liu, W., Holloway, P. & Fuertes, J.N. (1999). Appreciating similarities and valuing differences: Miville-Guzman Universality-Diversity Scale. Journal of Counseling Psychology, 46, 291-307.

does orientation promote learning and development? an assessment plan unearths the answers. megan...

Documents