life after the qep: leveraging assessment data for ......life after the qep: leveraging assessment...

Life After the QEP: Leveraging Assessment Data for Sustained Improvement

Timothy S. Brophy & Paloma Rodriguez, University of Florida

SACSCOC, Annual Meeting 2019

Getting to know you…

Your Role – Administration, Faculty, Staff

Are you currently concluding a QEP ?

Do you have plans for sharing your QEP assessment data?

Do you have plans for the continuation of your QEP initiatives?

Do you have experience with intercultural competence or study abroad surveys?

Goals for this Presentation

• Describe the University of Florida’s QEP and our findings• Share our survey instrument• Discuss how we leveraged assessment information to

ensure continuation of funding and inform leadership decisions

• Groups: Discuss strategies for institutionalizing your QEP findings

Qual i ty Enhancement P lan

Internationalization Task Force – 2010-2013

Leadership Team 2014-2019 • Leo Villalon, Dean, International Center• Paloma Rodriguez, Director, Office of Global Learning• Mary Kay Carodine, Assistant Vice President, Student Affairs• Timothy S. Brophy, Professor and Director, Institutional Assessment• Matthew Jacobs, Director, Bob Graham Center

Learning without Borders: Internationalizing the Gator Nation

Initiatives• Study Abroad• On campus course development• Campus Life• International Scholars Program• International Calendar & Marketing

Learning without Borders: Internationalizing the Gator Nation

Student Learning OutcomesInternationalization

Integration of global awareness and intercultural competence into student learning

SLO 1Identify & describe

global and intercultural conditions

SLO 2Analyze & interpret

global and intercultural issues

SLO 3Communicate effectively with members of other

cultures

Global AwarenessUnderstanding of

communities worldwide

Intercultural Communication

Effective interaction

The Measures

Direct assessment of content in the courses

Indirect and Direct Assessment of International

Critical Thinking and Communication

Direct measures in courses as determined by faculty

Indirect measures developed by UF experts

D a t a S o u r c e : T h e S u r v e y I n s t r u m e n t

Developing the Measures

We reviewed 7 existing instruments designed to measure components of

internationalization in undergraduates

We examined their alignment with our SLOs,

and evidence of their validity and reliability

We disqualified these for two primary reasons• Did not measure our SLOs• Did not meet our standards for

reliability (r = .90 or higher)

Developing the Measures

Institutional rubrics for the SLOs were developed based on AACU VALUE rubrics• Intercultural Knowledge and Competence• Critical Thinking• Written Communication• Oral Communication

International Critical Thinking (IntCRIT) and International Communication (IntCOMM) Attitudes and Beliefs Survey

Wilson, Timothy J. (2014). Assessing internationalization efforts: Utilizing Item Response Theory to validate intercultural competency and global awareness in postsecondary undergraduate students. (doctoral dissertation, University of Florida). Retrieved from: http://eds.b.ebscohost.com/eds/detail/detail?vid=1&sid=1563ae9d-bb48-458b-ae5d-a59011c14e08%40pdc-v-sessmgr04&bdata=JkF1dGhUeXBlPWlwLHVpZCZzaXRlPWVkcy1saXZl#AN=ufl.033650760&db=cat04364a

We reviewed all commercially available measures

Created our own, with Tim Wilson, a PhD candidate in Research and Evaluation Methodology (guided by M. David Miller)

http://eds.b.ebscohost.com/eds/detail/detail?vid=1&sid=1563ae9d-bb48-458b-ae5d-a59011c14e08%40pdc-v-sessmgr04&bdata=JkF1dGhUeXBlPWlwLHVpZCZzaXRlPWVkcy1saXZl#AN=ufl.033650760&db=cat04364a

S u r v e y

Psychometrics of the IntCRIT and IntCOMM

Scales

Item DiscriminationInternational Critical Thinking (reliability=.90)

1. I consider different perspectives before making conclusions about them. .696

2. I am able to manage when faced with multiple cultural perspectives. .687

3. I am open to different cultural ways of thinking in any international context. .681

4. I can make effective decisions when placed in different cultural situations. .666

5. Knowing about other cultural norms and beliefs is important to me. .661

6. I am able to think critically to interpret global and intercultural issues. .650

7. I actively learn about different cultural norms. .649

8. Understanding different points of view is a priority to me. .631

9. I can recognize how different cultures solve problems. .630

10. I can contrast important aspects of different cultures with my own. .623

11. Knowing about other cultural beliefs is important. .621

12. I am able to recognize how members of other cultures make decisions. .612

International Communication (reliability=.90)

1. I demonstrate flexibility when interacting with members of another culture. .692

2. I am confident that I can adapt to different cultural environments. .643

3. I am able to communicate effectively with members of other cultures. .632

4. I like working in groups with students from other countries. .620

5. I feel comfortable in conversations that may involve cultural differences. .616

6. When working on a group project, I enjoy collaborating with students from .610

7. I often ask questions about culture to members of other cultures. .602

8. I enjoy learning about other cultures. .588

9. I appreciate members of others cultures teaching me about their culture. .565

10. I am able to interact effectively with members of other cultures. .554

11. I appreciate differences between cultures. .542

12. I feel comfortable discussing international issues. .541

13. I can clearly articulate my point of view to members of other cultures. .538

S u r v e y

Administration: IntCRIT and IntCOMM Attitudes and Beliefs Survey

Sample Sizes

Year Random sample size

Received responses

Response Rate

2014-15 500 129 26%2015-16 12,000 1,084 9%2016-17 12,000 1,728 14%2017-18 16,000 2,202 14%2018-19 20,000 1,745 9%

Total 60,500 6,888 11%

Survey Administration 2014-2019Academic

YearEntry Year

2014 2015 2016 2017 20182014-15 14.12015-16 14.2 15.12016-17 14.3 15.2 16.12017-18 14.4 15.3 16.2 17.12018-19 15.4 16.3 17.2 18.1

F I N D I N G S

S U R V E Y R E S U L T S

Results by Component

3.70

3.80

3.90

4.00

4.10

4.20

4.30

4.40

4.50

Judgment Analysis Reasoning SolutionFinding

Sensitivity Production Awareness Adaptability Acceptance

4.37

4.13

4.004.03

4.38

4.12

4.27

4.17

4.25

Critical Thinking Communication

UF IntCRIT & IntCOMM Attitudes and Beliefs Survey


Results by Item


3.40

3.60

3.80

4.00

4.20

4.40

4.60

3 5 8 10 11 1 6 12 4 9 2 7

4.374.41

4.314.24

4.49

4.38

4.10

3.90

4.04

3.95

4.30

3.77

Critical Thinking

Judgement ReasoningAnalysis Solution Finding


Critical Thinking Lowest Scored Items


7. I actively learn about different cultural norms.

12. I am able to recognize how members of other cultures make decisions

11. Knowing about other cultural beliefs is important

Any thoughts?

9. I can recognize how different cultures solve problems

HIG

HEST

LOW

EST

5. Knowing about other cultural norms and beliefs is important to me


Results by Component: Communication Lowest & Highest Scored Items


Can you predict 3 highest & 3 lowest-ranked items?

21. I appreciate members of other cultures teaching me about their culture.

18. When working on a group project, I enjoy collaborating with students from other countries.

19. I often ask questions about culture to members of other cultures

20. I enjoy learning about other cultures

25. I can clearly articulate my point of view to members of other cultures

23. I appreciate differences between cultures


Results by Item


3.70

3.80

3.90

4.00

4.10

4.20

4.30

4.40

4.50

4.60

17 21 14 15 22 25 19 20 13 24 16 18 23

4.20

4.56

4.15

4.08

4.25

4.004.04

4.50

4.33

4.01

4.20

4.13

4.43

Communication


(24. I feel comfortable discussing international issues)


(15. I am able to communicate effectively with members of other cultures)


21. I appreciate members of other cultures teaching me about their culture



Any thoughts?HI

GHE

STLO

WES

T


Results by Gender & Race/Ethnicity





21. I appreciate members of other cultures teaching me about their culture



Who?

G E N D E R


Gender Comparison

To capture the gender effect, analysis of variance (ANOVA) was conducted on 25 items at the alpha level of 0.05

• Found significant gender differences (male/female) in 19 out of 25 items


• On significant items, women scored higher on all items but three.Who scored

higher?

Differences?

• Women scored higher on all items in the Judgment (Critical Thinking) and in Sensitivity, Awareness and Acceptance(Communication) .

Categories? • Male/Female/ Non-binary


Gender Comparison



Gender Comparison


Gender Differences by Item

5. Knowing about other cultural norms and beliefs is important to me.

16. I like working in groups with students from other countries


Any thoughts?

Some of the items with the widest differences are:


Gender Comparison


Gender Differences by Item

Any thoughts?

Two of the three items in which men scored higher than women are:

25. I can clearly articulate my point of view to members of other cultures.

24. I feel comfortable discussing international issues

R A C E & E T H N I C I T Y


Race/Ethnicity Comparison

An ANOVA was used again on 25 items at alpha level of 0.05 to see if there is any race differences. Then, since there are more than two categories within ethnicity/race, multiple comparison procedures were performed to find pairwise differences

• Significant race/ethnicity differences in all items


Differences?

Categories?


Race & Ethnicity Comparison


3.5

3.7

3.9

4.1

4.3

4.5

4.7

3 5 8 10 11 1 6 12 4 9 2 7

Ethnicity - Critical Thinking

Asian

Black

Hispanic

White

Hispanic-White

Asian-White

Critical Thinking

Judgement ReasoningAnalysis Solution Finding




3.5

3.7

3.9

4.1

4.3

4.5

4.7

17 21 14 15 22 25 19 20 13 24 16 18 23

Ethnicity - Communication

Asian

Black

Hispanic

White

Hispanic-White

Asian-White

Communication

Sensitivity AdaptabilityAwareness AcceptanceProduction




Who scored higher? • Hispanic students performed significantly better than White students in all items but one.

• Hispanic students performed significantly better than Asian students in most Communication components.

• Hispanic-White students performed significantly better than White students in all items but two.




7. I actively learn about different cultural norms.Hispanic

Hispanic-WhiteBlack

Asian WhiteAsian

White

Statistical significant differences:

(Lowest-ranked item in survey overall: 3.77 in 1-5 scale)

Race/Ethnicity Mean Difference P-valueHispanic/White 0.22 < 0.001Black/White 0.18 0.040Hispanic-White/White 0.17 0.009Asian/White 0.16 0.001




18. When working on a group project, I enjoy collaborating with students from other countries. Hispanic

Hispanic-White

Black

Asian White

Asian

White Statistical significant differences:Race/Ethnicity Mean Difference P-valueHispanic/Asian-White 0.31 0.004Hispanic/White 0.28 < 0.001Hispanic/Asian 0.21 < 0.001Hispanic/ Black 0.22 < 0.001Hispanic-White/White 0.16 < 0.001




19. I often ask questions about culture to members of other cultures. Hispanic

Hispanic-White

WhiteAsian White

Black

Asian

(This is the most varied question with 8 significant differences).

Statistical significant differences:Race/Ethnicity Mean Difference P-valueHispanic/Asian 0.16 0.001Hispanic-White/Asian 0.17 0.014Asian/White 0.11 0.050Black/Asian-White 0.33 0.022Hispanic/Asian-White 0.38 0.002Hispanic-White/Asian-White 0.39 0.001Black/White 0.22 < 0.001Hispanic/White 0.27 < 0.001White/Hispanic-White 0.28 < 0.001


Impact of Study Abroad


3.6

3.8

4

4.2

4.4

4.6

4.8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Cohorts without SA SA 18 post SA 18 Post

Study Abroad Pre- Post and Campus Cohort comparison

0

20

40

60

80

100

120

4Y Graduation 6Y Graduation

Graduation Rates FTIC 2012-2013 Cohort

Study Abroad Non Study Abroad


Leveraging the Data Any thoughts?

Who might be interested in knowing about your data besides SACSCOC?

Discussion:How can you use QEP assessment data to

ensure continuation of initiatives?

Institutionalizing the QEP

• Expand partnerships to ensure effectiveness

• Create and maintain momentum

• Use data to modify QEP during implementation

• Develop institutionalization plan building on success and targeting gaps on student learning

• Visualize student learning

Mutually reinforcing initiativesCurriculum

Internationalization

International Scholars Program

Assessment

Professional Development

OFFICE OF GLOBAL LEARNINGCONNECT ENGAGE INSPIRE COLLABORATE


Leveraging the Data

Developing programming to address gaps in student learning

S U R V E Y

Survey

International Critical Thinking (IntCRIT) and International Communication (IntCOMM) Attitudes and Beliefs Surveyhttps://internationalcenter.ufl.edu/about-ufic/assessment/intcrit-and-intcomm-surveys

Available free of chargeCan be downloaded from our website:• Question Items• Item descriptors

Can be used as:• Campus climate survey• Study Abroad Pre and Post• Pre-Post for individual courses

https://internationalcenter.ufl.edu/about-ufic/assessment/intcrit-and-intcomm-surveys

T H A N K Y O U !

Paloma RodriguezDirector Office of Global [email protected]

Tim BrophyDirector of Institutional [email protected]

life after the qep: leveraging assessment data for ......life after the qep: leveraging assessment...

Documents