reaffirmation 2014off-site review is completedon-site review is february 2014qep is ready to be...
TRANSCRIPT
![Page 1: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/1.jpg)
Developing a QEP Assessment Plan: Criteria and Process of Instrument Review and Development
M. David Miller, Director, QEPTimothy S. Brophy, Director, Institutional AssessmentUniversity of Florida
![Page 2: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/2.jpg)
Reaffirmation 2014
Off-site review is completed
On-site review is February 2014
QEP is ready to be submitted in January 2014
University of Florida Accreditation Process
![Page 3: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/3.jpg)
The institution has developed an acceptable Quality Enhancement Plan that includes an institutional process for identifying key issues emerging from institutional assessment and focuses on learning outcomes and/or the environment supporting student learning and accomplishing the mission of the institution. (Quality Enhancement Plan)
Core Requirement 2.12
![Page 4: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/4.jpg)
The Institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation and completion of the QEP; (2) includes broad-based involvement of institutional constituencies in the development and proposed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement.
Standard 3.3.2
![Page 5: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/5.jpg)
Purpose
Uses and interpretations
Studen
t
Learnin
g
(Outco
mes)
Begin with the end in mind
QEP Assessment
![Page 6: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/6.jpg)
Step 1. Establish topic and student learning outcomes (SLOs)
Step 2. Identify direct and indirect assessments to measure SLOs (and outputs to measure initiatives)
Step 3. If assessments are not available and sufficient, develop your own assessments
Step 4. Plan Assessment Schedule and follow it
Procedure for Planning Assessment
![Page 7: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/7.jpg)
Direct – Measures of Student Learning
Indirect – Measures of Attitudes, Behaviors, Beliefs, …
Outputs – Counts of the numbers of students participating in events, number of events, number of classes, etc.
Types of Assessments
![Page 8: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/8.jpg)
Direct and indirect assessments must measure progress on the SLOs
Outputs must measure participation in the initiatives
Each measure must have a planned use or interpretation (Validity)
Each measure must be reliably measured (with minimum error)
Our Guiding Principles
![Page 9: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/9.jpg)
It is easier to adopt an existing instrument than to develop a new instrument
Easier is not better if you violate your principles!
That is, existing instruments must: measure the SLOs; be valid for its planned uses and
interpretations; and be reliable for your purposes
Our Guiding Principles, Part II
![Page 10: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/10.jpg)
UF Theme : Internationalization
Internationalization is the conscious integration of global awareness and intercultural competence into student learning.
Step 1: Define Theme and SLOs
![Page 11: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/11.jpg)
SLO1 (Content):Students identify, describe, and explain global and intercultural conditions and interdependencies.
SLO2 (Critical Thinking): Students analyze and interpret global and intercultural issues.
SLO3 (Communication): Students communicate effectively with members of other cultures.
Step 1: Define Theme and SLOs
![Page 12: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/12.jpg)
Internationalization Measures Identified – Commercially Available
◦ Global Perspectives Inventory (GPI)◦ Global Competence Aptitude Assessment
(GCAA)◦ Intercultural Development Inventory (IDI)◦ Global Competencies Inventory◦ Cross Cultural Adaptability Inventory◦ Global Awareness Profile◦ Intercultural Effectiveness Scale
Step 2: Identify Existing Instruments
![Page 13: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/13.jpg)
International Task Force◦ Representation from all 16 Colleges◦ Student Representation◦ Representation from Faculty, Staff, Students,
Administration
Assessment Committee (5-6 with expertise on content and assessment)
Step 2a: Identify Experts for Review
![Page 14: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/14.jpg)
Measures the SLOs
Validity Evidence
Reliability Evidence
Feasibility to Use in Our Context
Step 2b: Defining Review Criteria
![Page 15: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/15.jpg)
Most commercial products had good reliability and validity evidence for their stated purposes
None matched our SLOs
Little evidence of feasibility for large scale use
Final Decision: Need to develop our own assessments
Step 2c: Factors in our decision to create our own assessments
![Page 16: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/16.jpg)
Consider Options Already Being Used as Part of Reporting System
Student Experience in the Research University (SERU) - biennial survey used by multiple universities that includes items on internationalization
Existing Surveys on Campus
![Page 17: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/17.jpg)
Items measure behaviors (e.g., courses taken, participation in Study Abroad, and other types of international experiences) and attitudes toward other cultures
We elected to add 10 items designed to
measure attitudes related to SLOs 2 and 3
SERU
![Page 18: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/18.jpg)
Direct Assessments focusing on SLO achievement
Must allow flexibility to measure learning
in any discipline
Needs to be aligned with Curriculum
Indirect Assessments of SLOs 2 and 3 only
Not appropriate for SLO 1 since Content is discipline-based
What Assessments do we need to Develop?
![Page 19: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/19.jpg)
Writing the Assessment Items
Ongoing review for fidelity with the SLOs (Validity)
Piloting to establish reliability and psychometric properties of items
Developing the Assessments
![Page 20: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/20.jpg)
• International Critical Thinking (IntCRIT) Attitudes and Beliefs
• International Communication (IntCOMM) Attitudes and Beliefs
• International Content not feasible with Disciplinary Differences
Two Assessments Developed
Step 3a: Indirect Assessments
![Page 21: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/21.jpg)
1.Development of item specifications based on the two SLOs and a literature review.
2.Writing items based on the item specifications.
Approximately 70 items were written for each SLO
Indirect Assessment Development Steps
![Page 22: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/22.jpg)
3.Review of the items by the ITF, the Assessment Committee and other
experts in assessment. (Validity and Match to SLOs)
4. Revision of items based on feedback from expert review.
Revisions were minor changes in wording.
Indirect Assessment Development Steps
![Page 23: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/23.jpg)
5. Pilot testing with undergraduate students at UF and eliminating items with poor discriminations.
◦ Initial piloting was completed with four forms to minimize the testing burden for students.
◦ Forms overlapped with ten items that expert agreed helped to define the construct.
◦ Each form was pilot tested with 70-100 undergraduates.
6. Item analysis of pilot data. ◦ The scale reliabilities exceeded .95 for all four forms.◦ Items were retained that had an item discrimination
of .25 or higher.
Indirect Assessment Development Steps
![Page 24: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/24.jpg)
7. Pilot testing (N=70-80) the retained items on a single form for each SLO.
8. Item analysis of pilot data. ◦ Recommended retaining items with the highest item discriminations
that would result in a scale with a reliability of at least .90. For IntCRIT, the recommendation was to retain 12 items. For IntCOMM, the recommendation was to retain 14 items.
9. Review of the items by the ITF, the Assessment Committee and other experts in assessment. (Validity and Match to SLOs)
Indirect Assessment Development Steps
![Page 25: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/25.jpg)
1. I consider different perspectives before making conclusions about the world.
2. I am able to manage when faced with multiple cultural perspectives.3. I am open to different cultural ways of thinking in any international
context.4. I can make effective decisions when placed in different cultural
situations5. Knowing about other cultural norms and beliefs is important to me.6. I am able to think critically to interpret global and intercultural
issues.7. I actively learn about different cultural norms.8. Understanding different points of view is a priority to me.9. I can recognize how different cultures solve problems.10. I can contrast important aspects of different cultures with my own.11. Knowing about other cultural beliefs is important.12. I am able to recognize how members of other cultures make
decisions
IntCRIT
![Page 26: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/26.jpg)
1. I demonstrate flexibility when interacting with members of another culture.
2. I prefer to socialize with people of my culture.3. I am confident that I can adapt to different cultural environments4. I am able to communicate effectively with members of other cultures5. I like working in groups with students from other countries.6. I feel comfortable in conversations that may involve cultural
differences.7. When working on a group project, I enjoy collaborating with students
from other countries.8. I often ask questions about culture to members of other cultures.9. I enjoy learning about other cultures10. I appreciate members of others cultures teaching me about their
culture.11. I am able to interact effectively with members of other cultures.12. I appreciate differences between cultures13. I feel comfortable discussing international issues.14. I can clearly articulate my point of view to members of other cultures
IntCOMM
![Page 27: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/27.jpg)
Use with curriculum
Must be flexible to allow faculty to define internationalization in discipline
Must provide a standard measure
Step 3b: Direct Assessments
![Page 28: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/28.jpg)
Define Rubric
Instructors develop or identify assessments in courses
Instructors will provide evidence of use and scores to University with examples of assessments
Adopt General Education model on campus
![Page 29: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/29.jpg)
The Association of American Colleges and Universities developed 15 Rubrics that can be used across programs and courses
VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics were developed by faculty and assessment expert teams across the country -- used by more than 2000 institutions (http://www.aacu.org/value/index.cfm)
Developing Rubrics
![Page 30: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/30.jpg)
UF QEP SLO VALUE Rubric Adaptation
SLO 1: Content
Intercultural Knowledge and Competence
Limit criteria to knowledge, dropping skills and attitudes. Modify descriptions for consistency across levels and ease of use.
SLO 2: Critical Thinking
Critical Thinking
Add language to reflect emphasis on international context for critical thinking. Modify descriptions for consistency across levels and ease of use.
SLO 3: Communication
Written Communication
Oral Communication
Combine rubrics to measure communication in multiple modes, andadd language to reflect emphasis on international context. Modify descriptions for consistency across levels and ease of use.
Adapting VALUE Rubrics for SLOs
![Page 31: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/31.jpg)
QEP Content Rubric
SLOComponents
Outstanding3
Satisfactory2
Unsatisfactory1
Not Applicable0
Concepts/Principles
Consistently and effectively demonstrates sophisticated understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, and beliefs and practices.
Usually demonstrates understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.
Rarely or never understands the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.
Not Applicable to Assignment or Course
Terminology
Consistently recognizes and effectively utilizes important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.
Usually identifies and implements important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.
Rarely or never understands important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.
Not Applicable to Assignment or Course
Methodologies
Consistently comprehends and effectively utilizes diverse and appropriate methodologies for understanding complex intercultural and global issues.
Usually comprehends and utilizes diverse and appropriate methodologies for understanding intercultural and global issues.
Rarely or never comprehends and utilize diverse and appropriate methodologies for understanding intercultural and global issues.
Not Applicable to Assignment or Course
![Page 32: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/32.jpg)
Review by Internationalization Task Force, Assessment Committee, and Experts for Match to SLOs and Validity
Piloting (in progress) in curriculum and Study Abroad
Additional Steps
![Page 33: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/33.jpg)
Number of participants at specific campus events with an international focus (QEP events).
Number of International Scholar courses. Number of students enrolled in International
Scholar courses. Number of Study Abroad courses offered. Number of students studying abroad.
Outputs
![Page 34: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/34.jpg)
University wide impact
Indirect Assessments
SERU Biennial, odd years
IntCrit and IntComm – Annual – Fall -
Sampling
Outputs
Monitor participation and implementation -
Annual
Impact on those taking courses or
Study Abroad
Direct Assessments - Rubrics
Indirect Assessments –
IntCrit and IntComm
Output – Number of students and
options
Assessment Plan
![Page 35: Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014](https://reader035.vdocument.in/reader035/viewer/2022062802/56649eb75503460f94bc0c5b/html5/thumbnails/35.jpg)
Annual analysis, review, and
interpretation of data to inform:
The implementation of the program
The degree of SLO achievement
When adjustments to program are
needed
The benefit of the program for
underrepresented populations
Closing the Loop