office of assessment july 11, 2012 non-academic assessment: developing an assessment plan
TRANSCRIPT
Goals of Workshop
Current assessment contextWhat’s new this yearIntroduce new Higher Learning Commission
CriteriaReview key items on the Assessment Plan
templateWalk through new online reporting form
2
Assessment Context
Assessment Context Last year participating in the Higher Learning
Commission’s (HLC) Academy for the Assessment of Student Learning
New HLC Criteria effective January 1, 2013 Campus focus on 2020 Strategic Plan alignment
Assessment Reporting Themes Focus on use of data Reflect on changes and impact on learning Streamline reporting process
Future Goals Longer reporting cycle Integration with annual reporting
4
What’s New This Year
Online reporting Fewer questions Questions “tagged” with New HLC Criteria Optional question on New HLC Criteria Report quality threshold to 70%
5
HLC New Criteria
Guiding Values, includes:1. Focus on student learning4. Culture of continuous improvement5. Evidence-based institutional learning and self-
presentation9. Mission-centered evaluation
Anatomy of Criteria Criterion—broad statement
Core Component—areas of particular focus Sub-Component—further elaboration on Core
Component, but not exhaustive (and not always included)
Some criteria now specifically mention certain units (e.g., advising)
6
HLC Criteria7
HLC Criterion One. Mission
The institution’s mission is clear and articulated publicly; it guides the institution’s operations. Core Component 1.A.
The institution’s mission is broadly understood within the institution and guides its operations. Subcomponent 2.
The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.
Criterion Title and
broad statement Core
Component statement—particular aspect of
criterion on which to
focusSub-component—specific area from
which evidence is expected
HLC New Criteria
The Five Criteria1. Mission2. Integrity: Ethical and Responsible Conduct3. Teaching and Learning: Quality, Resources, and Support4. Teaching and Learning: Evaluation and Improvement5. Resources, Planning, and Institutional Effectiveness
Assumed Practices Won’t check, but we are responsible to ensure/remediate Notable is B.2.c.1): Faculty participate substantially in:
analysis of data and appropriate action on assessment of student learning and program completion.
8
Upcoming Assessment Workshops
Developing an Assessment PlanThursday, July 19th, 10-11 am
Completing the Assessment Report Thursday, July 12th, 11-12 amTuesday, July 17th, 1-2 pm
Note: workshops/forms overlap
9
AN OPPORTUNITY TO STEP BACK AND PLAN EFFECTIVE ASSESSMENT AND ANALYSIS
Creating an Assessment Plan
10
Section 1: Current Unit Status
1. Explain how your unit supports 2020 Strategic Plan of YSU.
Removed mission; strategic plan operationalizes mission
11
HLC Criterion One. Mission The institution’s mission is clear and articulated publicly; it guides the institution’s operations.
Core Component 1.A. The institution’s mission is broadly understood within the institution and guides its operations.
Sub-component 2. The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.
HLC Criterion Three. Teaching and Learning: Quality, Resources, and Support The institution provides high quality education, wherever and however its offerings are delivered.
Core Component 3.E.The institution fulfills the claims it makes for an enriched educational environment.
Sub-component 1. Co-curricular programs are suited to the institution’s mission and contribute to the educational experience of its students.
YSU’s 2020 Strategic PlanYSU’s 2020 Strategic Plan
Office of Assessment (OOA)Office of Assessment (OOA)
The new University-wideStrategic Plan is organizedaround four cornerstones Student success The urban research
university mission/transition
Regional impact Institutional accountability
and sustainability
Two goals of the OOA (to increase campus engagement in assessment and to successfully demonstrate progress in the HLC Academy for the Assessment of Student Learning) are directly tied to 2 cornerstones of the 2020 Plan
Accountability and Sustainability which emphasizes shared responsibility.
Student Success since it documents student achievement in curricular and co-curricular areas.
The new Engagement in Assessment Portfolio will provide documentation.
Example: Strategic Plan Link12
YSU’s 2020 Strategic PlanYSU’s 2020 Strategic Plan
Office of Community InvolvementOffice of Community Involvement
Theme 2.2. Satisfaction with non-
academic experiences and campus life-including the importance of participation in service learning and community activities
The main goal of this unit is to increase participation in community activities as well as meaningful reflection on the impact of those activities on students’ development
Example: Strategic Plan Link13
Section 1: Current Unit Status
2. Describe the goals for your unit from the previous academic/fiscal year
Example (from Office of Campus Planning and Community Partnerships):
to annually update the Campus Master Plan and undertake such studies as may be required to implement specific recommendations in accordance with the University’s annual capital improvement program;
to complete the Campus Roadway Study initiated by the City of Youngstown and focused upon the major arterial roadways serving the campus;
to implement improved bus service to the campus in collaboration with the WRTA and the University’s Alternative Transportation Community; and
to collaborate with Student Government to develop a matrix of student volunteer requirements and opportunities.
14
Section 1: Current Unit Status
3. How do your goals support the learning environment at YSU?
Think big picture-not just the students, but also faculty, staff and the university as a system
Every unit contributes to the learning environment!
15
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.
Example
Support Learning Environment Example (from the Reading and Study Skills Center):
The Reading and Study Skills Center primarily serves first-year students….this Center is the “first stop” in building a foundation for these students’ success and eventual retention for YSU. By building said skills, instructors and faculty should be experiencing students in their classrooms who are capable, responsible students, who read and think more critically than students who may not have been exposed to the concepts and skills taught in one of the RSS courses.
16
Section 1: Current Unit Status
Describe your current assessment activities. 4. Provide a brief summary of unit and/or
program improvement data collected that document your unit’s assessment activities for the 2011-12 academic/fiscal year
Explain how you measured unit and/or program activities for continuous improvement
Does not need to be related to learning outcome, only focused on continuous reflection and improvement
Not data, but methods of collection (e.g., survey, performance evaluation)
17
Section 1: Current Unit Status
Data collection summary:
18
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.
Sub-component 2. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals. The institution assesses achievement of the learning.
Example
Current Assessment Activities Example (excerpt from Institutional Research and Policy Analysis):
Collected participation rate data on the CLA, NSSE, and the Graduate Plans Survey
Archived data requests from department and unit faculty and staff
19
Section 1: Current Unit Status
5. How was that data analyzed and used by your unit? Does not have to be a formalized approach Ex: a discussion of survey data during a staff meeting
20
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.
Sub-component 3. The institution uses the information gained from assessment to improve student learning.
Example
Data Analysis and Use Example (from Academic Advisors):
Survey responses were analyzed and reviewed. The data led to: Majors Exploration Fair was held to address concerns raised in
the survey indicating students were uncertain about their major.
A subcommittee formed to address needs of undetermined students
An advising syllabus was developed to address the students’ needs as a result of narrative comments from students on the survey
21
6. How did your unit utilize feedback from the OOA regarding last year’s assessment report?
o If this is your first time completing a plan or report, answer “first reporting year”
o This is an important part of completing a full assessment cycle/spiral/loop
Section 1: Current Unit Status22
Assessment process image credit: http://pdx.edu/studentaffairs/assessment-planning-and-practice.
Example
OOA Feedback Example (from Academic Advisors):
Based on feedback from OOA to develop a LO that could lend itself to direct method of measure, added a LO that included: Students will be advised in a timely manner Student will correctly identify their advisor (is being
considered for the future)
23
Section 2: Assessment Plan Development
7. State at least one learning outcome that is relevant
for your unit Keep to 1 (maybe 2 LO’s) Keep to 1 focused project (not a laundry list) What are you interested in?
24
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.
4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.
1. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals.
Section 2: Assessment Plan Development
Sources for LO: A particular project that your unit will be
focusing on in 2011-12 ORLO that staff from your unit develop ORYSU’s General Education learning outcomes ORThe Council for the Advancement of Standards
in Higher Education (CAS) General Standards OR
Learning Reconsidered
25
Section 2: Assessment Plan Development
Developing a learning outcome:Framing Question: What do you want
students/staff/faculty to know and be able to do as a result of interacting with your office or program you offer?
Should be SMART: Specific, Measurable, Attainable, Realistic, Timely
Two common methods to writing a learning outcome: ABCD Model SVP Model
26
Learning outcome material excerpted from GLACUHO Webinar on Writing Learning Outcomes by Jeff Long, May 2010
Section 2: Assessment Plan Development
ABCD Model of Learning OutcomesAudience-target of the outcomeBehavior-verb in future tenseCondition-activity causing learningDegree of Achievement (optional)
________ ________ ________ ________ Condition Audience Behavior
Degree
27
Section 2: Assessment Plan Development
ABCD Model Examples________ ________ ________ ________
Condition Audience Behavior (Degree-optional)
As a result of student staff training, student staff will be able to list (4 of the 5) department’s values.
Through publicity and training, staff will correctly complete online timesheets (90% of the time).
As a result of this workshop, attendees will understand the components of a well-written assessment plan.
28
Section 2: Assessment Plan Development
SVO Model of Learning OutcomesSubject = learnerVerb = cognitive processObject = content/knowledgeNote: removes condition (e.g. specific intervention);
somewhat more like academic learning outcomes.
Examples:Students will identify academic resources on
campus.Faculty will use DARS system in advising.
29
Section 2: Assessment Plan Development
Using Verbs in Learning OutcomesAvoid “know,” “understand,” “think,” etc. –
how will you measure?Consider the level of learning you expect
from your students/staff/faculty—do you want them to comprehend? Apply knowledge?
Verb list that considers the level of learning (aka Bloom’s Taxonomy)--very helpful!: http://www.clemson.edu/assessment/assessmentpractices/refe
rencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf
30
Example
Academic Advisors Learning Outcomes Example:
1.Students can independently seek out resources to aid in evaluating their progress toward degree completion
2.Students will be knowledgeable of the curriculum requirements for their chosen major
31
Section 2: Assessment Plan Development
8. How does this learning outcome fit your unit? Why did you pick this LO?
Example (for fictional Office of Community Involvement):
This is directly related to our unit’s purpose to
promote the benefit of community involvement
to our students
32
Section 2: Assessment Plan Development
9. How does this learning outcome fit in with the YSU
2020 Strategic Plan? Consider how the learning relates to the larger goals of the
university
33
HLC Criterion One. Mission The institution’s mission is clear and articulated publicly; it guides the institution’s operations.
Core Component1.A. The institution’s mission is broadly understood within the institution and guides its operations.
Sub-component 2. The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.
HLC Criterion Three. Teaching and Learning: Quality, Resources, and Support The institution provides high quality education, wherever and however its offerings are delivered.
Core Component 3.E.The institution fulfills the claims it makes for an enriched educational environment.
Sub-component 1. Co-curricular programs are suited to the institution’s mission and contribute to the educational experience of its students.
Section 2: Assessment Plan Development
Learning Outcome Relation to Unit Relation to YSU’s Strategic Plan
As a result of a unit publicity campaign, students will be more aware of the importance of community involvement
This is directly related to our unit’s purpose, to promote the benefit of community involvement to our students
Promoting the importance of community involvement helps to encourage participation, and the Regional Engagement Cornerstone outlines the importance of student involvement in the community to support quality learning experiences and positive community change (in particular Theme 1, Priority 1).
Staff demonstrate awareness of professional responsibilities of students serving in community organizations
By ensuring our staff are responsive to the needs of the community organization with which we partner…we are well positioned to expand our community service opportunities
By maintaining positive and responsive relationships with our community partners, we enable students to interact positively with community partners, and foster the collaborative relationships with community organizations outlined in the Regional Engagement Cornerstone (in particular Theme 1, Priority 1).
Example, Questions 7-9—how do they align?
34
Section 2: Assessment Plan Development
Measuring your learning outcome10. What is the way (or ways) you will measure
the status and/or progress in this learning outcome over the 2012-13 academic year?
Note: consider tools that would incorporate both direct and indirect measures
A minimum of one tool but we recommend no more than two measures
An “exemplary” answer would include a direct measure
35
Direct MeasuresDirect Measures Indirect MeasuresIndirect Measures
Direct evidence of learning outcome performance:
Factual knowledge questions embedded in a survey
Presentations Performances Simulations Internship supervisors’
evaluations
Self-reported achievement or proxy evidence:
Surveys, group discussions, focus groups, exit
Interviews Employment
placement rates
Examples: Direct vs. Indirect Measures
36
Methods: Both Direct and Indirect
Method Indirect Direct
Minute paper after a workshop on diversity
Perceptions of moral dilemmas regarding diversity
Factual question on the definition of diversity
Telephone calls to a Unit
Caller’s satisfaction with technology
A qualitative analyses of callers knowledge of technology based on their questions
Survey after a training Teacher and learner satisfaction
Factual question on knowledge of workshop content
37
Tools may be direct or indirect, depending upon how they are used and their purposes:
Section 2: Assessment Plan Development
11. What do you anticipate the outcome of your measures to be? Or, what do you hope to
find out?
This can be a check on whether you have chosen the right LO
Results do not need to be “positive” to be valuable
38
Example
Outcome Example (From Alumni and Events Management):
With our event changes we do receive positive and constructive feedback. That is what we are always looking to get from student and the public. No event is perfect. We are constantly trying to grow and develop our ceremonies and that is not possible without constructive feedback.
39
Section 2: Assessment Plan Development
Close the Feedback Loop12. What are the major constituency groups
for your unit? Note: think not just within your unit or your
supervisory structure (internal stakeholders), but about students/faculty/staff, the larger university, the surrounding community, and beyond (external stakeholders).
40
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.
Sub-component 4. The institution’s processes and methodologies to assess student learning reflect good practice, including the sub stantial participation of faculty and other instructional staff members.
Example
Constituency Groups Example : Internally: we share and seek
recommendations from all our staff as well as the Division VP
Externally: we share and seek recommendations students, campus units who do work on community involvement
University Honors and Scholars YSU’s Task Force on Civic Engagement WYSU Alumni Services
41
Section 2: Assessment Plan Development
13. How and when do you plan to involve your major constituency groups in the assessment process
e.g. data sharing and recommendation gathering)?
Mention both the venue and frequency of meeting share with groups outside your unit use active ways rather than passive ways to share findings seek recommendations from groups inside and outside the
unit
42
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.
Sub-component 3. The institution uses the information gained from assessment to improve student learning.
Example
Feedback Loop Example (from fictional Office of Community Engagement):
We will compile the results from our survey and share the information with the groups listed above through: monthly meetings with internal staff to update, present data, and solicit recommendations; soliciting a story about the results in the Jambar for students (spring 2013); and inviting other campus units to come in person for a presentation and discussion of the data (April 2013).
43
Section 3: Relation to HLC Criteria
14. (OPTIONAL FOR 2012) Please provide an example of evidence from your unit for the new HLC Criterion Five (Resources, Planning, and Institutional Effectiveness), Core Component 5A (The institution engages in systematic and integrated planning.), Sub-component 2 (The institution links its processes for assessment of student learning, evaluation of operations, planning, and budgeting.): Can help to begin familiarizing yourself with new HLC Criteria How will assessment activities you are planning impact your
planning and budget decisions? How does your work help contribute to YSU’s body of
evidence for this criterion?
44
Example
Aligning Assessment with Budget Example (from fictional Office of Community Engagement):
“While we didn’t think of it this way before reading this question, the 2011 survey results made it clear that more staff and financial resources needed to be devoted to a campus campaign. Through this assessment plan, we are devoting both fiscal and staff resources in order to carry out plans. Results from this year’s assessment will inform future planning and budgets.”
45
Section 4: Conclusion
Executive Summary15. Summarize the key ideas from Section Two
only Assessment Plan Development
one paragraph is fine a maximum length of one half page allows us to compile and develop themes from 40+
units
46
Section 4: Conclusion
16. (Optional) Is there anything else you would like to share regarding your assessment plan and/or is there any particular area on which you would like assistance or feedback?
This is a chance to share something that didn’t “fit” with our questions but is important to your assessment “story”
Or, it is something about which you could use our help or advice
47
Examples
We need help in having a forum to share our data from our prior year’s report. Can you help us set up a workshop or forum?
We are not certain how to incorporate a direct measure with this learning outcome-do you have any ideas
48
Example
We would love some examples of rubrics that have already been developed to measure community engagement. Do you have some instruments to share with us
We would like to use the institutional account for survey monkey. How do we go about learning more about using this survey?
49
New Online Reporting Format
Online Assessment Plan Submission Form: www.jotform.us/ysuassessment/2012naplan
50