1

Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question. Each polling question will have a new code.

Responding to Open-Ended Polls

Addressing the Needs of Assessment Professionals: Practical Recommendations for Higher Ed• Laura Ariovich• Conna Bral• Patricia Gregg• Matthew Gulliford• Jennifer Ann Morrow

3

Report Authors

Laura AriovichPrince George’s Community College

Conna BralMidwestern State University

Patricia GreggGeorgia State University

Matthew GullifordWatermark

Jennifer Ann MorrowUniversity of Tennessee

4

Project Purpose

and Resources

5

To analyze:

• practitioner and administrator

perceptions of assessment

• the roles/positions of the people in

higher education whose primary job

relates to learning outcomes assessment

• assessment activities on campus and the

level of assessment on campus, and

• professional development needs of

assessment practitioners in higher

education.

Purpose of Project

6

Watermark Survey (2016)

• practitioner and administrator perceptions of assessment

• purpose was to collect data to gain national insights into assessment and accreditation efforts in the

US.

Description of Datasets

Sample: 1074 respondents• 33% Faculty• 22% Department Chair/Coordinator• 21% Director• 9% Dean/Assistant Dean• 7% University Administrators (Pres, VP, Provost, etc.)• 6% Other Roles• 3% Analyst/Researcher

7

UKY Survey (2015)

• practitioner and administrator perceptions of assessment

• purpose was to examine institutional effectiveness professionals

assessment perceptions, activities, and job satisfaction.

Sample: 377 Assessment Professionals

○ 74% were highest level assessment person at their institution§ 52% Executive Director/Director§ 12% Asst/Assoc Vice President/Provost

Description of Datasets

8

Perceptions of

Assessment Professionals

9

Watermark(2016) participants were asked to describe how important assessment was

to their institution. Three primary themes emerged from the data:

Importance of Assessment

ImprovementAccountability

Implementation Concerns

10

Improvement

“How can we know what we do well and not so well without assessment? Assessment should guide our efforts to improve our outcomes.”

“There are so many legitimate but varying opinions on how to teach, what is going well and where improvement needs to occur. Assessments helps provide a common point of conversation.”

“Without sound assessment practices, we are only operating on "guess-work“ and convenience seekers who "like things the way they are.””

11

Accountability

“Assessment allows institutions to tell their story. It provides us a way to justify what we are doing is important to students, the community, and the state at-large. It allows us to show that we are making a difference.”

“We hold a duty of responsibility to those who entrust us with their future, as well as to companies who would employ them, to provide them with skills and abilities they can rely on throughout their life.”

12

Implementation Concerns

“Useful, well-designed assessments can guide an institution in worthwhile directions, but gathering numbers for the sake of just gathering numbers is a waste of valuable faculty time that ought to be spent teaching, working with students, or doing research.”

“It's needed, but it is not the purpose of higher education. That purpose is being eclipsed by "Assessment Madness.””

What do you like most about assessment?

Go to Menti.com and enter the code 556866.Type in up to three responses.

14

Watermark(2016) participants were asked what they liked the most about assessment.

Three primary themes emerged from the data:

What Assessment Professionals Like

Enjoyment of methodological aspectsEnjoyment of collaborative aspects

Appreciation for the impact/utility of assessment

15

Enjoyment of Methodological Aspects

“I love developing models to determine student engagement and achievement and then analyzing those models with data.”

“Finding a story within the numbers.”

“I like designing assessments because it is interesting to see if one can actually create something to measure something (like learning) that isn't readily observed.”

16

Enjoyment of Collaborative Aspects

“Providing faculty and administrators with support (e.g. data processing/analysis, assignment design suggestions, event coordination) and encouraging collaborative discussion to drive continuous improvement of process, instruction and student learning.”

“I love dialoguing with faculty about the assessment data they are collecting and how they can use the information they collect to continuously improve student learning.”

17

Impact/Utility of Assessment

“The opportunity to 'prove' that we are doing important work and contributing to student growth/abilities.”

“I enjoy the opportunity to see programs develop assessment plans that produce useful information, and to see that they are making changes with the information. This is rewarding because I know, ultimately, students are benefiting from these efforts.”

“It allows you to chart progress; understand strengths and weaknesses; provides guidance for decision making.”

18

What do you like least about assessment?

Go to Menti.com and click on “vote on another question.”

Enter the code 186360.

Type in up to three responses.

19

Watermark(2016) participants were asked what they disliked the most about assessment.

Three primary themes emerged from the data:

Issues with the methodological aspectsIssues with the need to persuade/teach others about assessment

Questions about the value of assessment itself

What Assessment Professionals Like Least

20

Issues with Methodology

“Sometimes assessment feels cumbersome, especially if we're starting from scratch or analyzing large data sets.”

“Dealing with ignorant and naive assessment boosters. I'm constantly amazed at how far folks with very little fundamental understanding of measurement, data analysis, organizations, and teaching get in this field.”

21

Need to Teach/Persuade Others

“Convincing other stakeholders involved in data gathering of the value of investing both time and money into the process.”

“My biggest frustrations are with assessment creators who don't understand how to build a good assessment, don't appreciate the information that can be derived from a well-constructed assessment and don't care to learn.”

22

Value of Assessment

“It is time-consuming and we don't have enough personnel to do it well all the time.”

“Assessment often turns into an additional layer of work that reveals only what one already knows about a program.”

“I least enjoy the pressure to quantify things that can't be quantified — the most important elements of any education — or the way assessment puts at the forefront things that can be measured as more important than things that cannot.”

23

Assessment Roles

and Activities

24

Assessment professionals who completed the University of Kentucky survey

(2015) were asked how much time they spent on assessment related work:

• 100% of their time (26%)

• 75-99% of their time (22%)

• 50-74% of their time (22%)

• 25-49% of their time (21%)

• <25% of their time (9%)

Assessment Professionals’ Activities

25

Assessment professionals who completed the University of Kentucky survey

(2015) were asked how much time they spent on assessment related work:

• 100% of their time (26%)

• 75-99% of their time (22%)

• 50-74% of their time (22%)

• 25-49% of their time (21%)

• <25% of their time (9%)

Assessment Professionals’ Activities

26

Assessment Professionals’ Activities

27

Assessment Professionals’ Roles

Assessment Professional

Assessment / Method Expert

Project Manager

Narrator/ Translator

Facilitator/ Guide <-> Political

Navigator

Change Agent

Jankowski & Slotnick (2015) Framework Watermark & AALHE Revised Framework

Project Manager

28

Assessment Professionals’ Roles

Assessment/Method Expert

• authentic assessments• curriculum mapping• data analysis• data cleaning• designing scoring protocols• determining the best indicators• qualitative analysis• quantitative analysis• reliability and validity issues• survey and assessment development• writing/developing rubrics

29

Assessment Professionals’ Roles

Narrator/Translator

• finding meaning within the data and communicating results• presenting data in visually-appealing or reader-friendly formats• providing a “big picture” perspective to institutional colleagues• support for evidence-based decision-making

30

Assessment Professionals’ Roles

Facilitator/Guide Political Navigator

• assisting faculty with improvement efforts• helping and mentoring others• interdisciplinary collaboration • managing accreditation• obtaining buy-in • overcoming resistance and obstruction

31

Assessment Professionals’ Roles

Change Agent

• active participation (not only provider but also end-user of data) • closing the loop • designing change• making a difference• redesign• reflection• responding to identified weaknesses• utilizing results

32

Assessment Professionals’ Roles

Project Manager

• overseeing the collecting, entry, and management of data• designing data gathering techniques• designing protocols• enforcing deadlines• finalizing analysis plans• follow-ups/reminders• managing scoring processes• organizing and archiving data• running reports• working with software

33

Professional

Development Needs

34

What areas/topics do you needprofessional development in?

Go to Menti.com and click on “vote on another question.” Enter the code 865232.

Type in up to three responses.

35

PD Method Preference (UKY Survey)

36

UKY (2015) survey were asked what professional development topics they would like to see offered:

staffing and resources

“managing a one-person office”

best practices in assessment

“research on what constitutes effective and meaningful practice in assessment.”

learning outcomes and co-curricular assessment

student affairs, general education, program evaluation

Professional Development Needs

37

UKY (2015) survey were asked what professional development topics they would like to see offered:

assessment tools

“adapting and writing rubrics”

collaboration

“more sharing of techniques and tips between institutions”

using data for change and/or strategic planning

“Now [that] we have the data...how do we enable change?”

Professional Development Needs

38

Respondents to the Watermark survey were asked what professional development topics they were interested in from a menu of nine topics:

• Respondents favored professional development topics that were practical in nature and/or those that would help them advance the use of assessment data in their institutions.

• Analyzing and interpreting data

• Documenting and reporting assessment results

Professional Development Needs

39

Conclusion and Recommendations

• Roles of assessment professionals in higher education vary widely and are evolving.

• What began as data collection for accreditation is evolving to change agent for organizational

improvement.

• Need flexible and effective approaches to professional development.

• Focus should be on equipping assessment professionals to improve teaching and learning.

40

Make sure to read the full report and a new infographic!

Download at watermarkinsights.com/aalhe-report