assessment on the ground

STUDENT AFFAIRS RESEARCH AND ASSESSMENT

ASSESSMENT ON THE GROUNDWhat you need to know when conducting assessment or working with an assessment professional so that the results help you do your best work!


SARAMission

Student Affairs Research and Assessment (SARA) provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes.

http://studentaffairs.psu.edu/hr/pdf/intorgchart.pdf

http://studentaffairs.psu.edu/hr/pdf/intorgchart.pdf


Why is assessment important?It will help you do a better jobIt will help ensure that you have

the resources to do that jobIt can help you prioritize your

effortsIt can contribute to the broader

conversation and understanding


What do you need to know?Not everyone has to be an assessment

expertBut you do need to know how to make

informed decisions ◦ To conduct an assessment◦ To use the findings effectively

Plan, Do, Check, Act (PDCA)


You are the content expertThings to consider:

◦ The parameters of the activity you are assessing◦ The conceptual model that underlies the activity ◦ The desired objectives/outcomes (learning or other)◦ Past assessment efforts and findings◦ How this activity/assessment relates to the strategic

goals of your unit and student affairs◦ What resources are available ◦ How you will use the results◦ Where/how will you disseminate the results


Basic Approaches (Schuh & Upcraft, 2001)1. 2. 3. 4. 5. 6. 7. 8. 9.


Key steps - PDCA Identify objectives Identify appropriate measuresChoose an appropriate assessment methodChoose an appropriate project designCollect the dataAnalyze the dataDisseminate the findingsTake actionStart all over again!


Identify & articulate objectivesYou should always be

able to tie what you are doing (programming, advising, etc.) to a specificobjective

Bloom’s taxonomy or similar learning constructs may be useful if you’re seeking to measure learning

The focus of a learning objective is not on what you will teach or provide, but on what students will learn or gain


One helpful approach to writing learning outcomes is ABC(D):

Audience: Who will accomplish the objective (e.g., students)

Behavior: What is expected; most helpful when overt and observable

Condition: Under what circumstances the behavior is expected

(Degree: The acceptable performance level - often unstated)


Avoid the “weasel words”• Learn• Know • Be aware of • Be familiar with• Have a firm grasp of• Understand• Appreciate


Example using Bloom Remember – After attending the Women's’ Resource Center open

house, students will be able to describe the resources available at the Center.

Understand – After completing their first-year seminar, participants will be able to interpret university policies related to academic integrity.

Apply – By the completion of the Study Skills series, participants will establish a realistic time management plan.

Analyze – By the end of their first elected term, student government participants can find and provide credible evidence to support a planned initiative.

Evaluate – After completing Leadership 101, class members will be able to debate the merits of various leadership styles and their application in different situations.

Create – By the end of their first elected term, student leaders will be able to develop and present detailed plans for proposed projects.

Audience BehaviorCondition


Determine appropriate measuresAttitudes?Experiences?Knowledge?Abilities?Persistence?Graduation?Employment?


More on measuresNorm-referenced = report

student performance relative to other students

Criterion-referenced = compares to an absolute standard of achievement (e.g., you pass or you fail)

Self-referenced = compares different scores/ratings from the same student


What kind of data?Quantitative

◦Can be economical, generalizable, reliable, valid

Qualitative◦Can provide rich data – can ask

about relationships rather than infer Mixed methods


Consider various types of data collection approachesTest of abilities or knowledgeSurveyInterviewFocus groupOther types


Tests◦Can be used to measure a specific skill or

knowledge base◦“Objective”◦Feasible?

Surveys◦Attitudes◦Experiences◦Self-reported abilities & knowledge◦Can be confidential/anonymous ◦Can reach large numbers of people


Interviews◦Rich & detailed information◦Body language can be helpful◦Respondents may feel inhibited◦Repeated interviews can build trust/comfort◦Time-consuming

Focus groups◦Many of the same benefits as interviews, but

not as in-depth◦Participants “feed off of” each other◦Can reach more people than interviews


Other ways to collect dataStudent self-assessments PortfoliosReflective journalsObservation-based

What others can you think of?


Project designGo back to your assessment question(s)

◦ What do you want to know? ◦ What are the limitations (e.g., time, resources, staff)?◦ How will the assessment be used?

The answer to these questions help you figure out:◦ What your purpose/question is◦ What approach to use◦ What type of information to collect◦ How to analyze your data


Data collection


Data analysisMake sure that your analysis is appropriate

to your question. ◦ Does your data meet the basic assumptions of

the statistical test you are using? ◦ Is your analyst free coding your transcripts when

your goal was to analyze your data using a pre-assigned coding system based on the theoretical framework underlying your program?

Make sure that you understand the assumptions and limitations of any analyses you rely on or report, whether you are the one who conducted them or not


Reporting & DisseminationThink about your audienceAdministrators don’t have time to

read a dissertation or even a journal-length article

Executive summary of the executive summary!

Put key points into easily digested tables, charts, graphs, and bullets

Don’t get carried away!


Dissemination continuedDO NOT FORGET to spell out the

implications of your work◦How do the findings improve our

understanding?◦What should we change?◦What else do we need to know?

Make sure the information gets to key decision makers – understanding the structure and politics of your institution is very important (Terenzini’s “contextual intelligence”)


You’re not done yet!Time for action.

What are you going to do with what you learned?

Plan, Do, Check, Act, Repeat


Theory to practiceIn practice, you will not always

have the resources to do assessment the way it is outlined in books and papers

What’s important? ◦Understand the impact of the

compromises you will make◦When you act, you are doing so

based on the best available information


QUESTIONS?


Your turnGet out a piece of paperChoose a student affairs area to

representWrite at least one learning

outcome for your unitHow would you assess it?


Think aboutType of approach (needs assessment, learning

outcomes, satisfaction, etc.)Type of measure(s) (attitude, knowledge, etc.)Type of data (quant, qual, mixed)How will you collect it? (focus group, survey,

etc.)What is your design? (pre-/post-test,

longitudinal, etc.)How will you analyze it? (thematic analysis,

mean comparisons, etc.)


Betty [email protected]

assessment on the ground

Documents