student affairs assessment committee training part 2
TRANSCRIPT
![Page 1: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/1.jpg)
Assessment Fellows Training
Brian Clark, Stan DuraAssessment Fellows Meeting
11/21/14
Brian Clark, Stan Dura 1/17/14
![Page 2: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/2.jpg)
Training Agenda
• Review of Foundational Concepts• Prediction, Control and Understanding– Increasingly complex models
• Univariate = 1 variable– Descriptive Statistics
• Bivariate = 2 variables– Correlations
• Multivariate = 3+ variables– Regresssion– Factor Analysis
• Discussion and Q & A
![Page 3: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/3.jpg)
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
![Page 4: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/4.jpg)
What “construct” are you measuring?
Survey and question design
and content
Responses
Is it really measuring it?Does it measure it the same each time?
Any errors in the questions or scales?Any bias in the questions asked or not asked?
Response set– -Are they representative?-How complete are they?-Is there non-response bias?
Over 80 cognitive and
memory biases
Surveys especially vulnerable
![Page 5: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/5.jpg)
Cognitive Stages Definition Vulnerabilities
Stage 1 Comprehension Understanding the question as author intended
Unknown or misused words, ambiguity, complexity, length
Stage 2 Retrieval Search of memory for relevant information
Memory bias, recall error, fatigue
Stage 3 JudgmentConsiders information
retrieved, makes “guestimations” and decides
Social or political bias, “fuzzy logic,” error in “guesstimation”,
personal sensitivity
Stage 4 Response Provides the information Human error, incomplete response, wrong format
Respondents Prone to Error
![Page 6: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/6.jpg)
Research, Assessment, & Program Evaluation
Research• Systematic collection of data• Generalize to larger populations• Based on Research question• Rigorous methodology• Used to test hypotheses• Contributes to the development
of theory and models
Assessment• Systematic collection of data• Generalize to larger population• Based on Assessment question• Borrows rigor when practical• Used to evaluate effectiveness• Contributes to judgments of the
quality of programs or activities
![Page 7: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/7.jpg)
Foundational concepts:• Entity – the things we examine and assess• Property – the characteristics that describe them• Variable – Properties that vary between people• Prediction and Control – Predict variable behavior• Relationship – Understand variable behavior• Statistical techniques – Analyze variable behavior
Foundational Concepts“Things and Stuff”
![Page 8: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/8.jpg)
Objects and properties– Humans
• Height, satisfaction, health, perceptions, emotions, beliefs, etc.
– Students (whether human or not )
• Age, GPA, knowledge, skills, engagement, class standing, etc.
– Physical Objects• Weight, chemical composition, density, etc.
– Forces• Magnitude, direction, etc.
Foundational Concepts“Things and Stuff”
![Page 9: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/9.jpg)
Some variables are straightforward-
–Height is often measured in Inches–Weight is often measured in Kilograms–Age is often measured in years
Foundational Concepts“Things and Stuff”
![Page 10: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/10.jpg)
Some variables are not-
– Satisfaction is often measured in… ???–Anger is often measured in… ???– Engagement is often measured in… ???
Foundational Concepts“Things and Stuff”
![Page 11: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/11.jpg)
Foundational ConceptsPrediction and Control
Performance
Haste Practice
![Page 12: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/12.jpg)
Foundational ConceptsPrediction and Control
Performance
Practice
If we did not Control for Haste, we would think all of this was
the result of Practice
![Page 13: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/13.jpg)
Foundational ConceptsPrediction and Control
Performance
Practice
When really, this was the impact.
Haste
![Page 14: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/14.jpg)
So…• Recognizing the relationships between
variables, and • controlling for them • is critical to understanding the impact
of any one of them.
Foundational ConceptsPrediction and Control
![Page 15: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/15.jpg)
Foundational ConceptsModeling
When we conduct quantitative or qualitative analyses, we are constructing models
• That model can be very, very simple• Mean – this is how the average person did…• Mode – The most frequent score was X
• Or complex… • For every hour engaged in co-curricular activities, when controlling for HS
GPA, Class standing, and Major, retention likelihood increases by .45%
• Descriptive stats = the weakest and least informative models
![Page 16: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/16.jpg)
Foundational ConceptsModeling
So when we assess, we need to Think of our Model
• What are the entities and properties we’re examining?• What are the variables involved? Operationalize them.• What is their relationship; how do they interact?• How simple or complex/accurate a model do I need?• How can we predict and control for these variables?
![Page 17: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/17.jpg)
One variable models = Univariate
This is when we look at 1 property of an entity.• Age of person• Height of a building• Responses to a single question
Prediction, Control and UnderstandingOne Variable Models
![Page 18: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/18.jpg)
One variable models = Univariate
These are generally very easy models to work with
Prediction, Control and UnderstandingOne Variable Models
![Page 19: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/19.jpg)
Mean
The average of all measures of a variable…
“The average score of all students was a 68”
Prediction, Control and UnderstandingOne Variable Models
What does this tell us of other scores?
![Page 20: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/20.jpg)
Mean
The average of all measures or scores
Prediction, Control and UnderstandingOne Variable Models
0 40 80 100
What does this tell us of other scores?
![Page 21: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/21.jpg)
Median
The measure in the exact middle of all scores
Prediction, Control and UnderstandingOne Variable Models
1, 3, 4, 4, 6, 6, 7, 7, 7, 7, 8
Mean = 5.5 Median = 6
![Page 22: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/22.jpg)
Mode
The most frequent measure observed
Prediction, Control and UnderstandingOne Variable Models
Mean = 5.5 Median = 6 Mode = 7
1, 3, 4, 4, 6, 6, 7, 7, 7, 7, 8
![Page 23: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/23.jpg)
Measures of central tendency
Mean, median and mode are all measures of central tendency because…
They attempt to describe the entire data set by identifying a central position.
How well do they help us predict & understand?
Prediction, Control and UnderstandingOne Variable Models
![Page 24: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/24.jpg)
Variance and Dispersion
When central tendency doesn’t help us understand, we can look at the dispersion or distribution of scores
Dispersion or Variance is the degree to which scores differ in relationship to each other
Prediction, Control and UnderstandingOne Variable Models
![Page 25: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/25.jpg)
Variance and Dispersion
• Low Dispersion – scores are grouped tightly
Prediction, Control and UnderstandingOne Variable Models
LOW
![Page 26: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/26.jpg)
Variance and Dispersion
• High Dispersion – scores are widely dispersed
Prediction, Control and UnderstandingOne Variable Models
HIGH
![Page 27: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/27.jpg)
Variance and Dispersion
• Frequency distribution
Prediction, Control and UnderstandingOne Variable Models
![Page 28: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/28.jpg)
Variance and Dispersion
• Probability distribution
• Grouped (frequency or probability)
Prediction, Control and UnderstandingOne Variable Models
![Page 29: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/29.jpg)
Normal Distribution
Layperson’s definition: Mean is exactly in the middle, 50% of scores on each side.
Perfectly Normal Distribution is RARE
Prediction, Control and UnderstandingOne Variable Models
![Page 30: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/30.jpg)
Normal Distribution
A critical assumption involved in most statistics is the concept that the scores are evenly distributed on each side of the mean
Thus, the more abnormal a distribution is, the more inappropriate it is to use inferential stats.
Prediction, Control and UnderstandingOne Variable Models
![Page 31: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/31.jpg)
Normal Distribution
Prediction, Control and UnderstandingOne Variable Models
![Page 32: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/32.jpg)
Distributions that are not normal
Prediction, Control and UnderstandingOne Variable Models
![Page 33: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/33.jpg)
Distributions
The shape of a distribution tells us more than one score…
It can show us the pattern of scores.
How well does it help us predict & understand?
Prediction, Control and UnderstandingOne Variable Models
![Page 34: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/34.jpg)
Standard Deviation
A measure that describes the dispersion of scores in relationship to their distance from the mean.
1 SD = 68.2% of all scores fall within 1 SD2 SD = 95.4% of all scores fall within 2 SD3 SD = 99.7% of all scores fall within 3 SD
Prediction, Control and UnderstandingOne Variable Models
![Page 35: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/35.jpg)
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
![Page 36: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/36.jpg)
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
![Page 37: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/37.jpg)
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
![Page 38: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/38.jpg)
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
How well does it help us predict & understand?
![Page 39: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/39.jpg)
When relying on descriptive stats
• Avoid making any significant inferences • If you must make inferences, evaluate the
measure in relationship to the standard deviation.
Prediction, Control and UnderstandingTwo Variable Models
![Page 40: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/40.jpg)
Performance
Practice
This also represents a correlational relationship
Prediction, Control and UnderstandingTwo Variable Models
![Page 41: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/41.jpg)
Correlation
Prediction, Control and UnderstandingTwo Variable Models
0.5 1 1.5 2 2.5 3 3.5 4 4.50
0.51
1.52
2.53
3.54
4.5
HS and College GPA
College GPA
HS
GPA
This is called a linear relationship
Notice tight grouping – indicates a stronger correlation
![Page 42: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/42.jpg)
Correlation
Prediction, Control and UnderstandingTwo Variable Models
How well does it help us predict & understand?
0.5 1 1.5 2 2.5 3 3.5 4 4.50123456789
10
GPA and Drinking Frequency
GPA
Drin
king
Fre
quen
cy
Notice looser grouping – indicates a weaker correlation
![Page 43: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/43.jpg)
Three or more Variable Models = Multivariate
Become increasingly complex as the number of variables increases, but generally provide much better data and understanding
Regression – A measure of the strength of a variable’s relationship with many other variables.
Factor Analysis – Groups inter-correlated variables into factors where the variables are most correlated together while being less correlated to the other groupings- reduces the # of variables and tend to be more stable
Prediction, Control and UnderstandingThree or more Variable Models
![Page 44: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/44.jpg)
Regression
Prediction, Control and UnderstandingThree or more Variable Models
Retention
Belonging
Academic Skills
Negative Social
Positive Social
Health
+.21
+.20
-.35
+.11
+.15
Retention
![Page 45: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/45.jpg)
Factor Analysis
Prediction, Control and UnderstandingThree or more Variable Models
![Page 46: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/46.jpg)
Strengths and Weaknesses
• Mean, Median, and Mode• Standard Deviation• Correlation• Regression• Factor Analysis
Discussion
![Page 47: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/47.jpg)
What inferences can you draw from:
• Mean, Median, and Mode• Standard Deviation• Correlation• Regression• Factor Analysis
Discussion
![Page 48: Student Affairs Assessment Committee Training Part 2](https://reader036.vdocument.in/reader036/viewer/2022062711/55c10b90bb61eb67348b463a/html5/thumbnails/48.jpg)
Questions?Concerns?
Snide Comments?
(Practical Application Next)