extracting useful information from the uk’s national student (satisfaction) survey mark langan,...

Extracting useful information from the UK’s National Student

(Satisfaction) Survey

Mark Langan, Alan Fielding and Peter Dunleavy

Manchester Metropolitan UniversityFaculty of Science & Engineering

Higher Education Academy Annual Conference 2010

What makes a student satisfied?


Structure


• Research evidence based on a range of quantitative approaches using national dataset for science subjects

• Discussion about implications of using NSS for decision-making in H.E.

• Compulsory process in UK (since 2005/6), conducted by Ipsos MORI on behalf of HEFCE. Uses a standard basic survey to monitor perceptions of final year students.

• Approach based upon an Australian survey called the Course Experience Questionnaire (CEQ; Ramsden 1991). Considered robust in terms of three statistical measures; internal consistency, construct validity and concurrent validity.

• Measures six dimensions: teaching; assessment and feedback (sometimes considered separately); academic support; organisation and management; resources; and, personal development.


• The NSS uses a 5 point scale, completed in the final UG year (usually online) survey.

• In addition to 21 ‘items’ there is a separate overall satisfaction rating (Q22).

• Thorough overview can be found in Surridge 2007 and Marsh and Cheng 2008. Take home message: the outputs are hierarchical in nature and not designed for simplistic league tables.

• Note: satisfaction is a complex concept to measure and there are many approaches.


With particular reference to…

• consistency of patterns between years• differences between subjects• factors associated with overall student

satisfaction.


Data

Level 3 (closest to Programme/Dept)NSS data froma)2007;b)2008; c) 2009.Pruned to remove subjects not taught at MMU(e.g. medicine). Still very large data sets (>40,000

cases per survey)


NSS QuestionsTeaching (Teach)Q1 Staff are good at explaining things.Q2 Staff have made the subject interesting.Q3 Staff are enthusiastic about what they are teaching.Q4 The course is intellectually stimulating.

Assessment fairness (Fairness)Q5 The criteria used in marking have been clear in advance.Q6 Assessment arrangements and marking have been fair.

Assessment feedback (Feedback)Q7 Feedback on my work has been prompt.Q8 I have received detailed comments on my work.Q9 Feedback has helped me clarify things I did not understand.


NSS QuestionsSupportQ10 I have received sufficient advice and support with my studies.Q11 I have been able to contact staff when I needed to.Q12 Good advice was available when I needed to make study choices.

Organisation (Org)Q13 The timetable works efficiently as far as my activities are concerned.Q14 Any changes in the course or teaching have been communicated effectively.Q15 The course is well organised and is running smoothly.

NSS QuestionsLearning Resources (Resources)Q16 The library resources and services are good enough for my needs.Q17 I have been able to access general IT resources when I needed to.Q18 I have been able to access specialised equipment, facilities or room

when I needed to.

Personal Development (PD)Q19 The course has helped me present myself with confidence.Q20 My communication skills have improved.Q21 As a result of the course, I feel confident in tackling unfamiliar

problems.

Overall satisfaction (Overall)Q22 Overall, I am satisfied with the quality of the course.


• Which of the areas surveyed do you think correlate with the Q22 overall satisfaction score?

• What do you think student perceptions of the questions are (i.e. what is going through their mind when they complete the questionnaire)?

Satisfaction


Satisfaction is % of students answering 4 or 5 to a question.e.g. Q 1 Biology MMU – 95% of students were satisfied

There are subject differences

Subject differences confound simple comparisons, examples from 2009.

Medians for Qs 7, 8 & 9 plus 13, 14 & 15.


90.087.585.082.580.0

92.5

90.0

87.5

85.0

82.5

80.0

2007 Satisfaction Scores

2008, 2009 S

ati

sfact

ion S

core

s

20082009

Variable

s25

s31

s59

s32

s12

s40

s11

s37

s35

s25

s31s59s32

s12

s40s11

s37

s35

Scatterplot of 2008, 2009 vs 2007

Biology results (2008)

What answers are correlated with Q22?

Approach• Use % in agreement with a question (answers

4 & 5 on 5 point scale).• Simple correlation (ignoring subject)• Correlation allowing for subject differences

(ANCOVA)• Repeat for each year.• Calculate nationally and within MMU


Annual national trends

Overall satisfaction is consistently related to:

Teaching Quality, Support and Organisation.

It only weakly related to Resources and Assessment, particularly feedback.


2009 2008 2007Teach 3.8 4.3 3.8 Assessment fairness 14.5 15.0 15.0 Assessment feedback 16.3 17.0 16.0Assessment 15.4 16.0 15.5Support 7.0 8.7 6.3Organisation 7.3 8.0 8.0Resources 20.0 14.7 20.0Personal development 11.7 13.0 11.7

Mean Rank

Subject differences (Feedback Qs)


Question

Subject prompt detailed explained

Biological Sciences r 0.018 -0.130 -0.205

Physical Sciences r *0.440 *0.385 *0.589

Physical Geography r *0.675 *0.377 *0.566

Mathematical Sciences r 0.328 *0.460 *0.533

Computer Sciences r 0.226 0.103 *0.353

Mechanically based Engineering r 0.117 -0.190 0.192

Electrical and Electronic Engineering r 0.249 -0.233 -0.150

Technology r *0.728 0.090 0.237

Human Geography r 0.274 0.348 *0.433

Predictive model (Forest Tree Analysis)


• Cluster analyses are unsupervised methods that take no account of pre-assigned class labels or values.

• Decision and regression trees use a supervised learning algorithm which must be provided with a training set that contains cases with class labels or values.

• We used a new variant of regression trees called ‘RandomForests’. Robust method with fewer constraints than traditional regression methods, for example allowing different factors to be explored in their influence on overall satisfaction within different subgroups.

Regression Trees (an example)based on http://www.dtreg.com/classregress.htm

Predicts property value

Effectiveness of Q1-21 to predict overall satisfaction (Q22)


Predicting questionnaire item Inc MSE (%)Q15 - The course is well organised and is running smoothly 119.89Q1 - Staff are good at explaining things 71.45Q4 - The course is intellectually stimulating 66.71Q14 - Any changes in the course or teaching have been communicated effectively 60.79Q10 - I have received sufficient advice and support with my studies 55.34Q11 - I have been able to contact staff when I needed to 43.40Q3 - Staff are enthusiastic about what they are teaching 40.08Q2 - Staff have made the subject interesting 38.26Q12 - Good advice was available when I needed to make study choices 35.27Subject 32.35Q6 - Assessment arrangements and marking have been fair 20.10Q17 - I have been able to access general IT resources when I needed to 18.73Q19 - The course has helped me present myself with confidence 17.35Q18 - I have been able to access specialised equipment, facilities or room when I 15.41Q16 - The library resources and services are good enough for my needs 15.34Q20 - My communication skills have improved 13.29Q13 - The timetable works efficiently as far as my activities are concerned 13.16Q7 - Feedback on my work has been prompt 10.49Q9 - Feedback on my work has helped me clarify things I did not understand 6.65Q5 - The criteria used in marking have been clear in advance 6.60Q21 - As a result of the course, I feel confident in tackling unfamiliar problems 3.32Q8 - I have received detailed comments on my work 3.04

Predictive model (Forest Tree Analysis)


Predictor 2007 (%)

2008 (%)

2009 (%)

Teaching 27.7 27.8 24.8 Fairness 10.7 13.6 6.9 Feedback 3.5 1.7 3.1Assessment 14.3 15.3 10.0Support 18.9 18.9 17.5Organisation 26.2 23.9 25.9Resources 2.3 4.2 6.8Personal Development 10.6 9.9 15.0

Q22 ‘under-performers’


Actual Predicted Residual SE1 SE2 SE3 Subjects

Q22 ‘as expected from Q1-Q21’


Q22 ‘over-performers’


University Groupings

Group n Mech Eng

ComSci

Allied Med

Elec Eng

Biol M&S EGS Hum Geog

Chem All

Million+ 85 72.6 71.2 74.2 71.9 75.9 92.0 84.3 91.1 75.7

Alliance 117 75.9 71.3 79.8 74.3 84.1 85.8 84.8 87.9 89.1 80.0

None 151 79.0 82.2 81.7 80.7 87.1 90.4 89.6 86.3 89.3 84.2

Russell 139 86.8 85.7 83.7 86.1 91.3 85.6 88.7 84.7 90.7 87.3

1994 84 83.4 88.0 89.8 90.0 89.7 90.3 88.6 91.5 92.7 88.9

All 576 78.7 79.7 80.6 80.7 86.4 87.6 87.6 87.7 90.5 84.5

n = 65 115 56 57 63 48 69 61 42

Mean overall Q22 for university groups

Conclusions


• Subject differences (e.g. mathematical content)• Institutional differences• False assumptions (e.g. enhancing feedback

directly enhances Q22)• Institutional effects• Satisfaction is a complex measure related to L&T

practices


1. Q15 The course is well organised and is running smoothly.

2. Q4 The course is intellectually stimulating.3. Q1 Staff are good at explaining things.4. Q21 As a result of the course, I feel confident

in tackling unfamiliar problems.5. Q10 I have received sufficient advice and

support with my studies.

2009 Top five predictors (best first)

Quotes from Ramsden (2007)


“… [The NSS] is not a measure of satisfaction so much as a window into how our designs for learning are experienced by students.

From these insights we assemble thepractical measures we may take toenhance the quality of theirexperiences.”



“... it is not simple to know what to do. Current experiences, unlike satisfaction, are a mixture of previous experiences and the environment as it is now...

... so sometimes we will need to adjust expectations or consider altering previous experiences in order to improve quality.”



“I cannot agree with the idea, for example, that because students are slightly less positive about feedback on assessed work in the NSS than about the quality of teaching...

... we should rush to bully academics into providing more feedback more quickly.”



“From this it also follows that students do not have a ‘right’ to be satisfied. They are themselves part of the experience. ..

... Students decide their own destinies and we can only add or subtract value at the margins.”


Does anyone have an example of direct change as a result of NSS surveys?

How can we use NSS ratings to enhance our practices?

extracting useful information from the uk’s national student (satisfaction) survey mark langan,...

Documents

teaching assessment

overall student satisfaction

nss questionssupportq10

q9 feedback

q2 staff

q3 staff

q6 assessment arrangements

australian survey