College of Science and Engineering

Evaluation of the Learning and Teaching Strategy:

The Way Forward?

Velda McCune

Centre for Teaching, Learning and Assessment

The current evaluation strategy

Questionnaire 1 - given at the start of the semesterresponsible learningpossible influences on responsible learning from the students

Questionnaire 2responsible learningperceptions of the course

Qualitative datagroup interviews in selected courses

Advantages of the present strategy

Allowed us to demonstrate correlations between -

responsible learning and student influences

(e.g. intrinsic motivation positively related to RL)

responsible learning and students perceptions’ of courses

(e.g. informative feedback positively related to RL)

Qualitative data offered richer insights into the students’ experiences

e.g. the value of opportunities to test understanding

(and there is more to come from these data)

Advantages of the present strategy (cont’d)

Provided an indication that the courses implementing the strategy were typically well received

Provided a positive view of students’ motivation

Allowed some specific feedback to course teams

(But there is more to do here)

Could be integrated with a College-wide evaluation process

Disadvantanges of the present strategy

There are often a lot of missing data from the questionnaires

It is difficult to infer causality because - courses are typically only modified in modest wayswe typically don’t have access to pre and post intervention datacourses not officially involved in the strategy are often quite in line with itmissing data makes comparing the same student across courses problematic

Struggling or failing students are the least likely to complete questionnaires

Qualitative data is really needed to make sense of the questionnaire findings but it is time consuming and costly to collect and analyse

Disadvantanges of the present evaluation strategy (cont’d)

The Learning and Teaching Strategy covers a wide range of possible changes to courses which makes it difficult to have a clear cut picture of its impact

It is rare to show clear cut findings with such measurement instruments in large first year courses

The current questionnaire does not cover all of the areas in the National Student Survey

We do not have the staffing levels to scale up this evaluation much more and our turn around time on the data could be better

There is a question as to whether more data collection of the same kind will really add to what we know

Alternative 1 - The same overall approach focused on fewer courses

This would allow more complete quantitative data to be collected(e.g. turning up in person to collect it, following up non-response)

This would allow richer qualitative data to be collected(e.g. more interviews with students, interviews with staff)

This would allow a more reasonable turn around time on the data

This might allow us to do additional types of analysis (e.g. looking at grades or prior qualifications)

Alternative 2 - In depth development and evaluation case work in very few courses

This would allow a very detailed picture of a small number of courses, which could be written up so as to show their potential relevance to other courses

This would allow a more integrated approach to educational development and evaluation work, which would make successful substantive changes more likely

The process can be tailored and iterative(e.g. questionnaires can be designed to tap into particular innovations)

This could only be done in very few courses e.g. one per semester

Alternative 3 - Focus on the experiences of particular groups of students

A focus on the most successful students would help us to understand the roots of responsible learning (e.g. in their prior experiences or attitudes)

Data from successful students could be worked up into guidelines for all students

A focus on struggling or failing students would help us to understand what might get in the way of responsible learning

It would be very important to look at the effects of courses through the eyes of these students (and not to assume a ‘good’ student ‘bad’ student model)

Alternative 4 - Focus on the experiences of academic staff

It may be that the most measurable impacts of the strategy will be on staff attitudes and expectations

Academics views of themselves as teachers are fundamental to how they approach their teaching

We could collect data about what supports or hinders educational change

Alternative 5 - Focus on a specific topic for a semester (e.g. feedback)

Would allow an in-depth approach to a topic which might support faster progress

Data collection could focus on courses which were of particular interest on the topic

Would allow time for wider literature search

Would allow time to make more of examples of good practice from other HEIs