Distance Learning Unit

Faculty of Engineering and Design

Towards Enhanced Feedback Practices in a Virtual Environment – a case study

Mick Hancock

Peter Mellett

Rachel Wood

Alex Copping

May 2011

Context ...• A modular distance learning M-level programme in Construction Management

• 300 students in 45 countries worldwide

• Blended learning format – teaching and learning materials include printed ‘workfiles’, textbooks and online activities.

• Students meet online and at residential schools to establish ‘collaborative communities of practice’

• Student support includes a full-time dedicated Bath-based Programme Administrator and external, contracted, module-specific online tutors who are usually academic subject specialists.

... and Generalisability• How do you attempt to answer questions of the sort ‘How can I improve the quality of my/our

practice?’

• How ‘distant’ are your colleagues and other critical staff?

• How can you bring them ‘closer’ so that change is owned by all within a sense of corporate endeavour?

The Distance Learning MSc programme International Construction Management (ICM)

ICM – Core Modules Assessment Profiles

• Module 1 – Assignment 60% / Open Book Exam 40%

• Module 2 – Assignment 60% / Open Book Exam 40%

• Module 3 – Online Negotiation Task 50% / CW 50%

• Module 4 – Assignment 50% / Closed Book Exam 50%

• Module 5 – Assignment 50% / Open Book Exam 50%

• Module 6 – Assignment 50% / Closed Book Exam 50%

NB. Assignments and examinations are written and marked by the relevant tutor. Assignments are updated/amended as circumstances dictate/allow.

ICM Core Units – tutors’ questionnaire response scores

Table 1: Start of the TDF project Adding value to summative assessment through enhancing feedback practices Question: To what extent do the assessment practices on

your module –Score / 10

M1 M2 M3 M4 M5 M6

1 encourage self-assessment? 1 2 8 8 4 2

2 encourage teacher and peer dialogue around learning? 5 5 3 5 7 3

3 help to clarify what good performance is (goals, criteria, expected standards)?

3 4 8 1 2 5

4 provide opportunities to close the gap between current and desired performance?

3 5 2 7 5 5

5 deliver high quality information to the students about their learning?

7 5 7 8 4 4

6 encourage positive motivational beliefs and self-esteem? 5 3 5 8 1 1

7 generate information about student learning that can be used to reshape the learning experience?

1 1 7 2 8 8

NB. modules are studied in the sequence 1 to 6. The assessment for Module 3 (Construction Law) is based on a Moodle-mediated negotiation exercise between student groups, whereas all others are based more directly on the study materials.

Appendix 1: Summary of Tutors’ responses to Moodle quiz Self-evaluation of Assessment Practice

Fred (M1) Jim (M2) Pete (M3) Arthur (M4) Pru (M5) Cynthia (M6)

Question 1: To what extent do the assessment practices on your unit encourage self-assessment?

1/10I don't see that the students are encouraged to do any of these things.

I am tutor for their first module and many struggle simply to write an 'academic' essay (for various reasons). As they have not yet had any experience of what is required of them, their ability to be self-critical is limited. Their wishes for feedback are almost certainly... whatever is needed to get a better grade.

They have no 'portfolio' of work to choose fromCurrent assignments are not designed to be developed in a way that would enable achievement milestones to be included. In any case, time on each module seems is short and I suspect that there is not enough money to pay the tutors to spend so much extra time on such an approach

Time is also probably too short for students to give feedback on each other's work... they are too busy trying to complete the module.

2/10I understand and respect the need for self-assessment tasks in a distant taught package such as ICM, but I have a F/t job and a life to keep going as well. Hence I do not always achieve all of these pedagogic goals.I have experimented with the idea of students giving feedback to one another before submitting the final (summative) assessment, but find it time consuming and relatively selective as many students do not participate.I would also suggest that the Multiple choice questions that conclude each section of the work file in Module 2 are a means of contributing to self assessment.

8/10I think the module is pretty good as getting the students to self -assess. There is the reflective part at the end of the negotiation exercise and there are milestones through the rest of the task as well. The practical scenario is less reflective.I think you can over do it - the dissertation forum where they talk about each other's work is having some success but I don't think it takes the place of student/tutor interaction.

8/10I seem to recall a fairly recent student survey which gave the students an opportunity to comment on the positive and negative aspects of their previously received feedback - but I might be wrong / confused with another institutionThe discussion forum indirectly enables students to reflect and make judgements about the quality of their work. This forum also provides an opportunity for students to learn from each other. Not sure that it is sold quite that way though.The M4 Handbook is very explicit concerning milestones and the need to build a portfolio or work and experience

4/10Module 5 has been set as per other modules.  No specific practice to encourage self assessment has been put into Module 5.

2/10The assessment is very traditional and they have criteria to set against but little in the approach to encourage this. Maybe core reason is I do not see this as my role as there is plenty on the site for them to self assess, I do not see that the assessment approach could or should help this.  

Question 2: To what extent do the assessment practices on your unit encourage teacher and peer dialogue around learning?

5/10This is a difficult one because I am not aware that the students are actually asked to do any of these things, but the opportunity is there for them to do at least some of them e.g.

share and discuss tutor feedback with their peers and suggest strategies to improve performance next time;identify feedback comments that they found useful and to explain how they helped give each other descriptive feedback on their work in relation to published criteria before submission;

Each of these can be done through the various forums etc.. within Moodle, but I am not aware of students being encouraged to do so.

5/10 I regard supporting the discussion forum as a key part of e-tutoring; and I commit at least two hours per week when the module is running. In my weekly response I aim to summarise the discussion that has taken place, confirm positive activity and contributions where appropriate, and encourage clarity and an academic approach.I also think that the way the programme material is presented and written helps students think about the scale of the problem. For example, one of the exercises that often raises questions of qualification requires students to find official construction data relating to the nation they live or work in.

3/10There is not a great deal of dialogue around the negotiation exercise which is that happy thing - something the students grasp and run with by themselves and then present their findings to us.The practical scenario used to have a stage where they needed approval of their scenarios but I am not sure if this is still used or how useful it actually was.

1-10/10I cannot be specific about this question for the following reasons:I imagine this is dictated by the extent of use / capability of Moodle. This being a Distance Learning programme, it is as good as the weakest link which might be the communications medium I think (but have no evidence to support) that student contact details are made available to one another I have seen messages referring to pub nights in BC but have never been and so cannot comment on their success rates

7/10A range of self reflection questions are built into Module 5 and students do submit responses and other students comment on these responses.One challenge is that only about 50% of students actually bother to post responses to the questions and there is no sanction to make all students contribute.

3/10there is some feedback on the assessment feedback but this is limited

Question 3: To what extent do the assessment practices on your unit help to clarify what good performance is (goals, criteria, expected standards)?

3/10Students are provided with carefully constructed criteria sheets and performance level definitions.andI do talk about criteria and required standards at the Introductory workshop (although unfortunately many students have already completed the module before they attend so it's too late).

However, i am not aware of any of the other examples given here being used.

4/10 This is something that I struggle with, since e-moderating appears to be moving the goal posts. I recognise a need to support and identify progress towards students producing their best results, but I shy away from explicitly telling them how to achieve this with model answers and the like.Having access to the marking criteria and a clear assignment brief is obviously essential; but providing exemplar assignments opens a can of worms! However, I can see room for discussing drafts and confirming the direction that students are making. I also can see a place for exercises relating to effective referencing but have not had the time to set this up electronically.

8/10 We use the grading sheets where they can see what a good mark is and what we reward. My only concern here is that there is some benefit it not disclosing these - a very clever student can tailor what they do to meet the criteria rather than perform how they would normally if they did not know the criteria. In some ways I prefer to give the feedback at the end when they can see how they did against undisclosed criteria - criteria which the right approach would have led them towards anyway.

1/10The Grading System provided to me is fine. I have posted a guide for students writing assignments. This seems to have worked, judging by the 14 assignments recently assessedI am not aware of any exemplar assignments or for the opportunity to assess one another's workAgain the discussion forum does offer the facility for discussion amongst students of results and feedback. I have previously been emailed with specific issues

2/10This is very weak.

5/10There is quite well developed performance criteria but limited to one way.

Good FeedbackA positive focus Start off with a positive encouraging comment.

Balance negative with positive comments.

Turn all criticism into positive suggestions.

 

Suggesting where nextMake general suggestions on how to go about the next assignment.

Suggest specific ways to improve the assignment.

Suggest follow-up work and references.

Ask questions which encourage reflection about the work.

 

Making connections with the student Be prompt.Use informal, conversational language.Offer help with specific problems.Offer the opportunity to discuss the assignment and comments.

 

Offering justification Explain all your comments.Include a brief summary of your view of the assignment.Relate specifically to the learning outcomes and assessment criteria.Explain the mark or grade, and why it is not better (or worse!)

Good Assessment for Learning…

…Engages students with assessment criteria

…Supports personalised learning

…Focuses on student development

…Ensures feedback leads to improvement

…Stimulates dialogue

…?

Good Assessment for Learning…

…Engages students with assessment criteria

…Supports personalised learning

…Focuses on student development

…Ensures feedback leads to improvement

…Stimulates dialogue

…?

Your plans…

• What must you keep?• What could you keep?• What could you change?• What must you change?• Justifications• Now, soon, and a bit further on• What evidence will you collect to tell the

team your ‘new’ thing has been a success?

• What must you keep?• What could you keep?• What could you change?• What must you change?• Justifications• Now, soon, and a bit further on• What evidence will you collect to tell the

team your ‘new’ thing has been a success?

The ICM Assessment Development Problem

• Tutors do not have sufficient ‘spare’ time to engage with this project as envisaged – and fulfil their obligations to ICM students as currently constituted – and meet their other mainstream professional commitments.

• External distance learning tutors operate as academics working in a semi-autonomous mode; there are dangers that change actioned internally may be perceived as rule by diktat and meet resistance.

The ICM Feedback Problem

• All feedback opportunities are limited where an assessment task may be intrinsically infertile or its intention is obscurely expressed.

The Tutor’s Assessment Mantra

Bearing in mind the requirement to ensure that all assessment briefs should allow students to give the best possible account of their learning ....

.... can I demonstrate that my assessment activities discriminate accurately and fairly?

Effecting change

• How do we bring about change in assessment practice and feedback practice – whereby internal staff do the development work and tutors hold the changes as their own?

• How do we establish an inclusive ‘us’ in the place of two lots of ‘them and us’ seen from opposing sides?

Buy two, get two (three) free!

Bearing in mind the requirement to ensure that all assignment briefs allow students to give the best possible account of their learning, can I show that my assessment activities discriminate accurately and fairly?

Assignment statement+

Marking criteria

Marking process+

Feedback process . . . . and feedforward

ICM – Core Module Assignment Profile

Table 2

Types of output required M1 M2 M3 M4 M5 M6

A commentary of your experience of engaging with the workfile

x

A written account x x

An academic discussion x

An academic essay x

A critical exploration x

Extended commentary on a scenario x

A discussion paper to be presented to the board of directors x

A paper x

Critique x

Essay x

A critical appraisal x

A report for the Chief Executive x

• Do tutors have a common understanding of the key terms? • Are the requirements made explicit to the students or do they depend on a tacit knowledge of academic terms, of which students may not share our understanding?

Bloom & Co.

ICM Core Module Assignments – Action Verb Analysis

Table 3

Action verbs M1 M2 M3 M4 M5 M6

Explore* x

Consider* x

Reflect on* x

Discuss* x x

Recommend* x

Remember List x

Describe x x x

Identify x x

Understand Elaborate on x

Explain** x x x

Comment upon x x

Analyse Analyse x x

Give an outline x

Explain the differences (i.e. contrast) x

Evaluate Justify x

Describe the benefits and disbenefits x

Evaluate x

Critically appraise x

Critically review x

Critically assess x

*These terms are not so easy to classify according to Bloom’s Taxonomy and may indicate a range of behaviours depending on the context.**Explain can require understanding or synthesis or evaluation.

Starting the glossary from existing terms• academic discussion• academic essay• account• analyse• comment upon• commentary• consider• critical appraisal• critical exploration• critically appraise• critically assess• critically review• critique• describe• discuss

• discussion paper• elaborate on• essay• evaluate• explain• explore• identify• Justify• list• outline• paper• recommend• reflect on• report

Gently coaxingWhat we would like you to do

1.Review and comment on the Glossary of terms so that we can develop a shared understanding of key terms (to be published to students in due course) by 1st April

2.Review your own assignment brief, taking account of the need for clarity of meaning and explanation of requirements so there is no room for ambiguity, and bearing in mind the range of skills required across the programme. Submit a new assignment brief and marking criteria by 1st April.

Below you will find links to each of the assignment briefs and assessment criteria.

These include annotations by Peter Mellett and Rachel Wood with potential points for clarification.

Key words and appropriate responses Key Words Appropriate response

Define Describe Identify Indicate

List Outline Recognise Review Select State

Remember, either from the previous text or from your own experience, recall data or information i.e. state:

•Terminology•Individual facts•Rules

•Categories•Structure•Procedure

etc. Understand data and information as you:

etc. etc. Apply your knowledge and understanding, demonstrating ....

Analyse Characterise Classify Compare Confirm Contrast

Correlate Differentiate Discriminate Distinguish Examine Explain Explore Identify Illustrate Outline Prioritise Relate Select

Analyse material/concepts/rules as you:

•Group data into identifiable or logical subgroups•Identify violation of rules or logic•Construct diagrams of the logical relationships between entities

•Explain the logic of a series of events•Separating materials or concepts into component parts so the organisational structure can be understood

etc. etc. etc. Evaluate etc. etc. Create etc. etc.

Summary

• The more ‘distant’ colleagues and other staff are, the more difficulty you will have acting as an agent of change.

• Be careful about the assumptions make when using the word ‘we’.

• Approach change from the perspective of an action-reflection cycle so that you do not make false assumptions about progress i.e. ask (and respond to) the sequential questions:

1. What is my concern?

2. Why am I concerned?

3. What do I think I can realistically do about it?

4. What kind of evidence can I collect to help me make some judgments about what is happening?

5. How can I check that my judgment about what has happened is reasonably fair and accurate?

• Affirm all contributions, however small, from ‘distant’ colleagues and clearly incorporate them into the evolving dialogue about change – and in the amended teaching and learning materials/processes that result and which those ‘distant’ colleagues ultimately will have to use.