chapter5.teaching mathematics in higher education - the basics and beyond

8/12/2019 Chapter5.Teaching Mathematics in Higher Education - The Basics and Beyond

1/44

Chapter 5

Student Assessment in Mathematics

A Note on How You Might Read this Chapter

Assessment is a major part of individual teaching and of departmental administrative affairs and in thischapter I have tried to cover the wide range this represents. If you are new to teaching and assessment, itmay be that you want to get down as quickly as possible to the actual nitty gritty of setting and markingexams, for example. In that case you can probably skip the first sections and jump straight in at Section5.5. However, the earlier sections give an overview of the importance and purposes of assessment andyou may like to return to them when you have nailed the practical side.

5.1 The Importance of Assessment

Student assessment is traditionally regarded as the bane of the academics life. Unlike schoolteachers,most of us mark our own students work, and the load it represents is not always welcome. But in fact thisshould be regarded as positive and valuable aspect of teaching in higher education, it is an opportunityto see how your students have progressed and the impact of your teaching. It is a serious business andtaken seriously it is difficult, but it is also important. If you are really interested in your teaching then it isinformative and (hopefully) rewarding. The skills of good assessment are developed over long and variedexperience. All we can do here is provide advice, theory, suggestions and good practice in assessmentin mathematics, that may give you ideas for assessing your own students. By far the most importantinput to this will however come not from literature such as this, but from talking to colleagues to get toas wide a range of views and experience as possible. The aim is to provide fair assessment for as manystudents as possible and so provide stakeholders such as employers with some idea of how the students

may be expected to perform in their professional roles. This is a tall order and requires careful thoughtabout the purpose and context of any assessment we undertake, including a wide range of human as wellas intellectual aspects of teaching and learning (Principle 2).

There are many issues to consider in assessing students. We will want to ensure good standards asviewed by a wide range of stakeholders, the department, the university, employers, government, thepublic, and so on. We will want to be fair to the students and give them every opportunity to show uswhat they can do. We will need to ensure that the assessment provides an accurate picture of what thebulk of the students have learned, so that colleagues can rely on that material in presenting their ownmodules. Sometimes these different drivers can pull in conflicting directions. For example an examina-

145


2/44

146

tion paper may appear unduly difficult (easy) to an external examiner, whereas in fairness to the studentsyou have made efforts, in the teaching, to take account of their strong (weak) background to prepare themappropriately for the examination. In each case of strong or weak background you may still be providingthe students with high standard challenges, but this may not be apparent to the outsider.

There is of course a vast literature on assessment, both generic and mathematics based. As overviewmaterial for generic assessment we mention the Generic Centre Assessment Series available on the HigherEducation Academy website. For assessment in the context of mathematics we mention the Good Practicein Assessment Guide for Students (Challis et al, 2004) available on the website of the MSOR Network ofthe HEA. Also see Supporting Good Practice in Assessment in Mathematics, Statistics and OperationalResearch on the same website.

In the spirit of assessment we will also look at the evaluation of our teaching in this chapter, in Section5.8. We have touched on this in the context of lecturing and tutoring, but give more general ideas in thischapter.

5.2 MATHEMATICS for Student Assessment

TheMATHEMATICSmnemonic of Section 1.7 can be applied to assessment, to remind us of the sortsof things we have to keep in mind when assessing students. TheMathematical content of the moduleobviously determines the content and nature of the assessment, and the main thing to ensure is thatwe assess an appropriate breadth and depth of the material, and that the students know the syllabuson which they will be assessed. And particularly in mathematics we need to be specific about what ismeant by content. The classic mathematics example here is the question of whether the students will beassessed on all of the proofs of theorems they are taught. Such content will be spelt out more preciselyin theAims and objectives of the course, and we have to ensure that the assessment is alligned with thelearning objectives through an appropriateTeaching and learning strategy and assessment strategy. Forexample you may set coursework that is an integral part of learning for the students. This coursework

may assess learning objectives different from the examination, allowing more extended developments.Or, if using numerical techniques on your course you may have an examination in a computer laboratory.The question ofHelp and support for the students in terms of assessment may for example include theissue of a specimen examination paper, particularly if it is the first time you have examined the module.You may also have a couple of revision classes before the examination.

Evaluation of assessment used to be just a matter for the external examiner, but now most modules willusually have a moderator to check the examiners work. Also there tends to be much more scrutiny ofthe outcomes of the assessment these days. Examination results that are way off the norm are scaled withmore reluctance these days and the emphasis tends quite rightly to be on ensuring that the examinationsare set properly in the first place to avoid such practices.

For the Materials for the assessment we will have to prepare the examination paper, or other forms ofassessment, and such things as specimen solutions and marking schemes. Assessment - well this is it, themain subject of the chapter!

Time and scheduling considerations arise throughout the assessment process - coursework distribution,submission and feedback times, length of examination and time allowed for questions, submission timesfor the papers, examination timetable and boards, etc. Indeed, assessment is one of the most fraught partsof the curriculum so far as scheduling and time constraints are concerned because the lecturer often haslittle control over it. If you get time estimates wrong in the actual teaching then you can usually handle ityourself in subsequent lectures. But in assessment, so much relies on or affects other people.

The issue ofInitial position of the students will come in of course when you set questions. Since the


3/44

147

students have just completed your module you know what their background is in your subject, and socan design questions accordingly. However, it is rare that questions do not also call on knowledge outsideyour particular course. For example in differential equations you will be relying on such basic knowledgeas integration and algebra. If the purpose is to find out what they know about differential equations thenyou might arrange for the parts of solutions relying on such background to be relatively straightforward.

Coherence of the curriculum can be reflected in coherence in the assessment. So for example questionsmight extend across the content to test that students have understood the linkages between topics. Asalways, we need to think about theStudents when considering assessment. For them, assessment is oneof their most worrying times - after all it is very important to them and can affect their confidence andpossibly their career prospects. So be considerate and understanding on this issue, this is one of thosetimes when they need all our support.

5.3 Assessment and the Basic Principles of Teaching Mathematics

As with everything in this book we take the basic principles in Section 2.4 to underpin our design and useof student assessment. Below we offer some suggestions about how these principles underpin studentassessment in mathematics.

Practicalities of providing the learning environment

P1. All teaching and learning must take place within limited resources (the most important of which istime) available to you and your students

One of the major requirements of good assessment is that it be practical. All assessment is a com-promise between ensuring the validity and reliability of the assessment and working within limitedresources. For example we could almost certainly get a much better idea of what a student reallyknows from regular interviews, and indeed much use of oral examinations is made in Europe andRussia, but this is simply too (wo)manpower intensive and beyond the resources of most UK depart-

ments with current student numbers. Also we have already noted the crucial issue of schedulingand time constraints in assessment.

P2. Teaching is a human activity that requires professional management of the curriculum, the groupand the interpersonal interactions that implies

Nowhere is it more important to balance professional position and ordinary human regard for stu-dents than in assessment. We of course want to help them all we can, but there are limits to theextent and manner in which we can do that. And we have to put aside any personal feelings toensure objective decisions on the outcomes of assessment.

P3. There must be clarity and precision about what is expected of the students and how that will bemeasured

Our expectations of the students are expressed in terms of the learning objectives. But it is a delicatematter. If you are over specific about objectives, then that is tantamount to telling the students pre-cisely what the assessment will be. The precision has to be in telling the students what is expectedof them and that the assessment is based on that alone.

P4. The teaching, learning and assessment strategies must be aligned with what is expected of thestudents

This is the issue of alignment between learning, teaching and assessment. For given learning ob-jectives, the teaching and learning strategies and activities must be designed to achieve them. The


4/44

148

assessment strategy and tasks must be designed to measure achievement of those objectives, takinginto account how we have taught the students.

How students learn

P5. The workload, in terms of intellectual progression, must be appropriate to the level and standardsof the course

As in the actual amount of material taught, so the assessment also needs to be reasonable in termsof the workload it represents. Not too much (or too little) coursework for example. Questions andexamination papers need to be of a reasonable length and difficulty. We need accurate assump-tions about prior knowledge, so for example some form of initial assessment might be necessary todetermine the real abilities of incoming students.

P6. Mathematics is best learnt in the away it is done, rather than in the way it is finally presented

And if this is how we teach, then it is how we should assess. If we have only taught students toregurgitate set proofs of specific theorems, then we have not trained them in the skills required togenerate their own unseen proofs and so should not examine this. Of course, the best option is toteach them how to produce unseen proofs, because this is what mathematicians really do, and thenwe can set such questions - tough ones in coursework, less demanding in examination questions.And we might have essay type questions that allow students to demonstrate their appreciation oftopics for which they have not yet mastered the details, or to test their command of higher orderskills. We might for example reward conjecturing skills in the exam, even if they do not lead to thecorrect conclusions. Excellent discussions of these issues can be found in Niss [59].

P7. Mathematics is most effectively learnt if the student reconstructs the ideas involved and fits theminto their current (corrected!) understanding

This requires more open-ended types of questions, which bring out how the student processes,analyses information and then synthesizes it into new forms. Of course this is easier to assess in

coursework or via project work but is possible in examination questions by for example testingunseen generalizations and abstractions.

P8. The meta-skills required for the previous learning points may need to be explicitly taught

Again, such skills are much easier to assess in projects or coursework, but it is not so much the skillsthemselves that need to be assessed as the deep learning that they produce. In an examination paperfor example one might ask a student to describe how they would approach a particular previouslyunseen problem, without actually solving it. They could describe the actions they would have totake, the new skills they might need to learn, and so on.

Teachers tasks (Explain, Engage, Enthuse)

P9. Good skills in explanation are required to assist students in learning efficiently and effectively(Explain)

One sometimes sees such instructions as Explain how you would ... in an examination question. Ifthis is a verbatim regurgitation of a set piece technique - say the adjoint matrix method of invertinga matrix - then it is hardly testing skills of explanation. Just as good explanation is a skill that thelecturer needs, so we should try to develop such skills in the students. We can test such things in anoral for a project, for example, or through an essay type question in the examination or coursework.And of course the teacher will need to exercise good explanation skills in order to set carefullyexplained and unambiguous questions.


5/44

149

P10. Students best learn mathematics if they are actively engaged in the process of doing mathematics(Engage)

No problem at all in a typical examination situation - it is one time when you can guarantee active

participation from the students! But of course coursework and project work are a different matter,and part of such assessment might indeed comprise the extent to which the student has contributedand engaged with the work.

P11. High levels of motivation are essential for effective learning (Enthuse)

As noted elsewhere we are quite confident of the motivatingqualities of assessment! However, weperhaps do not always think of making the assessmentinteresting. Generally we might describean interesting question as a nice question, meaning that it requires some clever twist, or perhapsleads to an unexpected result. Of course, we have to be careful that it is not what some mightdescribe as a trick question, so there is a balance between interesting and ingenious to be drawn!Also, we might construct an interesting question from a new or topical application (ensuring thatthe students will have the required background).

5.4 Definition and Purpose of Assessment

5.4.1 Definition of Student Assessment

It is worth being precise about what we mean by assessment. A typical dictionary definition is [1]

To assess: to estimate, judge, evaluate, e.g. persons work, performance, character.

Actually, the use of the term assessment in the context of student assessment is relatively new - circa1970s. Prior to that terms such as testing, examining, grading, tended to be used. Here, by assessmentwe mean the measurement of the extent to which students have met the learning objectives of a courseof study.

The importance of assessment and its influence on student learning has long been recognised. Indeed,like it or not, in practice most teaching and learning is assessment driven. For in-depth discussion of thedefinition of assessment in the context of mathematics, see Niss [59].

5.4.2 Purposes of Assessment

Assessment serves a number of purposes, some of the most important are listed below [13]:

to judge the extent to which knowledge and skills have been mastered

to monitor improvements over time to diagnose students difficulties to evaluate the teaching methods to evaluate the effectiveness of the course to motivate students to study to predict future behaviour and performance


6/44

150

to qualify students to progress.

The purpose intended for the assessment will naturally influence the form and conduct of the assessment,but whatever its purpose it is primarily a measure of student learning. Of course, it is not always a good

measure, and poor assessment can adversely affect student learning. So the outcomes of assessmenthave to be taken in context - for example, the sort of coursework assessment used to encourage studentlearning should be used carefully in predicting students future performance, if only because there is nocertainty that it is solely the students work.

Perhaps it is worth thinking about how the aims of assessment have evolved over the last few decades.When participation rate in HE was about five percent, university mathematics assessment was reallyabout finding the best of the best. The UK HE system has changed. With 40-50% participation rate theemphasis is on ensuring that as many students as possible achieve high levels of competence in math-ematics. First class results are now meant to reflect the highest level of achievement and are no longerintended to identify the very best, relegating the rest of the best to second class results. And indeed nowa-days there are frequently questions raised over whether the traditional assessment and awards systemis fit for purpose. We dont discuss such issues in this book, but focus on what is overwhelmingly the

mainstream practice in current mathematics assessment - coursework and examination.

Niss [59] goes into great depth on the purposes of assessment of mathematics, and makes the point that,uniquely, mathematics has often formed a filter for entry into many disciplines and professions. Thismakes assessment in mathematics particularly important. Niss also provides details of the history of thedevelopment of assessment in mathematics. He reminds us that the written exams we currently cherishso much were in fact introduced originally as an efficient way of assessing students as numbers increasedand made the previously universal oral examinations too costly.

5.4.3 Some Key Definitions in Student Assessment

Below are listed definitions of some commonly used terms in the study of student assessment. One can

find plenty of material on these in the books listed in the Bibliography ([4], [15], [40]) and depending onthe educational level and objectives of the book one can find various depths of treatment. Indeed, thereis even disagreement about the proper use of these terms. Here we cannot delve very deeply into therelevant topics, yet we must have a rough appreciation of the terms. We therefore provide definitionsand brief discussions that hopefully give an accurate common sense notion of what is intended, whilestill carrying adequate precision and rigour.

Formative assessmentThis is designed for developmental purposes and does not contribute to students marks or grades, allow-ing students to make mistakes without penalty. It includes such things as diagnostic tests, non-assessedcoursework, quick classroom quizzes. It is often argued that students wont do such work because it doesnot attract marks, and various devices are employed to get round this. For example, awarding marks forthe best 50% of such work, or an unspecified selection from a number of assignments. Many lecturers

issue weekly sheets of exercises designed to reinforce the lecture material and aid learning, and mayeven mark and provide feedback on these. But if there is also assessed coursework, then the studentsinvariably concentrate on that and may ignore the exercise sheets altogether. However, this should notdeter us from setting such formative exercises, provided the resultant workload does not exceed a rea-sonable level. Some students may do the exercises out of interest in the subject, and those who dont takeadvantage of the extra opportunities for learning that we have provided, well, that is their problem.

Summative assessmentSummative assessment is designed to establish students achievement at stages throughout a programmeand normally contributes to their marks and grades. It includes the usual examinations and assessed


7/44

151

coursework. Of course, any assessment that attracts marks has to give valid and reliable results (Seebelow). Examination conditions can usually guarantee this, but take away coursework cannot. In thewell-intentioned move to aid learning, coursework is used in many undergraduate courses. But of courseit is open to plagiarism, which can be difficult to prove in mathematics. For this reason marks awardedto such coursework are normally kept a low proportion of the whole assessment (say 10-20%).

Criterion-referenced assessmentAn assessment system in which students performance is marked and graded according to specified cri-teria and standards. In theory all students could fail completely or achieve the highest grade if they failto meet or achieve the required standards. Such assessment is sometimes used in situations where it isvital that the student actually has mastered key topics - say in medicine. The trouble is, it is costly foreveryone concerned, because inevitably many students will need a number of attempts before achievingall the required outcomes. An example of this type of assessment is a test given to new students whenthey start at the University of Warwick Mathematics Institute. The Institute identified a number of keyareas, differentiation, integration, trigonometry, inequalities, in which the first year intake lacked suffi-cient skills. These are regarded as so important that incoming students were given a test in each topic,with a requirement that they achieve at least 80% in order to be eligible for gaining full marks on one oftheir modules. The basic idea is that students are encouraged to master these key techniques. Of course,this serves the double purpose of bringing them up to speed in essential material and also emphasisesthe importance of these topics to all the students. Similar criterion-referenced assessment is used in manyvocational qualifications such as BTEC. Students are given phase tests incorporating key skills, whichthey have to take until they achieve them all. However, the crucial difference here is that they are allowedto take the same work away and rework it until they get it right. In such circumstances it is difficult toguarantee effective and lasting learning.

Norm-referenced assessmentAn assessment system in which students are compared with each other and placed in rank order on a(normally!) normal distribution curve. Only a proportion of students will obtain a particular grade orclass of degree. This is the conventional mode of assessment in undergraduate courses. It has of coursethe disadvantage that even a high mark cannot guarantee mastery of all aspects of the topic - it is only

evidence that they have learned selected material well enough to gather sufficient marks together.

Validity of assessmentThis is the requirement that the assessment measures attainment of the learning objectives set. In math-ematics it is usually possible to ensure this. For example it is easy to set a question that validly testswhether a student can solve a linear second order inhomogeneous differential equation. However, thereare some notable instances where validity can be more of a problem. For example, suppose one objectiveis that they will be able to construct mathematical proofs. Then a question that asks them to prove say theirrationality of

2would not be a valid test of this objective if they had thoroughly rehearsed the proof

in class.

Reliability of assessmentThis is the requirement that the outcome of the assessment is consistent for students with the same

ability, whenever the assessment is used, whoever is being assessed, and whoever conducts the assess-ment. Again, this is not too difficult to guarantee in mathematics because of the use of detailed markingschemes. However, as we will see later, even when these are used there can be quite significant variationsbetween different markers. Any scheme that is sufficiently detailed to always guarantee reliability is al-most certain to be complex and unwieldy, or to render the question so anodyne as to pose little challenge.


8/44

152

5.4.4 Framework for Discussion of Student Assessment

We will need some overall structure for discussing assessment, and in this chapter we will take thissimply as the order in which things usually have to be done in assessment, that is:

setting assessment tasks (Section 5.5) marking and moderation (Section 5.6) coursework and feedback to students (Section 5.7).

In this chapter we also look briefly at the issue of evaluation of our teaching (Section 5.8). We need not saymuch about this here because it will probably form part of your institutional and departmental training.However, we can say a few things about the mathematical aspects of such things, which you might finduseful. In particular this involves the way we use the outcomes of assessment.

Essential to developing skills in assessment of mathematics is a detailed study of the actual practice ofsetting and marking examination papers. To provide this we include in the appendices copies of papers,module specifications, mark schemes and sample students solutions for examinations in engineeringmathematics and abstract algebra. These are used in the following sections to provide nitty gritty ex-posure to the real thing. None of it is intended as good practice to imitate. Rather, regard them as casestudies designed to stimulate your own ideas, and to illustrate how you might discuss such papers inyour own department.

5.5 Setting Assessment Tasks

5.5.1 The need for Learning Objectives

By assessment tasks we mean any sort of assessment - examinations, coursework, projects, computeraided assessment, etc. Following Principles 3 and 4 we have to design the assessment to measure what thestudent is expected to have learnt, as expressed in the aims and objective of the course and to take accountof how it has been taught. The following exercise is intended to highlight this aspect of assessment. It isof coursenotan exercise in mathematics, but an exercise inteaching, indeed of any subject, which makesa crucial point that is sometimes overlooked, even in mathematics teaching at advanced levels.

Exercise - sum marking

Students have been asked to evaluate the product 1234 34without a calculator. Mark out of tenthis attempt by one student:

1 2 3 4 3 4

4 9 3 63 7 1 2 04 2 0 5 6

Compare your mark with those of a few colleagues. Do you all agree? Discuss any differences. Whyare there differences? What is the correct mark?

In fact, without further information about the objectives (which would define the context and purpose)of the question in the previous exercise there is no correct mark. If the objective is to test whether a


9/44

153

student knows how to multiply such numbers then although the answer is wrong, this student clearlyknows what to do and has simply made a small slip in mental arithmetic in one carry. A high mark wouldtherefore seem to be in order. But if the objective is to obtain exactly the right answer, with accuracy ofcalculation paramount throughout and the method assumed known, then this student has not achievedthe objective and a modest mark would be not unreasonable - some might even give a low mark.

The above exercise gets to the core of setting assessment tasks - we have to be quite clear that whatwe are asking the student to do will measure what we expect them to have learned. Furthermore, thestudent should have been told what we expect of them. All this could simply be summarised under therequirement that the assessment is fair, and taken very much as common sense, and indeed was takenin this way say twenty years ago. However, we are now expected to articulate much more precisely whatit is that is being assessed. This is essential these days because we are now assessing a much wider rangeof the population than a decade or two ago. As well as leading to a much more varied background in ourstudents it also means that with less personal contact with them (Modularisation, larger classes, etc) wehave to rely on assessment results to a much greater extent than in the past.

The common vehicle for specifying what we expect students to learn is the learning objective (Section

2.6). Learning objectives must be defined and published for everyone to see for a given module (See, forexample, Appendices 2 and 6), and then the assessment must measure the attainment of these objectives.We say the assessment must be linked to (or alligned with) the learning objectives . This is the mainsubject of this section, to design assessment tasks for various types of objectives. This amounts to theassessment strategy, in the way that the teaching and learning strategy links teaching activities to thelearning objectives (Section 2.7). For an in-depth critique of learning objectives in mathematics assess-ment, see Niss [59]. Like many books however, criticism of learning objectives is often on the basis ofintended learning that cannot easily be expressed in behaviourist objectives, neglecting the fact that thereis much intended learning that can and perhaps should be so expressed.

Example - the assessment of proofThis issue about clarity of objectives and its importance for high quality teaching and learningis illustrated very well by the assessment of proof in mathematics, referred to already in thisbook. If we set a question requiring a proof it makes a great difference whether the student hasbeen forewarned about the possibility of it coming up. If the student feels it might come upand they rehearse it verbatim in advance then the answer to the question reveals little abouttheir real abilities in unseen proof - the skills they need may in fact be at the lowest level. Onthe other hand, if it is intended that they will be able to handle unseen proofs in examinationsthen they must be made aware of that possibility, and must be trained accordingly. A commontype of compromise here is to teach a particular method of proof, such as induction, but in theexamination we might set new examples of this that they have not yet seen. The point is thatwe need to be clear about what skills our questions will aim to assess, and we have to informour students about this in the learning objectives of the module. A related issue is the extentto which we expect students to be precise about the statements of theorems. For mathematicsstudents we might require explicit statements of the conditions of theorems, whereas for say

engineers we might not be too fussy as long as they have the right idea and can use a theorem.Either way such issues need to be clarified in the learning objectives.


10/44

154

5.5.2 Designing Assessment Tasks for given Learning Objectives

Exercise

You have to write a question to test what first year students have learned about Pythagorastheorem. Construct three questions, easy, medium and hard for this purpose. Discuss yourquestions with a colleague, particularly focusing on your interpretation of easy, medium and hard inrelation to the skills that the students are expected to develop.

Almost certainly, any two attempts at this exercise by different lecturers will produce very different re-sults. It is just like the first exercise in this section, the objectives have not been specified clearly and sowe are all free to interpret the exercise as we wish and set questions accordingly. On the other hand, ifwe used the examples of a learning objective and the guidance on MATHKITgiven in Section 2.6 then itis likely that different attempts at the exercise would be more consistent. In practice of course we would

not go into anything like this detail, but instead would use intuitive notions of what we expect from thestudents and how we would assess them. The point of the exercise is to emphasize that in setting an as-sessment task we need to decide on the balance we want between K,I,Tand design the task accordingly(Or use your own preferred means of categorizing cognitive skills that you want to assess). As alreadytouched on above note thatKwill usually include a whole range of factual items or results or elementarytechniques, and any practical assessment will only sample this. Also care is needed in ensuring that theassessment ofI and T is authentic. (If either have been seen before to any great extent then they simplybecomeK - a drilled higher order skill becomes a lower order skill) - these days some lecturers ensurethis by stating what proportion of a question is bookwork or previously seen.

Contrary to the views expressed by some authors, most cognitive skills can be assessed by appropriatelyset unseen examinations. It has to be said that this becomes increasingly difficult as the duration of theexamination shortens. And it is a matter for debate whether this can be done to any significant extent

in the ninety-minute module examination that is becoming common these days. The traditional three-hour examination seems preferable here in that this is a more appropriate length of time to assess realcognitive skills. Of course for the so-called transferable skills of communication, time management,team working, etc, then one needs specialised forms of assessment such as projects or group exercises.And for practical topics such as numerical methods one might have computer laboratory assessment, butagain this is a specialised area not dealt with here.

StraightKcan be easily assessed by short objective questions, even by multiple choice or computer aidedassessment. Of course K can constitute a significant piece of work, albeit routine - for example in invertinga matrix or finding eigenvalues. So such skills may take up a sizeable proportion of a question. But onewould normally include also an aspect ofIT- in the case of matrix theory for example this is often quiteeasy to do in the form of unseen proofs, or application of routine techniques to novel situations. Anotherway of incorporating assessment ofIT is by essay type questions (see example below). Or one might

ask for a particular technique or method to be extended or generalised to an unseen situation. MostITquestions will be multi-step requiring students to weave ideas together (but again unseen).

An often neglected aspect of assessment of mathematics is the issue of how it is presented. You dont haveto be teaching long to come across the poor level of mathematical presentation and arguments used bystudents in their solutions to examination questions. Often this comes about because they simply copywhat they have seen the lecturer do on the board - write out a list of equations (sometimes minus theequals signs!) with little explanation. As long as they get the answer, they are happy. You may try toencourage better presentation skills by awarding some marks for this in your mark scheme - but makesure the students know about this.


11/44


12/44

156

5.5.3 Matching Assessment to the Student Profile

Just as we need to consider the background of the students when we are actually teaching them, so wealso have to think of this in assessing them. In mathematics there is one example of this issue that par-ticularly emphasises its importance - namely the service teaching issue. For example there are manyways that a question in elementary integration might look completely different for a first year engineer-ing class and for a first year mathematics honours class. Of course the language might look different.With the mathematicians we may be talking about continuous, differentiable, integrable functions, etc.For engineers it may be better to avoid such terminology - maybe put them under the well behavedumbrella. But there is more to it than that.

With engineers, for example, one might be more lenient on the level of rigour and precision expected (upto a point). Questions may be more oriented towards applying techniques, with less proof expectedfrom engineers. We may specify the method to help them, or we may allow them to use any validmethod if they have shown evidence of learning. Because they may have seen mathematics from a rangeof different perspectives (say in their other engineering subjects), we may be more tolerant of slips innotation, terminology and approach. The point is that with engineers our attitude to their abilities in

mathematics might be something like This is only a tool to them, has this student shown that (s)he canuse it provided they have the resources, and could in practice iron out the few slips they have made intheir solution? On the other hand for the mathematicians it might be This is fundamental material forthem, do they have a thorough understanding and facility in this, able to work quickly and accurately,adapting as necessary?. The difference between these two approaches will clearly influence the type ofquestions set.

ExampleIn complex analysis we often refer to regions of the form 0< |z| < , as in say the statementof Laurents theorem. This should be fine for the educated mathematics student. But forengineers for example, when doing complex variable, it might be better to simply refer to itas a punctured disc. In the heat of an examination a proliferation of symbols in a question

might wrong foot the engineer, when all we really want to know is whether they can actuallyuse the relevant result. On the other hand, mathematics students should be fluent in suchthings and one would expect them to be comfortable, even to prefer, a symbolic formalism.

Another area in which assessment may need to be geared to the background of students is if we have asignificant number of foreign students whose first language may not be English. Whilst mathematics isvirtually the only universal language it is also true to say that when a native language is used in math-ematics it is often to convey subtlety which may be lost on non-native speakers. Amongst all the otherthings we have to consider in assessment it is very hard to set papers that take such things into account.Usually it is only after the event, when the scripts come in, or students ask you questions in the exami-nation, that you realise a question may be ambiguous or even meaningless to a foreign student. Whenthis happens, you may need to take a lenient view in the marking (although of course with anonymousmarking you wont know who the foreign students are, so you will have to play this by ear!!).

And, of course, you may be a foreign lecturer. Your whole approach to such things as assessment maybe influenced by your own cultural background, as well as language. Some nationalities provide lesssupport for their students than might be the norm in the UK. Some may be used to quite severe policieson marking. So, if you are a foreign lecturer, new to UK teaching then discuss widely with colleagues tofind out as much as possible about UK conventions and other matters of assessment .

Another way in which you might need to learn about the students is in terms of your expectations oftheir abilities. Research on the expectations that lecturers have of their students abilities has shown[20] a significant gap between expectations and reality. And that was before the widening participation


13/44

157

agenda reached the levels of some 40% of the eligible age group! In some ways, being younger thanmany academic staff the new lecturer will be nearer to their students and perhaps understand betterwhat their backgrounds might really be. On the other hand the chances are that the new lecturer willcome from an institution with highly qualified entrants where the standards are much higher than forthe students they have to teach. Then older more experienced hands may have a better idea of what thebase-line is. Either way, we need to familiarise ourselves with the real abilities of our students and pitchour assessment accordingly. This is not lowering standards, but good teaching (Principle 5).

5.5.4 Validity and Reliability of Assessment Tasks

Ensuring that an assessment task measures the skills it is intended to (validity), and that it will do soconsistently in different situations (reliability) is important and difficult. These days we usually havemoderators, checkers, collaborators, etc to help us in setting our examination papers, and this may helpimprove validity and reliability. Other practical measures that have been suggested are given below, mostof these being self-evident in the context of mathematics.

Improving validity:

matching assessment to the learning objectives testing a wider range of objectives using a number of assessments that can be compared against each other testing under secure conditions to avoid cheating improving the reliability of the assessment.

Improving reliability:

setting clear unambiguous questions. insuring questions are at the right level and standard imposing realistic time limits use a rigorous marking scheme with precise criteria use moderation to check setting and marking minimizing the choices available on an examination

increasing the range of assessment methods and the duration of the assessment.

5.5.5 Methods of Assessment

As noted earlier, methods of assessment, viewed as measuring the level of attainment of learning objec-tives (i.e. what the students have learnt), must be matched to the learning objectives, to the student profileand be practical. Across the range of academic disciplines there is a wide choice of assessment methodsto use, but often practical issues determine the choice. Challiset al [18] describe the many assessmentmethods one might use in mathematics, and discusses the functions they may fulfil.


14/44

158

Overwhelmingly, the usual assessment method in mathematics is the unseen time-limited examination.There is a great deal of criticism of such assessment in the literature, but as mentioned earlier, despiteclaims to the contrary, a properly set unseen examination of sufficient duration can assess any cognitiveskill. No one would suggest that Olympiad test papers are in some way deficient in exercising cognitiveskills!

Between 1998 and 2000 the Quality Assurance Agency conducted what turned out to be the last detaileddepartment by department evaluation of mathematics provision in the UK [27]. This was the most thor-ough and detailed investigation of what mathematics departments did for their students in the UK, andit produced volumes of useful information. One of the aspects studied was student assessment [8], andin this there were many comments about the range of assessment, probably because this is regardedas one of the desirable features of good effective assessment in some disciplines. The argument is thatthere should be a range of different types of assessment methods, to assess the various types of skills onewishes to develop. In some subjects this principle is essential and has long been employed. For example,in a practical science such as Chemistry students have to develop practical skills in the laboratory, and sothey are also assessed in these.

In the past, mathematics has not been seen as a practical subject and has traditionally been examined byunseen examinations usually lasting three hours (Or module examination of 90 minutes, while Interna-tional Mathematical Olympiad examinations, for example, range from three to five hours). But in recentyears, with increased use of computers in core mathematical subjects, and the increasing emphasis ontransferable skills, mathematicians are expected to develop a wider range of skills, hence the call for awider range of assessment methods. However, it is not the wider rangeper sethat is necessary, but theneed to match assessment methods to the learning objectives. Thus, if a particular degree programmeis solely devoted to pure mathematics, then it is quite possible that traditional examination assessmentmethods are appropriate, although even here such things as coursework and projects are now becomingcommonplace [41]. In what follows we are going to focus mainly on the time-limited unseen examination,since that is probably what most of us will be involved with.

It is only right that in our discussion of examinations we should be aware of the reservations that some

experts have about the unseen time-limited examination as a method of assessment. The problem is thatit is easy to set bad examinations, and very difficult to set good ones, and this often brings them intodisrepute. While here we take this as an argument for thorough training (one should say education)in setting examinations, the case for replacing them should also be considered. One of the most schol-arly discussions along these lines is by Heywood ([40], page 274) who is pessimistic about traditionalexaminations. He notes that:

other than objective tests and grading systems, examinations have been shown to be relativelyunreliable

apart from errors in scoring, assessors are not always agreed on the purposes of testing or whatthey should be testing

question setting is often treated as an art rather than a science

few assessors have any knowledge of the fundamentals of educational measurement

little attention is paid to the way students learn or of how assessment should be designed to enhancelearning

if the processes of examining and assessment are to be improved then more explicit statements ofcriteria showing how these should influence learning will have to be made.


15/44

159

All this leads Heywood to propose a move towards outcomes based assessment. Heywood is actuallytalking about generic aspects of assessment, and his call for sharper statements of criteria may have lessforce in mathematics because of our widespread use of detailed marking schemes. But there is no doubtthat student assessment is an imperfect pseudo-science and much needs to be done to put it on a soundfoundation. This said, we still have to make the best of the current system, and any new lecturer willhave to learn to operate within that.

5.5.6 The Unseen Time Limited Examination

This, usually closed book (although there may be formulae sheets, for example), traditional form of as-sessment is easy to present, and each student has exactly the same assessment to complete. The usualformat for the unseen time-limited examination used to be the end of year three-hour paper, but withmodularisation this has sometimes been reduced, for example, to one and a half-hour examinations (Ortwo hour, particularly in the final year) for each module. Often such changes have been brought in with-out careful thought for their implications. For example, current modular examination schemes do not

always allow time for the assimilation of deep learning or for a student to demonstrate their in-depthunderstanding of a topic if the examination immediately follows the module. The assessment of higherorder thinking skills usually requires longer examinations. In this section we are however talking aboutthe unseen time-limited closed book examination in general, regardless of duration.

In putting together such a paper, we have to think about the number of questions, the choice open to thestudent, the duration of the paper and individual questions, etc. There is a wide range of formats - eg, 5from 8 three hour examination, 3 from 5 one and a half hour examination. Your department may have astandard format that you can adopt. Usually the examination is a compromise between what is desirablefor enabling the student to demonstrate what they have learnt, and what is practicable with the resourcesavailable. One could for example assess a 100 hour module with a half hour multiple choice computermarked test. This would be very efficient and save a lot of time, but it would not really give the students afair chance to show what they can do. On the other hand we could give each student a six-hour paper and

a one-hour viva. This would probably give every student a more than fair opportunity to demonstratetheir abilities, and would give a very accurate picture of what they know and can do. But it would beimpractical for most departments. The three-hour end of year exam evolved over a long period of suchcompromise and had probably got it about right (at least for mathematics). The assumption that two oneand a half examinations are the same as one three hour examination is however debatable.

One examination paper format sometimes used, particularly when we have a very wide range of studentability, such as in engineering mathematics classes, is the Section A and B type paper (See Appendix1). In this Section A contains a largish number of relatively straightforward mainly K type questionsspreading across the full syllabus, and all these questions have to be attempted. Then Section B containsa smaller number of longer, harder questions, from which there is some limited choice, and which containsignificantIT components. This gives the weaker students the opportunity to demonstrate some rangeof basic understanding across the module, while still providing a challenge for the stronger students.

Once an overall paper format has been decided we have to design the questions to test what we expectthe students to have learnt. This involves a lot of professional judgement and experience. Here we canonly give some rough ideas, and by far your best input on this will be from discussions with as manycolleagues as possible, looking at past papers, or those of other institutions. And of course as a studentyourself you will have seen many examination papers. But, this said, when you actually come to put thepaper together your main thought should be for the students you have actually taught and for whom theexamination is intended. It is not good practice to take questions off the peg, from books or other papers- they may be used for ideas, but should be rewritten with your students in mind. The questions shouldbe your questions, for your module, for your students. While you are designing the questions you will


16/44

160

probably have lots of ideas that you cant use all at once - just bank them for future use.

Overall the paper should cover the bulk of the main ideas of the module, and should test a range ofcognitive skills. How you measure this and ensure reasonable coverage is again a matter of judgement.

Using something like the KIT categorisation most questions should have a fair distribution of these typesof skills, and the paper as a whole should be structured to ensure that a high mark is only possible withconsiderable evidence of higher order skills. This can be a difficult balancing act.

ExampleSuppose you are examining a first year methods course comprising advanced calculus, dif-ferential equations, complex numbers, matrix theory, and vectors. Each of these topics wouldhave to have a corresponding question or part of a question. Also across the paper as a wholethe student should be called upon to demonstrate each type of skill from KIT. Itis well knownthat, at least at the elementary level, it is difficult to set high I and T questions in matrix andvector algebra, and equally difficult to set low I and T questions in calculus. As a consequence,if the paper is not carefully set it is quite possible for a student to concentrate on say the alge-

braic questions and achieve a good mark with relatively little use of higher order skills, andhaving done little calculus. One way round this is to mix algebra and calculus in a singlequestion, but then it is difficult to fit in significant higher order skills in both topics. This diffi-cult issue is ultimately a matter for your judgement, and whatever departmental mechanismsthere are for supporting you in designing questions.

A few general examples might help. Thus a typical basic knowledge objective question testing littlemore than memory would have mostly K, a good applied question, testing applications would havesomeK and a lot ofT. A good all round question might have a fair spread. A standard type of questionmight have say 8/20 K, 6/20 I, and 6/20 T. This is a sort of quantification of what an experienced lecturermight suggest - Start the question with some routine material to give all the students a chance, havesome more substantial material of medium difficulty for the good students, and them something hard forthe very best. It is a lecturers informed professional judgement that must decide such matters and for

example would take into account what (s)he has done in class with the students. Thus, as noted earlierthe proof of the irrationality of

2would be mainlyK if done and drilled in class, but mainly I if only

sketched and requiring much work from the student. Related to the format of the paper is the questionof what aids the students are allowed in the examination. If they are allowed a formulae sheet then thiswill influence the types of questions that can be set. For example we cannot set straight Kquestions suchas the derivative of a trig function if that is actually on the formulae sheet.

Once you have constructed questions and the examination paper to cover your required learning out-comes a final touch you might consider to the paper is to indicate the marks awarded for each part ofa question. This is now standard at A-level, and is becoming commonplace in HE, indeed it may evenbe departmental policy. My personal opinion however is that is one of the major culprits in eroding themathematical skills of students. It encourages the bitty approach we see so often in our students, themark hunting that replaces real learning. In a question for which a correct solution will take a good

student say 30 minutes I see no educational value in indicating the marks allowed for each part of thequestion - indeed the question should not be in parts at all.

ExampleIn some A-level papers there is a tendency to replace a straightforward question such asSketch the function f(x) = x3 3x2 x+ 3, indicating any intercepts with the axes andthe location of any stationary values with a sequence of sub-questions such as:

i) Factorize the functionf(x) = x3 3x2 x+ 3and hence determine its roots [4]


17/44

161

ii) Determine the stationary values off(x)[6]

iii) Discuss the behaviour of the functionf(x)for small and for large values ofx[4]

iv) Sketch the functionf(x)[6]

The marks are illustrated for each part of the question. Clearly this form of the question ismuch easier to answer than the shorter version. It leads the student through the question andremoves completely any skills of synthesis required to put the different steps together. It isoften argued that it helps the weaker student and gives them the opportunity to demonstrateat least parts of what they can do. But this is a misconception of what such questions aremeant to test. What it in fact does is replace higher order thinking and problem solving skills(Iand T) by low level routine tasks (K). It fails completely in educating the student, even theweak student, and provides them with nothing more than bitty, isolated techniques, whichwill easily be forgotten because the student has not had to initiate their use.

To illustrate the above ideas the next two subsections will look at some example papers in detail.

Exercise

Survey as many examination papers as you can, if possible from other institutions and on a range ofmathematical modules, not only those similar to yours. In each case try to determine the skills thatthe examiner is attempting to assess.

5.5.7 Example - a Sample Examination Paper for Service Mathematics

In Appendix 1 an examination paper for first year engineering mathematics is reproduced. In Appendix2 is the corresponding module description and in Appendix 3 the model solutions and marking schemeare given. Appendix 4 contains the scripts of three students. It is not suggested that these appendices

represent good practice that should be emulated, setting examination papers is and should be, withinprofessional reason a personal thing. Rather it is intended as a case history on which to base discussionand criticism illustrating points made in the text. The more such papers you discuss with your colleaguesand department the better.

The module is designed to consolidate students background knowledge to help in the transition to uni-versity. The entry requirement is A-level mathematics grade E or equivalent, but the actual range ofbackground knowledge is very wide and varies greatly from student to student. Also, the subsequentneeds of the students vary greatly - some will go on to relatively mathematical rich programmes such aselectronic engineering and some will proceed to programmes with modest mathematical requirements,such as some chemistry or biology programmes. So the examination has to be designed to ensure thata certain basic level of mathematics is assessed, while students who need them can demonstrate higherlevel skills. These requirements determine the rubric. There is a compulsory Section A earning 50% of

the marks and then a Section B has a selection of three from five longer questions. It is not difficult forthe conscientious student to pass this examination (40%) so long as they have command of basic skills inalgebra, trigonometry and calculus, but it is also not easy to get 100% even for the best students. No aidsof any kind are allowed, no calculators or formulae sheets, so what the student produces is what theyreally know. Pretests give a good idea of the incoming students background and the examination canbe used to measure the added value during the module. A substantial proportion of the incoming stu-dents have non-standard vocational equivalents to A-Level mathematics, or lower qualifications suchas BTEC and their mathematics is sometimes relatively weak. This accounts for the elementary natureof the requirements in Section A particularly. As noted earlier, this is not a lowering of standards. We


18/44

162

accept these students in full knowledge of their background and it is our job to provide curriculum andassessment that takes account of this.

One of the great problems with this sort of module is ensuring that well prepared students cant rely

entirely on what they have met before coming to university, since most of the material revises what theyhave already seen at school. For example in the binomial theorem we want to wean them off the Pascalstriangle. And in calculus we want them to develop sufficient facility that they no longer need to rely onexplicit formulae for things such as the product rule. This is reflected in the mark scheme - for exampleno marks are given for the use of Pascals triangle, and time allocated for questions is intended to assumethat they have the required speed and facility. Of course, it is impossible to be sure that all this works forall students, but at least the attempt is made.

There is no breakdown of marks on the questions. As noted above I regard that as bad practice, encourag-ing a fragmented approach to learning and teaching. Even first year students should be able to cope witha 30 minute question without having to be led through it. In fact, itemising marks simply distracts thestudent, especially since they only have ninety minutes. They will only waste time in trying to approacha question strategically, weighing up which bits to try according to the marks allocated. And then there

is no guarantee that they will get it right. Also it is an essential skill for them to be able to mentally skimthrough a question and assess what they can and cant do, and how much that contributes to the overallsolution. Certainly, when they go into the workplace they wont be given tasks broken down into Somuch for this part, so much for that, they will just be given a job and told to get on with it. The introduc-tion of part marks on examination papers has probably contributed more than any other innovation tothe lack of learning of higher order skills in schools (and now universities). Also, the higher the order ofskill you try to assess in an examination, the greater the likelihood that you will get the marks allocationwrong and may need to adjust your marking scheme after the event when you find a question is too hardor too easy.

There are some points worth making about particular questions. For example, Question 11 is not a goodquestion because if students cannot do the first part, or make a slight slip then that effectively denies themfrom the rest of the question. This is a common mistake in setting papers. It could have been avoided by

actually giving the answer to the first part in the show that form. However, the good student will checkthe result they obtain by multiplying the results out, if they sense something is wrong for the subsequentparts. In this sense one could regard the question as it is as largelyIT. Question 12 is straightforwardand may appear to be rather generous, but in fact these students have great difficulties with the infinitebinomial series and its applications and very few attempt or complete such questions (It is interestingthat many foreign students cope easily with such things however).

Question 13 might not look it, but is all IT. In the module a point is made with trigonometric identitiesthat they only need to remember(very well!) two - Pythagorean and the compound angle identity forsine. The students are then encouraged to derive all the others themselves, so they are not trotting out roteproofs in this question. This is different to what most of the students will have seen at school where theytend to use formulae sheets or booklets of identities. Here they are not expected to learn a lot of differentidentities, rather they are expected to learn one or two very well and then develop the higher order skills

to deduce the rest for themselves. Question 14 is quite hard for students at this level. Although they haveseen this sort of question - doing integration by parts twice to evaluate an integral - in class, it is unlikelythat they would have remembered it and they will find the plethora of symbols difficult to deal with.However, even if they cannot do the first part, they can still get marks for the fairly routine ITof the lastpart. Finally, Question 15 is largely IT since the students have to decide on the methods to use for quitesimilar looking integrals.I3is a test ofTfromI2but is the sort of thing that few students at this level cancope with.

The important point from the above discussion is that to most lecturers the questions on this paper willappear elementary and all very much K and routine, but viewed from the students perspective, taking


19/44

163

into account their limited background and the way they have been taught, the skills involved are in somecases quite sophisticated.

5.5.8 Example - a Sample Examination Paper for Pure Mathematics

Appendices 5-7 give an examination paper in Abstract Algebra, the module specification and solutionsand marking scheme. Three student scripts are given in Appendix 8. This is a first course in AbstractAlgebra taken by mathematics students in their second year. This is another example of matching theassessment to the level of the students. Even students with good A-levels have (at least in the UK) hadlittle exposure to abstract structures and ideas. Some are not even capable of separating inductive anddeductive thinking and will sometimes prove things by giving an example (They are not so quick atdisproving things by giving a counter example however!). So they have to be eased into this sort of workvery gently. There is certainly no remorseless definition, theorem, proof format. They only meet keytheorems, such as Lagranges Theorem, and the proofs of these are dealt with fully and carefully. Also,they get practice on fairly simple proofs such as the uniqueness of the identity. The overall objective

of this course is to expose them to the basic ideas of Abstract Algebra - elementary formal set theory,relations and functions, simple algebraic structures, with the focus mainly on groups. A large proportionof the course is spent on an important application intended to fire the students imagination and thatdemonstrates clearly how applicable abstract algebra really is - the RSA Encryption algorithm. Most ofthe students are excited by the fact that with what they have learnt on the module they can understandthis very powerful coding system, can use it to code and decode and can prove all the necessary theoremson which it is based.

For many of the students this module is the most sophisticated mathematics that they have met to dateand is more abstract than they normally meet. So to ease it in gently most of the abstract ideas areintroduced and exemplified through concrete numerical examples, and this is reflected in the examina-tion questions. In fact, there is relatively little straight K in this subject - definitions, standard theoremsand proofs, but demonstrating understanding of these and applications is largely IT. Thus, Question 1

commences with a confidence boosting straightforward exercise using set operations on simple sets ofnumbers. This is routineK. However, this is followed by formal proof and some riders, all of which isIT, as it is unlikely that they will have memorised the particular proofs involved. Predictably, few stu-dents complete this question, despite the easy start. Equivalence relations, examined in Question 2 causeproblems for many students, and again they do better on concrete numerical examples such as modulararithmetic. There is no harm in catering for this. The students have two years of mathematics ahead ofthem where formal abstract thinking will be extended and consolidated, and this module is only a start.In fact, most of Question 2 is IT, albeit at a modest level. Virtually all of Question 3 will be new to thestudents. Apart from the well-drilled proof of the uniqueness of the identity, this is largely IT again. InQuestion 4 the proof of Lagranges Theorem is (hopefully!) Kas it has been done carefully in class andthe students know they are expected to know it. The rest isIT. Question 5 splits into too halves. The firstpart is largelyIT, since the students really have to understand the RSA algorithm to be able to express itin their own words (Some regurgitate the summary of the actual algorithm, but few marks are awarded

for this). The second half is, provided the students have thoroughly mastered the algorithm, mainlyK,and the students invariably get most of their marks on this part.


20/44

164

Exercises

Draft an examination paper for one of your modules, giving thought to the issues raised above.

Compare at least five examination papers from courses in your department, with theappropriate module specifications and marking schemes. Analyse the papers bearing in mindthe issues discussed in this section. If possible discuss with the setters of the papers.

5.5.9 Administrative and Organisational Matters

The traditional end of year/module examination gives the impression that we only need to think aboutassessment near the end of the course or module. This is far from the case however - assessment isone of the prime considerations during the planning of the module. As soon as we have decided on thelearning objectives we have to start thinking about how and when we will assess the students. Apart from

the actual intellectual aspects of the assessment this also includes a great deal of tiresome but essentialadministrative drudgery to complete, and we will get this out of the way in this section.

There are two main administrative and organizational aspects to attend to in assessment. Firstly thereare the regulatory arrangements that your department/institution have for the assessment process, withschedules, paper-trails, etc. All this should come to you via your institutional induction and/or throughyour department assessment officer, so what we say here is a summary of the sorts of things that mostof us have to deal with in this respect. However, it covers the main stages that most assessment regimeswould have to conform to these days, and the checklist below can be easily adapted to your own envi-ronment if you find it useful.

The main milestones are:

match assessment to objectives during course planning

decide on the nature, weighting and timing of the various components of assessment find out the dates and venues for submission of examination papers, examinations, and examina-

tion boards

inform students in writing about the assessment process prepare assessment, with solutions, marking schemes, etc submit papers to examinations officer/ external examiner for moderation respond to moderation of papers

attend examinations/invigilations mark examination scripts/assessment outcomes forward results to appropriate officer attend examination boards learn any lessons from the outcomes other - that we have omitted and that you can add yourself.


21/44

165

Secondly there is the process of informing the students on all relevant aspects of the assessment process.This is very important and you should make sure that they have all the information they need. Examplesof the sort of information that the students should have, in writing, are:

the units of assessment of their degree programme the status of your course/module as a unit of assessment the mode of assessment of your course (e.g. coursework/exam) the examination requirements (duration of examination, rubric, etc) the relation between units of assessment and the final degree classification format of the examination or other mode of assessment materials allowed in the examination or other mode of assessment

date, time, location of examination date, time location of publication of results assessment schedule for coursework coursework feedback on coursework Other?

One stressful administrative issue that might arise is what to do when a student disputes the way in whichyou have marked their work. Now of course this depends on the degree of the dispute. But, if it cant

be settled amicably refer to the departmental processes for dealing with such things. Dont be bulliedor intimidated, but also dont be dismissive of the students case. There might be a cultural componenthere - some nationalities consider it reasonable to barter and negotiate for their marks - others believe themarkers word is law and wouldnt dream of questioning your decisions! Of course, you should alwaysbe prepared and able to justify your own marking decisions. In fact, a good test of the rapport you havewith the students is how smoothly such matters can be dealt with.

Exercise

Devise a checklist for your own purposes, enabling you to keep track of the administrative andorganizational requirements for your assessments, such as course-work and exams.

5.6 Marking and Moderation

Marking time, the moment of truth. Most of us become apprehensive at this time, not only because of theintense work required, but because we worry about how our students have done. You dread that yourpaper was too hard, or too easy, and you will get embarrassing results - average of 80% or, more likely,60% failure rate. When the students scripts arrive on your desk you cant help but feel moved - one wayor the other! In this section we are going to look at writing marking schemes and marking the traditionalexamination.


22/44

166

5.6.1 Writing Marking Schemes

The Learning and Teaching Support Network Generic Centres Assessment Series 3 [13] gives a lot ofadvice on writing criteria for assessment, which in the case of mathematics usually takes the form of amarking scheme. The essentials of good criteria are that they:

match the assessment tasks and learning objectives enable consistency of marking can pinpoint areas of disagreement between assessors can be used to provide useful feedback to students.

In fact, the generic literature on criteria is confused and contradictory, and less than helpful for practi-cal purposes. It is similar to the issue of taxonomies of learning objectives - there are a lot of differentapproaches, and some contradictory advice in the literature. Your department may have a standard

format/convention/practice, which may make the choices easier for you. Here we will suggest quite de-tailed practical advice on writing the standard types of mathematics marking schemes, which hopefullywill be easily adapted to your own needs. Essentially, we are trying to allocate marks/reward to differentparts of questions in a way that reflects what we value about the skills that have been learned by thestudents. There is however a range of opinion about the need and the format for this.

Firstly, some would argue that one only needs a quick run through the solution to check it is satisfactory- in many cases this can be done in a few scribbles by the experienced lecturer. The production of a de-tailed marking scheme is held as being unnecessary and even impossible because students may producedifferent solutions. You may want to change the scheme with hindsight after the examination, it is notalways clear how best to allocate marks to parts of questions, and a global overview of what studentshave done is necessary before deciding on this. There are elements of sense in all this, but there are alsogood reasons why it is better to at least produce a first draft of a scheme. It is acceptable to fine tune the

scheme after seeing what the students have done, and usually only a small proportion of students willproduce a completely different version of the solution for which you may have to bin the scheme andtreat the case on its merits.

One advantage of preparing a marking scheme is that it should force you to think through the solution asthe students are likely to do it in the light of what you have taught them. This focuses your mind on howdifficult it really is for them and exactly what it tests. Also, it is a way of communicating to a third party,such as the examination moderator, or the external examiner, exactly what is expected from the students,what you have done with them, what value you place on various ideas, concepts, techniques, skills, etc.And of course it is now standard practice to provide marking schemes for internal and external qualityreview processes.

So how does one construct a mark scheme for a given question? It is quite possible for a moderator todisagree with aspects of any scheme that you produce. They might say too many marks for this, too few

for that, etc. Now, in a way, they are placed in a difficult position, because they havent sat through yourteaching, and they dont know what you have covered with the students, or what you expect from them.However, if you have stated the learning objectives for the module and the criteria for assessment thenit should be easier for a moderator to understand your intentions. The learning objectives and criteriashould be sufficiently transparent for a fellow mathematician, or for the students, to judge whether theexamination covers appropriate material. But there is more to it than that. We all know that one of thefirst questions of clarification a moderator will ask is Have you covered this is the lectures - have theyseen an example like this before? Usually this is because they are concerned about the difficulty of partof a question. For example, if part of a question asked for the proof of the result for the sum of the first n


23/44


24/44

168

learning objectives are covered ([7], page 194; [63], page 14). Your part marks will depend on this, andon how much you expect students to develop each skill. Again, this may take a number of iterations.And you will need to compare this across the different questions, as well as across the possible choicesthe students can make ([7], page 194, classifies the difficulty of examinations using a similar idea) to getan overview of what various overall marks for the paper would mean. For example if a paper asks for 3questions out of five, and two of the questions have 12/20 marks for Kalone, then a student could pass(at 40%) this paper employing no HOTS at all. So allocation of marks is clearly a delicate optimisationproblem, guided by common sense!

Having done all this, tidy up the paper, the solutions, with your mark scheme, and ask someone else tolook at it. If it passes this hurdle, then it can go off to the examinations office. But, that may not be the endof it. It takes a lot of experience to set a good, fair, challenging paper that wont result in a high failurerate (above 10%). You dont really know how fair a paper is until you see how the students find it, atwhich point you should be prepared, in consultation with colleagues, to change your views if need be!

Finally some summary advice for writing mark schemes:

consult other marking schemes to get ideas first write out a model solution - as the student would do it allocate each mark so that it is associated with something that is there or not there, right or wrong check the overall balance across the paper aim for a scheme that is usable by the educated layman make it reliable, so that anyone can mark the answers and broadly agree on the marks penalise once only for each error - carry through consequential marks pilot the marking scheme be flexible in accepting solutions that dont fit into the marking scheme.

A frequent concern in constructing a marking scheme is the level of detail to employ. For sake of argumentlet us focus on a typical half-hour examination question, and suppose we have to award this a total of20 marks. This is a common level of award, the usual intention being that one would only award to thenearest integer. This is about the maximum level of resolution that is possible when assessing how thestudents have performed on such a question - and some would argue that this is too optimistic. Of coursethis can still lead to problems. The most trivial mathematical statement can be broken down into partsalmost indefinitely and we might feel a need to split our mark scheme further. Dont bother - assessmentis simply not that exact a science and if our mark scheme is a blunt instrument there is little point tryingto sharpen it! The same applies when we actually come to mark the scripts - see later.

5.6.2 Example of a Marking Scheme in Service Teaching

In Appendix 3 we give the marking scheme for the Engineering Mathematics examination of Appendix1. Again it is not intended to demonstrate good practice, but simply to form a basis for discussion. Asalways the dry numbers conceal a great deal of hidden assumption and background knowledge of thestudents, and here we make a few of the more important points.

Most Section A questions are not broken down in submarks. These questions are so straightforwardthat one can view attempts at them overall and make judgements on the hoof. As we will see later, the


25/44

169

variation in approach to these questions, and the types of errors made can be so wide that any finer markscheme would simply get in the way. Questions in Section B are broken down into clear submarks, butthis will not always be appropriate for all students solutions, so can be overridden by judgement onthe fly, if need be. Also, there are a number of hidden assumptions of which only the setter would beaware. For example, in Question 6 it is well known that many students will leave the answer in the formsin(2x+x)- and indeed if you ask them to differentiate something like this some will retain the 2x+xsplit in the process. Now although it is not explicitly stated in the mark scheme I would certainly deducta mark for this. The point is that no marking scheme can cover all such eventualities.

In Section B questions the breakdown of marks is very crude. Question 15 nicely illustrates the folly ofputting part marks on the examination paper. Part of the test here is to see whether the student has anoverall appreciation of the different methods involved and can judge for themselves the likely amountof work in each case. How does it benefit the student to know that there are the same marks forI2 andI3? Do they then assume they are equally difficult, which for most students they are not, of course? Andin some cases displaying the marks can give away the game - for example when a question involvesevaluating contour integrals by a range of methods such as Cauchys integral formulae, parametrisationor Cauchys Theorem. The last is much the easiest and if you reveal that by displaying the marks thestudents will know to use that method. We are assessing the students understanding of mathematics,not their strategic skills in harvesting marks.

Overall then, a marking scheme has a significant personal dimension, informed by what and how youhave taught the material, what you expect from your students, what you know about them and so on.And it is a framework only, not a straitjacket.

5.6.3 Example of a Marking Scheme in Abstract Algebra

Appendix 7 gives the marking scheme for the Abstract Algebra paper of Appendix 5. This is quite de-tailed in some cases, reflecting specific tasks set in each question. It does make the marking harder, buton the other hand we are looking for quite specific knowledge and skills. The marks are fairly self-

explanatory. Some people might break down the first part of Question 5 into more detail as one can withessay questions - listing the points required, perhaps allocating some marks for clarity and presentation,etc. However experience shows that it is very difficult to bend such schemes to fit the attempts of differentindividuals - it is better to treat each attempt on its merits and rely on overall judgement for consistencyand fairness. In any case, marking essays is notoriously unreliable.

The overriding point about the above examples is that the marking scheme is very personal and reflectsthe setters interests and priorities, always tempered of course by the usual collegial agreements on whatthe students need to know and be able to do. For example in a first year methods class a lecturer shortof time may decide to skip through hyperbolic functions, regarding them as not very fundamental orimportant, setting them, if at all as minor riders on an examination question. But if a subsequent course,say in partial differential equations or complex analysis, makes significant use of them then they mightneed a more thorough foundation. So, for the same questions, different lecturers may have different

marking schemes.


26/44

170

Exercises

Construct a marking scheme for the examination paper you drafted in Section 5.5 explainingyour rationale for the scheme.

Discuss the marking schemes for the five papers you looked at earlier with the setters of thepapers.

5.6.4 Marking Examination Papers

With a good marking scheme at hand marking is not too bad, except that it almost invariably has to bedone quickly and accurately, which some people find quite stressful, because after all it can mean a greatdeal to the students. You have to be fair without being too harsh or too generous (This actually dependson whether the marking really matters - for example you can afford to be a bit harsh in coursework, and

maybe less so in a final examination). You need to be sensible - time spent anguishing over a precisemark is rarely fruitful, student assessment is simply not that rigorous a process and it may be betterto step back and look at a question globally to get an overall feel for what the solution is worth. If astudent makes a minor error early on in the question which renders it more difficult to continue than youintended then try to find clues that the student did know how to proceed but got bogged down in thecalculations. This is another area where your best source of advice is discussion with your colleagues. Thesuggestions discussed here may be useful, but again you will find varying opinions between lecturers.However most will agree on one aspect of being sensible about marking, which is the policy of revisitingborderline cases. For example few would return a mark of 39%, and indeed nowadays most departmentshave automatic replacement by 40% in such cases.

Marking a question at a time (horizontal marking) improves reliability although it is wise to reorderscripts between each question so that the same students are not repeatedly victims of any weariness and

irritation you might have after marking the same question many times! On the other hand papers of shortquestions can be done in one go, and indeed this might be the most efficient method if there are a lot ofsuch questions on the paper. For example Section A of the paper in Appendix 1 could be marked in onego for each script. These days most departments have anonymous marking, at least for examinations, andthis minimises any personal bias in marking. Avoid the halo effect in marking (allocating excessive/lowmarks for reasons other than objective interpretation of the marking scheme). An example here is toavoid being unduly influenced by untidy work, poor handwriting, etc. Only up to a point of course, butif you can read the material and the student clearly knows what they are doing, then unless you havespecifically indicated that there are marks for presentation it seems churlish to penalise for such things.Also, as you mark your mood can change - dont let it influence your marking.

Sometimes you are really not able to read a students handwriting, how are you going to mark his/herwork? It depends on the degree here. One unfortunate consequence of widening participation has been a

greater variety of presentation styles,

chapter5.teaching mathematics in higher education - the basics and beyond

Documents