14 june 2011par symposium1 the assessment, marking and feedback inventory (amfi): a tool for...

14 June 2011 PAR Symposium 1

The assessment, marking and feedback inventory (AMFI): a tool forexploring the practitioner perspective

Lin Norton, Bill Norton, Kamel Mansi


Research Background

This earlier research only tells part of the story in two respects: 1) it concentrated on ’new’ lecturers’ 2) it only looked at their views on assessment design

The current study was designed to look more widely at the whole issue of assessment, marking and feedback with lecturers with a range of experience all working at Liverpool Hope University

Previous research carried out by the Write Now CETL research team focused on newly qualified lecturers’ views about assessment design which found that constraints could hinder sound assessment design being put into practice (Norton, Norton & Shannon, in press).


Theoretical background: the effects of the discipline Biglan (1973) classification: Hard-soft, pure applied Neumann Parry & Becher (2002): hard (pure and applied) - strongly committed to research and less

committed to teaching, (generally seen as relatively straightforward and unproblematic), collaborative research and teaching

soft (pure and applied) - greater emphasis on scholarly knowledge that translates readily into teaching, more emphasis on individualistic enquiry and not so much acceptance of joint teaching

?? Since assessment is so closely aligned to learning and teaching, might there be similar differences in lecturers’

assessment practices and attitudes ??


The concept of signature pedagogies

Lee Shulman, former president of the Carnegie Foundation for the advancement of teaching

10 year study to understand how people areprepared for practice in law, engineering, the clergy, teaching, nursing and medicine.

Argued that education for professionals has to include more than knowing and understanding the discipline is only part of what we should be teaching

Signature pedagogies are not stable and unchanging and are as likely to change as professional practices themselves


Characteristics of the signature pedagogies Shulman a mode of teaching that has become inextricably

identified with preparing people for a particular profession”. says there are three defining characteristics he describes as

1. “… it’s distinctive in that profession… 2. … it is pervasive within the curriculum. So that students

learn that as they go from course to course, there are certain continuities that thread through the program that are part of what it means to learn to “think like a lawyer,” or “think like a physician,” or “think like a priest.””

3. The third feature is another aspect of pervasiveness, which cuts across institutions and not only courses. Signature pedagogies have become essential to general pedagogy of an entire profession, as elements of instruction and of socialization.


So where is assessment in signature pedagogies? Shulman writes about teaching processes but appears silent on the

subject of assessment processes Taking the very broad concept of signature pedagogies as a proxy

for teaching process differences, then there presumably must be assessment process differences

This was found in the ADI work where Norton et al findings (in press) on assessment design:

Differences found on ‘desirable assessment practice’ factor: lecturers from soft applied disciplines scored higher on the overall total than

lecturers from soft pure disciplines, hard pure disciplines and hard applied disciplines

Differences found on ‘constraints to desirable practice ’ factor lecturers from hard pure and hard applied disciplines scored higher on the

overall total than lecturers from soft pure and soft applied


Can signature pedagogies be applied to subjects studied at Hope? We thought it could, given the strong government

agenda that university education should be linked to employability skills BUT Hope is mainly liberal arts; nevertheless we thought subjects that could be counted as professionally-orientated might include Education Music, Drama, Fine Arts Business studies, Marketing Psychology, Sociology, Health

So we were keen to analyse our AMFI results to see if there were any subject differences, but…


Research questions

1. What do lecturers think about assessment methods, marking and feedback practices?

2. Are there any differences between the subject disciplines?

3. Can a questionnaire on assessment, marking and feedback be developed that would be useful for both research and practitioner purposes?


Development of the Assessment, Marking and Feedback Inventory (AMFI)

A pilot version of the AMFI was developed by Dr Kamel Mansi (former Write Now CETL postdoctoral researcher)

This consisted of four main sections:1. methods of assessment (52 item checklist from Bloxham & Boyd

(2007))

2. rationale for choice of assessment method (13 statements)

3. marking attitudes and practices (22 statements)

4. feedback attitudes and practices (31 statements)

The AMFI was made available online to Hope lecturers. 45 completed inventories from 17 disciplines.


…not enough participants

Out of the 45 completed inventories, there were not sufficient numbers to make any meaningful comparisons throughout, as many of the subjects were represented by one or two lecturers

In subjects where there were more, such as Business (N=5), not all of them completed all the questions.

This left us with the option of assigning the subjects to broad categories of hard applied, hard pure, soft applied, soft pure


Which subjects did the participants represent?

HARD APPLIED SOFT PURE SOFT APPLIED


Section 1. Methods of assessment 52 methods presented and participants were asked to

look at each one and respond: Have used Would like to use Not Familiar No response

See handout


Assessment methods continued

Of the 52 methods of assessment presented, only one (patchwork texts) had not been used by any of the respondents, although 3 participants said they would like to use this method

Many respondents expressed interest in using methods of assessment they did not currently use.


Most popular assessment methods used (N=45)

Essay 98%

Dissertation 93%

Research project 93%Presentation 91%Reflective learning assignments 80%In-class tests 76%Examinations (unseen, seen, open-book, case study, take-away) 75%Portfolio (written) 71%

A study we did at Hope, eight years ago, with 22 lecturers from Biology, Information Management & Communications, Management & Business, Sociology, and, Psychology showed most common methods of assessment being used then were: exams, essays, class tests, group presentations, reports, & time-constrained case studies. There were some differences between these five subjects but all of them had some form of exam (Steward et al, 2003)


Most popular assessment methods respondents would like to use

Method N=45

Electronic presentation (web pages) 38%

Writing abstracts 31%

Debate speech 29%

Film or radio programmes 29%

Simulation exercises 24%

Computer based assignment 22%

Annotated bibliography 22%

Fieldwork reports 20%

Review of book, article, website, etc. 20%

Placement reports 20%


Section 2: Rationale for choice of assessment method 13 items were presented, e.g.

Takes into account students’ preferences Prepares students for employment

Participants were asked to rate how important they felt each item was: Very important Important Not important Completely unimportant


Most important rationales for choice of assessment & comparison of subjects

Very important/important Overall(45)

Drama(4)

Education(7)

Provides a sample of what students are able to do with the knowledge they have acquired

98 75 (3) 100 (7)

Develops students’ writing skills 97 100 100

Prepares students for employment 90 75 (3) 86 (6)

Prevents students from plagiarising 89 100 (4) 86 (6)

Allows students to develop oral as well as written skills

86 100 (4) 72 (5)


Section 3: Marking attitudes and practices

This section contained 22 items e.g. The external examiner system does not ensure grading fairness

(attitude) I have learned to mark myself without any professional training

(practice)

Participants were asked to respond to each one by choosing one of the following responses: Strongly agree Agree Disagree Strongly disagree


Marking items with strongest levels of agreement (& comparison between subjects)Strongly Agree/Agree Overall

(45)Drama

(4)Education

(7)I have learned from my colleagues how to mark (practice)

93 100 (4) 71 (5)

It is important to maintain objective standards in marking (attitude)

93 100 100

Grading student assignments is open to subjective interpretation (attitude)

88 75 (3) 100 (7)

Marking is a lecturer’s own professional judgement (attitude)

86 75 (3) 71 (5)

Certification/grading shows that students have achieved a certain level (attitude)

85 100 (4) 57 (4)

Marking is not an exact science (attitude)

85 50 (2) 86 (6)


Section 4: Feedback attitudes and practices

This section of the AMFI contained 31 statements e.g. Face to face communication with students is the best way to give

feedback (attitude) I seek advice from colleagues for providing clear information to

students (practice)

Participants were asked to respond to each one by choosing one of the following responses: Strongly agree Agree Disagree Strongly disagree


Feedback items with strongest levels of agreement (& comparison between subjects)Strongly Agree/Agree Overall

(45)Drama

(4)Education

(7)

I have developed my feedback skills through experience (P) 100 100 100

Students should receive feedback regularly during the academic year (A)

100 100 100

Feedback is as important as a grade (A) 97 100 (4) 86 (6)

The best formula for giving feedback is to start positively, then give negative comments and end on positive note (A)

96 100 100

I comply with the external examiner’s requirements on providing good feedback (P)

95 100 100

External examiners should read the feedback provided to students (A)

95 100 100

Knowing students personally helps lecturers give constructive feedback (A)

82 75 (3) 43 (3)

Face to face communication with students is the best way to give feedback (A)

80 100 (4) 58 (4)


Some thoughts on these preliminaryfindings

Disappointed we didn’t have bigger numbers to draw conclusions about differences between several subjects

The two subjects we did have (Drama and Education) did show some distinct differences in methods of assessment, and in their rationale for these methods which might suggest the existence of signature pedagogies BUT the numbers are very small

When it came to marking and feedback, again there were a number of differences (particularly in attitudes) which supports previous research that stresses the primacy of the subject discipline in our

professional practice as lecturers ( Becher & Trowler, 2001; Knight & Trowler, 2000; Neumann, Parry & Becher, 2002)

Will be looking to further analyse these data, using Biglan’s (1973) broader categorisation of hard/soft and pure/applied


References Becher, T. and Trowler, P.R. (2001) Academic Tribes and Territories: intellectual

enquiry and the cultures of disciplines (2nd edition). Open University Press/SRHE, Buckingham.

Biglan, A. (1973) The characteristics of subject matter in academic areas. Journal of Applied Psychology, 57, 195-203

Bloxham, S. and Boyd, P. (2007). Developing Effective Assessment in Higher Education. Open University Press.

Knight, P.T. and Trowler, P.R. (2000) Department-level cultures and the improvement of learning and teaching, Studies in Higher Education, 25, 1, 69-83.

Neumann, R., Parry, S. & Becher, T. (2002) Teaching and learning in their disciplinary contexts: A conceptual analysis, Studies in Higher Education, 27, 4, 405-417.

Norton, L., Norton, B. & Shannon, L. (in press) The Assessment Design Inventory: A tool for research & staff development. In C..Rust (Ed) Improving Student Learning: Proceedings of the 2010 ISSOTL/ISL conference. Oxford: Oxford Centre for Staff and Learning Development.

Shulman, L (2005) Signature pedagogies in the professions, Daedalus,134,3,52-59 Steward, S., Norton, L.S., Evans, I. & Norton, J.C.W. (2003) Lecturers! What are you

assessing? Paper given at the Learning & Skills Research Network Annual Conference ‘Research for all’ GMB National College, Manchester, 6 June, 2003

14 june 2011par symposium1 the assessment, marking and feedback inventory (amfi): a tool for...

Documents

teaching processes

advancement of teaching

mode of teaching

signature pedagogies

lecturers assessment

issue of assessment

teaching process differences

sound assessment design