Evaluation

Principles and

Objectives

MEU WORKSHOP

Evaluation Principles and Objectives

Objectives

Teachinglearning methods Evaluation

Relevance

• Why

• What

• When

• Whom

• How

Is evaluation necessary?

• To make sure that the student will be able to do the job competently

• Exams encourage the students to work harder

• To guide the teacher and student about which part of the course has been successful and which part needs to be improved

Student evaluation

• Measurement: application of mathematical tools for finding the degree of achievement

• Example: awarding marks for MCQ

Assessment

• This term is used for those attributes which do not lend themselves to precise measurement and where some subjective decisions are involved

• Example: marking of essay type questions

Evaluation

• Involves passing a value judgement based on the information obtained from measurement and assessment

• It is a major component of the curriculum

Functions of evaluation

• Feedback for improved learning

• Predictive

• Selection

• Grading

• Program evaluation

Types of evaluation

• Formative: helps the learner and the teacher to know the progress of the student

• Summative: end of term evaluation

What is the role of internal assessment?

Abilities to be evaluated

• Evaluate all three domains of learning

• Weightage allotted to each component will vary depending on the subject area under consideration

• Evaluation is not the end – rather it is the means to further the effectiveness of an educational programme

• It has to be planned keeping the educational objective in mind

Evaluation tools

• Should be appropriate for the learning outcomes to be evaluated

• The characteristics of an evaluation tool are:• Validity

• Reliability

• Feasibility

• Relevance

• objectivity

Validity

• A tool is valid if it measures what it is supposed to measure

• Example: to determine whether a student can prepare a thin blood smear.

Types of validity

• Content validity

• Criterion validity and

• Construct validity

• Content validity:

• indicates synchronization between content of the test and content of teaching

• To ensure that content validity is measured,

– prepare a list of content matter to be tested

– Assign weightage for each

– Prepare table of specification

– Create test for each

How to build-in content validity

Prepare a table of specifications

Topic Weightage

A 15%

B 10%

C 5%

D 20%

E 30%

F 20%

Total 100%

Table of specifications

• Items required 01

• Use of gloves 01

• Size of the drop of blood 01

• Location of blood drop on the slide 01

• Holding the spreader 01

• Way the smear is spread on the slide 02

• Quality of the smear 02

• How the smear is dried 01

• Max marks 10

Factors influencing validity

• Unclear directions

• Ambiguous phrasing of questions

• Inappropriate level of difficulty

• Inappropriate question for the outcome to be tested

• Too short a test

• Insufficient time

Reliability

• Consistency of measurement

• The degree of reproducibility determines the reliability of an evaluation tool

– Test-retest reliability

– Equivalent-forms

– Split halves reliability

– Marker reliability

Measures to improve reliability

• Optimum time of test

• Use appropriate levels of difficulty

• Maintain conditions of test constant

• Ensure objectivity of scoring

• Ensure validity of the instrument

• A reliable test need not necessarily be valid!

Objectivity

• Structuring of questions

• Preparing model answers

• Agreeing on the marking scheme

• Having papers independently valued by two examiners

Feasibility

• Time and resources required

Relevance

• Appropriateness of the process of evaluation with reference to the tasks to be performed by the students after qualification

• Reflect the health needs of the society

How do you choose an appropriate evaluation tool?

• Purpose

• Domain to be tested

• Number of students

• Time allotted

• Resources

• Ease of administering the test

• Ensuring validity and objectivity

Steps of evaluation

Define objectives

Provide teaching-learning experience

Select measuring instrument

Decide marking

Administer test

score test

Analyse result

Take decision

Domain-wise matrix of evaluation methods

• Cognitive domain:

– Written and oral

• Essays- Modified essays

• Short answers and very short answer questions

– Objective items

– Simulated patient management problems

• Psychomotor domain:

• Observation

• Practical Clinical

• OSPE / OSCE

• Affective Domain:

• Observation

• Rating scales

• Checklist

• Questionnaires

• Logbook

• Group discussion

What makes a good evaluation?

• Does the evaluation comply with the regulations of

the course

• Does it test important skills and abilities ( is it valid)

• Are you sure that the marks gained by each student

is accurate ( is the marking reliable)

• Does the evaluation give you information that

will help the students to learn better and help

you to improve your teaching

• Is it reasonably economical in terms of

material and time

EvaluationEssay and SAQ

Evaluation of knowledge

• As a general rule, early in the course, factual

knowledge is presented to the student and as the

course develops higher domains of learning come

into picture

Current system of examination

• Evaluation of knowledge: Theory paper:

– Two long essays 2 X 10= 20 marks

– Six short notes: 8 X 5 = 40 marks

– Short answer questions: 10 x 2 = 20 marks

Evaluation of Cognitive skills

• Commonest tool is the traditional essay question

– Unstructured essay question

• Student has the freedom within the subject context, to

determine the nature and scope of the answer

• Example: write an essay on protein energy malnutrition

• Free response questions

• Subjectivity in marking is high, hence unreliable

• Advantages:

– Evaluates higher domains of learning and comprehension

of the student

– Relatively easy to frame

– Allows students free and effective expression

– Ability to organise ideas

• Disadvantages

– Low validity and reliability

– Limited range of application

– Lack of objectivity

– Inconsistent and time consuming scoring pattern

– Hardly any feedback to the teacher

– Scope for guess work by the students

– First answer and its assessment influences the subsequent ones

Structured essay questions

• Restricted response questions

• Example: write an essay on PEM covering the causes, classification, clinical diagnosis and initial management in the hospital

Modified essay question

• Problem solving question

• Short history is given based on which questions are asked

• Requires the student to apply what he has learnt, in the context of a given situation

Problem solving question

• A 4 year old boy presents with H/O epistaxis on and off since 3 years. On examination his knee joints are swollen. Mother gives H/O similar complaints in the older sibling

• What are the possible causes for this type of presentation?

• What investigations will you order to arrive at a diagnosis?

• What advise will you give to the mother?

Guidelines to write better essay questions

• Match the question to the specific learning outcome.

• The objective of learning and objective of testing

should be the same

• The more precise and clear we are with our learning

objectives, the easier and better it will become to

formulate good essay questions

• The question should clearly specify to the student

what is expected of him

• Phrase the question in a simple, clear language

• Indicate the weightage to different parts of the

question. This helps the student to effectively plan

his answer

• Prepare a checklist and allot marks to different points

to be included in the answer

• Mark essays question by question rather than

student by student, to compare the performance of

each student on that particular question and not as a

whole

• Mark the essay question by more than one examiner

and take average marks

• Grading instead of marking will further minimize the

variation between examiners

• All students should answer the question since

construction of optimal questions of equal difficulty

is not possible. Hence comparison of scores of two

students becomes invalid

• Avoid distractions during scoring

Short answer questions

• Tests lower domains of learning

• Student may be required to write a word, phrase, a

sentence to complete a statement

• Longer the expected answer, more is the subjectivity

involved

Types of SAQ

• Completion type

• Open SAQ

Merits of SAQs

• They are easy to construct when compared to MCQs

• They are more specific than essays and hence more

reliable

• They are quicker to answer. The students can be

tested on a wide range of topics in a short period

• They can be marked quickly

• SAQs are less prone to guessing than MCQs

• It is possible to construct a checklist for responses to

ensure intermarker reliability and thus objectivity

• Time taken to answer SAQ is more than for MCQ

• Number of SAQ in a given time is reduced compared

to MCQ

• Reliability of a test is directly proportional to the

number of questions

• Subjectivity in open SAQ

Group Activity I

• Each group is given two essay questions and two short answer questions

• Discuss within your group and write your comments

• Time allotted 10 minutes

• Presentation: 4 X 6 minutes: 25 minutes

• Total time: 35 minutes

Group Activity II

• Prepare two structured essay questions

• Two short answer questions

• Time allotted 10 minutes

• Presentation by each group: 4 X 5 = 20 min

• Total time: 30 minutes

Group activity I – Group A

• Essay questions:

• 1. Write in detail about oedema

• 2. Give a comprehensive account about the thoracic part of the oesophagus

• SAQ:

• 1. DNA

• 2. Cholecystitis

Group activity I – Group B

• Essay questions:

• 1. Write what you know about family planning

• 2. Discuss the role of endocrines in the regulation of human growth

• SAQ:

• 1. Right atrium

• 2. List a few haemoparasites. Can you draw them

Group activity I – Group C

• Essay questions:

• 1. Write an essay on SLE

• 2. Amyloidosis

• SAQ:

• 1. Rifampicin

• 2. PCV tubes

Group activity I – Group D

• Essay questions:

• 1. Write an essay on Senile cataract

• 2. Inguinal hernia

• SAQ:

• 1. POEMS

• 2. R S cell