EDM 311 – Educational Testing, Measurement and Program Evaluation

Karen Luz Y. Teves, PhD. kartevs@yahoo.ca 

Course DescriptionThis graduate course is about the concepts, principles, and procedures of measurement and evaluation, proper classroom test construction and administration, criterion-referenced and norm-referenced measures, technical and practical issues related to test development and use, marking procedure/grading systems, testing program and the use of statistical procedures in evaluation.

•  

Learning ObjectivesThe graduate student should, upon completion of the course, be able

to : 

1. understand the nature, characteristics, differences, and uses of measurement and evaluation.

2. Understand the types of tests and uses of norm-referenced and criterion referenced tests.

3. learn and be able to apply the learnings of proper construction and administration of tests

4. understand the purpose of item analysis and be able to demonstrate the basic skills and interpretation of item analysis results.

5. understand the different types and uses of assessment and demonstrate ability to conceptualize, construct, and score alternative assessments.

6. learn and use basic statistical concepts, calculations, and analyses

7. identify and practice the qualities desired in measurement and assessment procedures

8. understand and appropriately apply various marking systems within the context of classroom assessment.

9. understand the major strategies for: assigning student academic achievement grades, factors that influence grading, and the influence of student assessment strategies on student performance

10. interpret the meaning of derived scores from standardized tests and other measures and review test content with respect to intended use.

11. distinguish between measurements of validity and reliability and strategies for estimating validity and reliability.

Learning Objectives

Course ContentI. Introduction to Educational Assessment,

Measurement and EvaluationII. Tests, their Uses in Educational Assessment,

Constructing, Administering and Scoring of TestsIII. Assessment of Cognitive, Psychomotor and

affective Domains of learningIV. Test Item Data Use and AnalysisV. Grading, Correlating, Organizing, Summarizing

Test Scores and Test ReliabilityVI. Assessing Student Outcomes, Grading and

Reporting Student Performance

Learning Activities and Requirements

1. Class Participation and Practice Exercises – in between/after the lecture, the professor will design different types of activities for such as topical situation critique-discussion, practice exercises from selected data sets, testing and statistical computations.

2. Development of a Criterion Referenced Test - students will develop a Criterion Referenced Test for use in the subject, grade/year level and topic of their choice following the practices and methods as discussed in the lectures. A rubric will be provided for students to follow in the design of their tests.

3. Development of an Alternative Assessment - students will develop an Alternative Assessment for use in the subject, grade/year level and topic of their choice following the practices and methods as discussed in the lectures. A rubric will be provided for students to follow in the design of their assessments.

4. Pass the Final Examination.

Educational Assessment, Measurement and Evaluation - An Introduction Measurement – in education, is the quantification of what

students learned through the use of tests, questionnaires, rating scales, checklists and other devices.

E.g. – A teacher gave his class a 10-item quiz is undertaking measurement of what his students learned from the lesson

Assessment – the full range of information gathered and synthesized by teachers about their students and their classrooms by observation, giving of assignments, tests, report/output

Evaluation – a process of making judgements, assigning value or deciding on the worth of students’ performance

E.g. – teacher assigns a grade to the quiz score obtained by the student

Test: 1. Measurement instrument 2. Designed to elicit specific sample of behavior

Measurement: 1. Quantification: assigning numbers

2. Characteristics: abilities, traits, attributes, constructs

3. Rules and procedures: must be replicable

Measurement – how much does a student learn or know?

Assessment – how much change has occurred on the student’s acquisition of skill, knowledge or value before and after a given learning experience?

Evaluation – how good, adequate or desirable is the student’s performance?

Evaluation: 1. Systematic 2. Gathering of information 3. Making decisions: value judgments

Measurement = Testing+ Quantitative Data

Evaluation = Testing + Quantitative Data + Qualitative Data + Judgment (Bachman, 1990)

Purposes of Educational Assessment, Measurement and Evaluation

1. Improvement of student learning2. Identification of students’ strengths/weaknesses3. Verification of accomplishment of an instructional

objective through the use of a particular teaching strategy4. Verification of effectiveness of curriculum5. Assessment and improvement of teaching effectiveness6. Communication with and involvement of parents in their

children’s learning7. To diagnose nature of difficulties8. To measure student’s achievement9. Prediction of an individual’s level of achievement in future activities or

predict one measure from another measure10. Motivation

Types of evaluation

1- Formative evaluations: It is an ongoing classroom

process that keeps students and educators informed of students’ progress toward program learning objectives.

The main purpose of formative evaluation is to improve instruction and student learning.

2- Summative evaluations•It occurs most often at the end of a unit.

•The teacher uses summative evaluation to determine what has been learned over a period of time, to summarize student progress, and to report to students, parents and educators on progress relative to curriculum objectives.

3- Diagnostic evaluation

•It usually occurs at the beginning of the school year or before a new unit.

• It identifies students who lack prerequisite knowledge, understanding or skills.

•Diagnostic testing also identifies student interests.

•Diagnostic evaluation provides information essential to teachers in designing appropriate programs for all students.

Bloom’s Taxonomy - provides an important framework for

teachers to use to focus on higher order thinking. By providing a hierarchy of levels, this taxonomy can assist teachers in designing performance tasks, crafting questions for conferring with students, andproviding feedback on student work

Cognitive Objectives

•Knowledge•Comprehension

•Application

•Analysis•Evaluation•Synthesis

SimpleSimple

ComplexComplex

KnowledgeThe remembering of previously learned materialExamples of learning objectives:

-know common terms-know specific facts-know methods and procedures-know basic concepts-know principles

– who, what, why, when, omit, where, which, choose, find, how, define, label, show, spell, list, match, name

ComprehensionThe ability to grasp the meaning of materialExamples of learning objectives:-understand facts and principles-interpret verbal materials-interpret charts and graphs-translate verbal material to mathematical formulaejustify methods and procedures

- compare, contrast, demonstrate, interpret, explain, extend, illustrate, infer, outline, relate, rephrase, classify

Application•The ability to use learned material in new and

concrete situations•Examples of learning objectives:• -apply concepts and principles to new situations•apply laws and theories to practical situations•solve mathematical problems•construct graphs and charts•demonstrate the correct usage of a method or

procedure

- apply. build, choose, construct, develop, interview, make use of, organize, experiment, with, plan, select, solve, utilize

Analysis• The ability to break down material into its

component parts• Examples of learning objectives:• - recognize unstated assumptions• -recognize logical fallacies in reasoning• distinguish between facts and inferences• evaluate the relevancy of data• analyze the organizational structure of a work

- analyze, categorize, classify, compare, contrast, discover, dissect, divide, examine, inspect, distinguish, simplify

EvaluationThe ability to judge the value of material for a given

purpose based on definite criteriaExamples of learning objectives:- judge the logical consistency of written material-judge the adequacy with which conclusions are supported

by datajudge the value of a work by the use of internal criteria

(organization) or external standards of excellence

- build, choose, combine, compile, compose, construct, create, design, develop, estimate, formulate, imagine, invent

Synthesis• The ability to put parts together to form a new

whole • Examples of learning objectives:• -write a well organized theme• -give a well organize speech• write a creative short story• propose a plan for an experiment• integrate learning from different areas into a plan

for solving a problem

- award, choose, conclude, criticize, decide, defend, determine, dispute, evaluate, judge, justify, measure, appraise

Synthesis (Example)

Write a paragraph summarizing the text you have read. Your

summary should have a topic sentence defining the problem,

some of the causes, some of the effects, and a conclusion.

Principles of Evaluation

Evaluation should be1. Based on clearly stated objectives2. Comprehensive3. Cooperative4. Used Judiciously5. Continuous and integral part of the teaching

– learning process

Qualities of a Good Measuring Instrument

Validity: the extent to which the instrument measures what it is intended to measure.

Reliability: the consistency with which an instrument measures a given variable.

Objectivity: the extent to which independent and competent examiners agree on what constitute a good answer for each of the elements of a measuring instruments

Practicability: the overall simplicity of the use of a test both for test constructor and for students.

Validity:Does the test measure what it is supposed

to measure?

Reliability:Will this test yield stable scores over

repeated administrations?

•Criterion-referenced (c-r) standard is used to determine if someone has attained a specified standard

•Norm-referenced (n-r) standard is used to judge an individual’s performance in relation to the performances of other members of a well-defined group

Approaches to Evaluation

•Criterion-referenced (c-r) standards are useful for setting performance standards for all

•Norm-referenced (n-r) standards are valuable for comparisons among individuals when the situation requires a degree of sensitivity or discrimination in ability

•Norm-referenced standards- developed by testing a

large group of people- using descriptive statistics

to develop standards- percentile ranks are a

common norming method

•Major concern - groupcharacteristics used to develop norms may not result in desirable norms

•Criterion-referenced standards- predetermined standard of

performance shows the individual has achieved

a desired level of performance

- performance of individual is not compared with that of other individuals

“Evaluation is the process of giving meaning to a measurement by judging it against some standard”