11

Issues in Assessment in Issues in Assessment in Higher Education: ScienceHigher Education: Science

Higher Education Forum on Scientific CompetenciesHigher Education Forum on Scientific CompetenciesMedellin-Colombia Nov 2-4, 2005Medellin-Colombia Nov 2-4, 2005

Dr Hans WagemakerINTERNATIONAL ASSOCIATION FOR THE EVALUATION OF EDUCATIONAL ACHIEVEMENT

Theme: Validity in AssessmentTheme: Validity in Assessment

What can/should be Assessed.What can/should be Assessed. How can it be AssessedHow can it be Assessed Frameworks, Curriculum and Frameworks, Curriculum and

TrajectoriesTrajectories From TIMSS to Higher EducationFrom TIMSS to Higher Education

Purpose of Assessment: Purpose of Assessment: Assessment for Assessment for

Learning/Assessment of LearningLearning/Assessment of Learning

Validity/Reliability- Primacy of Validity/Reliability- Primacy of ValidityValidity

Construct ValidityConstruct Validity Consequential ValidityConsequential Validity

TIMSS what is Assessed:TIMSS what is Assessed:Content DomainsContent Domains

Grade 4 – Life, Physical and Earth Grade 4 – Life, Physical and Earth ScienceScience

Grade 8- Biology, Chemistry, Physics, Grade 8- Biology, Chemistry, Physics, Earth ScienceEarth Science

TIMSS DesignTIMSS Design

What is Assessed: Cognitive What is Assessed: Cognitive Assessment DomainsAssessment Domains

KnowingKnowing ApplyingApplying ReasoningReasoning Scientific InquiryScientific Inquiry

Implies different strategiesImplies different strategies

KnowingKnowing 1. Recall/Recognize- statements, facts, 1. Recall/Recognize- statements, facts,

relationships, processes, concepts.relationships, processes, concepts.

2. Define- provide or identify definitions of 2. Define- provide or identify definitions of scientific terms, vocabulary, symbols, scientific terms, vocabulary, symbols, abbreviations, units, scales in appropriate abbreviations, units, scales in appropriate contexts.contexts.

3. Describe- organisms, physical materials, 3. Describe- organisms, physical materials, scientific processes that demonstrate scientific processes that demonstrate knowledge of properties, structure and knowledge of properties, structure and relationships.relationships.

KnowingKnowing

4. Illustrate with Examples – support 4. Illustrate with Examples – support or clarify statements of facts or or clarify statements of facts or concepts with appropriate examples- concepts with appropriate examples- provide specific examplesprovide specific examples

5. Use tools and procedures – 5. Use tools and procedures – demonstrate knowledge of the use of demonstrate knowledge of the use of science apparatus, equipment , tools, science apparatus, equipment , tools, procedures, measurement devices.procedures, measurement devices.

ApplyingApplying

1. Compare contrast classify- 1. Compare contrast classify- identify or describe identify or describe similarities/differences between similarities/differences between groups of organisms, materials, groups of organisms, materials, processes –distinguish, classify or processes –distinguish, classify or order objects, materials or organisms order objects, materials or organisms and processes based on given and processes based on given characteristics and properties.characteristics and properties.

ApplyingApplying

2. Use Models – use a diagram or 2. Use Models – use a diagram or model to demonstrate science model to demonstrate science concept, structure, relationship, concept, structure, relationship, process, biological or physical process, biological or physical system or cycle ( e.g.. Atomic system or cycle ( e.g.. Atomic structure, electrical circuit, water structure, electrical circuit, water cycle)cycle)

ApplyingApplying

3. Relate – relate knowledge of an 3. Relate – relate knowledge of an underlying biological or physical underlying biological or physical concept to an observed or inferred concept to an observed or inferred property, behavior, or use of objects, property, behavior, or use of objects, organisms or materialsorganisms or materials

4. Interpret information - interpret 4. Interpret information - interpret textual, tabular or graphical textual, tabular or graphical information in light of a scientific information in light of a scientific concept or principleconcept or principle

ApplyingApplying

5. Find solutions – use a science 5. Find solutions – use a science relationship, equation, or formula to find a relationship, equation, or formula to find a qualitative or quantitative solution qualitative or quantitative solution involving the direct involving the direct application/demonstration of a conceptapplication/demonstration of a concept

6. provide and explanation for an 6. provide and explanation for an observation demonstrating understanding observation demonstrating understanding of the underlying science concept, of the underlying science concept, principle, law or theory.principle, law or theory.

ReasoningReasoning

1. Analyze problems to determine 1. Analyze problems to determine the relevant relationships, concepts the relevant relationships, concepts and problem solving steps – develop and problem solving steps – develop problem solving strategiesproblem solving strategies

2. Integrate/Synthesize – provide 2. Integrate/Synthesize – provide solutions to problems that require solutions to problems that require consideration of a number of consideration of a number of different factors or related concepts – different factors or related concepts – integrate mathematical conceptsintegrate mathematical concepts

Reasoning Reasoning

3. Hypothesize/predict – combine 3. Hypothesize/predict – combine science knowledge with information science knowledge with information from observation/experience or from observation/experience or formulate questions that can be formulate questions that can be answered by investigation – answered by investigation – formulate testable hypotheses – formulate testable hypotheses – make predictions about effects of make predictions about effects of changes in biological or physical changes in biological or physical conditions in light of evidence and conditions in light of evidence and scientific understandingscientific understanding

ReasoningReasoning 4. Design Plan investigations appropriate 4. Design Plan investigations appropriate

for answering questionsfor answering questions 5. Draw conclusions – detect patterns in 5. Draw conclusions – detect patterns in

data trends – interpolatedata trends – interpolate 6. Generalize – make general conclusions 6. Generalize – make general conclusions

that go beyond experimental or given that go beyond experimental or given conditionsconditions

7. Evaluate – weigh 7. Evaluate – weigh advantages/disadvantages about advantages/disadvantages about alternative processes, materials, alternative processes, materials, alternative explanations etc.alternative explanations etc.

ReasoningReasoning

Justify – use evidence and scientific Justify – use evidence and scientific understanding to justify explanations understanding to justify explanations and problem solutions…and problem solutions…

Scientific InquiryScientific Inquiry

1. Demonstrate a formalized 1. Demonstrate a formalized approach to scientific investigationapproach to scientific investigation

Core Principles of Effective Core Principles of Effective AssessmentAssessment

1. assessment that guides and 1. assessment that guides and encourages to effective approaches to encourages to effective approaches to learninglearning

2. assessment that validly and reliably 2. assessment that validly and reliably measures expected learning outcomes, in measures expected learning outcomes, in particular higher-order learning that particular higher-order learning that characterizes higher educationcharacterizes higher education

3. assessment and grading that defines 3. assessment and grading that defines and protects academic standardsand protects academic standards

Assessment for Learning- Assessment for Learning- FormativeFormative

Validity of Assessment for learning is Validity of Assessment for learning is high when assessment procedures high when assessment procedures help the student toward achieving help the student toward achieving the real goals of the learning activitythe real goals of the learning activity

Requires, committed participants, Requires, committed participants, trust, understanding of what is being trust, understanding of what is being asked, effective feedbackasked, effective feedback

Assessment for Learning – Assessment for Learning – Formative AssessmentFormative Assessment

Task Factors – Knowledge of key Task Factors – Knowledge of key aspects and difficulties of taskaspects and difficulties of task

Criteria – students told ( and Criteria – students told ( and understand) clearly what is needed.understand) clearly what is needed.

Standards – set standards are Standards – set standards are appropriate to studentsappropriate to students

Feedback – balance of attention – Feedback – balance of attention – both strengths and weaknesses- is both strengths and weaknesses- is useful for studentuseful for student

Assessment of Learning – Assessment of Learning – Summative AssessmentSummative Assessment

Done under conditions that give Done under conditions that give confidence it is students own work.confidence it is students own work.

Marking is standardized across Marking is standardized across several of many teachersseveral of many teachers

Most up to date information possibleMost up to date information possible

Assessment of learningAssessment of learningSummative AssessmentSummative Assessment

Quality ( and appropriateness) Quality ( and appropriateness) should take precedence over should take precedence over quantityquantity

Things that can be assessed in final Things that can be assessed in final exam should be - those that can not exam should be - those that can not should not.should not.

If multiple assessments of same If multiple assessments of same material – final grade should reflect material – final grade should reflect final level of performancefinal level of performance

Assessment of LearningAssessment of Learning- Summative Assessment- Summative Assessment

Disadvantage – time sampling Disadvantage – time sampling problemproblem

May not be valid for admitting May not be valid for admitting student to next stage of educationstudent to next stage of education

Indicators of affective assessmentsIndicators of affective assessments

1. assessment which is integral 1. assessment which is integral component of teaching and learning component of teaching and learning processprocess

2. multiple roles of assessment 2. multiple roles of assessment recognized.recognized.

3. departmental/faculty policy as 3. departmental/faculty policy as guideguide

4. alignment between what is taught 4. alignment between what is taught and what is assessed.and what is assessed.

Indicators..Indicators..

5. Assessment tasks assess capacity 5. Assessment tasks assess capacity to analyze and synthesize new to analyze and synthesize new information.information.

6. Variety of assessments used6. Variety of assessments used

7. Assessment designed to assess 7. Assessment designed to assess generic skills as well as subject generic skills as well as subject specific skillsspecific skills

IndicatorsIndicators

8. steady progression of complexity 8. steady progression of complexity and demands in later yearsand demands in later years

9. student choice (paper/exam/timing)9. student choice (paper/exam/timing)

10. workloads considered10. workloads considered

11. not excessive assessment11. not excessive assessment

12. task weighted to balance 12. task weighted to balance formative and summative rolesformative and summative roles

IndicatorsIndicators

13. Grades calculated on basis of 13. Grades calculated on basis of clearly articulated learning clearly articulated learning outcomes.outcomes.

14. Students provided with 14. Students provided with explanatory feedbackexplanatory feedback

15. Tasks checked for bias15. Tasks checked for bias

16. Plagiarism minimized 16. Plagiarism minimized

Assessment formatsAssessment formats

Multiple choiceMultiple choice Extended ResponseExtended Response Individual ProjectIndividual Project ThesisThesis Laboratory workLaboratory work Group workGroup work Computer basedComputer based PresentationsPresentations Many others: oral, portfolioMany others: oral, portfolio