Assessing student competencies in experimentation Marcus HammannWestfälische Wilhelms Universität, Münster

&

Maike Ehmer, Phan Thanh Thi Hoi, IPN, Kiel

• Introduction

• Theory: SDDS-Modell (Scientific Discovery as Dual Search)

• Research Questions

• Hypotheses

• Design und Methods

• Results

• Discussion

Outline

How well do students experiment?

„In Biology as well as in Physics instruction, students‘ understanding of the method of experimentation and the principles of evidence-based argumentation remain underdeveloped. According to TIMSS, only 10-15 % of the students understand simple experimental designs at the end of grade 8.“

BAUMERT (1997)

How do students plan experiments?

A student hypothesizes that minerals are important forplant growth

TIMSS 1997

In order to test her hypothesis, she needs one more plant. Which of the following should she choose?

Students do not distinguish between different types of variables . They choose an

incorrect experiment as experimental control.

Grade 7: 60%

Grade 8: 58%

How do students plan experiments?

Hammann, M. et al. (2006) Fehlerfrei experimentieren. MNU 59/5, S. 292-299

How do students plan experiments?

Water

The seeds need water in order to grow.

Air

The seeds need air in order to breathe.

Nutrients

They are in the soil. The seeds feed on them.

How do students plan experiments?

„failure to seek disconfirmation“ (KLAYMAN & HA 1987)

„from an engineering model to a science model of experimentation“ (SCHAUBLE & KLOPFER 1991)

I put the seeds ca. 3 cm deep in the soil. The seeds need water to grow, that‘s why I add water. The seeds use the air and the nutrients and grow.

How convincing is an experiment without an experimental control?

• Can a student say after this experiment, which of the following factors are necessary for seed germination?

Selected student answers (n=15, age 10-12)  

• S 1 No, because each variable must be tested individually. • S 2 No, because the air is invisible.• S 3 No, because the seeds also need light and warmth. Then the

experiment would turn out well. • S 4 No, because the seeds need also warmth. • S 5 No, a plant needs no nutrients.• S 6 Yes,because the student added all the things that are necessary for

seed germination.• S 7 Yes, because the seeds grow and when the plant grows, he sees

that the plant needs the things that he added.

Hammann, M. et al. (2006) Fehlerfrei experimentieren. MNU 292-299.

OECD PISA 2006

Assessing Scientific, Reading and Mathematical Literacy: A Framework for PISA 2006.

www.pisa.oecd.org

TIMSS 1997

butter

whole meal

white floursugar

honeymargarine

The original recipe for a cake

John‘s recipe:

(Tschirigi 1980)

How do students plan experiments?

Great cake?Hypothesis: =

Which one experiment can be used to test John‘s idea?

Experiment 2

Experiment 3

+

+

+

+

+

+

Experiment 1

„change all“

„HOTAT = hold on to one thing at a time“

correct

How do students test hypotheses?

Klayman & Ha (1989)

Restricted search in the hypothesis space,

Confirmation bias

Do student relate hypotheses logically in a series of experiments?

Schauble et al. (1991)

Do students revise hypotheses if confronted with disconfirming data?

Zelltyp Mutiertes Gen

Lactose Input

[Microgramm]

-Galactosidase-Output

[Microgramm]haploid kein 100 50

haploid P 200 100

haploid I 200 876

haploid O 200 527

Dunbar (1993). Concept Discovery in a Scientific Domain. Cognitive Science 17, 397-434.

How do students analyse data?

Mike: „Because the seeds in tray 2 did not have any light and only 6°C they did not germinate, [...] because, I think, it was not warm enough for them, only 6°C, not very warm and also no light. And in tray 3, the seeds also did not have any light, but instead they had a temperature of 22°C, and that was enough for the beans to germinate. Well, this proves that [...] bean seeds need light as well as

warmth for germination.“ conclusion is in conflict with the evidence

Mike: „well, what is written here is not quite correct. He should write [...] that bean seeds need light as well as ... no ... light as well as ... no ... One can‘t say ‚as well as‘, but light and perhaps also warmth.“I: „But the seeds do need light, or not?“Mike: „They need light. But in this experiment [tray 3] they don‘t need light. [...] exceptions prove the rule It works with light, but also with warmth. So it does not matter.“

I: „So, do you think that this experiment shows that the seeds need light or not? Mike: „Well, not really. In fact they only need special warmth of 22°C [...]. But only if this photo is correct.“ [...] calls into question that the representation of the experiment may be incorrect

How do students analyse data?

•Oliver: „And in tray 3 ... there was no light ... but instead there was warmth, and the seeds germinated anyway ...“ (makes a face)•I: „ And this does not seem to make you happy? •Oliver: „No, well, I‘d rather say that warmth and light are necessary for this.“ explicit statement of student conceptions

•I: „And this can also be seen in the experiment?“•Oliver: „In tray 1 (laughs), but not so much in tray 3 ... (thinks)“•I: „And is it enough, if the results show in tray 1?“•Oliver: „Yes, I would say so, yes. One must add light and warmth.“• looks at individual components of the experiment

that confirm his expectations

How do students analyse data?

Theory: SDDS

Top-level components of the SDDS-Model

SDDSScientific Discovery

as Dual Search

search hypothesis

space

test hypothesis

evaluate evidence

David Klahr (2000). Exploring Science: The Cognition and Development of Discovery Processes. MIT Press.

Hammann, M. (2007). Das SDDS Modell. In: Handbuch der Theorien der Biologiedidaktik. Springer Verlag.

Research Questions

1. testability

Can the three dimensions of the construct “experimentation as problem solving” (SDDS Modell) be tested with paper-and-pencil test Items?

2. number of dimensions

How many dimensions should be tested? Do the three dimensions correlate?

3. domain-specifcity of the dimensions

Do the three dimensions correlate with the biological pre-

knowledge?

Design und Method

Brot backen

Kleine Küken

Samen-keimung

Apfel-wein

Knowledge

Test

8-10 Items (SMC)

8-10 Items (SMC)

8-10 Items (SMC)

8-10 Items (SMC)

Competency test

„Search in the hypothesis space“

2 Items (SMC)

2 Items (SMC)

2 Items (SMC)

2 Items (SMC)

Competency test

„Search in the experiment space“

2 Items (SMC)

2 Items (SMC)

2 Items (SMC)

2 Items (SMC)

Competency test

„Data analysis“

2 Items (SMC)

2 Items (SMC)

2 Items (SMC)

2 Items (SMC)

Design und Method

  pretest 1 pretest 2 pretest 3 

Main study

Test knowledge-testCompetence-

test (2 Vers.)

knowledge-test knowledge-testcompetence-

test (1 Vers.)

knowledge-TestCompetence-

test (1 Vers.)

students (n) Version 1: n = 252Version 2: n = 547

Version 1: n = 122

Version 1: n = 77

Version 1: n = 1006

Test booklet 111 (n = 73) 121 (n = 82)131 (n = 97)211 (n = 131)221 (n = 216)231 (n = 94)241 (n = 106)

111 (n = 122) 111 (n = 77) 111 (n = 1006)

Answering format

knowledge-Test: CMC

competence-Test: SMC

knowledge-Test: CMC

knowledge-Test: SMC

competence-Test: SMC

knowlegde-Test: SMC

competence-Test: SMC

Design and MethodeSample unit: Seed germination-Version 2

1. Search in the hypothesis space

A student named Jan did an experiment about seed germination. He used two pots with soil (pot1 and pot 2) and one pot with cotton wool instead of soil (pot 3). He sowed bean seeds in the three pots and put all three pots in the sunlight at a temperature of 22°C. He watered pot 1 and pot 3, but did not water pot 2.

Why does Jan do this experiment ? A Because he wants to make the seeds germinate faster. (level 0)B Because he thinks that light and warmth are necessary for seed germination. (level 1)C Because he thinks that water, soil, light and warmth are necessary for seed germination. (level 1)D Because he thinks that water and soil are necessary for seed germination. (level 2)

2. Data analysisAfter some days Jan had the following results: The seeds in pot 1 and pot 3 germinated, whereasthey did not germinate in pot 2.

Which one is the best explanation of the findings?A The experiment did not work because the seeds in pot 2 did not germinate (level 0)B The experiment showed that seeds need light and warmth to germinate (level 1)C The experiment showed that seeds need soil and water to germinate (level 1)D The experiment showed that seeds need no soil, but water to germinate (level 2)

soil / water/ light/ 22°C

soil / no water/ light/ 22°C

no soil / water/ light/ 22°C

soil / water/ light/ 22°C

soil / no water/ light/ 22°C

no soil / water/ light/ 22°C

Sample unit: Samenkeimung-Version 2

3.Search in the experiment space

Jan thinks bean seeds will germinate faster if they are put at a warm temperature. He plans an experiment to test this idea. This is Jan’s experiment. He puts the bean seeds in a pot with soil (pot 1), waters them, and keeps the pot at a temperature of 22°C in the sunlight.

Design and Methode

Soil/ light / 22°C

Pot 1

Jan needs another pot in order to compare with pot 1

Which one in the following pots (A-D) should he chose?

Pot B Pot C Pot D

soil / light/ 10°C

no soil /no light / 10°C

Pot A

no soil / light / 10°C

soil / no light/ 10°C

Pot A (level 0)

Pot B (level 1)

Pot C (level 1)

Pot D (level 2)

Item difficulties

Item difficulty for

items

Mean Item difficulty for

units

Mean Item difficulty for

booklet

Item difficulty for knowledge

test10.3% -84.6% 43% - 60% 51.7%

Item difficulty for competency

test46% - 77% 54% - 69% 62.3%

reliabilities

Reliability at unit level

Reliability at booklet

level

Unit 1: Seed

germination

Unit 2: Chicken

eggs

Unit 3: Apple wine

Unit 4: Baking bread

Reliability for knowledge

test .3201 .2742 .4845 .4097 .6307

Reliability for competency

test.6895 .6303 .7724 .7110 .8840

Cronbach’s alpha for scale “forming

hypothesis”(8 items)

Cronbach’s alpha for scale

“data analysis”(8 items)

Cronbach’s alpha for scale “planning

experiment”(8 items)

.7822 .6879 .7122

Reliability of the three scales of the competency - test

Correlations between the biological pre-knowledge and the three dimensions

Biological pre-

knowledge

forming hypotheses

analysing data

planning experiments

0.385*

0.353**

0.400**

* Correlation is significant at the 0.01 level (2-tailed)

** Correlation is significant at the 0.05 level (2-tailed)

0.785**

0.615**

0.607**

Correlations between the three dimensions of experimentation

Forming hypotheses

Planning Experiments

Analysing data

Latent class analysis

Item profile of each model

Two-class model

50

60

70

80

90

100

1 2 3

Three dimensions in experimentation (1: Search in the hypothesis space; 2: Data

analysis; 3: Search in the experiment space)

Per

cen

tag

eClass 1

Class 2

Three-class model

50

60

70

80

90

100

1 2 3

Three dimensions in experimentation (1: Search in the hypothesis space; 2: Data

analysis; 3: Search in the experiment space)

Per

cen

tag

e Class 1

Class 2

Class 3

Thre relationship between three groups of students and three dimensions in experimentation

50

100

Search in the hypothesisspace

Data analysisSearch in the experiment

space

Class 1

Class 2

Class 3

Findings und Discussion

• Assessment of student competencies in experimentation should take into consideration well-documented problems / student conceptions.

• According to an empirical model of experimentation as problem solving, three dimensions should be assessed: forming hypotheses, planning experiments, analysing data.

• In the paper-and-pencil test presented, the three dimensions test form reliable scales that are fairly highly correlated.

• A possible reason for this may be a common factor that influences performance in the three dimensions (e.g. intelligence, methodological knowledge)

• The role of the biological pre-knowledge cannot be assessed with a great amount of certainty in this study.

• Qualitative studies yielded important insights into students’ thinking and argumentation when they form hypotheses, plan experiments and analyse data.

Thank you for your attention.

hammann.m@uni-muenster.de