hans niedderer mälardalens högskola institutionen för matematik och fysik

Generalisability Generalisability in Science Education Researchin Science Education Research

FontD Seminar 2004 in Mälardalens Högskola

Hans Niedderer

Mälardalens HögskolaInstitutionen för Matematik och Fysik

Part 1

Theoretical aspects about generalisability

Aspects of generalisability

Aspect 1: Statistics and classical quantitative design Aspect 2: Generalisability as essential feature of theory Aspect 3: Generalisability and paradigm (Kuhn)

Aspect 1: Statistics and classical quantitative design

Petri 1996: ONE Student Hake 1998: 5000 students

Aspect 1: Generalisability guaranteed by design?

Weaknesses of case studies (Wirth & Leutner 2004)

… No discovery of a universal, generalizable truth … No discovery of a cause-effect relationship … Not appropriate for testing hypotheses

I could think of cases: with/without computer, with/without electronium

General empirical approach (Wirth & Leutner 2004)

GeneralizationInduction

Modification

Deduction of hypothesis

Scientific observationTheory

QuestionProblem

Internal Validity

General empirical approach (Wirth & Leutner 2004)

GeneralizationInduction

Modification

Deduction of hypothesis

Scientific observationTheory

QuestionProblem

Internal Validity

=> "External validity"

External validity

Extent to which conclusions drawn from a scientific observation can be generalized to other persons, situations or points of time.

Control environmental conditions, real life setting, representative sample, replication (in different contexts), theory use

Possibilities in science education

Where we have strong hypotheses from previous qualitative research ... Investigate the effect of using an electronium model of the

atom compared to a course using a propability model of the atom

Doing similar courses in optics with and without using different kinds of computer software

Example: In Roger's dissertation project at KAU, we try to develop a classical design, using different treatments with different use of ICT in optics

Aspect 1: Statistics and classical quantitative design

Generalisability ...seen as a problem of design and statistical evidence.

Aspect 2: Generalisability as essential feature of theory

Reif, F., Larkin, J. H. (1991) Cognition in Scientific and Everyday Domains: Comparison and Learning Implications, JRST Vol. 28, NO. 9


Reif, F., Larkin, J. H. (1991) Cognition in Scientific and Everyday Domains: Comparison and Learning Implications, JRST Vol. 28, NO. 9

Little generalisability in EDL thinking ("cluster concepts")"Generalisability as definition of science" (Reif, Larkin, Schecker, Niedderer)

Incl. Science education !

Example of working for more generalisability

In Margareta's work about ownership we are working hard on a better theoretical definition of the concept of ownership, with maximal Generality Parsimony Precision Consistency

Furthermore we try to define in such a way that it has predictive power.

No statistics will be needed in this task.


Generalisability ...seen as amount and quality of use of theory.

Theory with general concepts

Try to come to general definitions of concepts: use the same definition for every case, not one definition in one case and a different definition in an other case (as we would all do it in everyday life context!)

Similar: work on general claims or hypotheses Try to build up confidence by telling frequencies -

how often you were able to apply this concept - in qualitative work - and how often you were unsure about it. A negative statement - category does clearly NOT fit - is a positive statement in this sense!

Aspect 3: Generalisability as acceptance in the scientific community

T. S. Kuhn distinguishes three phases of development of a scientific theory:

the pre-paradigmatic phase: Many different questions, many "theories", little generalisability

the paradigmatic phase, high generalisability of paradigmatic research

the revolutionary phase: generalisability of the new paradigm takes time to build up.


In the paradigmatic phase of research, there are agreed questions concepts repeated and agreed results meanings

In SER we have at least one such field:alternative conceptions of learners Many Swedes contributed to it (B. Andersson, ..., F. Marton)

This gives a body of agreed knowledge which gives the highest amount of generalisability(Generalisability by cummulation in the scientific community)

Number of investigations in Pfund&Duit [email protected]

1998

> 4000 entriesalltogether

TopicMechanics 281

Electricity 146

Heat 68

Optics

69Particles60

Energy 69

Astronomy 36

Quantum Physics 11

421

218

111

35

1991 1994 1998687

379

159

190162

151

57

R. Duit:Bibliography "Students' Alternative Frameworks and Science Education"Aug. 2002

Part 2

Cases

Cases

Case 1: 6000 students study by Hake (1998) Case 2: Doctoral study of Bethge (1988) Case 3: Doctoral study of Petri (1996)

Case 1: Empirical Results Hake 1998

Interactive-engagement vs traditional methodsA six-thousand-student survey of mechanics test data for introductory physics coursesR. Hake, Am.J.Phys. 1998 (1)

Traditional methods

"Traditional" (T) courses as those reported by instructors to make little or no use of IE methods, relying primarily on

passive-student lectures recipe labs, and algorithmic-problem exams

“Interactive Engagement” (IE)

Methods as those designed at least in part to promote conceptual understanding through interactive

engagement of students in "heads-on" (always) and "hands-on" (usually) activities which yield immediate feedback through discussion

with peers and/or instructors

Interactive-engagement vs traditional methods

gain vs pretest - universities

4832 studentsg < 1

G in %

Generalisability in case 1

Aspect 1: Statistics and design 6000 students is very impressive Control of variables?

"interactive teaching as reported by the teachers" and "better understanding as measured by the FCI"?More qualitative tasks related to alternative conceptions about force?FCI measures only one aspect of competence, namely qualitative, multiple choice tasks related to many different alternative conceptions (not only Newton's force)


Aspect 2: Generalisability as essential feature of theory General theoretical claim Overgeneralised? Other factors not taken into account?


Case 2: Students' alternative conceptions in atomic physics (Bethge 1988)

Case 2: Students' alternative conceptions in atomic physics (Bethge 1988)

Methods 1) Audio recordings of current physics lessons were our main

data source. 2) A pair-relation questionnaire with associative elements. In this

type of questionnaire students were asked to make statements using two given concepts, for example: wave - energy level wave function - trajectory trajectory - energy level position - wave function electron - wave trajectory - probability

3) A questionnaire with seven "thinking type" tasks 4) Interviews with nine pairs of students

Conceptions related to orbits (trajectories) in quantum physics after teaching

(O1) Classical orbits (about 50%)(O2) Only special orbits allowed

(O3) Smeared orbits

The concept of "trajectory" is combined with notions of "probability" and "wave function" from wave mechanics in several ways to form a new "intermediate" conception:

- the orbits are "smeared", not exactly determined, "fuzzy"

- the probability for a special orbit is given- the probability of parts of the orbit is given

(O4) Trajectories do not exist in quantum physics (about 25%)

about 25%


Aspect 1: Statistics and design Criteria of Wirth & Leutner

external validity, generalisability to other persons, situations or points of time, representative sample: This research was with data from many students and classes,

real life setting: with data from real teaching The results seem thus be generalisable from a

statistical view with respect to 17 to 19 age students, after teaching in atomic physics with more than Bohr's model


Aspect 2: Generalisability as essential feature of theory General theoretical claim: students also in quantum atomic

physics show a limited number of alternative conceptions. Some of these are ...

Aspect 3: Generalisability as acceptance in the scientific community Replication in different contexts (Aspect 1) and

cummulation (Aspect 3) … was done to some extent later by dissertations (Lichtfeldt

1992, Mashhadi 1996, Deylitz 1999) and other research (Harrison et al. 1999, Müller et al. 2002)

So the results are - today ! - partially generalisable

Case 3: Learning pathways in atomic physics (Petri 1996)

Building on theoretical ideas in the community Driver 1989, Scott et al. 1991: conceptual pathways Bremen workshop 1991: need to describe learning pathways

"(1) Need to document learning pathways for different content areas in physics""(2) Need to construct ways of describing cognitive systems that are useful to researchers in physics education"

Niedderer (1991 to 1996): Prior example electric circuits Psillos 1999: Conceptual evolution

Giving ONE example in great detail (analysing huge amount of data from ONE student)

Carls Learning Pathway “Modell of the Atom"Petri 1996; Petri&Niedderer 1998

Example of working for more generalisability

In developping a new dissertation project at MdH, Peter Gustavsson and I are planning to have a new doctoral student working on a learning pathway for a whole class in distant education.

An advanced model of the learning process

Planetary view

Quantum cloud view

Smeared orbits view

Quantum particle view

Example: conceptions of an atom

time

strength


Aspect 1: Statistics and design 1 student: No generalisability to other students

Aspect 2: Generalisability as essential feature of theory Showing new, general model of a "learning pathway" as a

theoretical idea (GENERALISABILITY-claim!) in ONE example in great detail, thus making it work!Holzkamp: An experiment is a realisation of theory.

Aspect 3: Generalisability as acceptance in the scientific community The theoretical ideas were later used - and cited - by other

researchers(Psillos 1999 "conceptual evolution"; Taber (2000) "manifold conceptions in cognitive structure")

Part 3

Conclusions

Final conclusion

Try to come to theoretical definitions of concepts and claims, thus building up potential generalisability

Generalisability is a decision of the community of researchers, developping a paradigm (Kuhn), coming to paradigmatic research

This decision is based on empirical research in combination with normative decisions about relevant questions and theoretical approaches

To formulate it the other way round: I do not believe in generalisability from one study.

Final conclusion (ctd.)

In this view, generalisability means similar questions are asked with similar theory with similar results by (many) other researchers

This is why literature search and writing a "state of the art" in a doctoral dissertation and relating this to own results is so important!


In one field we are in a revolutionary phase:the basic understanding of teaching and learning

Old paradigm about teaching and learning

Paradigm shift about teaching and learning


Acceptance in the community of practioners: still high

Acceptance in the scientific community: high,but consequences are still under development


Acceptance in the community of practioners: still high

Example of a generalisable statement, not yet fully accepted:A learner's constructions are different from the teaching input - as a rule, not as an accident

Case 4: Mixed methods approach (Deylitz 1999, Budde 2003)

Pre- and post test results of special items

Results

Learning environment Cognitive system of student

Problem with quantum interpretation

Uli: Well, actually I completely put aside the concept of trajectory in the area of atomic physics. The function you have is nothing but the probability of presence of an electron. ... you can't say it moves on an orbit. To explain - well, the motion cannot really be explained anymore ... It's however, not an orbit any more. .. The electron must move somehow - very strange, it is now here and then there .. That gets crazy ..

Damned, it could theoretically move in between. Just that it moves in a strange zigzag, but that would mean again something like an orbit. And that's crazy again. Well, somehow I can't get that clear.

Diagram of probability distribution shown during

interview

Elke: If they didn’t move, a probability distribution would make no sense at all . The electron would stay in one position - that’s all ... As it shows up at different places it must move! Otherwise, it would be present always at only one point.

Summary of findings

25% of students use conceptions near to modern physics

25% use some typical intermediate conceptions such as "smeared orbits"

50% stick to classical orbits ("Bohr model")

... even after teaching

Part 2

Jürgen Petri (1996)

Research on learning processes

Research questions

How can we describe a learning process from a cognitive point of view?

Related theory International workshop on physics learning in 1991 Learning processes studies Cognitive modelling of physics learning Conceptual or learning pathways (Driver, Scott;

Niedderer, Petri; Psillos et al.) Conceptual change (Hewson, Duit, ...)

Data

1 student, 80 lessons on QAP, grade 13 gymnasium 6 interviews at different points in time Audio tapes from group work of students, partially

transcribed Written artefacts (reports etc.) from 1 student

Methods: Scheme for interpretive analysis of qualitative and quantitative data

xxx

Carls LearningPathway “Modell of the Atom"Petri&Niedderer 1998

Carls initial conception about an atom

C: The electrons, negatively charged tiny balls, move around the nucleus in definite orbits or circles, even several electrons in one orbit or shell.

orbits

Planetary model

Carls second conception about an atom

C: If I take the nucleus and think of a field around, a wave-like probability-field, then I think of drawing a psi-function as a wave that spreads out equally in all directions. And everywhere, where the probability is higher, there is an orbit. I don't know, I can't get rid of these orbits, though I don't know, where I got them.

Smeared orbits

IC smeared orbit in an atom ("cognitive attractor")

Found by different authors, with different teaching approaches:Bayer 1985, Bethge 1988, Petri 1996

Final state “model of an atom“ (Petri&Niedderer 1998)

Final state of Carl‘s cognitive system “atom"

An advanced model of the learning process

Planetary conception

Quantum cloud conception

Smeared orbits conception

Quantum particle conception

Example: conceptions of an atom

time

strength

Part 3

Stefan Deylitz (1999)

Research about evaluation

Erg

Testaufgaben Einzelne Items

Research questions

Evaluation of a new approach to quantum atomic physics (QAP)

Related theory/literature: different types of evaluation Comparative E. Formative E. Summative E.

==> How far were our own main goals achieved?

Data

26 students, 3 courses, grade 12 gymnasium New test constructed according to goals Pre and post test Post interviews with all students

Selected questions of the questionnaire

1a Draw a picture of an atom and label it!

1b Describe your model with a few sentences.

1e Can you determine the size of an atom in your model of an atom?

3b Given are three drawings. Describe commonalties and differences between these three atomic models and use the notions "electron orbit", "probability density", and "charge cloud".

Methods

Performance levels 0, 1, 2, 3Because of open-ended questions , we had to take answers to different questions related to the same knowledge domain. So, item combinations were defined to determine performance levels.

Evaluation: To what extent have the main goals been achieved by the students?

A quantum mechanical model is dominating

A classical model is

dominating

The psi-function is the main component of the model of an atom

No connection between psi-function and

model of an atom

The concept of state has a new meaning with relation to quantum physics

Students formulate the process of application without help

Equation is hidden behind

STELLA models

Observed phenomena are not related to a

quantum model of atom

A qualitative understanding of shielding; used in own STELLA models

No understanding of shielding effects

to be seen

Total performance level 0

Model of an atom

Psi-function

Notion of state

Schrödinger equation

Relating measurements to theory

Higher order atoms

0 1 2 3

50% -

50% -

50% -

50% -

50% -

50% -


The concept of state has no meaning with relation

to quantum physics

Observed phenomena are explained by relating them to a quantum model of atom

The first three goals

Deylitz 1999

Evaluation: To what extent have the main goals been achieved by the students?

The second three goals



dominating



model of an atom




STELLA models





to be seen


Model of an atom

Psi-function

Notion of state



Higher order atoms

0 1 2 3

50% -

50% -

50% -

50% -

50% -

50% -



to quantum physics




dominating



model of an atom




STELLA models





to be seen


Model of an atom

Psi-function

Notion of state



Higher order atoms

0 1 2 3

50% -

50% -

50% -

50% -

50% -

50% -



to quantum physics


Deylitz 1999

Part 4

Marion Budde (2003)

Research about learning effects of

content specific teaching inputs

Research questions

What effects has a certain teaching input on the learning process of an individual student?

Theoretical perspective:

==>Resonance

Detail of teaching approach ("teaching input")

The electronium modelThe probability model

Data

2 students: Audio tapes from teaching (five lessons of 45 minutes per week, for approximately18 weeks)

1 student attended private lessons (approximately 50 lessons), which were also audio-recorded.

Pre and post test for understanding of atomic models.

Methods

Criteria for statements showing a learning effect: Statements made spontaneously Statement with different content Contradicting statement Consequent statement in a complex discussion Use of own formulation or in a new context

Development of different types of resonance No resonance Congruent/non-congruent resonance

Overview of the observed resonances

Continuum /Continuousdensity

No motion

Student “Klaus” Student “Thomas”

Not intended non-resonanceParticle conception is toostrong: even liquids are seenas consisting of particles

Slightly delayedcongruent intendedresonance

Not intended non-resonanceDistance conception of density

Spontaneous congruentintended resonance

Spontaneous congruentintended resonance

Liquid Slightly delayed congruentintended resonance

Research about resonances during teaching

Trial of summary 1

Expect strong mechanistic thinking Particle Orbits, trajectories Planetary model Stability from equal forces: F-coulomb = F-centrifugal

Trial of summary 2

Expect intermediate conceptions(self constructed in students heads,not intended by the teacher)

Trial of summary 3

The "electronium" interpretation of psi in atoms seems ... to be intuitive ... to develop a meaning of a distribution ... to help to replace the particle conception ("cognitive tool")

by a similar powerful conception of a compressible gas or liquid

Cognitive attractor "smeared orbits"

Bayer (1986) describes it as a spherically smeared shell with a certain thickness, coming from a kind of wave orbits.

Bethge (1988) describes it as a new and broader understanding of orbits, taking into the notion of orbits some elements of smeared or most probable orbits or probabilities along the orbit.

Kuehnen (1994) describes conceptions which are no more purely mechanistic seeing the electron with some probability in a special space region, where they move along orbits. But they still are particles.

These different investigations show that the conception of ”smeared orbits” probably is what could be called a cognitive attractor in this content area, which describes a cognitive construction using a conception of orbits to make sense of a new teaching input with the idea of probability.

hans niedderer mälardalens högskola institutionen för matematik och fysik

Documents