asme 09 keynote

8/6/2019 ASME 09 Keynote

1/22

www.ioe.ac.uk

Research perspectives

and formative assessment

ASME Conference: Researching Medical

Education, November 2009: RIBA, London

Dylan Wiliam


2/22

OverviewThe nature of educational research

What should educational research try to do? How should it try to do it?

Formative assessment

Definitions

Implementations

Researching formative assessment


3/22

Pasteurs quadrant


4/22

Educational researchAn elusive science (Lagemann, 2000)

A search for disciplinary foundationsMaking social science matter (Flyvbjerg, 2001)

Contrast between analytic rationality and value-rationality

Physical science succeeds when it focuses on analytic rationality

Social science

fails when it focuses on analytic rationality, but

succeeds when it focuses on value-rationality


5/22

Research methods 101: causality Does X cause Y?

In the presence of X, Y happened (factual) Problem:post hoc ergo propter hoc

Desired inference: If X had not happened, Y would not have happened

(counterfactual)

Problem: X did happen

So we need to create a parallel world where X did not happen Same group different time (baseline measurement)

Need to assume stability over time

Different group same time (control group)

Need to assume groups are equivalent

Randomized contolled trial


6/22

Plausible rival hypotheses Example: Smoking cigarettes causes lung cancer

Randomized controlled trial not possible

Have to rely on other methods

Logic of inference-making

Establish the warrant for chosen inferences

Establish that plausible rival interpretations are less warranted


7/22

KnowledgeNot justified-true-belief

Discriminability (Goldman, 1976)

Elimination of plausible rival hypotheses

Building knowledge involves:

marshalling evidence to support the desired inference

eliminating plausible rival interpretations

Plausible determined by reference to a theory, a community of

practice, or a dominant discourse


8/22

Inquiry systems (Churchman, 1971)

System Evidence

Leibnizian Rationality

Lockean Observation

Kantian Representation

Hegelian DialecticSingerian Values, ethics and practical consequences


9/22

The Lockean inquirer displays the fundamental data that allexperts agree are accurate and relevant, and then builds a

consistent story out of these. The Kantian inquirer displays

the same story from different points of view, emphasising

thereby that what is put into the story by the internal mode of

representation is not given from the outside. But theHegelian inquirer, using the same data, tells two stories, one

supporting the most prominent policy on one side, the other

supporting the most promising story on the other side

(Churchman, 1971 p. 177).

Inquiry systems


10/22

The is taken to be is a self-imposed imperative of the community. Taken in the

context of the whole Singerian theory of inquiry and progress, the imperative hasthe status of an ethical judgment. That is, the community judges that to accept its

instruction is to bring about a suitable tactic or strategy [...]. The acceptance may

lead to social actions outside of inquiry, or to new kinds of inquiry, or whatever.

Part of the communitys judgement is concerned with the appropriateness of these

actions from an ethical point of view. Hence the linguistic puzzle which bothered

some empiricistshow the inquiring system can pass linguistically from is

statements to ought statements is no puzzle at all in the Singerian inquirer:

the inquiring system speaks exclusively in the ought, the is being only a

convenient faon de parler when one wants to block out the uncertainty in the

discourse. (Churchman, 1971: 202).

Singerian inquiry systems


11/22

Educational researchcan be characterised as a never-ending process of assembling

evidence that: particular inferences are warranted on the basis of the available evidence;

such inferences are more warranted than plausible rival inferences;

the consequences of such inferences are ethically defensible.

The basis for warrants, the other plausible interpretations, and the

ethical bases for defending the consequences, are themselvesconstantly open to scrutiny and question.


12/22

Effective learning environmentsA prevalent, mistaken, view

Teachers create learning The teachers job is to do the learning for the learner

A not so prevalent, not quite so mistaken, but equally dangerous view

Only learners can create learning

The teachers job is to facilitate learning

A difficult to negotiate, middle path

Teaching as the engineering of effective learning environments

Key features:

Create student engagement (pedagogies of engagement)

Well-regulated (pedagogies of contingency)

Develop habits of mind (pedagogies of formation)


13/22

Formative assessment: a definitionAn assessment functions formatively to the extent that evidence about

student achievement elicited by the assessment is interpreted and usedto make decisions about the next steps in instruction that are likely to

be better, or better founded, than the decisions that would have been

taken in the absence of that evidence.

Formative assessment therefore involves the creation of, and

capitalization upon, moments of contingency (short, medium and long

cycle) in instruction with a view to regulating learning (proactive,

interactive, and retroactive). (Wiliam, 2009)


14/22

The formative assessment hi-jackLong-cycle

Span: across units, terms

Length: four weeks to one year

Impact: Student monitoring; curriculum alignment

Medium-cycle

Span: within and between teaching units

Length: one to four weeks

Impact: Improved, student-involved, assessment; teacher cognition about

learningShort-cycle

Span: within and between lessons

Length:

day-by-day: 24 to 48 hours

minute-by-minute: 5 seconds to 2 hours

Impact: classroom practice; student engagement


15/22

Unpacking assessment for learningKey processes

Establishing where the learners are in their learning Establishing where they are going

Working out how to get there

Participants

Teachers Peers

Learners


16/22

Five key strategiesClarifying, understanding, and sharing learning intentions

curriculum philosophyEngineering effective classroom discussions, tasks and activities that

elicit evidence of learning

classroom discourse, interactive whole-class teaching

Providing feedback that moves learners forward

feedbackActivating students as learning resources for one another

collaborative learning, reciprocal teaching, peer-assessment

Activating students as owners of their own learning

metacognition, motivation, interest, attribution, self-assessment

(Wiliam & Thompson, 2007)


17/22

and one big ideaUse evidence about learning to adapt instruction to better meet learner

needs


18/22

A model for professional changeContent

Evidence Ideas

Process

Choice

Flexibility

Small steps Accountability

Support


19/22

KMOFormative Assessment Project24 teachers, each developing their practice in individual ways

Different outcome variablesNo possibility of standardized controls

Polyexperiment with local design

Synthesis by standardized effect size


20/22

5

0

5

6

0 64 6 8 8 8

0 3 4 5 6 6 7 9

5 9

3

4

1.5

1.4

1.3

1.2

1.1

1.0

0.9

0.8

0.7

0.6

0.5

0.40.3

0.2

0.1

0.0

-0.1

-0.2

-0.3

-0.4

Jack-knife estimate of

mean effect size: 0.32;

95%C

.I. [0.16, 0.48)


21/22

Effect size by comparison typeI Parallel set taught by same teacher

in same yearS Similar set taught by same teacher

in previous year

P Parallel set taught by different

teacher in same year

L Similar set taught by different

teacher in previous yearD Non-parallel set taught by different

teacher in same year

N National norms


22/22

SummaryEducational research is a never-completed process of assembling

evidence that: particular inferences are warranted on the basis of the available evidence;

such inferences are more warranted than plausible rival inferences;

the consequences of such inferences are ethically defensible.

The basis for each of these is constantly open to scrutiny and question

asme 09 keynote

Documents