on quality in educational research

ON QUALITY IN EDUCATIONAL RESEARCH

Experiences from evaluations and assessments of projects funded by

UVK(CES) Prof. em. Berit Askling

2008-05-06

• UVK has been allocating research funds since 2001.

• There has been a great pressure on UVK to allocate its funds in the best way possible.

• Many studies have been carried out on projects funded by UVK.

My purpose is not to judge

- the conclusions, relevance and truthfulness of each study

but to

• reflect on the observations drawn from the studies and

• discuss possible improvements with regard to research quality

DATA 1

Researchers production (applications and progress reports).

• Surveys:

Applications 2001-2003 Kim and Olstedt

Applications 2001-2005 Aasen et al

Applications 2006-2007 Askling

Data 2

• Evaluations:

Progress reports Björkqvist et al

Progress reports +

Interviews Arnot et al

Data 3

• UVKs peer review work: grade and priority

Applications 2001-2005 Aasen et al

Applications 2005-2006 Askling

SOME FIGURES 1

Surveys:• Applications 2001-2003 178• Applications 2001-2005 182• Applications 2006-2007 58

Evaluations Progress reports 28 Progress reports + interviews 14

SOME FIGURES 2

• UVKs peer review work:

Applications 2001-2005 182

Applications 2005-2006 58

Some general criticism; many researchers…

• are imprecise and vague when they present what they want to do,

• are unclear in their approaches and arguments when selecting theory and method,

• seem to gather around more or less the same subjects and research areas and work within the same theoretical frames of reference.

Research is domestic and relates to domestic needs and problem areas.

• Surveys and evaluations have

- focused on the scientific and academic dimensions, and

- have been carried out without consideration of the context of the projects concerned.

On the other hand• UVK´s peer-review groups have taken into

account a variety of special conditions listed in the Handbook for Reviewers.

The evaluations and surveys undertaken by UVK raise fundamental questions about;

- quality in educational research

- how quality is evaluated and assessed

- how quality in educational research correspond to expectations of different stakeholders

The surveys and evaluations bring to the fore some issues that ought to be discussed

• What is quality in educational research?

• How to assess and evaluate quality in educational research?

• Who is responsible for quality in educational research?

• On what national and local arenas can quality be assessed and evaluated?

Some questions for discussion:

• Do we really have a problem with research quality or is it a question of research communication?

• Should UVK have higher demands on scientific criteria?

• Are the evaluators too critical (and too decontextualised) in their assessments?

• Is UVK´s instruction – to support basic research of high quality and at the same time support an infrastructure for research – unrealistic?

Some questions…

• Is there a lack of academic quality control within educational research (at institutional, faculty board, and department levels)?

• Are relevance criteria outplaying scientific criteria in educational research?

• Are departments too dependent on external funding (and on “markets”) for keeping a high level of standard in research?