assessing progression in creativity and critical thinking skills by stéphan vincent-lancrin (oecd)

ASSESSING PROGRESSION IN CREATIVE

AND CRITICAL THINKING SKILLS

Stéphan Vincent-Lancrin Deputy Head, Senior Analyst and Project Lead, OECD Centre for Educational Research and Innovation

Paris, 20 June 2016

objectives of the meeting

• What are we talking about when we talk about creativity and critical thinking in the higher education context?

• Peer learning about pedagogies trying to intentionally foster students’ creativity and critical thinking – Can we group them in categories?

– What characterises them?

• How are students assessed to show the acquisition of these skills?

• What advice to start an action research in education aligned with what we are doing in schooling?

Objectives of the meeting

• First day mainly about pedagogies:

– Design thinking

– Entrepreneurship

– Other approaches

• Second day morning mainly about evaluation and assessment:

– Standardised tools

– Faculty-designed asssessments

– Qualification frameworks

• Second day afternoon

– Discussing a possible international action research to showcase how higher education institutions can foster their students’ creativity and critical thinking

Structure of the meeting

skills for innovation

Skills and education for innovation « 21st Century Skills »

Innovation

Skills

Education and

training

Skills that tertiary-educated professionals

report as very important in their job

Percentage of employees reporting the following skills as very important in their job

Source: OECD, based on REFLEX and HEGESCO data

22.7

30.6

40.2

40.3

40.4

40.4

40.5

41.8

46.5

48.0

50.0

53.4

54.2

55.0

56.5

56.9

58.6

60.8

61.7

0 10 20 30 40 50 60 70

knowledge of other fieldswrite and speak a foreign language

assert your authoritynegociate

alertness to opportunitiespresent ideas in audience

willingness to question ideasmobilize capacities of others

analytical thinkingcome with news ideas/solutions

write reports or documentsacquire new knowledge

master of your own fieldcoordinate activities

use computers and internetmake your meaning clear

work productively with othersperform under pressure

use time efficiently

Critical skills for the most innovative jobs

(according to tertiary-educated workers) Likelihood (odds ratios) of reporting the following skills: people

in the most innovative jobs vs. least innovative jobs

Source: Avvisati, Jacotin and Vincent-Lancrin (2013), based on REFLEX and HEGESCO data

1.83

2.05

2.08

2.09

2.15

2.19

2.19

2.20

2.35

2.36

2.36

2.42

2.51

2.58

2.60

2.71

2.81

3.00 3.90

1

assert your authorityknowledge of other fields

negociateperform under pressure

use time efficientlywork productively with others

use computers and internetwrite and speak a foreign language

write reports or documentsmaster of your own field

make your meaning clearmobilize capacities of others

acquire new knowledgecoordinate activities

analytical thinkingalertness to opportunitiespresent ideas in audience

willingness to question ideascome with news ideas/solutions

Top 10 skills for future jobs according to

employers (2016)

2020

1. Complex problem solving

2. Critical thinking

3. Creativity

4. People management

5. Coordinating with others

6. Emotional intelligence

7. Judgment and decision making

8. Service orientation

9. Negotiation

10. Cognitive flexibility

2015

1. Complex problem solving

2. Coordinating with others

3. People management

4. Critical thinking

5. Negotiation

6. Quality control

7. Service orientation

8. Judgment and decision making

9. Active listening

10. Creativity

World Economic Forum

What individual skills should

education systems foster?

Technical skills (know-what and know-

how)

Creativity and critical thinking

skills (Critical thinking,

observation, curiosity, ability to make

connections, imagination,...)

Social and behavioural skills (Self-confidence, energy,

perseverance, passion, leadership, collaboration,

communication)

Some comments on these skill

categories

• They are domain-specific

– Skills are generally domain-specific: one is creative in a field, one knows how to behave/communicate in a specific context, one has problem-solving skills in a field, one has content knowledge in a field

• They can become « domain-generic »

– A skills becomes « domain-generic » when one has gained it in a number of domains or settings, so that it becomes a « habit of mind » (a disposition or a stabilised skill) that one can apply to new fields

• They overlap and may reinforce each other

But

• They are different and cannot be reduced to a single skill (or measure)

Technical skills

Skills in thinking and

creativity

Behavioural and social skills

(character)

Skills and education for innovation « 21st Century Skills »

Well-being

Skills

Education and

training

higher education for innovation

Higher education for innovation

Innovation

Skills

Education and

training

• Which graduates get highly innovative jobs?

• Do higher education systems foster skills for innovation?

• Which pedagogic practices are associated with highly innovative jobs?

What share of graduates of a given field

have a highly innovative job?

20.3

22.9

23.4

25.4

27.6

28.1

28.1

28.4

28.8

30.9

32.7

36.5

37.6

0 10 20 30 40 50

law

health

humanities

others

business

sciences & maths

education

average

social sciences

architecture

agriculture

arts

engineering & computing

Product /service innovation



12.4

12.6

13.2

15.2

16.0

18.4

18.6

21.0

23.9

24.2

28.6

28.8

38.4

0 10 20 30 40 50

law

others

humanities

social sciences

education

health

business

average

arts

agriculture

architecture

sciences & maths


Technology, tool, instrument innovation



30.4

31.5

35.1

36.1

37.6

38.0

38.3

38.5

39.5

39.9

41.7

41.7

48.3

0 10 20 30 40 50

law

others

business

humanities

social sciences

arts

average

health

agriculture

architecture


education

sciences & maths

Knowledge / method innovation


Innovation

Skills

Education and

training




Strong points of higher education

0 2 4 6 8 10 12 14 16 18 20

assert your authoritymobilize capacities of others

negociatealertness to opportunities

willingness to question ideascoordinate activities

write and speak a foreign languagemake your meaning clear

come with news ideas/solutionspresent ideas in audienceknowledge of other fields

use time efficientlyuse computers and internet

perform under pressurework productively with others

write reports or documentsacquire new knowledge

analytical thinkingmaster of your own field

Ranking of 3 top strong skills by graduates

Source: OECD, based on REFLEX, HEGESCO and PROFLEX data

Weak points of higher education

0 2 4 6 8 10 12 14 16 18 20

acquire new knowledgemaster of your own field

make your meaning clearcoordinate activities

work productively with otherswillingness to question ideas

analytical thinkingperform under pressure

use time efficientlyalertness to opportunities

come with news ideas/solutionswrite reports or documents

mobilize capacities of othersknowledge of other fields

use computers and internetpresent ideas in audience

assert your authoritynegociate

write and speak a foreign language

Ranking of 3 top weak skills by graduates

Source: OECD, based on REFLEX, HEGESCO and PROFLEX data


Innovation

Skills

Education and

training




Innovation and the relative emphasis on

practice- and theory-based instruction

0.95

1

1.05

1.1

1.15

1.2

engineering business health education science others

practice score theory score

Odds ratios between innovators and non-innovators, by field of study


Innovation and the relative emphasis on

practice- and theory-based instruction

0.95

1

1.05

1.1

1.15

1.2

any innovation technology, tools product, service knowledge,methods

practice score theory score

Odds ratios between innovators and non-innovators, by type of innovation


ICT-enhanced pedagogic models to

foster « skills for innovation » in STEM

• Virtual and remote laboratories

• Educational games

• Technology-enhanced cooperative learning

• Real-time formative assessment

• Skills-based assessment

Problem-based learning in higher

education

Please cite this paper as:

Hoidn, S. and K. Kärkkäinen (2014), “Promoting Skills forInnovation in Higher Education: A Literature Review on theEffectiveness of Problem-based Learning and of TeachingBehaviours”, OECD Education Working Papers, No. 100,OECD Publishing.http://dx.doi.org/10.1787/5k3tsj67l226-en

OECD Education Working PapersNo. 100

Promoting Skills forInnovation in HigherEducation

A LITERATURE REVIEW ON THEEFFECTIVENESS OF PROBLEM-BASEDLEARNING AND OF TEACHING BEHAVIOURS

Sabine Hoidn, Kiira Kärkkäinen

• PBL more beneficial regarding long-term retention and application of knowledge and skills

• PBL has positive impact on students’ motivation, satisfaction, and attitudes toward learning

• Traditional approaches seem to have an edge for test scores and knowledge acquisition, but not significant

fostering and assessing creativity and critical thinking in education

• 2 networks:

– Higher education

– School (primary and secondary)

• 3 possible domains:

– One STEM discipline

– One arts education discipline

– One generic approach to innovation/creativity

• Activities in primary-secondary education:

– Develop a pedagogical toolkit (rubric, pedagogical activities and assessments, student work)

– Document pedagogies to foster the selected skills/dispositions

• Activities in tertiary education:

– Share about pedagogies and identification of skills

– Pedagogical intervention like in primary and secondary?

Ongoing OECD project on assessing

progression in creativity and critical thinking

Participants in 14 countries (June 2016)

• Brazil

• China

• Finland

• France (3 teams)

• India

• Italy

• Hungary

• Netherlands

• Russia

• Slovakia

• Spain

• Thailand

• UK (Wales)

• USA (4 teams)

Initial version of the rubric (not to be shared)

CREATIVITY

(Coming up with ideas and solutions)

CRITICAL THINKING

(Questionning and evaluating ideas and

solutions)

Progression

INQUIRE

Feel, empathise, observe, describe relevant experience and information

Explore, seek and generate ideas

Understand context/frame and boundaries of the problem

Review alternative theories and opinions and compare/find perspectives on the problem

Level 1 Level 2 Level 3 Level 4

IMAGINE

Make connections, integrate other disciplinary perspectives

Stretch and play with unusual/risky/radical ideas

Identify strengths and weaknesses of evidence, arguments, claims and beliefs

Challenge assumptions, check accuracy, analyse gaps in knowledge


DO / SHARE

Envision / Express / Produce / Prototype new product / solution / performance

Appreciate the novelty of solution and/or possible consequences

Appraise / Base / Justify opinion/products on logical, ethical or aesthetic criteria/reasoning

Acknowledge own bias (as perceived by others) and uncertainty/limits of endorsed opinion/solution


Creativity

Inquire

Imagine

Do / Share

Feel, empathise, observe, describe relevant experience and information

Explore, seek and generate ideas

Make connections, integrate other disciplinary perspectives

Stretch and play with unusual/risky/radical ideas


Appreciate the novelty of solution and/or possible consequences

Critical thinking

Inquire

Imagine

Do / Share

Understand context/frame and boundaries of the problem

Review alternative theories and opinions and compare/find perspectives on the problem


Challenge assumptions, check accuracy, analyse gaps in knowledge

Appraise / Base / Justify opinion/products on logical, ethical or aesthetic criteria/reasoning


A rubric to help develop lesson plans

• Mapping of the different steps of a lesson plan on the rubric

• Not all dimensions need to be covered

• Generally task are mainly about creativity or critical thinking (but can cover some habits of both)

• Develop scoring rubrics

• Develop student self-assessment rubrics

• Develop domain-specific rubrics

• To be further simplified

A rubric to…

The NCU rubric on creative thinking

• A, B: Defining problem, use of evidence

• C: Identifies/challenges assumptions

• D: Logical and inventive combination of ideas

• E: Contradictory perspectives

• F, I: Multiple solutions to problem

• G, J: Originality

• H: Changes when necessary

• K: Concludes by weighing evidence

• L, M: Shares coherently

Creativity / match with NCU rubric

Inquire

Imagine

Do / Share

Feel, empathise, observe, describe relevant experience and information - A

Explore, seek and generate ideas – F, G

Make connections, integrate other disciplinary perspectives - D

Stretch and play with unusual/risky/radical ideas - G


Appreciate the novelty of solution and/or possible consequences – K, L?

Critical thinking/ match with NCU rubric

Inquire

Imagine

Do / Share

Understand context/frame and boundaries of the problem - A

Review alternative theories and opinions and compare/find perspectives on the problem – E, F


Challenge assumptions, check accuracy, analyse gaps in knowledge - C

Appraise / Base / Justify opinion/products on logical, ethical or aesthetic criteria/reasoning – B, M


• What is missing?

• What would you omit?

• How would you simplify?

Your feedback on the current version of

the OECD rubric

a controlled pedagogical intervention in higher education?

A pedagogical toolkit

A. The toolkit A1.

Rubric for

assessment

Dimensions

Levels of progression

A2.

Pedagogical activities

Specific for each domain

Designed to test

dimensions and levels of progression

A3.

Set of exercises

Specific for each domain

Designed to prepare for

the assessment

A4.

Portfolio of student work

For each domain

As examples of different skill levels

Contextual data collection

B1.

Subject-based assessment

Standardised assessment of

academic achievement

(maths and science; visual

arts and music)

B2.

Creativity assessment

Standardised test for

creativity

(EPoC)

(domain-specific)

B3.

Survey questionnaires

School principals

Teachers

Students

B4.

Interviews / focus groups

Teachers

Students

School-based intervention

School year

A A A A

Primary school group - Ages 8-9. STEM, Arts or interdisciplinary domain

Secondary school group - Ages 13-14. STEM, Arts or interdisciplinary domain

A

What factors influence the outcomes?

• Pre-tests and questionnaires at the beginning of the intervention: – Are differences related to baseline in achievement, creativity, to student

beliefs, to pedagogies, to socio-economic bacground, etc.?

• Observations and discussions within the network

B

B

School-based intervention

School year

A A A A

Primary school group - Ages 8-9. STEM, Arts or interdisciplinary domain

Secondary school group - Ages 13-14. STEM, Arts or interdisciplinary domain

A

What effects for the intervention?

• Measures after the intervention: – Post-tests and -questionnaires

– Qualitative observations of pedagogies

– Interviews, ESM, executive functions

• Matched control group (with some kind of intervention as well)

Control group

Control group

B

B

B

B

• What are the differences across country school networks in terms of beliefs/attitudes and pedagogies (and baselines)?

• Is there change in student and teacher beliefs/attitudes and practices?

• What effect on standardised achievement tests?

• What effect on standardised creativity tests?

• Any difference by socio-economic background, by achievement level, by type of pedagogy, etc.?

• What are the perceived hurdles to implementing these pedagogical approaches?

Questions we would like to answer

What domains (ideally)?

Domains

• Engineering

• Teacher training/education

• Humanities / Social science

• Innovation projects

• Arts

• Science

• Professional / technical education

Levels

• 20 year olds

• First year students

• Last year of bachelor

Institutions

• Top tier

• Average

• Short cycles

• Questionnaires (beliefs/attitudes, pedagogy, anchoring vignettes) – Students

– Teachers

– Deans

• Domain-specific achievement tests (with embedded questions on interest, curiosity, attitudes, and domain-specific pedagogies)

• Domain-specific creativity tests

• Domain-specific critical thinking tests

• (Good domain general instruments?)

What instruments?

• Under what conditions could HEI be interested?

– Should we work with a few institutions?

– Could we have a large network of institutions?

– Could we build on existing initiatives?

• What funding model?

– Would HEIs cover their own costs? Who else could fund

their costs?

– Who could fund the international costs (OECD development and coordination work)?

Size of pilot and funding model

other suggested ways to support the agenda

www.oecd.org/edu/innovation

www.oecd.org/edu/universityfutures

www.oecd.org/edu/internationalisation

[email protected]

THANK YOU

http://www.oecd.org/edu/innovation

http://www.oecd.org/edu/innovation

http://www.oecd.org/edu/universityfutures

http://www.oecd.org/edu/internationalisation

mailto:[email protected]

