Why ‘outcomes’ aren’t enough

Tim Oates

Group Director

Assessment Research and Development (ARD)

Some system characteristics (indicators?)

1 buoyancy in the labour market for young people in their late teens, until the recent recession

2 a high level of ‘lifelong learning’ after cessation of compulsory schooling

3 decline in employer use of FE relative to the volume of employer-funded training

4 accompanying high flexibility in the structure and content of FE provision

5 persistence in structural skill shortages

6 collapse in provision of ‘classical apprenticeship’

7 the use of imported labour in skilled segments of the economy

8 partial success in the attempt to use HE as a dominant VET route (including variability in return)

9 poor resilience and responsive in the system in respect of offering places during a period of recession and high youth unemployment

Qualifications reform as a principal focus

Schools

Known locations of learning (schools)

Relatively easy to govern

Ownership clear – although parental/state responsibility an issue

Clear populations and cohorts

Legal obligation to attend

Solid research on pedagogy

Powerful instruments of curriculum control

VET (note HE VET ambiguity)

Diverse and shifting locations of learning (workplaces, colleges, PTPs)

Difficult to control, governance institutions unclear

Ownership of/responsibility totally unclear – employers, State, individual

Constantly shifting participation patterns

Complex patterns of requirements and incentives regarding participation

Very little work on workplace pedagogy

So qualifications reform will fix it? …. Unlikely

A reductive framework

QCF

- Notional learning time (size and weight)- Level- Sector

Rationale - Reduction in number of qualifications- Rationalisation of elements; new combination of common elements

Stimulated explosion in qualifications – 22,000? No….6,450 VQs

Increase in QCF category; no commensurate reduction in others

06-07 07-08 08-09 09-10 10-11

QCF 1000 1050 2480 6100 9700

Total 8150 9700 11500 15300 18100

‘Curriculum thinking’

Aims

Content

Methods

Assessment

Evaluation After Michael Eraut

Curricular and extra-curricular

Taught and untaught curriculum

Unstated curriculum (ethos, culture)

The constructed curriculum

Outcomes versus formation

Conceptual ground-clearing 1

The term ‘system’ – as in ‘education and training system’ is a metaphor

‘Systems-analysis’ is a theoretical and practical means of understanding the overall operation and affects of a diverse set of education and training arrangments

Should we aspire to manage these arrangements as a system – do we have the theoretical and practical tools to do this? Are we prepared for the constitutional consequences of a ‘systems management’ approach?

The implications of ‘systems-thinking’

Singapore and Korea – the characteristics of modern industrial strategy

UK attitude to industrial strategy

With the race for international investment going global, how should Britain address its manufacturing sector in order to boost the economy? And how much is this the government’s responsibility, rather than that of industry and businesses? Industrial strategy has been considered an unfashionable term in the past, with some unable to see beyond British industry being in a slow decline since the 1980s. This view could hamper the focus of policy-makers on manufacturing as crucial to better-balanced growth. Coupled with a devastating manufacturing skills gap, this should be carefully considered in the lead-up to the Budget.

Total politics Issue 56 March 2013

Conceptual ground-clearing 2

Implicit routes exist in seemingly ‘unified’ systems – you need to make choices of subject and focus

Seemingly ‘routed’ systems possess different levels of real flexibility – issues of outcomes (attainments), destinations and switching between routes

Conceptual foundation 1Situated cognition

Lave & Wenger GeeNewble & Clarke Seely Brown, Collins & DuguidSociolinguists - context and language acquisition BhatkinBrownBoreham and Samurcay

Anderson Reder & Simon (1998)

-issues of context –bound nature of learning, arguing that this is dependent on the precise content of knowledge being acquired

-issues regarding the extent to which abstraction is inappropriate for initial acquisition; iteration of abstract concepts and concrete exemplification is vital

-knowledge (and skill) is not resolutely context-bound; generalisation is possible

-context is characterised by complexity; there are issues of orientation and selective engagement

Specific skills associated with managing equipment, materials and compounds, measuring, observing, reporting

Levels of analysis

Generic issues of reliability

Occupational

Enterprise

Job role

Individual issues of currency

False distinction in that functional competence (held by individuals) can be described as generic but is necessarily constituted in an individual manner and is linked to the circumstances of its formation

Work process knowledge and distributed competence

Conceptual foundation 2Learning as well as outcomes

French VET system – ‘formation’ German Dual System – ‘Beruf’ English apprenticeship system – ‘timeserving’

‘Curriculum thinking’

Aims

Content

Methods

Assessment

Evaluation After Michael Eraut

Curricular and extra-curricular

Taught and untaught curriculum

Unstated curriculum (ethos, culture)

The constructed curriculum

Outcomes versus formation

‘Curriculum thinking’

Aims

Content

Methods

Assessment

Evaluation After Michael Eraut

Curricular and extra-curricular

Taught and untaught curriculum

Unstated curriculum (ethos, culture)

The constructed curriculum

Outcomes versus formation

The National Curriculum 1995

Chemical Reactions

i. that when chemical reactions take place, mass is conserved;j. that virtually all materials, including those in living systems, are

made through chemical reactions;k. to represent chemical reactions by word equations;l. that there are different types of reaction, including oxidation and

thermal decomposition;m. that useful products can be made from chemical reactions, including

the production of metals from metal oxides;n. about chemical reactions, e.g. corrosion of iron, spoiling of food, that

are generally not useful;o. that energy transfers that accompany chemical reactions, including

the burning of fuels, can be controlled and used;p. about possible effects of burning fossil fuels on the environment.

Science - key stage 3Materials and properties

National Curriculum 2007

Chemical and Material BehaviourIn their study of science, the following should be covered:

a. chemical change takes place by the rearrangement of atoms in substances;

b. there are patterns in the chemical reactions between substances;

c. new materials are made from natural resources by chemical reactions;

d. the properties of a material determine its uses.

Securing precision of the construct – science practical work

specific outcomes – the HE requirement for specific preparation – mastery?

persistence (Deakin Crick)

precision (Fotheringham)

methods of measurement

experimental design

experimental control and threats to validity

psychomotor co-ordination

principles and practices of safe working

ethics of investigation

observation

recording

teamworking and collaboration

‘beruf’

concepts of proof, adequacy, verification, falsification

The number of qualifications in England

Comparison with Germany 2005-2007

National Database of Accredited Qualifications (source: QCA-DfES) May 2007 5830 total

938 mainstream academic qualifications

4800 vocational and ‘other’ qualifications

Vocational qualifications in the German system (source: BIBB)

4070 total

150 FT State-specific qualifications

420 vocational qualifications in the Dual System of Apprenticeship (3yr employment-based programmes in which around 50% of the 16-19 cohort participate)

3,000 chamber of commerce qualifications

500 continuing education qualifications

Note – this is a careful comparison of ‘like with like’ – in neither case does it include industry-owned specific qualifications

The number of qualifications in England

My conclusion in 2006-07:

 

There is no basis for using 20,000 as the baseline figure for the English system. There is no compelling argument that the system is incoherent, compared with other benchmark nations

Comments collected from industry at that time: there are issues regarding gaps – particularly maths - and some duplication in information technology qualifications.

Rapid growth in NVQ2 care as a result of change of labour market requirement

Oxford work on Skills Survey 2000

Qualifications supply and combination

QCF implied combination and saliency

Policy maker: ‘There are hundreds which are not used…’

Me ‘So what?...they’re not used...’

Policy maker: ‘Remove all with fewer than 500 candidates…’

Me ‘what about the 50 who qualified in bomb disposal?...and…”

QCF lacked principles of combination

QCF lacked criteria of need (utility and saliency)

Material impediments to shared units

Relatively unfettered escalation in qualifications

Failure of principle or a failure of implementation?

Neurotic obsession with system tidiness.

Combination: purpose and needs

It is vital to differentiate:

 •school-based VET (including work experience) as a component of compulsory general education

•initial VET in a full-time educational setting

•employment-based for young entrants to the labour market – focused VET for comprehensive labour market preparation

•continuing VET for adult employed workers

•VET for unemployed adults

‘Frameworks’ as a policy instrumentThe importance of higher order criteria

- Education and training ‘system’ is a metaphor- Qualifications arrangements - Qualifications frameworks

- Clear progression routes - Absence of inefficiencies- Absence of problems of knowledge and skill supply - High levels of recognition and trust- High levels of motivation and attainment

Frameworks have emerged as a relatively recent policy instrument

Understanding relations as well as objects

- Curriculum coherence - Curriculum control

- Explanatory factors- Control factors

Control factors

1 curriculum content (nc specifications, support materials, etc)2 assessment and qualifications3 national framework for qualifications4 inspection5 pedagogy6 professional development 7 institutional development8 institutional forms and structures (eg size of schools, education phases)9 allied social measures (linking social care, health care and education)10 funding 11 governance (autonomy versus direct control)12 accountability arrangements13 labour market/professional licensing 14 allied market regulation (eg health and safety legislation; insurance regulation)

Oates T 2010 Could do better: using international comparisons to refine the National Curriculum in England Cambridge Assessment

Curriculum control

A system must exercise control, it is not that individual agencies should take control:

‘…our purpose in introducing alternative ways to govern curriculum…is not to advocate one approach or another. As analysis by Cochran-Smith and Fries (2001) indicates, disagreements about teaching and, by implication, curriculum, often divides along ideological lines, an outcome that occurs no matter how pragmatic the veneer. A functional approach, by specifying in advance the criteria that an effective curriculum-governance system must meet, lessens the tendency to judge these systems in terms of the political values they represent (eg regulation vs deregulation, public interest vs private

interest…’

Schmidt W & Prawat R 2006 p656

Qualification frameworks as policy instruments

Key distinctions

Descriptive frameworks (accommodation – ‘relating’ frameworks

Prescriptive frameworks (change) – ‘realigning’ frameworks

30 levels in CINOP project

12 levels in Scotland – the rationality shown by empty cells

Functions

-showing relationships (level, ‘weight’, worth)

-indicating progression routes

-rationalising qualifications

Qualification frameworks as policy instruments

EQF – metaframework

Analytic function – labour market mobility as prime aim)

NQF – national framework

Developing equivalences and rationalising the system as awhole

QCF – formula for recognition in addition to framework

Reducing the number of qualifications, duplication and redundancy

Who is a framework for, exactly?

QCA website statement: ‘for learners to navigate the system’

Erosion of concept of formation

Ditch lessons, schools are toldGeraldine Hackett, Education Correspondent 24 06 2007

‘State secondary schools are being told to ditch lessons in academic subjects and replace them with month-long projects on themes such as global warming.

Quote from official “At the moment most schools are in the traditional mindset, which means they organise the day into a meaningful form take content and divide it up into fragments called timetables. They do it as it has always been done…The idea is to offer less education prescription and more opportunity to interpret the curriculum. Cutting across all subjects are curriculum dimensions; a set of themes including creativity, cultural understanding and diversity.”

Classical apprenticeship

Internal economics of classical apprenticeship Merits of long duration Formation through long duration Formation as a removal of the need to specify and assess externally Material efficiency of inflexibilities

Re-establishing curriculum thinking in general education as well as VET

QCF impacts and issues

High cost of QCF compliance

Small quals with genuine utility removed from the system

Some ABs forced out of business - undue market restriction

Material impediments to shared units

Relatively unfettered escalation in qualifications

Comments from contracted developers

Failure of principle or a failure of implementation?

Neurotic obsession with system tidiness.

Insights from German VET

External appearance of incoherence:

364 training lines (high labour market mobility)

Each set of standards revised on a sector basis (High saliency)

Qualification is associated with ‘formation’ not with certification

Restriction vital to the system

Low complexity in QA arrangements around qualifications

Social partnership essential to form of system (the institutional form of curriculum control)

(Reuling on modularisation 1995)

Is this for England? No naïve policy borrowing.

Modualisation and ‘outcomes orientation’

In general education

Greater clarity in learning outcomes

Early feedback on performance

Gains in motivation and engagement

Clear conceptual sequencing in material

But

A very significant increase in assessment load

A truncation of learning processes

Problems in establishing comparability

A narrow focus on assessment objectives, contributing to narrow instrumentalism in schools and colleges

Modualisation and ‘outcomes orientation’

In vocational education and training

Greater linkage between assessment and requirements of work

High specificity, precision and transparency in required outcomes

Validity leading to higher reliability

More flexible provision, reduced programme cost, meeting LM needs

Recognition of prior attainment

Qualifications capable of being used across all VET routes

But

Escalation in cost and complexity of quality assurance

Significant problems in reliability and comparability

Reducation of employers’ incentives to train (erosion of apprenticeship)

Failure to recognise more subtle ‘work process’ skills

The 40 functions Oates & Cole CEDEFOP ‘Changing Qualifications’ report (2010)

1 Documenting the outcomes of learning

3 Signalling

6 Controlling flows into specific occupations

7 Empowering citizens

8 Re-aligning control of professions

10 Measuring the performance of the education system

14 Guaranteeing the quality of provision, for funding agencies

16 Affecting the identity of learners

25 Influencing the content of learning programmes

35 Providing feedback to learners (formative and diagnostic function)

Post-16 Mathematics – an important case study

An emergent gap in qualification arrangements identified through domestic research and international comparisons

Research undertaken by King’s College London, sponsored by the Nuffield Foundation

An increase in the maths requirements of a broad range of science and social science degree courses

A level and AS qualifications too specialist and high level

GCSE repeat or retake demotivating, higher level, refined content required

‘Functional skills’ qualis inappropriate; higher level, refined content required

The NQF/QCF did not indicate or highlight deficit in qualifications ‘catalogue’ – specific research on needs and requirements was required

Concluding remarks 1 (of 3)

System tidiness is a misleading preoccupation

Vertical progression is vital; clarity in signalling and opportunity

Coherence and knowledge symmetry in sectors is vital

‘Level 2’ hairdressing, ‘Level 2 Engineering’ – why is this not immediately a scandal and a problem in the economy?

Labour market flexibility and skills supply vital – lessons from Germany

Bridging at key points in the system are critical

Real systems (‘arrangements’) are messy

Concluding remarks 2 (of 3)

High quality qualifications arrangements are not dependent on the existence of a qualifications framework

Dependence on qualifications frameworks can lead to over-simplistic approaches to the development of effective qualifications arrangements and effective learning provision – signs from the Richards Review

Multiple purposes for qualifications are likely – but curriculum coherence is vital; it requires sophisticated and evidence-based policy formation, management of ‘over determination’ and adjuvant policy, combined with subtle monitoring of impact and system performance

Concluding remarks 3 (of 3)

The assets of outcomes:

- drive to remove dysfunctional duplication - drives analysis of the composition of competence

But

- ‘level’ of analysis is crucial – theory matters - robust and explicit method and aim are vital - can encourage reductivist move from ‘formation’ and ‘beruf’ - can drive policy towards a consideration of ‘policy objects’ and a

neglect of ‘complex relations’