dpme evaluations - south african qualifications authority presentation.pdf · evaluation: what is...

NQF Impact Workshop,

Midrand

Antonio Hercules (DPME)

09 February 2016

DPME Evaluations

Outline

1. Evaluation: what is it? [2 minutes!] 2. NES and Types of Evaluation. what these are [6

mins!] 3. NQF Impact Study vs. NES Impact Evaluation.

the study and impact evaluation [5 minutes!] 4. 2017/18 NES Implementation Evaluation

current NES evaluation [5 minutes!] 5. Presentation Key messages

2

Evaluation – easy word, specific meaning

As a field, its value and its shortcomings 1. Field: about 30-40 years, imprecise but improving, specifically deals with policy/programme/project as the object, specialised, mixed methods, sensitive to bias

2. Value: offers “objective” external and “expert” internal approach options. great window for renewal/improvement, highly flexible, decades of practice strongly evidence-based

3

Evaluation – easy word, specific meaning

3. Shortcomings Complex Duration Financial investment Must have buy-in Needs good and committed managers/political principals to

take forward Evaluation is different from “normal”

evaluative reflection in policy analysis/research

4

National Evaluation System and Evaluation

Six (6) guidelines (for policy/prog./proj.) 1. Diagnostic*: … “identification”

“research” – info. foundations 2. Design*: “formulation” 3. Implementation: performance

(results), everything short of impact 4. Impact: using quantitative RCT

approach conventionally 5. Economic*: …economic benefits 6. Synthesis*: …synthesis of pre-existing

evaluations

5

NQF Impact Study vs. NES Impact Evaluation

1. NQF Impact Study has value in its own right 2. Key question: would Study pass in the NES? 3. Main reasons: Largely impact evaluation TOR, but methodology does not

deliver accepted evaluation design More of a “synthesis” evaluation pulling together existing

bodies of knowledge -> produces overview, trends, insights, but doesn’t subject underlying evaluation design of supporting reports to data verification/validation testing

Does not appear to comply with accepted large system impact evaluation design (correlations, regressions, etc.)

Still very useful, and well-timed for NQF Act policy implementation evaluation (NES – NEP 2016/2017)

6

Typical Impact Evaluation Approach

7

• Policy • Goals • Results

1 Results Frame-work

• Theory of Change

• Logframe • Operational

Strategy • “Nesting”

2 Logic

• Literature • Interviews • Focus

Groups • Survey • Databases

3 Evidence

• Policy Narrative

• Evidence • Analysis • Results • Conclusions

4 Assess-ment

• Sound • Specific • Based on

evidence

5 Recom-mend-ations

2017/18 NES Implementation Evaluation

Purpose: to assess the implementation of the NQF Act relative to its goal(s) and objectives in the period of review, including its associated policies and regulations. Successes and challenges in its implementation will be identified, and recommendations offered regarding improvements to implementation of the NQF Act in the future.

8

1. (Relevance and Appropriateness) To what extent is the theory of change (intervention logic) of the NQF Act adequately robust, including its main underlying assumptions?

2. (Effectiveness) To what extent has the implementation of the NQF Act been effective in achieving its policy goal(s), objectives and intended outcomes?

3. (Efficiency) To what extent has the implementation of the NQF Act been efficient?

4. (Impact) What is the emerging impact of the NQF, if any? 5. What needs to be done to improve the implementation

of the NQF Act, and which aspects of the NQF Act need to be reviewed?

9

2017/18 NES Implementation Evaluation

Sufficient teaching

time

Nutritious meals delivered to

schools on time

Nutritious meals eaten

Improved health and nutritional status of South African primary school children

Better educational performance

Improved levels of

primary school attendance

Improved concentration

in class

Activities

Local business prospers

Department of Education pays

for food delivered

Outputs

Children have adequate nutrition

Outcomes Impact

Assumptions

Local producers provide the

food/ingredients at fair price

Inputs

Funds

Local SMMEs

appointed to render

services

Decentralised purchasing

system Fair

procurement Payments to producers/suppliers

made on time

Food prepared locally and safely

Immediate Intermediate

Schools have adequate

food storage facilities

National School Nutrition Programme TOC

10

TOC example: Aspen Institute (2003)

12

Thanks! Antonio Hercules: Director: Evaluations & Research.

antonio@presidency-dpme.gov.za

Head: Evaluation and Research, DPME. Ian Goldman. Ian@presidency-dpme.gov.za

www.thepresidency-dpme.gov.za