1

A Monitoring and Evaluation

Framework for the ETDP SETA

Prepared by: Research Chair – Evaluation

UKZN

October 2016

2

TABLE OF CONTENTS

1. Executive Summary ………………………………………………………………………. 3

2. The Purpose of M & E ……………………………………………………………………. 5

3. Scope of M & E………………………………………………………………………………. 6

4. ETDP SETA Approach to M & E…………………………………………………… …. 6

5. Levels of M & E………………………………………………………………………………. 9

6. Key Indicators and Priority Targets ………………………………………………… 11

7. Evaluations …………………………………………………………………………………… 12

8. Data Collection ……………………………………………………………………………… 14

9. Reporting ………………………………………………………………………………………. 14

10. Annexures ……………………………………………………………………………………….16

10.1 The Results Framework of the ETDP SETA’s strategic outcome-

oriented goals 2015/16 – 2019/20 …………………………………………… 16

10.2 The Proposed M & E Plan and Institutionalization Strategy ………..20

3

Executive summary

The M & E Framework proposed here sets out the minimum monitoring and

evaluation requirements that will enable collection and provision of indicators of

progress, or lack thereof, and effective review of the ETDP SETA’s programmes. It will

guide the overall monitoring and evaluation functions of the SETA during

implementation of its current Five Year Strategic Plan 2015/16 to 2019/20.

The M & E activities will essentially be focused on the performance of six programmes

that have been formulated by the ETDP SETA to respond to the national priorities of

seeking that there should be quality basic education (outcome 6.1) and a skilled and

capable workforce to support an inclusive growth path (outcome 6.5) that the SETA

aims to contribute to. The proposed participatory M & E system is underscored by the

belief that effective implementation of M & E strategies requires the buy-in and a sense

of ownership by those who will be practising it. For the strategies to succeed, M & E

should be institutionalized to involve people and their capacities, processes and

systems.

In essence the M & E activities proposed in this framework will take place at two

distinct but closely connected dimensions: one level will focus on the outputs, which

are the specific products and services that emerge from processing inputs through

programme activities, whilst the other focuses on the outcomes and impacts which are

the changes that the ETDP SETA aims to achieve through the delivery of its

programmes. As the custodian of and resource for M & E in the ETDP SETA, the M &

E Unit will continue to conduct monthly and quarterly performance monitoring and

conduct field monitoring visits to verify the information gathered. Working closely

with the Research Chairs that have been established, the M & E Unit in collaboration

with the Research and Skills Planning Unit and the Programme Managers will

continue to develop and/or upgrade existing monitoring tools to ensure their fit-for-

purpose with current programmes and plans that will be monitored and evaluated.

Risks that have been identified per each of the six programmes should be monitored

simultaneously with the activities, outputs and outcomes by the programme

staff/managers and the M & E Unit and these should be reflected upon in the quarterly

reports that will be produced.

A results framework gives an indication of what will be monitored and evaluated at the

programme level and it includes indicators, baselines, targets and means of

verification that will be tracked. Special care should be taken though to ensure that

demand for objective verification does not lead to exclusive focus on quantitative

4

indicators at the expense of more qualitative indicators which may be harder to verify

but may better capture the essence of the changes that are taking place.

Evaluations will not be one-time events; instead they will be exercises involving

assessments of differing scope and depth carried out at several points in time in

response to evolving needs for evaluative knowledge and learning. A key requirement

for all evaluations will be that they should be linked to the identified outcomes as

opposed to only implementation or immediate outputs. It is also expected that

evaluations will not only answer questions generated by the monitoring process but

will importantly also draw heavily on data generated through monitoring which

include baseline data, information on the programmes’ implementation processes,

and measurement of progress towards the planned results through indicators. Several

criteria are put forward for consideration when decisions on what to evaluate are

made, and these include factors such as: use, purpose and timing of an evaluation,

resources invested, the likelihood of future initiatives in the same are or programme,

anticipated problems, need for lessons learnt, and alignment and harmonization to

national, regional/provincial and the ETDP SETA’a corporate priorities.

The data collected in monitoring and evaluation processes will need to provide the

information that is necessary to measure the indicators. The M & E team must identify

how to collect the information necessary to measure the performance indicators,

whether it will be through existing data collection systems or whether new systems will

be developed. Currently all structures of the ETDP SETA have some M & E functions

and therefore report on M & E.

Finally an M & E Plan is presented together with an institutionalization strategy as an

integral component of the M & E Framework. The institutionalization strategy

proposes that a gradual approach to phase in participatory approaches to M & E in the

ETDP SETA be adopted in order to prevent disruptions and instability that may result

when new methods and systems are abruptly introduced. The proposed participatory

methods will be introduced as part of the institutionalization of M & E process in the

capacity building training that is scheduled to start in 2017.

5

1. The Purpose of M & E

Monitoring is conventionally defined as a continuing function that aims primarily to

provide the management and main stakeholders of an ongoing intervention with early

indicators of progress, or lack thereof, in the achievement of results. Evaluation, on

the other hand, is a selective exercise that attempts to systematically and objectively

assess progress towards achievement of results in an intervention. It involves

assessments of differing scope and depth carried out at several points in time in

response to evolving needs for evaluative knowledge and learning during the effort to

achieve results. The overall purpose of monitoring and evaluation (M & E) is the

measurement and assessment of performance in order to more effectively manage the

outcomes and outputs known as development results. Performance is defined as

progress towards and achievement of results. Information from regular and planned

monitoring and evaluation activities is required for an effective and timely decision

making. Monitoring provides real-time information on ongoing programme or project

implementation required by management, whilst evaluation provides more in-depth

assessments. Whereas questions that are answered by evaluations are usually

generated by the monitoring process, evaluations on the other hand often draw heavily

on data generated through monitoring, including baseline data, information on the

programme or project implementation process, and measurement of progress towards

the planned results through indicators.

The ETDP SETA is a state agency charged with promoting and facilitating skills

development in the education, training and development sector in order to raise the

skills levels of employees and achieve a healthy balance between supply of and demand

for skills in the country’s labour market. In the context of its general mandate of

promoting skills development in the country, the ETDP SETA has developed a sector

skills plan (SSP) that is annually updated to guide implementation of its Five Year

Strategic Plan (2015/16 to 2019/20) which is the overarching programmatic

framework through which it aims to achieve its goals in the coming five years. Short to

medium term annual performance plans ( APPs) are simultaneously developed to

provide detailed annual targets and plans that will be implemented cumulatively to

build up to the goals that will be achieved in five years. To ensure that the intended

results in its interventions are achieved as planned and corrective actions are taken

timeously in the delivery of its programmes, the ETDP SETA has set up monitoring

and evaluation (M & E) systems and processes that are fully integrated in the planning

and implementation of its programmes. Alongside its Skills Planning and Research,

and the M & E Units, six Research Chairs (assisted by 11 Research Interns) have been

set up with selected higher education institutions (HEIs) to beef up its research

capacity to ensure that sound evidence-based policy decisions guide its endeavours.

This M & E Framework sets out the proposed minimum monitoring and evaluation

requirements to enable collection and provision of indicators of progress, or lack

thereof, and effective review of the ETDP SETA’s programmes. The framework will

guide the overall monitoring and evaluation functions of the ETDP SETA including its

6

current Five Year Strategic Plan 2015/16 to 2019/20 together with the short to

medium term subsidiary plans such as the current Annual Performance Plans

(2016/17), the Service Delivery / Implementation Plan, and the units’ operational

plans which incorporate individual employees’ work plans. The core purpose of the M

& E system1 proposed here, therefore, is to timeously provide information needed for

impact-oriented project management that not only ensures compliance with the

programmes’ strategies and approaches, but also improves responsiveness, efficiency

and effectiveness by providing constant feedback from programmes’ implementation.

2. Scope of M & E

This framework is primarily aimed at guiding the M & E activities that will be carried

out by the ETDP SETA in the coming five years. M & E activities will essentially be

focused on the performance of six programmes that have been formulated to enable

the ETDP SETA’s delivery on its mandate. Alongside monitoring and evaluating

activities, processes and systems carried out to achieve the goals and objectives of the

six programmes, the fit between policies, structures and processes will be

simultaneously assessed / reviewed periodically to ensure that the ETDP SETA’s goals

and objectives are achieved. On one hand activities, outputs and immediate outcomes

will be monitored and reported internally (i.e. by staff and managers within the

organization) to track implementation progress, whilst, on the other hand, periodic

and systematic reviews and evaluations will be commissioned2 to assess progress and

help improve performance by helping the organization to extract, from past and

ongoing activities, relevant information that can subsequently be used as the basis for

programmatic fine-tuning, reorientation and planning.

3. The ETDP SETA’s Approach to M & E

The approach of the ETDP SETA to the monitoring and evaluation functions is

informed by the need to establish a sustainable system that has the buy-in of all staff

in the organization where the system will be recognized as the critical yardstick or

instrument to measure the effectiveness of the organization’s service delivery as well

1 Monitoring as used throughout this document refers to ongoing activity that focuses on progress in terms of activities and it tracks mainly the use of inputs (activities), outputs, and also tracks (intermediate) outcomes. In contrast and complementing monitoring, evaluation refers to an assessment of a programme / project’s progress over a longer period of time and it takes place at specific moments during the lifespan of a project (see The PPD Handbook: A Tool kit for Business Environment Reformers:2005). 2 Although some internal evaluations are useful and recommended to strengthen the capacity of organizations, the international best practice of using both internal and externally conducted evaluations will be adopted since commissioning evaluations to external service providers carries an added advantage of minimizing subjective bias and leads to credibility which often results in better utilization.

7

as its units’ and individuals’ performance. The management of the ETDP SETA has

adopted the logic model that links inputs (what resources will be used to do the work),

to activities (what actions or processes will be done to produce results), to outputs

(what goods or services will be produced or delivered), to outcomes (what medium-

term results and /or change for specific beneficiaries will be achieved as a logical

consequence of delivering specific outputs), and finally to impacts (the long term

developmental results at societal level that achievement of specific outcomes will

contribute to) as the M & E framework that will guide implementation of its M & E

strategy. Table 1 below summarises the key questions that will be posed per each level

of the objectives hierarchy in this logic model:

Table 1: Information needs in the objective hierarchy that will be

sought by M & E

Level in objective

hierarchy

What to Monitor and Evaluate

Activities What needs to be done to produce the intended / planned outputs? Have

planned activities been completed on time and within budget? What unplanned

activities took place?

Outputs What direct tangible products or services has the programme delivered as a

result of activities? What are the things that different stakeholders must provide?

Key outcomes What changes have occurred as a result of the outputs and to what extent are

these likely to contribute towards the programme purpose and desired impact?

Where do we want to be in five years? What are the things that must be in place

first before we can achieve our goals and have an impact?

Purpose Has the programme and/or the ETDP SETA achieved the changes for which it

can realistically be held accountable?

Goal (Impact) What is the programme’s and ETDP SETA’s overall goal? To what extent has the

project contributed towards its longer-term goals? Why and why not? What

unanticipated positive or negative consequences did the project have? Why did

they arise?

(Adapted from: International Fund for Agricultural Development (2002): A Guide for Project M & E; www.ifad.org).

In order to ensure that a sustainable M & E system that has the buy-in of staff is

effectively utilized by the whole organization to track progress and take decisions that

will lead to timely corrective actions where necessary, a participatory approach to M &

E where both managers and staff keep track of their operations and continuously learn

from what they are doing, is proposed. The participatory M & E system that is

proposed here is undergirded by the belief that effective implementation of M & E

strategies is centred around effective institutionalization which comprises people and

their capacities, processes and systems (see The M & E Plan and Institutionalization

Strategy in Annexures). Hence this M & E Framework should be correctly viewed as a

work-in-progress since in the participatory M & E approach it is requisite to involve

8

people who are involved in collecting and using the data in the design and formulation

of the M & E system3.

The M & E system will provide regular reports on the progress of programmes in order

to support managers and implementers (including staff, technical service providers

and skills development facilitators) to ensure that interventions lead to desired results

and adequately address the needs of the targeted beneficiaries. In addition to the

conventional principles commonly used as parameters for M & E practice, specific

principles that will be used to guide implementation of the M & E strategy will include

the following:

Emphasis of the M & E system should be on benefits and rewards (carrots) rather than sanctions (sticks) since project managers and staff should see the value of M & E in order to appreciate and commit themselves to it.

Baselines – or alternatives to these – will be specially considered in setting up the M & E system as they make it possible to see change.

The M & E system should always ensure that it includes information which helps answer the five core evaluation questions: relevance, effectiveness, efficiency, impact and sustainability, and which corresponds to each of the levels of the objectives hierarchy.

Enough operational information is included to know if optimal use of resources is being made and operations are of good quality. To be able to manage for impact, an M & E system needs to track progress in relation to targets.

M & E should seek information that helps not only to check targets, but to also explain progress. Only by knowing why something is happening or why not, can we have a basis for deciding what corrective action is needed.

The M & E system should also seek out unintended positive and negative impacts in order to enable taking any corrective action that might be necessary.

The M & E system should try to only collect information that is actively used and avoid the common M & E trap of gathering too much data of limited quality and not analyzing it. Less data may lead to more useful information.

Information needs and indicators should be reviewed as the programmes evolve in the implementation of ETDP SETA’s mandate to include new information needs and delete those that are no longer relevant.

M & E integrity and trustworthiness should be underpinned by evidence-based information, methodological soundness and the expertise and professionalism of the evaluation teams.

3 This aspect (of involving both managers and staff involved in the operations in the design and formulation of the M & E system) will be incorporated in the training workshops of staff and managers that are scheduled to be carried out starting in September 2015 and will be facilitated by the Research Chair on Evaluations.

9

4. Levels of Monitoring and Evaluation

The ETDP SETA is focused on delivering on two of the fourteen key priorities of the

National Development Plan, and these are: 6.1: Quality Basic Education and 6.5: A

skilled and capable workforce to support an inclusive growth path. To respond to

these outcomes the ETDP SETA has set for itself the following strategic goals that

must be achieved by March 2020:

Strategic Outcome Oriented Goal 1: Research focused on improving sector

skills planning, employability and productivity for economic growth;

Strategic Outcome Oriented Goal 2: Highly skilled, transformed and

productive personnel for improved teaching and learning in schools, TVET

colleges and higher education institutions

Strategic Outcome Oriented Goal 3: People centred and service oriented

public workforce for improved service delivery within the post school

education and training

Strategic Outcome Oriented Goal 4: A seamless post school education and

training system that addresses access and progression into high level skills

and work placement for career and occupational development

Strategic Outcome Oriented Goal 5: Enhance capacity in non-schooling

constituencies such as non-governmental organisations (NGOs), library and

archival services, political parties and trade unions through skills

development

Strategic Outcome Oriented Goal 6: Strengthen the ETDP SETA corporate

governance, leadership and operational effectiveness and efficiency

In essence monitoring and evaluation proposed in this framework will take place at

two distinct but closely connected dimensions / levels: one level focuses on the

outputs, which are the specific products and services that emerge from processing

inputs through programme activities. The other dimension focuses on the outcomes

which are the changes that the ETDP SETA aims to achieve through delivery of its

programmes’ outputs. The main focus of monitoring (which is largely a function

carried out internally by managers and staff) will thus be on tracking progress in the

delivery of outputs and the immediate outcomes whilst evaluations (both internal and

external) will focus mainly on outcomes and impacts and also include process

evaluations to assess the extent to which programmes are operating in the way/s

intended.

The M & E Unit will continue to conduct simultaneously monthly and quarterly

performance monitoring for each programme. The M & E Unit report will be based on

unit reports and the reports from field monitoring visits which it conducts periodically

to verify the information gathered. Monitoring tools to track programme performance

will continue to be developed by the M & E Unit in collaboration with the Research

10

Chairs that have been established with the higher education institutions (HEIs).

Development of monitoring tools will naturally entail upgrading and updating of

existing tools to ensure that there is purpose fit with the current programmes and

plans that will be monitored and evaluated.

4.1 Annual Performance Plan

Although the participatory M & E of this framework promotes that everyone in the

organization keeps track of the operational and strategic aspects of their work (as

monitoring is a daily and spontaneous activity), the responsibility for accounting on

the planned activities and outputs as per the annual performance plan targets still lies

with the managers as monitoring programme performance is part of the management

function. The main thrust of M & E will be on the monthly, quarterly and annual

monitoring performance of the six programmes in terms of the planned outputs and

outcomes that are in the APP. In terms of programmes, programme managers

continue to be responsible for performance monitoring. Through the use of indicators,

baselines and targets they will monitor performance, keep track of the changes and

progress attained, gather and report information together with the staff through their

monthly and quarterly reports. Annual reports will thereafter assess whether the

outputs and outcomes attained in the year’s performance take the organization any

closer to achieving the five year targets outlined in the Five Year Strategic Plan.

4.2 The Five Year Strategic Plan

The monthly and quarterly performance monitoring outlined above for tracking the

Annual Performance Plan culminate in an annual assessment which measures the

year’s performance to determine if such performance takes the organization any closer

to attaining the programme goals intended to be achieved in five years. A mid-term

review will be held in year 3 of implementation to assess overall performance and take

stock of progress or lack thereof which should guide the organization to taking

informed decisions that may lead to corrective measures and adjustments where

necessary. It is therefore apparent that performance monitoring of both the Annual

Performance Plan and the Five Year Strategic Plan will take place concurrently during

implementation. In addition to these plans, the ETDP SETA Service Delivery /

Implementation Plan which has been developed will be monitored and reviewed as

well which should logically entail the evaluation of units’ operational plans and

individual employee plans as part of the organization’ Performance Management and

Development System (PMDS). Since the latter fall within the domain of programme

or project management, the ETDP SETA managers will report on these aspects as part

of their conventional management functions. The focus of the M & E Unit at this

(strategic) level will be to ascertain to what extent has the ETDP SETA’s Five Year

Strategy contributed to the realization of the National Development Plan outcomes.

This will entail examining the extent to which the various goods and services produced

by the SETA in that period have had an impact on the priority outcomes. It is

11

anticipated that at the end of the five year period of implementation major outcome

and impact evaluations will be conducted by the Research Chair on Evaluations not

only to determine the extent to which intended outcomes and impacts were achieved

but also to generate lessons to inform future planning.

4.3 Unintended outcomes or effects and risks

Since it is important to note that sometimes well intentioned actions may lead to

negative results, it will be necessary to devote time to thinking through various risks

and possible unintended effects or outcomes. Several risks and risk mitigation

strategies have been identified per each of the six programmes that are currently being

pursued by the ETDP SETA and these are encapsulated in the Five Year Strategic Plan.

These risks will be monitored simultaneously with the activities, outputs and outcomes

by the staff, programme managers and the M & E Unit and will be reflected upon

especially in the quarterly reports that will be produced. Similarly, implementation of

programmes may lead to unintended results or consequences. In reality these are

another form of risk which will happen and may lead to undesirable results. All

members of the organization are urged to note, discuss and document any possible

unintended results or consequences.

5. Key Indicators and Priority Targets

Indicators are signposts of change along the path to development. They describe the way to

track intended results and are critical for monitoring and evaluation (UNDP Handbook on

Planning, Monitoring and Evaluating for Development Results; p. 61)

Performance indicators are units of information that, when measured over time, will

document change. When supported with sound data collection, analysis and reporting,

indicators enable managers to track progress, demonstrate results, and take corrective

action to improve service delivery. Good quality results, i.e. well formulated impacts,

outcomes, outputs, activities and indicators of progress, are crucial for proper

monitoring and evaluation. Monitoring and evaluation will be challenging, making it

difficult for staff and managers to know how well plans are progressing and when to

take corrective actions if results are unclear and indicators are absent or poorly

formulated. Similarly without baseline data it is very difficult to measure change over

time and set targets or to monitor and evaluate. Hence indicators and baseline data

are considered to lie at the heart of the monitoring and evaluation system. It is in this

context that the importance of both indicators and targets in the monitoring and

evaluation encapsulated in this framework should be understood.

Both the APP and the Five Year Strategic Plan have identified key indicators that will

be monitored to track performance and achievement of progress or lack thereof in the

programmes’ implementation. To track performance in these measures, performance

measurements, and performance standards a key question that will be posed is: How

12

will we know if we are on track to achieve what we have planned? With regard to the

accompanying means of verification key questions that will be posed are: What precise

information do we need to measure our performance? How will we obtain this

information?

Table 2 (in Annexures below) presents a results framework that gives an indication at

the programme level of what will be monitored and evaluated. It includes indicators,

baselines, targets and means of verification that will be tracked. Both output / process

indicators (that measure the degree to which activities are being implemented) and

outcome / impact indicators (that measure changes resulting from the interventions)

will be closely monitored in monitoring and evaluation. M & E should take special care

that the demand for objective verification does not lead to exclusive focus on

quantitative indicators at the expense of more qualitative indicators that are harder to

verify but may better capture the essence of the changes taking place that are sought.

6. Evaluations

Evaluations of the ETDP SETA’s interventions will be carried out to assess,

systematically and objectively, progress towards attainment of the identified outcomes

that the SETA is seeking to achieve. These will not be one-time events, instead they

will be exercises involving assessments of differing scope and depth carried out at

several points in time (during and after implementation of the five year strategy) in

response to evolving needs for evaluative knowledge and learning in the efforts to

achieve the identified outcomes. A key requirement for all evaluations, including

process evaluations that assess relevance, performance and other criteria, is that all

evaluations need to be linked to the identified outcomes as opposed to only

implementation or immediate outputs. As briefly alluded to earlier in the document,

it is expected that evaluations will not only answer questions generated by the

monitoring process but will importantly also draw heavily on data generated through

monitoring, including baseline data, information on the programme/s’

implementation process, and measurement of progress towards the planned results

through indicators. The ETDP SETA has conventionally focused on commissioning

process, outcome and impact evaluations to ascertain that its interventions are leading

to the desired results and outcomes. It is envisaged that the M & E Unit within ETDP

SETA will develop and/or update an evaluation plan (if there is an existing one) which

will become an integral component of the M & E framework and will be compiled from

an aggregate of individual programmes’ evaluation plans which will have been

developed by the programmes’ managers and implementers.

In this framework we propose that, amongst the criteria used to commission

evaluations, the following be considered as well in deciding what to evaluate:

Uses, purpose and timing of an evaluation – Evaluations should be proposed

only when the commissioning units in the organization are clear at the onset

13

about why the evaluation should be conducted (purpose), what the information

needs are (demand for information), who will use the information, and how the

information will be used. The intended use determines the timing of an

evaluation, its methodological framework, and level and nature of stakeholder

participation, whilst the timing of an evaluation should be directly linked to its

purpose and use. To ensure its relevance and effective use of evaluation

findings, the evaluation should be made available in a timely manner so that

decisions can be made informed by evaluative evidence.

Resources invested – An area or programme in which the ETDP SETA has

invested significant resources may be subject to an evaluation as there are

usually greater accountability requirements concerning such a programme or

area.

The likelihood of future initiatives in the same area or programme –

Recommendations to guide future work often emanate from evaluations. As it

is an evaluation which enables programme units to take stock of whether

outputs have contributed to desired outcomes, it will therefore be necessary

when selecting an intervention to be evaluated to look for one in an area or

programme that is likely to continue to be supported by the ETDP SETA.

Anticipated problems – Evaluations can help prevent problems and provide an

independent outlook on existing problems. When selecting an outcome for

evaluation, it is therefore prudent to look for those with problems or where

complications are likely to arise as a result of an outcome being in a sensitive

area possible with other stakeholders or partners.

Need for lessons learnt – A consideration of what kinds of lessons are needed

to guide further implementation is critical when evaluations are to be

commissioned.

Alignment and harmonization to national, regional/provincial and the ETDP

SETA’s corporate priorities – Planned evaluations should, on one hand, be

aligned with national and regional/ provincial development priorities and, on

the other hand, should be aligned with the ETDP SETA’s corporate priorities

and be harmonized with national evaluations. Opportunities for joint

evaluations with other government departments and agencies should be

actively pursued to minimise costs and make effective use of existing capacities.

In determining the timing of evaluations, the ETDP SETA should consider

various decision-making points that exist (in the government departments and

agencies whose work affect the SETA) such as budget decision making,

development framework or strategy setting, and existing review processes for

development programmes and projects. For instance, if the national

government is undertaking an evaluation of the NDP outcomes that the ETDP

SETA is contributing to, ETDP-managed evaluations should enhance

complementarities and minimize duplicated efforts.

14

7. Data collection methods and sources

The data collected will need to provide the information that is necessary to measure

indicators in both monitoring and evaluation processes. The types of data collected

will, to a large extent, be determined by the feasibility of collecting such data given the

constraints of time and resources. The primary data that will be collected from the

intended ETDP SETA beneficiaries, service providers, employers and other

stakeholders through direct observations and other methods such as interviews and

focus group discussions, will be complemented (and not replaced) by secondary data

where literature, reports and analyses will be extensively studied to deepen and

strengthen understanding of particular issues and phenomena being monitored and

evaluated at any given time. During the data collection process, the M&E Unit should

assess the strategic information needs and the existing information systems and

capabilities to address those needs to determine what is feasible and what is not. The

M & E team must identify how to collect the information necessary to measure the

performance indicators, whether it will be through existing data collection systems or

whether new systems will be developed. In addition, it must determine how

information will be recorded and reported, including identifying any tools or forms

that will be needed.

8. Reporting including M & E roles and responsibilities

As a public entity the ETDP SETA is subject to the government departments’ and

public entities’ reporting procedures which, among others, include monthly, quarterly

and annual reporting. In these processes the importance of performance information

is increasingly being recognized as part of public sector monitoring and evaluation,

playing a significant role in the allocation of budgets and in the monitoring of service

delivery and value for money. Monthly reports that are presented by line managers

and unit heads to the Broad Management Team (BMT) meeting that is chaired by the

CEO capture the information aimed at addressing operational performance, while the

ETDP SETA’s quarterly performance reports provide information or progress report

on the implementation of the SETA’s Annual Performance Plan. These quarterly

reports embody procedures that have been set according to the National Treasury

Regulations that are aimed at facilitating performance monitoring, evaluation and the

taking of decision on corrective action. The ETDP SETA publishes its Annual Report

which are also tabled in Parliament and are made available to the ETDP SETA

stakeholders, constituencies and the general public. In terms of M & E the information

that is reported annually forms the basis for the annual performance review and

planning for the following year and the MTEF period. This annual review and planning

culminates in the Annual Performance Plan. In the meantime the ETDP SETA is also

required to report on financial and programme performance to external statutory

bodies such as the Auditor General (on quarterly basis), the National Treasury (on

15

quarterly basis), Portfolio Committee on Higher Education & Training (on quarterly

basis and ad-hoc reporting upon request), and the Department of Higher Education &

Training (quarterly and annual performance reporting).

8.1 The ETDP SETA’s M & E roles and responsibilities

Currently all structures within the ETDP SETA have some M & E functions and

responsibilities. The Accounting Authority monitors and evaluates the effectiveness

and efficiency with which the SETA utilizes public resources in its activities. It also

utilizes M & E findings to provide strategic and governance oversight and direction for

the SETA as a whole to ensure that: the desired outcomes and impact are achieved. In

relation specifically to M & E the Chief Executive Officer (CEO) has the responsibility

to ensure that the ETDP SETA maintains appropriate and valid performance

information systems; management and employees comply with M & E prescripts;

prompt action is taken in response to M & E findings; that unit/ provincial resources

are allocated to conducting M & E; M & E information is communicated to the relevant

audience; and that appropriate capacity exists for monitoring and evaluation and the

M & E system is operating effectively. Senior Managers are similarly charged with

ensuring, among others, that: an M & E Plan is developed for their areas of

responsibility and M & E functions are included in the performance agreements of

their staff. Line Managers/ Provincial Managers are similarly required to ensure that

information is available within the stipulated time and format to the M & E Manager

with the means of verification, whilst members of staff are expected to capture the

required data and ensure that the means of verification submitted are relevant to the

indicators.

The M & E Unit, on the other hand, is regarded as the custodian of the M & E

Framework and works to ensure its effectiveness through providing M & E services to

the ETDP SETA. The Unit acts as an M & E resource to the organization by: providing

advice and insight on good practices and support for effective use of M & E as a

performance management tool; coordinating, promoting and disseminating M & E

findings and information to support service delivery improvement and report on

follow-up action; assessing the M & E capacity within the ETDP SETA; and

coordinating training to build the requisite skills internally and to create a culture of

self-assessment among all staff.

16

ANNEXURES

Annexure One: The Results Framework

Table 2: The results framework of the ETDP SETA’s strategic outcome-oriented goals

2015/16 – 2019/20

Strategic

outcome-

oriented

goal

Planned

results /

outputs

Indicators Baseline Target Means of

Verification

Programme

1: Research

focused on

improving

sector skills

planning,

employability

and

productivity

A 5-year sector

skills plan (SSP)

An approved 5

year SSP

A 5-year SSP

has been

developed by

the ETDP

SETA and is

updated

annually

A complete

and

approved

SSP

submitted to

DHET and

Parliament

on due dates

Workplace

Skills Plans;

DHET’s

datasets (e.g.

EMIS, HEMIS

and TVETMIS)

and

commissioned

empirical

research

Programme

2: Highly

skilled,

transformed

and productive

personnel for

improved

teaching and

learning in

schools, TVET

colleges and

higher

education

institutions

25 000 primary

and secondary

school teachers

trained in subject

content,

methodology and

ICT skills in order

to improve their

competency

levels;

5000 teachers in

full service and

LSEN schools

enrolled for

braille and sign

language

facilitation

Number of

teachers

trained in

subject

content,

methodology

and ICT skills

Number of

teachers in full

service and

LSEN schools

enrolled for

braille and

sign language

facilitation

5680

teachers have

been trained

on subject

content

knowledge;

461 on ICT

learnerships.

394 teachers

in full service

schools

trained on

special

needs.

25000

primary and

secondary

school

teachers in

public

schools

5000

teachers in

full service

and LSEN

schools

Letters of

awards and

attendance

registers

Letters of

awards/

learners’

documents

including

agreements

and MIS

reports

17

5000 learners

assisted with

funding to study

towards Grade R

or Foundation-

phase teacher

qualification

Number of

learners

assisted with

funding to

study towards

Grade R or

Foundation-

phase teacher

qualification

278 have

been trained

for Grade R

or

Foundation-

phase

teacher

qualification

5000 or

more

learners

assisted with

funding to

study

towards

Grade R or

Foundation-

phase

teacher

qualification

Letters of

awards/

learners

documents

including

agreements

and MIS

reports

2000 TVET

college lecturers

and managers

have completed

relevant

programmes to

enhance their

capacity

Number of

TVET college

lecturers and

managers who

have enrolled

and completed

relevant

programmes

to enhance

their capacity

720 lecturers

have been

trained in

PIVOTAL

programmes

and 475 have

been exposed

to industry

workplaces

2000 or

more of

identified

TVET

college

lecturers and

managers

Letters of

awards/

learner s’

documents

including

agreements

and MIS

reports

500 HEI

lecturers trained

to gain

appropriate

competencies in

order to improve

student

performance

25 HEIs receive

financial support

for staff

development

Number of

academics in

research and

HE

institutions

enrolled at

Masters or

PhD levels

Number of

HEIs receiving

financial

support for

staff

development

15 in 2013/14

500 or more

25 HEIs

Letters of

awards/

learner s’

documents

including

agreements

and MIS

reports

HEMIS and

individual HEI

sources on

skills

development

Programme

3: People

centred and

service

oriented public

4000

unemployed

youth trained and

placed in public

Number of

unemployed

youth trained

and placed as

administrators

1000

estimated in

2014/15

4000 Signed

internship

agreements

and letters of

awards

18

workforce for

improved

service delivery

within the

post-school

education and

training

schools as

administrators

2000

unemployed

youth trained and

placed as library

assistants

Number of

unemployed

youth trained

and placed as

librarians

--------- 2000 Letters of

awards

confirming

placements

5000 national,

provincial and

district officials

are trained to

increase the

administrative

capacity of

critical office-

based personnel

Number of

national,

provincial and

district

officials

enrolled in

relevant

learning

programmes

---------- 5000 Letters of

awards/

learner s’

documents

including

agreements

and MIS

reports

Programme

4: A seamless

post-school

education and

training

system that

addresses

access and

progression

into high level

skills and work

placement

8000 youths

placed in

workplace

experience

programmes

(graduates and

undergraduates)*

Number of

youths placed

in workplace

experience

programmes

718

estimated for

2014/15

8000

Signed

workplace

agreements

and letters of

awards

Programme

5: Enhanced

capacity in

non-schooling

constituencies

such as non-

governmental

organizations

(NGOs);

library and

archival

services;

political

parties and

trade unions

through skills

development.

3 Trade union

federations are

supported *

Number of

trade union

federations

supported

-------

3 trade

union

federations

Letters of

awards and

signed

agreements

19

Programme

6: Strengthen

the ETDP

SETA

corporate

governance,

leadership and

operational

effectiveness

and efficiency

Train ETDP

SETA employees

through skills

programme*

Number of

ETDP SETA

employees

trained

through skills

programmes

-------

All ETDP

SETA

employees

are provided

with skills to

fill the gaps

identified in

their PMDS

HR Unit,

learner

agreements

= only one planned result / output is shown amongst several outputs planned. Comprehensive

indicators per each of the updated programmes are in the proposed M & E Matrix in Annexure

Two.

20

ANNEXURE TWO

A tentative4 M & E Plan and Institutionalization Strategy

2.1. The M & E Plan

2.1.1 Introduction to the M & E Plan

The M & E Plan sets out the proposed minimum monitoring and evaluation

requirements to enable collection and provision of indicators of progress, or

lack thereof, and effective review of the ETDP SETA’s programmes. The Plan

outlines the overall monitoring and evaluation functions of the ETDP SETA

including its current Five Year Strategic Plan 2015/16 to 2019/20 together with

the short to medium term subsidiary plans such as the current Annual

Performance Plans (2015/16), the Service Delivery / Implementation Plan, and

the units’ operational plans which incorporate individual employees’ work

plans.

The standards to which the M&E plan should be held include the following:

M & E must be useful and serve the practical and strategic information

needs of the intended users for decision making purposes (from

assessing program performance to allocating resources, etc.) Intended

users may include those at the highest central levels making decisions

about national programs to those making decisions to improve programs

at the district level.

The M&E plan should be realistic and practical. To the extent possible,

it should make the best use of existing data collection systems. If new

data collections systems or studies are involved, resources (cost and

technical capacity) must carefully be considered. Data collection, special

studies included in the M&E plan must abide by regulations and

protocols that consider the rights of those involved in and affected by

M&E activities.

4 The M & E Plan is designated tentative at this stage because it is still going to be work-shopped with staff and managers during the scheduled capacity building training workshops and the subsequent M & E institutionalization within the ETDP SETA to ensure that the information and results sought are guided by the Plan and are consistent with their expectations. The M & E Unit will need to consult especially with programmes’ managers to ensure that the indicators in the M & E Plan reflect actual data being collected and validate whether the information is the most useful for them in order to make decisions for the improvement of their programmes.

21

Finally, the M&E plan should provide technically accurate and useful

information for decision making and program improvement.

2.1.2 Capacity and Resources for M&E

The ETDP SETA currently has an M&E unit with staff trained in M&E methods.

The Unit is responsible for leading the development and coordination of the

M&E plan. This existing capacity for M & E in the ETDP SETA is complemented

by the Research and Skills Planning Unit which coordinates research and

planning with regard to skills. The ETDP SETA has also beefed up its internal

capacity through establishment of several research chairs one of whom – the

Research Chair on Evaluations – directly contributes to the SETA’s monitoring

and evaluation needs through several evaluation studies that it has conducted

and is currently coordinating M & E institutionalization within the SETA itself.

Above all, programme managers and implementers are charged with M & E

related work in their operations as they are responsible for developing and

implementation of monitoring in their line duties. Programme managers are

expected to analyze programmes’ performance and map out trends and

patterns of efficiency in the organization, and also use evaluative knowledge for

learning, accountability and service delivery improvements.

2.1.3 Problem statement

South Africa is experiencing skills shortages in critical sectors of the economy

and the education and training sector is no exception as it has been plagued

with inefficiencies in the performance of the system. The twelve constituencies

that are serviced by the ETDP SETA in the education and training sector are all

experiencing different skills development needs.

2.1.4 Translating Programme Goals and Objectives into M&E

Frameworks

In responding to the skills shortage in the education and training sector, the

ETDP SETA has set itself the following six strategic goals which can be

objectively measured:

Strategic Outcome Oriented Goal 1: Research focused on improving sector

skills planning, employability and productivity for economic growth;

22

Strategic Outcome Oriented Goal 2: Highly skilled, transformed and

productive personnel for improved teaching and learning in schools, TVET

colleges and higher education institutions

Strategic Outcome Oriented Goal 3: People centred and service oriented

public workforce for improved service delivery within the post school

education and training

Strategic Outcome Oriented Goal 4: A seamless post school education and

training system that addresses access and progression into high level skills

and work placement for career and occupational development

Strategic Outcome Oriented Goal 5: Enhance capacity in non-schooling

constituencies such as non-governmental organisations (NGOs), library and

archival services, political parties and trade unions through skills

development

Strategic Outcome Oriented Goal 6: Strengthen the ETDP SETA corporate

governance, leadership and operational effectiveness and efficiency

For each of these goals several corresponding objectives have been developed,

the pursuit of which will be measured through indicators (see Annexure 1: The

Results Framework above). These indicators will importantly provide answers

to two key questions: “What do we want to know at the end of the programme?”

and “What do we expect to change by the end of the programme?” (See attached

M & E Plan that incorporates the M & E Matrix with comprehensive indicators

that will be used to measure attainment of set objectives and expected results).

23

2.1.5 The Monitoring Plan

The following template captures the essence of what will be captured, how,

when and by whom in the monitoring plan

INDICATOR BASELINE

What is the

current value?

TARGET

What is the

target value?

DATA

SOURCE

How will it

be

measured?

FREQUENCY

How often will it

be measured?

RESPONSIBLE

Who will measure it?

REPORTING

Where will it be

reported?

Goal:

Outcomes:

Outputs:

Activities

Inputs

2.1.6 Information dissemination and use

This section of the Plan will detail how the information gathered will be stored,

disseminated and used. This will help ensure that findings from M&E efforts

are not wasted because they are not shared.

The various users of this information should be clearly defined, and the reports

should be written with specific audiences in mind.

Dissemination channels can include written reports, press releases and stories

in the mass media, and speaking events.

2.1.7 Implementation and mechanism for update

Here a mechanism for reviewing and updating the M&E plan will be included.

This is because changes in the programmes can and will affect the original plans

for both monitoring and evaluation.

2.2. The M & E Institutionalization Strategy

As briefly alluded to above, it is the belief of the authors of this document that effective

implementation of M & E strategies should be centred around effective

24

institutionalization which comprises people and their capacities, processes and

systems. In line with this we recognize the need for all staff of the ETDP SETA to have

a sense of ownership of the M & E system. Whilst we advocate for a participatory

approach to ensure that everybody in the organization buys in in the M & E, we are

aware of the fact that a haphazard introduction of such a system without preparing the

people affected will likely result in disruptions and cause instability which will be very

costly to the ETDP SETA and derail the organization from achieving its targets and

objectives.

In line with this reality, we therefore propose a gradual approach where participatory

M & E will be phased in hand-in-hand with the scheduled capacity building/ training

on M & E that will commence shortly with operational staff and managers. It is

envisaged that the M & E Plan (a component of the M & E Framework proposed above)

will be reviewed and validated with managers and staff in the training workshops. This

is particularly important in view of the fact that the views of people involved in actual

data collection for M & E are critical in the design and formulation of the data

collection instruments. Aspects such as frequency, feasibility, and availability as well

as responsibility for data collection methods and tools are particularly pertinent to

validate through a participatory approach.

Owing to the importance and urgency of institutionalizing the participatory approach

to M & E in the ETDP SETA, we therefore propose that the training sessions (four in

all) be aligned to this process which, ideally, should be done during this early years of

implementation of the Five Year Strategic Plan. This will importantly enable the

validation process which is critical in the efforts to institutionalize M & E and will

accompany several processes to realize this goal. Such validation will also inform

planning for the coming years of implementation.

References

ETDP SETA (2011) Monitoring and Evaluation Framework 2011/12 to 2015/16;

Higher Education and Training, Pretoria

___________ (2015) Annual Performance Plan 2015/16; Higher Education and

Training, Pretoria

____________ (2015) Five Year Strategic Plan 2015/16 to 2019/20; Higher

Education and Training, Pretoria

_____________ (2015) The ETDP SETA Performance Monitoring, Evaluation and

Reporting Policy; Higher Education and Training, Pretoria

UNDP (2009) Performance Monitoring and Evaluation Handbook; New York

25

UNISDR (2009) Monitoring and Evaluation Framework; Bangkok