International Vocational Educational Organization (IVETA) Conference - December 3 & 4, 2013, Las Vegas, Nevada, USA

QUALITY ASSURANCE IN TECHNICAL VOCATIONAL EDUCATION & TRAINING (TVET)

HEART TRUST/NTA (JAMAICA)

Approaches toMonitoring &

Evaluating TVET

Programmes

Presenter: J Wallder

Jamaica • An island nation surrounded by the Caribbean Sea. • Located 90 miles south of Cuba; 118 miles west of Haiti.• 145 miles in length, up to 50 miles in width and 4,243 sq. miles

in area. • Population – approx. 2.7 million.

TVET – CARIBBEAN REGION IMPERATIVES

Dec. 2013 2

United Nations Educational, Scientific, and Cultural Organization (UNESCO) & International Labour Organization (ILO), 2002:

Those aspects of educational process involving the study of technologies and related sciences (including general education) and the acquisition of practical skills, attitudes, understanding and knowledge relating to occupations in various sectors of economic life.

TECHNICAL VOCATIONAL EDUCATION & TRAINING (TVET) – DEFINITION

Dec. 2013 3

Caribbean Community (CARICOM) Council for Human and Social Development (COHSOD), at its 19th Meeting, April 2010, established plans to review the 1990 CARICOM Regional Strategy for TVET to consider new realities of the world of work.

CARICOM Secretariat stated “it has become appropriate at this time to review the strategy primarily because global trends have indicated a need for a knowledge-based and skilled labour market”.

TVET – CARIBBEAN REGION IMPERATIVES

Dec. 2013 4

Implications:

Education & training – more relevant to meet market needs

Emphasis on lifelong learning/continuous retooling

Renewed strategies to be aimed at workers at all levels of operations – operational, managers, professionals, technical experts.

TVET – CARIBBEAN REGION IMPERATIVES

Dec. 2013 5

Established as a statutory body under the HEART Act in 1982 to govern the development and delivery of TVET in Jamaica.

The Human Employment and Resource Training Trust/National Training Agency (HEART Trust/NTA)

THE HEART TRUST/NTA - JAMAICA

VISION STATEMENT: The HEART Trust/NTA is committed to the systematic design, development and delivery of an integrated, flexible and responsive TVET system, enabling a productive workforce for national priorities and global competitiveness

Dec. 2013 6

Key Drivers of the HEART Trust/NTA

UNESCOILO

Board of Directors

Govt. of Jamaica

(2030 Plan)

Key Ministries: Education,

Labour

CARICOM

THE HEART TRUST/NTA

Dec. 2013 7

• Core business - training and certification of Jamaican citizens

for employment, while regulating and setting standards for

the TVET system in Jamaica.

• 30 Training institutions – current focus towards tertiary level

status; integration of levels 1 & 2 programmes into secondary

schools.

• Partnering training institutions (CTIs: 79);

• (Private Providers : 12)

• 2010 – 2013: Approx. 100 skill areas offered (levels 1 – 5)

Average # certified per year: 18,300

THE HEART TRUST/NTA - JAMAICA

Dec. 2013 8

Key Objectives:• Framework to guide M&E activities• Performance standards for TVET design, operations and

delivery• Monitoring and evaluation data collection and feedback

mechanisms/tools to facilitate strengthening of processes• Monitoring and evaluation audits (across the TVET system)

against the relevant standards towards programme accreditation and quality improvement

• Evaluations to ascertain programme effectiveness, impact and customer satisfaction within the TVET system • Provision of timely information to inform strategic decisions.

ORGANIZATIONAL RESTRUCTURING APRIL 2013 - NEW MONITORING & EVALUATION DEPARTMENT ESTABLISHED

Dec. 2013 9

Current Internal Quality Systems; Org. Commitment, Etc.

External Quality Assurance Agencies

Global Trends Re M&E

Best Model/Approach to Monitor & Evaluate Jamaica’s TVET System

SETTING UP THE M&E SYSTEM

FACTORS TAKEN INTO ACCOUNT

Dec. 2013 10

HEART TRUST/NTA – INTERNAL QUALITY SYSTEMS

The National Council for Technical Vocational Education & Training (NCTVET)• Responsible for quality assuring the TVET system • Established quality framework for developing competency standards,

along with guidelines/procedures for delivery and assessment activities• Accredit TVET training providers and training programmes

Training Institutions• Required to establish comprehensive Policy and Procedures Manuals

(PPM)• Management/administrative structures• Internal Quality Assurance Committees – monitors activities/alignment

with PPM

The Organization • Guided by Act of Parliament/sound mandate; other key drivers• Planning system premised on Balanced Scorecard methodology• Comprehensive structures to support TVET programmes

Dec. 2013 11

HEART Trust/NTA – Internal Quality SystemsStandards used by the NCTVET • Quality Assurance & Control/Governance• Staff Resources• Physical Resources• Client Services & Activities• Training• Assessment• Programme Evaluation

Dec. 2013 12

HEART TRUST/NTA – EXTERNAL QUALITY SYSTEMS

Quality standards used by the UCJ:• Governance and mission• Academic programmes being offered• Admission policies and procedures• Staffing• Student assessment• Student support services• Physical accommodation• Library/resource centre• Laboratory/computer facilities• Finances

The University Council of Jamaica (UCJ) • statutory body – established by the University Council of Jamaica

Act, 1987. • Primary focus – tertiary educational institutions; Institutional

registration and programme accreditation

Dec. 2013 13

Globalization - increased pressures on governments and organizations around the world to be more responsive to the demands of internal and external stakeholders for:• Good governance• Accountability • Transparency• Greater development effectiveness• Delivery of tangible results.

Translated into attendant need for results-based monitoring and evaluation of policies, programmes and projects.

MONITORING & EVALUATION – GLOBAL TRENDS

Dec. 2013 14

MONITORING & EVALUATION – GLOBAL TRENDSDefinitions - Organization for European Co-operation and Development (OECD)

A continuous function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress, and achievement of objectives in the use of allocated funds.

Monitoring

The systematic and objective assessment of an ongoing or completed project, programme, or policy, including its design, implementation, and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.

Evaluation

Dec. 2013 15

Monitoring & Evaluation – Global Trends

Monitoring & Evaluation - Two Definitions Distinct but Complementary

Monitoring Gives Information on Where a Policy, Programme, or Project is at Any Given Time:• Clarifies programme

objectives• Routinely collects data on

indicators; compares actual results with targets

• Reports progress to management and alerts them to potential problems

Monitoring

Evaluation Gives Evidence of Why Targets and Outcomes are/not being Achieved:• Analyzes why intended

results were or were not achieved

• Explores unintended results • Provides lessons, highlights

significant accomplishments or programme potential, and offers recommendations for improvement

Evaluation

Dec. 2013 16

Training delivery of TVET programmes; executed through all HEART training institutions, partnering/Community Training Institutions, secondary school system, private providers.

Must ensure standardization of inputs, processes, outputs, outcomes and impact across training entities.

Rationale for a Monitoring and Evaluation Framework for Jamaica’s TVET System

Why Monitor

& Evaluate?

Dec. 2013 17

The NCTVET (and other accreditation/certification bodies) - QA for accreditation.

M&E - critical support function – seeks to monitor/track progress of delivery entities.

Ensuring systems and processes are established against quality/accreditation standards; In state of readiness for accreditation/verification activities.

RATIONALE FOR A MONITORING AND EVALUATION FRAMEWORK FOR JAMAICA’S TVET SYSTEM

Why Monitor

& Evaluate?

Dec. 2013 18

Monitoring and Evaluation Framework established, guided by a Systems Approach, to facilitate effective monitoring and evaluation activities of Jamaica’s

TVET system.

A SYSTEMS APPROACH TO M&EMonitoring and Evaluation Framework – A Process/Results Model

INPUT PROCESS OUTPUT & OUTCOME

IMPACT

QA & CONTROL; M&E

Dec. 2013 19

SYSTEMS APPROACH TO MONITORING AND EVALUATION

Inputs

Financial, human, and material resources used for implementation of TVET programmes

Technical Expertise, Approved Curricula,Equipment, Funds

Processes

Systems established; work performed 

Strategic planning, Organization structure, Roles/responsibilities established; quality assurance systems; Training programmes

implemented

Outputs The products and services that result from the

TVET system 

Number of persons trained and certified

 Dec. 201320

SYSTEMS APPROACH TO MONITORING AND EVALUATION

Outcomes

The likely or achieved short-term and medium-term effects or changes arising from persons

trained and certifiedIncreased skilled labour force

More persons obtaining employment

Impact

The long-term results of the implementation of TVET programmes

 Persons enjoying improved standard of living

Improved national productivityPositive impact on GDP

Dec. 201321

M&E aligned with goals/objectives of the Organization Captures context of the TVET system Tracks progress on inputs, processes, outputs and outcomes of the system Conducts impact evaluations to measure actual results Documentation & reporting mechanisms; inform stakeholders & mgtment decisions.

A SYSTEM RESULTS-BASED APPROACH M&E MODEL

Strategic Goals & Objectives of

the HEART Trust/NTA;

TVET System

3. Outcomes & Impact Evaluation: Analysis of Outputs/Results & Impact Assessment

2. Performance Tracking: Appraisal & Adjustment

1. Design of M&E System: Based on FEA – Context of the TVET System

Documentation of Accountability & Lessons Learned

Dec. 2013 22

Monitor and evaluate systems and processes of the

TVET system to facilitate conformance with

educational quality standards

Provide ‘on the ground’ capacity building activities through

guidance/coaching, sharing of best practices, sharing of resources, follow-up M&E

activities

Facilitate preparation for accreditation

activities

Provide timely information to

stakeholders of the TVET system

M &E Framework in accordance with

international standards

developed and implemented

ALIGNMENT OF OBJECTIVES WITH GOALS OF ORGANZATION/TVET SYSTEM Org. Goals: A Robust Quality Management System Implemented & Maintained

Dec. 2013 23

DESIGN OF THE M&E SYSTEM: Context of TVET System, Stakeholders, Resources , Etc.

Dec. 2013 24

The Ministry of Education The HEART Trust Board of

Directors The Senior Executive Team

Provide strategic direction and resources

KEY STAKEHOLDERS OF THE M&E SYSTEM; ENGAGEMENT

Directors of the HEART Trust/NTA

Senior Director (Workforce Development & Employment Division)

Directors/Principals

Instructor/Teachers Employees of training

entities Learners

Accreditation bodies: NCTVET UCJ

Dec. 2013 25

The TVET Development and Support Systems Division Learning Resources Development (LRD) Department TVET Consultancy and Institutional Capacity Building (TCICB)

Department Vocational Training Development Institute (VTDI) Monitoring & Evaluation Department

KEY STAKEHOLDERS OF THE M&E SYSTEM; ENGAGEMENT

The Monitoring and Evaluation Department – Process Owner Its mandate: to effectively implement and maintain the

M&E system Responsible for engaging all stakeholders to facilitate

successful implementationDec. 2013 26

• Sound understanding of organizational and wider TVET objectives.

• Obtain requisite resources (through budgetary system) – staffing, funding for operations, accommodation & equipment.

• Critical areas of competence of M&E staff - working knowledge of quality assurance systems and measures.

• Familiarization with TVET training programmes, including competency-based delivery and assessment methodologies.

CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION

Dec. 2013 27

Development/imp. of macro plan for M&E

• Target training institutions for M&E• Timelines• M&E personnel assigned, including collaborations

with partnering entities (e.g., CTIs and MOE schools)

• Budget to facilitate M&E operations.

f

Educational quality standards (consistent with NCTVET/UCJ) Document review Portfolios of workQuestionnaires InterviewsObservation rating scale inventory check-list status reports

CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION

Rationalize/dev./imp. appropriate M&E tools and strategies

Dec. 2013 28

Documentation Facilities

• Template (incorporating QA standards) for M&E audits & reporting

• Results-based evaluations – standard report structure

• Monitoring - via monthly score card reporting process; information is fed upwards from individual, to dept, to executive/board/Ministry levels.

• Needs assessment currently underway to inform integrated IT system for M&E

CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION

Dec. 2013 29

IMPLEMENTATION OF THE M&E SYSTEM: Performance Tracking; Use of QA Standards; Evaluation & Documentation

Dec. 2013 30

PERFORMANCE TRACKING

Performance Tracking Monitoring the TVET System Checking for consistency between what was intended against what is actually taking place. Tracking progress of the implementation process. Identifying system strengths and weaknesses. Documenting experience and appraising performances, e.g., assessing training performances against objectives; & making recommendations for adjusts where necessary

Dec. 2013 31

IMPLEMENTATION - PERFORMANCE TRACKING

Standards for Monitoring the TVET System

NCTVET – primary QA body for TVET programmes - accreditation standards are adopted Provide general orientation/reinforcement on QM issues Conduct M&E audits to facilitate readiness for accreditation/NCTVET external verification evaluations. Reinforce quality-consciousness among TVET stakeholders.

Dec. 2013 32

IMPLEMENTATION - PERFORMANCE TRACKING

Example of Utilization of Standards in Reporting Template1. Standard: Quality Assurance & Control/GovernancePerformance Indicator/Clause 1.1

Example of Evidence

Findings Statement

The organization has a mission statement that reflects the defined goals/aims and objectives as they relate to TVET and the needs of the target population and is communicated to staff, clients and to the general public effectively.

Mission statement: Documented Clarity; Addresses TVET Articulates outcomes re client

benefitsStatement of goals/aims and objectives: Clarity; Relates to Org. mission

Communication of mission to target group: Copies of advertisements used for

the programmes or organizationOrganization’s brochure, client and staff handbooks, and related docs.Mission statement mounted and placed at strategic locations, such as the administrative office and the library

FindingsConformances xx xx Non-conformances yy yy Recommended Actions for Non-conformances zz zz

Dec. 2013 33

IMPLEMENTATION - PERFORMANCE TRACKING

Types of Tools for M&E1. Standard: Quality Assurance & Control/GovernancePerformance Indicator/Clause 1.1

Example of Evidence

Example of Types of Tools for Data Collection

The organization has a mission statement that reflects the defined goals/aims and objectives as they relate to TVET and the needs of the target population and is communicated to staff, clients and to the general public effectively.

Mission statement: Documented Clarity; Addresses TVET Articulates outcomes re client

benefitsStatement of goals/aims and objectives: Clarity; Relates to Org. mission

Communication of mission to target group: Copies of advertisements used for

the programmes or organizationOrganization’s brochure, client and staff handbooks, and related docs.Mission statement mounted and placed at strategic locations, such as the administrative office and the library

• Document review (policies and procedures, reports, records, etc.)• Checklists• Interviews• Observation

Dec. 2013 34

Completed M&E

Reports

Director, TCICB – for Capacity-buidling Issues

Relevant Senior

Directors

Directors, Principals,

Managers of Training Insts. Ministry

of Education

M&E Department

INFORMATION FROM M&E TRACKING PROCESS INFORMS FINAL REPORTS

Dec. 2013 35

M&E Officer required to follow-up with the training institution to:• Confirm receipt of M&E

report• Establish convenient time

to meet with the wider management team of the training institution

• Review content of report• Develop action plan with

timelines to address gaps identified during the M&E audit

• Agree on schedule to facilitate follow-up reviews ensure action items are

being addressed (inc. recommended capacity building issues)

M&E FOLLOW-UP PROCESS

Dec. 2013 36

EVALUATION

Performance tracking/monitoring – not enough to measure real results.

Evaluative information required to determine whether or not the intended outcome/result/impact is being achieved.

Aims to determine the relevance and fulfillment of objectives, effectiveness, impact, and sustainability of the TVET system.

Dec. 2013 37

EVALUATION

Sub processes to consider to inform Evaluation process:• Diagnosing needs, capabilities and capacities. • Determining baseline/starting-point

performance.• Pilot-testing/trial-run to validate tools and

strategies.• Formative evaluation of inputs, processes,

outputs and impact.• Summative or outcomes evaluation. • Impact assessment (main effects and side

effects).

Dec. 2013 38

EVALUATION

Strategies for Evaluating the TVET System:• Directives from Senior Management – e.g.,

evaluations on the effectiveness of the restructured training system.

• Evaluations on selected training interventions to analyse possible impact that has been made towards quality improvement.

• Evaluations on certification rates.• Tracer studies. • Training impact evaluations.

Dec. 2013 39

• Reports generated from M&E activities appropriately documented and disseminated

• The M&E model - serve to foster/promote learning and enhance knowledge management processes.

stakeholders are able to effectively apply information in their daily work for accountability, improvements in performance, decision making and learning.

Provide critical support for the TVET system’s strategic process thereby strengthening sub-processes to facilitate the achievement of objectives.

Improve the overall performance and quality of results of ongoing and future programmes, projects and strategies.

DOCUMENTATION AND DISSEMINATION OF M&E INFORMATION

Dec. 2013 40

.

Info/updates shared at meetings: dept.; div.; executive; Board; MOE

Existing ICT infrastructure; NA currently being done to inform Integ. M&E system

Reports emanating from M&E activities – to stakeholders

Action plans generated via M&E audits – Follow-ups & Feedback

Quarterly progress reports (M&E audits)

Examples of documentation

and dissemination of M&E information

DOCUMENTATION & DISSEMINATION OF M&E INFORMATION

Updates via Balanced Scorecard reports: monthly/quarterlyannually

Dec. 2013 41

MONITORING THE M&E SYSTEM

• Continuous feedback from TVET system:- Training Institutions- Directors/Principals/Instructors/ Learners- Senior Management, etc.

• Comprehensive questionnaire administered annually

Information gathered - basis for learning/inform improvements in M&E systems/processes

CONTINUOUS FEEDBACK

Dec. 2013 42

JAMAICA’S APPROCH TO MONITORING AND EVALUATION

OF ITS TVET SYSTEM

THANK YOU

Dec. 2013 43