The Diploma in Technical and Vocational Education

Tracer Study

November 2008

Research Team:

Margaret Nare, FCTVE

Montlenyane Selma-Robertson, FCTVE

Tebogo Angel Moreetsi, FCTVE

George Herd, TA

Klavs Dahl Christensen, TA

Candy Mbongwe, TA

Sepako Seosenyeng, TA

Table of Contents

1List of Abbreviations4

2Acknowledgements5

3Introduction6

3.1The Diploma in Technical and Vocational Education (DTVE)6

3.1.1Aims of the programme7

3.1.2Outcomes of the Programme8

3.1.3The DTVE programme structure.9

3.1.4The facilitation of the programme.10

4Policy background11

4.1Research Team11

5Methodology12

5.1Project Implementation Plan (PiP)12

5.2Purpose and Objectives of the Survey12

5.2.1The research areas12

5.2.2Target Groups for the Data Collection13

5.2.3The research questions14

5.2.4Detailing the research questions15

5.3Contact details of the former DTVE Students15

5.4Development of Data Collection Instruments16

5.5Development of Templates for Registration of Data17

5.6Data Collection Procedures17

5.6.1Data from DoE lecturers17

5.6.2Data from former DTVE student-teachers not employed in brigades/TCs17

5.6.3Data from brigades/TCs17

5.7Quality Aspects18

5.7.1Size of Samples19

5.7.2Reliability21

5.7.3Validity21

6Data presentation23

6.1Questionnaire for DoE lecturers. (Annex D)23

6.1.1Relevance of the content of the DTVE23

6.1.2The structure of the DTVE programme25

6.1.3Portfolio Assessment27

6.2Questionnaire for former DTVE student-teachers (Annex C)29

6.2.1Relevance of the content of the DTVE and its use by DTVE graduates.29

6.2.2The structure of the DTVE programme33

6.2.3Mode of facilitation used in the DTVE programme.35

6.2.4Quality of facilitation by DoE lecturers.37

6.2.5DTVE student-teacher perception on portfolio assessment39

6.2.6Effectiveness of the DoE support system for the DTVE programme40

6.3Comparing responses from DoE lecturers with responses of the DTVE student-teachers.44

6.3.1DTVE programme content44

Table 7 Average relevance ratings for content of modules DN01, DP01 and DP02 by DoE lecturers and DTVE student-teachers.44

6.3.2DTVE programme structure46

6.3.3DTVE portfolio assessment system47

6.3.4DTVE programme support system48

6.4Structured focus group interview: management and supervisors of DTVE graduates48

6.5Structured focus group interview: learners taught by DTVE graduates50

6.6Completing the DTVE programme51

6.7Concerns raised.54

7Conclusions and Recommendations58

7.1Curriculum58

Integrated Key Skills (KS)58

Facilitation/Facilitators59

The Programme Structure/Portfolio Assessment60

Learning support to students61

Teaching Placement (TP)62

8Literature Reference64

8.1Exit Survey DTVE Diploma in Technical and Vocational Education Programme64

8.2BOTA survey64

8.3Consultancy to Conduct a Tracer Study of Graduates of the Botswana Technical Education Programme, 200765

Table of tables

Table 1: DTVE Enrolment 2001 - 2008-11-247

Table 2: DTVE modules and credit values9

Table 3: Response rate19

Table 4: Representation of categories among respondents20

Table 5: Representation of cohorts among respondents20

Table 6: Frequencies of responses on item 2.1636

Table 7: Average relevance ratings for content of modules DN01, DP01 and DP02 by DoE lecturers and DTVE student teachers44

Table 8: Average overall relevance ratings of each module and the integrated key skills (KS) by DoE lecturers and DTVE student-teachers45

Table 9: Average overall relevance ratings of each module and the integrated key skills (KS) by DoE lecturers and DTVE student-teachers46

Table 10: Comparing ratings of related to programme structure by DoE lecturers and DTVE graduates47

Table 11 Comparing of average ratings on items related to portfolio assessment by DoE lecturers and DTVE graduates47

Table 12 Colleges and brigades visited and number of respondents in the focus group interviews with management / supervisors of the DTVE graduates48

Table 13: Colleges and brigades visited and number of learners participating in the focus group interview50

Table 14: Data on withdrawal, supplementation, pass and fail. DTVE cohorts 1-4.52

Table 15: Reasons for withdrawing from DTVE programme52

Table 16: Pass/fail rates as percent of completers53

Table of diagrams

Diagram 1: Bar chart illustrating the relevance rating (scale 1 to 5) by DoE lecturersrs24

Diagram 2: RELEVANCE of module content as perceived by DoE lecturers25

Diagram 3: RELEVANCE of module content as perceived by DoE lecturers26

Diagram 4: Structure of the DTVE programme27

Diagram 5: Portfolio assessment mode and process as perceived by DoE lecturers29

Diagram 6: Relevance and use of modules compared.31

Diagram 7: Modules content relevance and use as perceived by DTVE graduates32

Diagram 8: Modules content relevance and use as perceived by DTVE graduates33

Diagram 9: Structure of the DTVE programme views of DTVE graduates35

Diagram 10: Facilitation modes used in the DTVE programme rating of36

effectiveness and frequency of use36

Diagram 11: Pie chart illustrating views on variety in facilitation37

Diagram 12: Pie chart displaying student-teachers views on constructivist,38

learner-centred facilitation modes used by DoE lecturers.38

Diagram 13 Former DTVE student-teachers views on the quality of their facilitators39

Diagram 14: Bar chart of average agreement ratings on items on portfolio assessment mode nad process as perceived by the former DTVE student-teachers40

Diagram 15: Bar chart of average agreement ratings on support related issues in the DTVE programme as perceived by former DTVE students42

Diagram 16: Average agreement ratings on items related to support during supplementation and support ro distance learners as perceived by former DTVE students44

DTVE SurveyPage 2

List of Abbreviations

APP

Annual Performance Plan

ATTC

Automotive Trades Technical College

BOTA

Botswana Training Authority

BTEP

Botswana Technical Education Programme

CTVE

College of Technical and Vocational Education

DoE

Department of Education

DTVE

Diploma in Technical and Vocational Education

DTVET

Department of Technical and Vocational Education and Training

FCTVE

Francistown College of Technical and Vocational Education

GTC

Gaborone Technical College

ICT

Information and Communication Technology

JTeC

Jwaneng Technical College

MTC

Maun Technical College

NCC

National Craft Certificate

NCQF

National Credit and Qualification Framework

NDP #

National Development Plan #

ODeL

Open, Distance and eLearning

OBE

Outcomes Based Education

PaTeCo

Palapye Technical College

PiP

Project Implementation Plan

RNPE

Revised National Policy on Education

SPTeCo

Selebi Phikwe Technical College

TA

Technical Assistant

TC

Technical College

TEC

Tertiary Education Council

TP

Teaching placement

TS

Tracer Study

TVET

Technical and Vocational Education and Training

VBA

Visual Basic Applications

Acknowledgements

The tracer study group would like to express sincere gratitude to everyone who contributed to the collection of data for this study. We are highly indebted to the Brigade Coordinators and Principals of the Technical Colleges (TCs) who assisted by facilitating and organising the visits of the data collectors to their institutions. We especially appreciate all learners and staff who contributed by either completing the lengthy questionnaire - the former students on the Diploma in Technical and Vocational Education (DTVE), or who participated in a focus group interview - institutional managers and learners. We thank them all for their time and input.

Introduction

The main objective of this Tracer Study (TS) is to provide data and information to support a review of the Diploma in Technical and Vocational Education (DTVE) which is the responsibility of the Department of Technical and Vocational Education and Training (DTVET) and the management of the Francistown College of Technical and Vocational Education (FCTVE). The study is timely, in view of the significant changes envisaged for the Technical and Vocational Education and Training (TVET) sector of Botswana.

DTVET will become responsible for the management and funding of the 41 Brigades to support the expansion and standardisation to the Botswana Technical Education Programme (BTEP) qualification framework. A significant staff development requirement will arise since many of the brigade instructors do not have formal teacher training qualifications and are unfamiliar with the BTEP.

The Tertiary Education Council (TEC) is in the process of taking management and funding responsibility for the TVET sector with significant operational implications for the TCs

The National Credit and Qualification Framework (NCQF) is under development in an attempt to define and standardise the qualification levels of all education sectors in Botswana.

National Craft Certificates (NCC) units are under review

Proposals for a certificate and an advanced certificate in Teacher training, within the BTEP framework, have been presented to DTVET.

The study involved the tracing of former students of the DTVE programme from 2001 onwards to gather data on their experiences, both during the programme, and their subsequent involvement within the TVET sector. This report presents the data gathered, and suggests conclusions and recommendations.

The Diploma in Technical and Vocational Education (DTVE)

The DTVE is an 18 month, full-time teacher training programme offered by the Department of Education (DoE) at FCTVE. The entry requirement is normally a diploma or first degree in a vocational field. The programme is modularised, credit and outcomes-based.

Until the end of 2007 The DTVE programme was originally offered by the autonomous institution The College of Technical and Vocational Education (CTVE) located in Gaborone. The first cohort of students was enrolled in 2001. At the end of 2007, the college relocated to Francistown to become the DoE at FCTVE.

The primary aims of the original college were defined by the RNPE Government Paper Number 2 of 1994 recommendation 62 (a), (see NDP 8 chapter 15, paragraph 15.138) as:

to provide pre-service training for teachers / instructors in the TVET sector

to provide in-service ,staff-development training for the persons working in the TVET sector

NDP 9 in section 15.122 describes the major task of the DoE at FCTVE as

to provide lecturers with professional teaching skills using both full-time course and distance/e-learning delivery

Currently, the only programme offered by the DoE is the DTVE full-time programme in affiliation with the University of Botswana (UB).

Since 2001, the enrolment numbers are as follows:

Table 1: DTVE Enrolment: 2001 - 2008

Sept 2001 - May2003

Cohort 1

July2003 - Dec2004

Cohort 2

Jan2005 July2006

Jan 2005 May 2008

Cohort 3

Sept2006 - June2008

Cohort 4

TOTAL

Full time programme

Full time programme

Full time

Block release

Full time programme

Pre-service

In-service

Pre-service

In-service

Pre-service

In- service

12

3

27

18

28 Pre

23 In

15

33

17

15

45

66

50

176*

The cohort of 80 students enrolled in 2008 and currently studying is not included in the table and was not used to gather information for this TS.

A total of 174 students, in 4 cohorts have started the DVET programme.

*The above total is 176 - 2 students starting in cohort 2 left due to illness and continued in cohort 3.

Aims of the programme

The DTVE curriculum aims to provide opportunities for student-teachers to:

a) become reflective and critical professional practitioners in the vocational education and training sector;

b) acquire knowledge of relevant educational theory related to the system in which they already work or plan to enter following the programme;

c) develop a wide range of strategies, tactics and expertise necessary for planning, preparing, implementing, assessing and evaluating teaching and learning sessions for the subjects and groups of learners or trainees with which they will be working;

d) integrate in the facilitating of learning, studies relating to inter-personal relationships and the development of key skills associated with their intended or actual work role;

e) acquire knowledge and skills to respond to the needs of their learners;

f) identify, select and use a range of resources to support learning sessions, including ICT;

g) acquire confidence in their professional knowledge and problem-solving skills, developing their own personal philosophy of education and commitment to and critical awareness of their professional situation;

h) commence the development of a positive and professional attitude to the philosophy of life long learning and professional development;

i) gain access to a ladder of awards, each of which matches their training and professional needs and those of their employers and learners;

j) identify barriers faced by learners in education and training such as disability, age, race and gender and promote professional practice that does not exclude groups of learners but recognises and values their diversity;

k) study teaching and training situations within the social, psychological and philosophical context and to relate these to changes in industry and to Government policy.

Outcomes of the Programme

On successful completion of the DTVE Programme the student-teacher will:

a) exhibit non-discriminatory beliefs, values and behaviour, in respect to disability, race, gender, sexual orientation, culture or ethnicity in all written work, teaching practice, work based experience and their day to day involvement with the Programme tutors, peers, learners, and others with whom they work or study;

b) have developed skills and understanding in relation to the use and application of information and communication technologies, sufficient to carry out their intended or expected role in technical/vocational education and training;

c) have demonstrated appropriate skills in researching, analysing and using information gathered from a variety of sources such as libraries, internet, intranet, world wide web, CD-ROMs, newspapers, journals and educational texts such as syllabuses and unit specifications;

d) be able to design, plan, implement, assess and evaluate learning programmes which take into account the diverse needs of the vocational learner or trainee and the specialist area in which they will teach;

e) demonstrate appropriate skills and competence in relation to supporting their own learners, monitor progress and provide effective guidance which enhances learning;

f) evaluate their own teaching effectiveness and establish their own development targets;

g) understand the role of assessment in relation to the evaluation of learning, the vocational curriculum and the award of national qualifications related to their own teaching area;

h) understand the factors which influence the development and structure of the vocational education and training curriculum and play an effective part in the planning implementation and evaluation of this curriculum, in relation to their own specialist teaching field;

i) understand the role of vocational education and training in relation to the Botswana Government, economic, social and cultural policy, in relation to their own professional practice;

j) be able to plan and execute work based projects aimed at enhancing the quality and effectiveness of the vocational curriculum, through appropriate levels of enterprise and collaboration with peers, mentors and managers;

k) evaluate their own learning during the programme and contextualise this with their own values and beliefs in relation to their own professional role as a teacher or trainer;

l) develop the knowledge, understanding and skills, needed for effective and unsupervised work in a wide range of contexts in vocational education and training.

The DTVE programme structure.

To achieve the above stated aims and outcomes an 18 month modular programme was designed and piloted with a group of 15 student teachers beginning September 2001. Teaching Placement (TP) at an vocational institution i.e. attachment to a brigade or TC offering BTEP, forms an important, integral part of the programme of which 60% is college based and 40% is TP

The programme structure consists of 10 modules. Each module has a credit value which is an indication of the time an average student-teacher will need to complete the module.1 credit corresponds approximately to 30 hours of study time. This includes contact hours with the tutor, with peers, study in library, work in computer room, individual study, etc. The table below gives an overview of the modules in the programme.

Table 2: DTVE modules and credit values

Module

Credit value

DN01 Assess learners needs

6

DP01 Plan for effective learning

6

DP02 Prepare resources to support learning

6

DF01 Facilitate learning

8

DS01 Provide learners with support

4

DA01 Assess learning

4

DA02 Evaluate learning sessions

6

DE01 Reflect upon roles of a teacher

6

DE02 Improve own professional practice

6

DE03 Undertake an action research

8

TOTAL

60

The complete programme carries a credit value of 60, being equivalent to approximately1800 hours of study. The following key skills or generic skills are fully integrated into the 10 modules:

personal and interpersonal skills;

information gathering and processing skills (Information and communication technology, ICT);

communication skills;

numeracy skills;

quantitative data processing skills (numeracy);

improving own professional performance.

These skills are developed by ensuring they are applied to activities in each and every module. The Key Skill level of each student is assessed at start of the programme and, whenever necessary, student-teachers are assisted and provided with (online) resources to practice and upgrade their skills.

The facilitation of the programme.

The DTVE is a practical, activity-based programme founded on the general principles of adult education. Module activities and tasks are related the vocational area of specialisation of each student-teacher where possible. It is intended to be facilitated in a mode exemplary of that specified by the BTEP i.e. learner centred, based on a constructivist view of learning, supported by appropriate use of Information and Communication Technology (ICT).

The evidence for the outcomes, again similar to some BTE programmes, requires portfolio building. The portfolios are assessed in line with the Quality Assurance and Assessment Unit (QAA) following the procedure: unit assessment, internal verification and external verification. The only difference from DTVE is that the external verification is not done by QAA but by UB. In line with all BTE programmes the DTVE programme has assessed work placement periods. DoE refers to these work placement periods as Teaching Placement (TP

The DTVE programme is flexible and individualised. Student-teachers, within limits, can determine the pace of their own progress and the pace at which evidence are gathered for each module portfolio. For this to be successful, an effective support and mentoring system is in place. A college based personal tutor is assigned to each student teacher to provide support and guidance the student-teacher throughout the programme. During TP each student-teacher is attached to a mentor from the institution concerned.

Policy background

The Government is committed to providing quality technical and vocational education and training NDP9, section 15.112). The emphasis is on equipping learners with the knowledge, skills and attitudes to contribute to the socio-economic and technological advancement of the country. The Government in its drive to diversify the economy seeks to create a competent, innovative and internationally competitive workforce. To provide quality education one needs quality teachers / instructors.

DTVETs Strategic Plan (2005 2009), FCTVEs Strategic Plan (2007 2009) and FCTVEs Annual Performance Plan, (APP) 2008-2009incorporated the Governments objectives mentioned above. One of the initiatives mentioned in the APP, as a quality improvement initiative, is to conduct a tracer study of DTVE graduates. The DTVE programme has been running, with few changes, since its inception in 2001. A review of the programme is required to establish whether the stated aims and objectives are being achieved, and are still relevant to the current needs of DTVET and the vocational sector as a whole.

In a meeting with DoE in February 2008 called by the FCTVE Management attended by the Technical Assistance (TA) Team, the FCTVE Management instructed DoE to run a tracer study (TS) of former students of the DTVE programme with the purpose of informing the review of the programme, which has been offered unchanged since September 2001. The DoE was instructed to present the TS report not later than 30 June 2008. At the same meeting the TA Team committed itself to support DoE in its undertaking of the TS by developing a plan for implementation of the TS including survey instruments and assistance with analysis and reporting. The TAs estimated the DoE staff input in order to meet the deadline stated by the FCTVE Management - to the TS to be 2 lecturers -time for 3 months. It was also underlined by the FCTVE Management that the TA assistance should be seen as capacity building of DoE staff with the objective of enabling DoE staff to undertake future similar surveys. Carrying out surveys of tertiary education institutions, like FCTVE, is an expectation that has been expressed by Tertiary Education Council (TEC) members during their visit to FCTVE in relation to accreditation of the college and its programmes

Research Team

Following the meeting with the FCTVE Management, DoE nominated the following lecturers from their Department to be responsible for the planning and carrying out of the tracer study:

Margaret Nare

Montlenyane Selma-Robertson

Tebogo AngelaMoreetsi

To assist DoE the following TAs were nominated:

Jan Deurwaarder

George Herd

Klavs Dahl Christensen

The three DoE lecturers and the three TAs formed the TS task force. The task force was in September 2008 joined by the following short term TAs:

Candy Mbongwe

SepakoSeosenyeng

Methodology

This tracer study used quantitative as well as qualitative data. The main study is based on a survey questionnaire to two groups: 1. former DTVE student-teachers and 2.lecturers in the DoE at FCTVE. Qualitative data were collected through structure interviews with two groups 3.management and supervisors of the DTVE graduates employed in brigades and TCs and 4.Learners now being taught by the graduates. Interview guidelines were developed for groups 3 and 4.

The established TS task force had its first meeting in March 2008. Following that, several meetings were planned. However, some meetings had to be cancelled due to high workload of the DoE members.

A separate folder was created on the FCTVE Moodle platform to cater for the communication among the task force members. All reference material, ideas, suggestions and decisions were uploaded to the platform.

Due to a high workload on the part of the DoE members of the TS task force, nearly all the work was carried out by the TAs. This meant that the request formulated by the FCTVE Management about capacity building among the DOE members of the TS task force did not occur.

Project Implementation Plan (PiP)

The first output from the task force was a PiP (Annex A), outlining tasks to be undertaken, responsibility and timeframe. The PiP took its point of departure in the request formulated by FCTVE Management in February 2008 that the survey report should be presented by June 30, 2008. This was due to the inability of DoE members of the task force being unable to meet the various deadlines and contributions stated in the PiP, resulting in a delays in completion of the TS report.

Purpose and Objectives of the Survey

Following the development of the PiP, the task force focused on defining the purpose and objectives of the survey. The purpose of the survey was already defined by FCTVE Management as:

Provision of information for review of the DTVE programme

The research areas

In order to meet the stated purpose the task force identified the areas that should be the target for the survey.

The relevance of the content of the DTVE programme for the teachers / instructors in order to competently deliver their vocational subject area should obviously be addressed in the survey. But the content should not be assessed in isolation. The content only becomes meaningful if the graduates are able and willing to implement the knowledge, skills and attitudes developed during their study. Consequently, the survey would also investigate to what extent the DTVE graduates were actually applying the skills and knowledge in their daily work as lecturers or instructors.

The quality of any face-to-face programme as perceived by the learners also depends on the quality and mode of facilitation. This is especially the case, when the objective of the programme is to equip the student teachers-with planning, learning resource development, facilitation, assessment and evaluation skills, using an outcome based, and learner centred approach. The mode of facilitation of the DTVE programme should be exemplary (do what you preach).

The learning resources and facilities available to the student-teachers are important in order to enhance their learning in a technological enhanced environment. Then, when in employment in colleges, they can act as examples of good teaching practice to colleagues on the use of technology in learning and facilitation.

The assessment mode of the DTVE programme is through continuous portfolio building for each of the learning modules. Portfolios contain the evidence of having achieved the learning outcomes and document the process towards their achievement. This mode of assessment is unfamiliar to many of the DTVE student-teachers more used to end examinations. It is a mode of assessment that requires continuous support from the DoE lecturers and which serves at the same time as an example to them, as they must be able to use portfolio assessment when they join TCs teaching BTE programmes at advanced and diploma level.

The support system in place for student-teachers was also identified as an area for evaluation, being a key aspect of any educational programme. The views of student teachers on the support system of the DTVE programme form an important aspect of the data collection. Views of those that completed the programme successfully (obtaining their DTVE), those that left before the normal end of the programme, and of those that completed the programme but failed to achieve the DTVE are all relevant for evaluation of the support system.

The final research area identified, focused on what happened to learners on the DTVE programme after they completed or left the programme i.e. post course destinations. Did the newly qualified student-teachers find employment? What happened to those who failed or left the programme early?

In conclusion it was decided by the TS task force that in order to meet the purpose of the survey that data should be gathered in the following seven areas:

1. Content, its relevance and extent to which it is used by graduates, and programme structure

2. Mode of facilitation of the programme

3. Quality of facilitation

4. Learning resources and environment

5. Mode of assessment

6. Support system

7 Post course destinations

However, an issue was raised concerning item 4 above. With the exception of the groups, cohort 4 and cohort 3 (block release), none of the other graduates of the programme would have experience of FCTVE facilities, its learning resources and environment, since they completed their programme at CTVE in Gaborone. Cohort 3 (block release) and cohort 4 were at FCTVE for a relatively few months only, and during a time when the campus was not fully operational. Therefore, it was decided to exclude the above item 4 from the survey as student-teachers would not be in a position to give views on the facilities and resources a full operational FCTVE would offer.

Target Groups for the Data Collection

Discussions in the TS task force indicated that there was a need to collect data not only from those who completed the DTVE programme successfully the graduates, but also those who did not. Consequently, the student-teacher population targeted consisted of all student-teachers enrolments to the DTVE programme since its inception. In this document this group is referred to, not quit appropriate, as DTVE graduates.

Another group targeted to provide data, were current and former lecturers in the DoE (formerly CTVE). With their experience in facilitating the programme as well as their theoretical background they would be in a position to provide useful information related to the relevance and usefulness of programme content, its mode of facilitation and assessment, and the student support system.

The purpose of the DTVE programme is to produce skilled teachers / instructors for the TVET system in Botswana. It can therefore be expected that former DTVE graduates will be working in a brigades, government TCs or other vocational training institutions as instructors/lecturers[footnoteRef:2]. In order to establish whether DTVE graduates use the knowledge, skills and attitudes acquired in the DTVE programme two groups were identified to provide data. [2: The data did confirm this - 116 (67%) of the 174 graduates identified were employed in either a brigade or TC]

a. learners in brigades/TCs currently taught by the DTVE graduates

b. the management of the institution in which the graduates are working, including their immediate supervisors

This data would focus on the relative performance of DTVE graduates compared with teaching staff with a non DTVE background.

In summary, the target groups for the survey were:

1. Former students enrolled on the DTVE programme since its inception (referred to as DTVE graduates)

2. Lecturers from DoE (current and/or former)

3. Learners in brigades/TCs currently taught by the graduates

4. Supervisors in brigades/TCs of the graduates

The research questions

The TS posed the following the questions:

1. a. How relevant is the content of the DTVE programme to the work of facilitators of BTEP/ other vocational programmes within the vocational sector of Botswana?

b. Do the graduates apply the skills and knowledge which they have developed?

c. How do graduates perceive the structure of the programme (Duration, workload, level, sequencing)and the content of modules?

2. Where the modes of facilitation used in the programme conducive to learning?

3. b. How do those taught by the graduates rate the quality of their facilitation?

c. How do managers/supervisors in brigades and TCs rate the quality of the graduates?

4. a. How do graduates perceive the portfolio assessment system?

b. How do DTVE facilitators perceive the portfolio assessment system?

5. What are graduates views on the support received during / after the programme?

6. a. What is the completion rate / pass rate?

b. What are reasons for dropping out from the programme before completion?

c. Where are graduates employed after completion?

d. What is the time gap between completion and employment?

Detailing the research questions

For each of the research questions a key performance indicator (KPI) with a standard was formulated. The data source / respondents to supply data and the data collection instrument to be used to collect the data were identified. The person(s) responsible for collecting these data were identified.

To ensure that the questions included in the data collection instruments address the objective a pro forma document justifying the questions was designed (Annex B). The DoE members were to complete these justification documents with the Department as a form of validation that the correct questions were asked and nice to know questions, not related to the set objectives were not included. Time constraints did not allow this part of validation of the questions to take place.

Contact details of the former DTVE StudentsThe records of former students of the DoE were requested. However, the lists did not include the latest contact details for graduates or their completion status (passed with or without supplementation, failed). There was also no indication of DTVE student teachers that withdrew from the course before the normal end date.

As graduates tend to be mobile, contact details current when the graduates left the DoE become quickly outdated. Those not in employment within the TVET system were therefore particularly difficult to trace. Considerable energy and time was therefore needed to trace each individual. Strategies for generating a complete list of DTVE graduates were as follows:

Records of the DoE on the 4 cohorts were the starting point, though they were found subsequently to be incomplete. Each student teacher admitted to the DTVE programme receives a unique ID number. The list of former student-teachers submitted to the tracer study group had some IDs omitted.

Admission records for cohorts 1 to 3 from a former DTVE lecturer were examined to supplement, where appropriate, the records by DoE management. The Deputy Principal resources at FCTVE (former HoD in the DoE) provided a fairly comprehensive for the student teachers in cohort 4.

The above strategy produced a list of 174 graduates. The list has a few missing IDs i.e. ID numbers without a name attached. However the missing IDs could be accounted for as it was found that for some cohorts (especially 3 and 4) IDs were given to all student-teachers offered admission to the DTVE programme. Some having been offered a place on the programme did take up the offer i.e. never started the programme resulting in empty IDs. The TS working group is confident that all DTVE graduates were however properly identified. The next task was to trace the current location and contact details of the graduates.

a. Strategy for tracing graduates working in the TCs or brigades

Staff records from the TCs and the database containing the data of all persons working in the brigades were used to identify the graduates currently working in these institutions. This allowed 116 (66%) of the graduates to be located, 77 in brigades and 39 in TCs.

A snowball technique was used whereby graduates who had been located were asked for assistance in tracing other graduates from their year or other cohorts. A further 36 graduates were traced.

22 graduates (13% of the total) could not be traced.

In summary, a total of 152 (87%) graduates were traced and contacted.

Development of Data Collection Instruments

The following data collection instruments were developed|:

a. A questionnaire for former DTVE student-teachers (graduates). (Annex C)

b. A questionnaire for former and/or current lecturers in the DoE facilitating the DTVE programme. (Annex D)

c. Guidelines for a structured focus group interview with managers / supervisors of DTVE graduates. (Annex E)

d. Guidelines for a structured focus group interview with learners currently taught by DTVE graduates. (Annex E)

The steps in developing the instruments were the following:

a. Documents detailing the research questions in each of the six areas included a section with draft / suggested questions that should be answered by the identified populations in order to collect data for answering the research question. . These 6 documents were made available to members of the TS workgroup for scrutiny and review, together with a pro forma checklist to justify each question (See annex B) suggested for inclusion in the data collection instruments. The workgroup members based in the DoE were tasked with presenting the documents within the Department and include Departmental feedback and review. This is a low level form of establishing the validity of the data collection instrument. No feedback was received and hence it was assumed that the documents could be used for the next stage without any change.

b. The suggested questions in the 6 documents were collated into 4 separate documents: the first draft of the 2 questionnaires and the 2 guidelines for the structured focus group interview. These draft documents were made available, through the Moodle platform to the members of the TS workgroup for review and dissemination in the DoE, together with two checklists. The first checklist was to be used to check each questions, whether it was correctly formulated, short, simple language, not a leading question, etc. The second checklist addresses the layout and structure of the whole questionnaire e.g. where similar questions grouped together, does it start with relatively straight forward questions, etc. No feedback was received from the DoE, but feedback was received from TA members in the TS workgroup.

c. The TA members of the group produced, based on the feedback, a final version of the documents. These were presented to the principal FCTVE for approval, and to the HoD of the DoE for discussion within the Department and approval. Approval was received from principal and HoD of DoE.

Development of Templates for Registration of Data

Once the questionnaires were developed, templates for entering of the quantitative data were developed. In order to construct the two dimensional XY (Scatter) charts a Microsoft Visual Basic Applications (VBA) macro was developed.

Data Collection Procedures

Four of the TS workgroup members were involved in the collection of the data working in pairs. To ensure that data were collected in the same way by each pair of data collectors, guidelines for the structured focus group interview with management/supervisors and learners of the DTVE student-teachers were developed (Annex E). Data collection visits to TCs and brigades were structured in the same way to increase the reliability of data collected.

Data from DoE lecturers

The developed and approved questionnaires (Annex D) were distributed to all 14 lecturers in the DoE at FCTVE. Five former lecturers were contacted by e-mail and requested to complete the questionnaire.

Of the 14 DoE lecturers 9 (75%) returned completed questionnaires. Three completed questionnaires were received from former DTVE lecturers. Relatively, this is a fair representation of DTVE lecturers involved in facilitating the DTVE programme.

Data from former DTVE student-teachers not employed in brigades/TCs

58 former student teachers (33%) are not working in the TVET system (TC or brigade). Of these 58 former DTVE student teachers 22 could not be traced (13% of the total 174 former DTVE student-teachers). The 36 former DTVE student teacher traced were approached by phone and/or e-mail and questionnaires were dispatched to them through fax, e-mail or by hand. All 36 received the questionnaire. 22 (61%) were received back, some after numerous follow ups through telephone and/or e-mail. A few were collected by visiting the persons at their workplace / house. The collection of these 22 questionnaires turned out to be a tedious and time consuming exercise.

Data from brigades/TCs[footnoteRef:3] [3: List of all institutions visited is found in Annex G]

A data collection plan was drawn up by the TS task force, including visits to all TCs and brigades. There are 20 brigades with 39 DTVE graduates. 13 (85%) were visited and data collected from the former DTVE student-teachers management / supervisors and learners.

A time plan, scheduling visits to two institutions per day was produced. A standard programme[footnoteRef:4] for the visits was drawn up. All institutions were contacted by telephone, informed about the purpose of the survey and the programme for the visit. Once the details had been agreed on, the programme was faxed to the institution. Generally, all institutions contacted were very cooperative, and interested in receiving the data collection teams. [4: The programme is found in the Annex H]

The 7 brigades not visited by the data collection teams were contacted by phone and fax. Questionnaires for the former DTVE student-teachers were supplied through fax. The completed questionnaires were to be faxed back. A total 34 (87%) completed questionnaires were obtained from DTVE student-teachers in the brigades and 49 (64%) from DTVE graduates working in the colleges.

Former DTVE students

At the brigades/TCs the graduates of the DTVE programme were addressed as a group. A brief introduction explaining the purpose and the objectives of the data collection exercise was given. Following that, the questionnaires were handed out to the instructors/lecturers for completion. The time for completing the questionnaire was approximately 1 1 hour. This is generally too long for a questionnaire, however as the DTVE former student-teachers were sitting together, discussing issues, recalling good and bad experience during their stay at the college and given the time by management to complete the questionnaire in all but one instance this worked well. In GTC the former DTVE student-teachers refused to complete the questionnaire as they had, according to them, raised issues with DTVET to which no response was received and hence they felt that there was no need for them to complete the questionnaire as nobody will attend to what we say and we completed a questionnaire when exiting the college, so this is a duplication.

With 49 (64%) of the 77 former DTVE student-teachers currently employed in the TCs completing the questionnaire, sufficient data were collected. After, and during completion of the questionnaires the respondents were encouraged to raise issues not covered in the questionnaire. A number of concerns related to recognition of the DTVE, BTEP phase training, the BTEP, continuous staff development, and other were brought forward. These are presented in section 7 of this report.

Management, supervisors of the instructors/lecturers in the brigades/TCs

While the instructors/lecturers completed their questionnaires, the research team interviewed the supervisors. In the brigades the supervisors comprised the brigade coordinator, the training coordinator and in some cases HoDs. In the TCs the supervisors generally were the HoDs. In several instances members of the brigade/college management participated, e.g. the centre contact, deputy principal academic. The structured interview was conducted, following the guidelines produced (Annex E). One member from the data collection team would read the questions and both researchers would write down answers for later comparison and compilation. Purpose of the interview was explained to the managers / supervisors emphasising that in answering the questions they should make comparison between those of their staff who had followed the DTVE programme and those who had not. The interviews lasted approximately 45 minutes.

Learners in the brigades/TCs currently being taught by DTVE graduates

After interviewing the supervisors, learners taught by the DTVE graduates were gathered and interviewed. On average 3-4 learners of each DTVE graduate were requested to participate in the focus group interview. In institutions with many graduates the number of learners per. graduate could be smaller to keep the total number of learners in around the 15 range for a manageable group and ensure participation of all the learners were briefed on the purpose of the interview and its confidence. The names of the DTVE graduates were mentioned to them and they were requested in their answers to keep these instructors / lecturers in mind and compare them with those of their instructors/lecturers who had not completed the DTVE programme. During the interview learners raised issues not covered in the guidelines and/or not directly related to the facilitation of the DTVE graduates. Yet some of the concerns are relevant to the TVET system in general and hence included in section 7 of this report.

Quality Aspects

In this section issues of sample sizes, reliability and validity will be discussed

Size of Samples

The sample sizes measured by respondents as compared to the populations are as follows:

Table 3: Response rate

Target group

Population size

Number of respondents

%

DoE lecturers

Lecturers in DoE: 14

Former lecturers traced 5

9

3

75%

60%

Former DTVE students

174

105

60%

Learners taught by DTVE graduates

Not known

195

-

Management / Supervisors of DTVE graduates

Not known

59

-

A 60% response rate from all the DTVE former students and a 69% response rate from those traced and contacted is high for tracer studies. The data collected are more than sufficient to draw conclusions as they may be considered representative for the whole population.

The tables 4 and 5 present the data on how the different categories in the population of 174 DTVE former students are represented in the sample of 105 respondents.

Table 4 Representation of categories among respondents

Population

Number

Number respondents

Male

94

58 (62%)

Female

80

47 (59%)

In service

75

56 (75%)

Pre service

99

49 (60%)

Full time

159

93 (58%)

Block release

15

12 (80%)

In the total population of 174 DTVE graduates three categorisations can be used

By gender: male or female

By background: in service (employed in the TVET sector and send for teacher training) or pre service (mainly following completion of diploma course in a vocational area but without work experience in the TVET system)

By mode of studying: in full-time face to face mode or in block release mode

Full-time face to face student-teachers train for 18 months at the DoE. The block release student-teachers (all in-service) train at DoE for one academic term (with a front load of two terms) and work at their institution the next term. It takes these student-teachers 3 years to complete the programme.

Table 4 gives the data on the number of respondents in each category as a percent of that category in the population. With a representation of about 60% to 70% in the respondent groups all categories are well represented.

Table 5 Representation of cohort among respondents

Cohort

Number in population

Number responding

C1

15

5 (33%)

C2

43

(actual 45, but2 transferred to C3)

24 (56%)

C3

51

32 (63%)

C3 block

15

12 (80%)

C4

50

32 (64%)

Total

174

105 (60%)

Table 5 gives the percent of each cohort among the 105 respondents. Cohort 1 is not surprisingly least represented as they are most difficult to trace. 6 (40%) could not be traced of the 9 traced and supplied with the questionnaire 5 (56%) responded. The other cohorts have a 60% representation, with the block release group 80%. As this group left in May 2008 and an exit structured focus group interview was carried out with this group, data on the group were available. In addition all student-teachers in this group were in service and hence could be easily contacted in their workplace in a brigade or a TC.

From the above data one might conclude that the 105 respondents are a good representation of the population of 174 on the variable mentioned.

Reliability

A reliable data collection instrument is one that would give the same result if repeatedly used with the same or an identical group.

In order to assess whether the data collection instruments were reliable the data collection instruments were piloted at Palapye Technical College and Selebi Phikwe Technical College. The pilot did not reveal any conceptual problems with the instruments. When the data collection instruments were rolled out to other colleges and brigades, no significant difference between the responses from the two colleges and the responses from the two pilot colleges were observed.

More systematically the internal reliability of the two questionnaires was estimated by applying Cronbachs as an estimator[footnoteRef:5] that quantifies the reliability of a score of several questions in a questionnaire. The estimator will take values between 0 and 1. In most cases an acceptable minimum score is 0.7. The estimator was calculated for all sections in the two questionnaires with numerical data. The estimator provided values to underline that the two questionnaires can be regarded as having acceptable reliability. [5: Detailed description is found in Annex I]

For the calculated means (average rating) of the scores on a nominal 1 to 5 scale the 95% confidence intervals were computed. These intervals will with 95% certainty contain the true value in the population.

Validity

It is only after establishing the reliability of the instrument that validity is to be considered.

Validity of a data collection instrument refers to the question Is the instrument or measure used measuring what it is intended to measure? To assess the validly of the data collection instruments the method of consensual members checks was used.

The data collection instruments were presented by the DoE members of the task force for feedback, comments and alterations to all DoE staff members, together with the question justification document(Annex B) Each question in the instrument was to be justified: why is the question asked? To what research question is it attempting to find an answer to? As no feedback was received it was assumed that the instruments were valid. As professional educators staff in the DoE wants to collect valid data from the DTVE student teachers in order to review content, mode of facilitation, assessment and support of the programme offered. The lecturers in DoE being the experts on the DTVE programme are therefore the most knowledgeable persons to assess the content validity of the data collection instruments.

The data collection instruments do allow to collect data on the relevance and use of the DTVE programme content, its mode of facilitation, its facilitators, its mode of assessment and the support provided as perceived by stakeholders.

This is one aspect of the DTVE programme. The aspect NOT addressed is how the DTVE programme compares to vocational teacher training programmes offered in the region and internationally. Is the programme in line with current developments and thinking related to the training of teachers for the TVET sector in the 21st century? This is an important but different aspect. This however, raises the question whether the survey actually is providing answers to all the relevant questions. An independent researcher with relevant background in pedagogical training of vocational lecturers might need to look at the DTVE programme from these different angles. This is further elaborated under Recommendations.

Data presentation

In this section the data collected are presented.

Questionnaire for DoE lecturers. (Annex D)

The facilitators in the DoE have on average 3.3 years of experience with the programme. 33% of the lecturers have facilitated 6 or more of the 10 modules in the DTVE programme.

Relevance of the content of the DTVE

On a scale from 1 (very irrelevant) to 5 (very relevant) the relevance of each content aspect of the programme has an average rating from 4.3 to 5.0. As key performance indicator was set the value 4. Hence the content of the DTVE programme is considered to be relevant for student-teachers to become competent and efficient facilitators.

The average relevance rating for the 10 modules is in the range 4.6 to 5.0. Lecturers view the 10 modules as very relevant.

The lowest overall rating is scored on the integrated key skills (average score 4.3, closer to the partly relevant score of the relevant rating scale.

Diagram 1 illustrates lecturers ratings of the relevance of each of the 10 modules and of the incorporated key skills (KS)

Diagram 1: Bar chart illustrating the relevance rating (scale 1 to 5) by DoE lecturersrs

The relevance rating of the elements of content of each of the modules, and of integrated key skills is presented in diagram 2 and diagram 3

Diagram 2: RELEVANCE of module content as perceived by DoE lecturers

Diagram 3: RELEVANCE of module content as perceived by DoE lecturers

The lecturers identified in their individual comments 4 content areas that, in their opinion, need to be included and/or strengthened in the programme

Integrating of HIV/AIDS related issues

Use of modern technology in facilitation and learning (e-learning)

Underpinning knowledge related to adult learning principles and psychology of learning

Aspects of school management and administration (professional conduct, academic rules)

The structure of the DTVE programme

Lecturers consider the DTVE programme as a very relevant teacher training programme. However, as for the structure of the programme on several statements the key performance indicator level of 4 (on the 1 to 5 scale) was not attained. The average rating on the 1 to 5 agreement scale (from strongly disagree to strongly agree) are displayed in diagram 4

Diagram 4: Structure of the DTVE programme

Just half of the responding lecturers feel that the programme is well planned. The weakness of the programme reported is time related or support related.

In relation to time the majority of lecturers stated that

Time to complete the programme by learners is inadequate

Workload for learners and lecturers is too high

Time for teaching placement is insufficient

In relation to support the majority of lecturers feel that the support for student teacher

on the programme is inefficient

failing the programme (and having to complete in distance mode) is not effective

The items related to the above all score 3 or below. Time is a major issue in the programme (work overload) and there is need when reviewing the programme to look for ways to make the programme more time friendly.

Portfolio Assessment

The views of the 12 DoE lecturers related to the portfolio assessment mode and process indicate that the lecturer consider the portfolio assessment system to be (average scores 4.1 to 4.8)

the most appropriate for OBE

encouraging self directed learning and reflective practice

motivating

valid

fair in demonstrating the progress towards the achievement of theLOs

transparent to student-teachers

Diagram 5 illustrates the agreement rating on the statements in the questionnaire on the 1 to 5 agreement scale.

Diagram 5: Portfolio assessment mode and process as perceived by DoE lecturers

The majority consider the portfolio assessment mode more time consuming than other more traditional ways of assessment of learning. About one third of the lecturers doubt whether the assessment is facilitator independent and can ensure authentic of the work produced by learners.

The above responses are amplified in the responses to the open questions in this section of the questionnaire. The motivating aspect, quality assurance (IV/ EV) process and the learner centred nature of portfolio assessment are mentioned as the strength of this form of assessment.

As weaknesses are mentioned that portfolio assessment is time consuming, brings in issues related to authenticity of the documents placed in the portfolio (plagiarism) by the student-teachers, and for it to function w needs lecturers very familiar with OBE and its assessment.

To combat the weaknesses it is suggested that more, well trained teacher trainer, familiar with OBE should be employed and staff development workshops should be facilitated for new staff not familiar with OBE portfolio assessment.

Questionnaire for former DTVE student-teachers (Annex C)

104 former DTVE student-teachers returned the questionnaire which was made available to 154 (80%) of the DTVE former student that could be traced from 174. The questionnaire was lengthy and as a result most of the questionnaires returned missed responses to some of the questions.

The data as collected are summarised in annex C .

Relevance of the content of the DTVE and its use by DTVE graduates.

The content of the DTVE programme was rated by respondents on its relevance and its use in the work situation. As one might expect the USE ratings are all lower than the RELEVANCE ratings. The content of the DTVE programme is considered by DTVE graduates as (highly ) relevant all content being rated above the performance indicator (4) set. The relevance rates are in the range 4.2 5.0.

The use frequencies were rated on a nominal scale from never used (1) to very often (5) and fall in the range from 2.5 to 4.8. Diagram 6, 7 and 8 display relevance and use ratings of content

Diagram 6: Relevance and use of modules compared.

Diagram 6 clearly illustrates the lower rating on use than on relevance of the content of the 10 modules and the in-cooperated keys kills.

The DTVE graduates ratings on relevance and use of content was analysed across the following three categorisations of the population

gender: male or female

background: in service (employed in the TVET sector and send for teacher training) or pre service (mainly following completion of diploma course in a vocational area but without work experience in the TVET system)

mode of studying: in full-time face to face mode or in block release mode

Average ratings for each group in each category were computed in Excel and a t-test to test for significant difference at the 95% confidence level was carried out (Annex H).

No significance differences were found. The expressed views and given ratings on aspects of the DTVE programme are independent of gender, mode the programme was taken by the student-teacher or whether the student teacher was in service or pre service.

Diagram 7: Modules content relevance and use as perceived by DTVE graduates

Diagram 8: Modules content relevance and use as perceived by DTVE graduates

The USE of following content was rated 3 or below

use of manipulatives in learning sessions

use of PowerPoint presentations

keeping of personal developmental log

action research

The low use of PowerPoint presentations is mainly due to lack of the appropriate technology in the TVET institutions, as reported during interviews.

Content that was mentioned as to be included or strengthened can be place in the following categories

Underpinning theory / knowledge

Curriculum development

Psychology (adult educational)

Learning theories

Botswanas educational laws and policies

Key skills

Stress management

Time management (2)

Data analysis (2)

Presentation skills

ICT

Software use such as PowerPoint, Excel (6)

Internet use and search (3)

Video production

Introducing and training of vocational area specific software (e.g. AutoCad)

Management and administration in education

Principles of educational administration and management

Guidance and counselling

School resource management

Use of modern technology in education

E-learning

Video conferencing

E-assessment

BTEP phase 1 - 3

Full coverage (and recognition by DTVET) of BTEP phase training

Three of the above areas, underpinning knowledge, use of modern technology and management/administration in education, were also mentioned by the DoE lecturers

The structure of the DTVE programme

The responses to the items in the questionnaire (item 1.63 1.78) related to the structure of the DTVE programme indicate that the respondents agree that the programme is enjoyable, interesting, motivating, well planned, of high quality. The average score on the agreement scale on these items was in the range 3.2 to 4.3. Diagram 9 illustrated the strength of agreement to the items in the questionnaire.

Diagram 9: Structure of the DTVE programme views of DTVE graduates

Two items score below the neutral point of 3. Both are time related. Workload is seen as not acceptable and time to complete LOs as insufficient. In the general comments (question 1.79) this is amplified. None of the cohorts completed in the 18 months scheduled and a number of months were added to each course.

In the general comments, remarks related to the structure of the programme restate in many different forms the two low rated statements

The workload is too heavy and

Time is insufficient

it can argue that both statements express the same. The time is insufficient to

Fully benefit from TP as TP is too much evidence collection focussed instead of developing classroom management and facilitation skills in a range of situations

To learn, the focus is on evidence collections and document production with insufficient time to digest the underpinning theory. This results in copying / plagiarism / buying of evidence from other groups and/or cohorts

One suggestion forwarded was to make all the learning resources for each module available online (e-learning) so student-teachers can work at own pace.

Mode of facilitation used in the DTVE programme.

The items 2.01 to 2.17 requested the former student teachers to give their opinion on the facilitation mode used. The DTVE programme is designed to do what it preaches: using learner centred, blended approaches to supporting student-teachers to acquire knowledge, skills and attitudes. Items 2.01 to 2.11 ask learners to rate effectiveness of various methods used and to rate the frequency these methods are used.

Diagram 10 shows a bar chart displaying the average ratings on effectiveness and frequency of use.

Diagram 10: Facilitation modes used in the DTVE programme rating of

effectiveness and frequency of use

The methods mentioned are all considered to be (very) effective with ratings in 4.1 to 4.6 range. The uses of the various methods range from 2.8 to 4.2. Least used (rate 2.8) are video based sessions. As the method used will depend on the content it is not surprising that certain methods are used less frequent than others.

Average ratings across gender, programme mode, and in/pre service entry were checked for significant differences using t-test. No significant differences at the 95% confidence level were found.

Respondents prefer mainly co-operative learning approaches and dislike teacher centred, lecure type of facilitation mode.

The variety in facilitation mode is found to be fine by 82% of the respondents. The pie chart displays the responses.

Diagram 11: Pie chart illustrating views on variety in facilitation

As for the mode of facilitation catering for the preferred learning style of the student-teacher, the majority (about 60%) feels that that is true to some extent. The results are in table

Table 6: Frequencies of responses on item 2.16

Mode of facilitation catered for preferred learning style

Frequency

Percent

Not at all

2

2%

To some extend

61

59%

Very well

40

39%

The mode of facilitation used by DoE reflects very well (70%) a constructivist, learner centred approached according to the student-teachers. Diagram 12 illustrates in a pie chart student-teachers views.

Diagram 12: Pie chart displaying student-teachers views on constructivist,

learner-centred facilitation modes used by DoE lecturers.

Views of former DTVE student teachers on variety in facilitation, link between mode of facilitation and own preferred learning style and whether facilitators use a learner centred constructivist approach do not differ significantly across gender, learning mode (FT or BL) or across in/pre-service student teachers.

Quality of facilitation by DoE lecturers.

Section 3 of the DTVE graduates questionnaire collected data on the views of the former students as to the quality of the facilitation they received. Customer satisfaction surveys are important for the service provider in order to find out whether or not customers are satisfied. The information obtained can guide well informed improvement plans. In the educational setting the student-teachers are the customers and the lecturers the service providers.

The rating on a scale from very poor (1) to very good (5) are displayed in the bar chart in diagram 13.

Diagram 13 Former DTVE student-teachers views on the quality of their facilitators

Student teachers average rate on each item is in the adequate to very good range. Student teachers generally express the opinion that their lecturers are doing a professional job. Student teachers are least satisfied with the turn over time of assignments (rated 3.6). Across the categories used in this survey the average ratings are not significantly different (Annex H).

In the open question space student-teachers were positive about the facilitation of their lecturers. Positive critical observations were clearly (i) referring to isolated cases (ii) aimed to improve the learning environment and interaction between student-teachers and lecturers.

Issues mentioned by more than one student-teacher

Give prompt feedback *6

Ensure timely feedback well before submission deadline *2

Give constructive feedback, not destructive *3

Treat student teachers with respect / as adults *5

Employ qualified lecturers familiar with the DTVE programme / OBE *7

Employ DTVE graduates as SDFs *6

Lecturers need to come to sessions well prepared *4

Lecturers should be familiar with / inducted into the DTVE programme *7

Lecturers should be open to criticism *2

Inconsistency in assessment among lecturers need to be addressed *3

Lecturers should also be specialist in a particular vocational area *4

Address authenticity / plagiarism *2

Act professional, avoid bias, favouritism

DTVE student-teacher perception on portfolio assessment

DTVE is assessed continuously through portfolio building. Particularly for teacher trainin programmes, portfolio assessment is very appropriate, provided it is used effectively and efficiently. The DTVE portfolios should contain the evidence for the learning outcomes and document the process that lead to the final evidence. Student-teachers, were to rate their agreement with statements related to portfolio assessment. Diagram 14 illustrates the average rating on the portfolio assessment related items

Diagram 14: Bar chart of average agreement ratings on items on portfolio assessment mode nad process as perceived by the former DTVE student-teachers

Student teachers were in agreement (score 4 of above) with the statements that portfolio building as used in the DTVE programme is most appropriate, ensures all learning outcomes are covered, gives responsibility to learners for own learning, enhances reflective practice and critical thinking, is motivating, transparent and effective to monitor own progress. These aspects are mentioned in the open items 4.17 and 4.19 where as strength of the portfolio assessment system were mentioned:

Continuous nature of portfolio assessment, the cycle: submit evidence receive feedback implement feedback resubmit

Portfolio assessment (PA) allows to monitor growth and progress

PA encourages reflective and critical thinking

PA places responsibility for learning and evidence collection with the student teacher

PA ensures all LOs are covered

PA process is transparent as the portfolio verification document describes in detail the evidence to be submitted

The below 4 agreement ratings scores are on exactly the same aspects as identified by the DoE lecturers:

Reliability

Authenticity

Time consuming mode

The three aspects being problematic are described in a variety of ways in item 4.18

Reliability of assessment

Marking is subjective

Different lecturers use different criteria (You pass with lecturer A, but will have to resubmit with lecturer B)

Authenticity of submitted evidence

Copying from other students (as work will be submitted to different lecturers, hence copying / plagiarism goes unnoticed in nearly all instances)

Plagiarism from web

buying portfolios from previous cohorts

Focus on evidence production / assessment at the cost of learning encourages copying due to time pressure and deadlines to be met

Time consuming nature of PA

Too much work within a short time

not achieved for minor short comings [could be resolved using oral evidence as in BTEP]

The expressed views of the DTVE student teachers on portfolio building are independent (at 95% significance level) of gender, whether they are in- or pre-service student teachers and whether they studied in the full time face to face mode or block release mode (See annex H).

Effectiveness of the DoE support system for the DTVE programme

In all modes of programme delivery the support available and given to learners to achieve the learning outcomes is crucial. One might produce the most engaging, motivating learning materials but without effective and efficient support to learners the materials by itself will not produce the desired outcome i.e. that learner will learn and achieve the set learning outcomes. Within the DTVE programme the student teacher may aspect support and guidance from (i) the module facilitator on academic aspect and evidence production for the module (ii) his/her personal tutor to receive guidance and support in a holistic way. The (weekly) meetings with the personal tutor will discuss not just a single module but look at the overall picture of the programme and set attainable targets on weekly basis. Action planning is a key feature, as is the addressing of (personal) issues that might negatively impact on achieving set goals. (iii) The TP mentor, the experienced lecturer / instructor in the institution where the student teacher is placed for TP. The mentor inducts supports and assists the student teacher throughout the TP period by observing the student-teacher, giving feedback and supporting in evidence collection.

The questionnaire items 5.01 5.12 addressed the above issues. Diagram 14 displays in a bar chart the average agreement ratings on the items related to the availability and receiving of support.

Diagram 15: Bar chart of average agreement ratings on support related issues in the DTVE programme as perceived by former DTVE students

Average agreement ratings on the item seeking view on the support available and given to the DTVE student teachers falls within the 3.4 4.2 range. The lowest average agreement rating (3.4) is on the item relating to a personal guidance/counseling, a system to address personal problems negatively affecting the performance of the student teacher. This is not surprising as the DoE has for long been depending on guidance and counselling officers NOT located within the Department. With DoE at FCTVE the current student-teachers can make use of the Guidance and Leaner support system in place within FCTVE.

Overall student teachers perceive the support system available and the support given by their lecturers as adequate for their needs. Among strengths are mentioned (item 5.19):

Facilitators are very supportive throughout the programme *18

Facilitators are committed and readily available for support *3

Strong support given before and during TP

Close cooperation between student-teachers and facilitators, and constructive feedback ensures effective evidence collection *2

Counselling available when needed (if the going was tough) *2

Personal tutor system effective to overcome academic problems

Learning resources accessible and available *6

As weaknesses in the support system student-teachers mentioned:

Facilitators are unskilled/less experienced in OBE *6

Feedback provided

Not timely *8

Inconsistent across facilitators (biased /reflecting favouritism of lecturers) *6

Plagiarism / copying goes undetected *2

Insufficient briefing and support for student-teacher placed in distance learning mode

Supplementation rates of DTVE students are high. Few pass after final submissions of their portfolios. The supplementation rate for the first cohort was 33%, for the second cohort 10%, while for cohort 3 and 4 the supplementation rate was 100% - no student teacher passed on first attempt. Those who supplement are provided with extensive feedback on what is to be done in order to meet the evidence requirements; this is part of the support system. Those failing the programme are referred to distance mode they can complete the programme in distance mode. For those having failed and continuing in distance learning mode a support system is crucial, without it they will not be in a position to successfully complete the programme. Diagram 16 is a bar chart displaying the average agreement rate of DTVE former students that were supplementing and/or referred to the distance learning mode.

Diagram 16: Average agreement ratings on items related to support during supplementation and support ro distance learners as perceived by

former DTVE students

The average agreement ratings by supplementing DTVE students on the questionnaire items 5.13 5.15 are between 3.5 and 4. This is in the partly agree region. It suggests that the support system during supplementation needs to be critically reviewed to make it more effective and efficient.

This corresponds to the rating of the DTVE facilitators on the statement 2.17. The programme has efficient support systems in place with an average agreement rating of 3.0 half of the responding facilitators (strongly) disagreed with the statement.

DTVE student-teacher that failed the programme and were placed in distance mode rated the items related to support in the range 1.8 to 2.0. In other words they perceive that there is no effective operational system in place to support them. 50% of the facilitators concur with that perception (item 2.18 in the lecturers questionnaire score average rating of 2.7)

The expressed views of the DTVE student teachers on the DTVE programme support system are independent (at 95% significance level) of gender, whether they are in- or pre-service student teachers and whether they studied in the full time face to face mode or block release mode (Annex H).

Comparing responses from DoE lecturers with responses of the DTVE student-teachers.

The questionnaire for lecturers and for DTVE graduates had many identical items. This allows the perceptions of the two groups to be compared and tested for significant differences. (Annex H)

DTVE programme content

The average relevance ratings of content and modules in general by DoE lecturers and DTVE graduates were compared. A t-test was applied at 95% confidence level no significant difference were found. The ratings of DoE lecturers and of DTVE student-teacher are not in any way different.

Table 7 compares the average relevance rating by DoE lecturers and DTV graduates.

Table 7 Average relevance ratings for content of modules DN01, DP01 and DP02 by DoE lecturers and DTVE student-teachers.

DTVE graduates

DoE lecturers

1.01 Diagnose Learners Needs

4.6

5

1.02 Determine Learning Style

4.5

4.8

1.03 Methods for different needs and learning styles

4.7

5

1.05 Interpret LU specifications

4.6

5

1.06 Prepare scheme of work

4.8

5

1.07 Prepare session plans

4.7

5

1.09 Different activities to meet LOs

4.7

4.6

1.10 Written LM to support learning

4.9

4.9

1.11 Audio/visual LM to support LOs

4.7

5

1.12 Visual learning aids to support LOs

4.8

4.8

1.13 Manipulatives to support Los

4.5

4.3

1.14 OHT to support Los

4.6

4.5

1.15 PowerPoint to to support LOs

4.6

4.5

1.16 Internet/Web based resources to support LOs

4.7

4.6

The differences in ratings are very small and not significant at 95% confidence level (Annex H) The content of the modules is rated as (very) relevant by both student-teachers and DoE lecturers. Diagram 16 in a double bar chart illustrates that ratings of the two groups are extremely close to each other or identical in some instances.

Table 8: Average overall relevance ratings of each module and the integrated key skills (KS) by DoE lecturers and DTVE student-teachers.

DTVE graduates

DoE lecturers

1.18 Group discussions in facilitation

4.8

4.8

1.19 Cooperative learning spproaches

4.6

5

1.20 Experimental learning approach

4.7

5

1.21 Individualised learning approach

4.6

4.8

1.22 Range of motivational techniques to support LOs

4.7

5

1.23 Manage conflict situations

4.7

4.7

1.24 Effective questioning techniques

4.8

4.9

1.25 Promotion of equal opportunities

4.8

4.5

1.27 Induction programme

4.8

4.9

1.28 Support to meet LOs

4.7

4.6

1.29 Personal support and guidance

4.6

4.5

1.31 Different formative assessment activities

4.8

5

1.32 Feedback to support learning

4.9

5

1.33 Keeping record of learners achievements

4.9

4.9

1.34 Fair and unbiased assessment

4.8

5

1.36 Evaluate own facilitation

4.6

4.9

1.37 Evaluate teaching methods

4.6

4.9

1.38 Evaluate learning resources

4.6

4.9

1.40 Being a reflective practitioner

4.2

4.6

1.41 Keeping personal develoment log

4.0

4.4

1.43 Evaluate own practice using action plans

4.4

4.5

1.44 Effective working relationships

4.7

4.5

1.45 Adhering to high professional standards

4.8

4.5

1.46 Championing teaching and learning methods

4.7

4.9

1.47 Enterprise activities with institutions

4.2

4.7

1.48 Proactivity on own further development

4.7

4.8

1.50 Undertake action research

4.4

4.9

1.52 Using Word

4.8

4.7

1.53 Use speadsheet, e.g. graphing

4.8

4.6

1.54 Use power point

4.8

4.5

1.55 Use data analysis techniques

4.8

4.6

1.56 Use Web search

4.7

4.5

1.57 Use APA referencing

4.5

4.5

1.58 Use discussing skills (participating)

4.8

4.5

1.59 Use discussion skills (leading)

4.8

4.4

1.60 Use presentation skills to audience

4.9

4.7

Note that on the key skills content the average ratings by DoE are ALL less than the average ratings of the student teachers.

The overall relevance rating of each module by DoE lecturers and TVE student-teachers are displayed in table 9.

Table 9: Average overall relevance ratings of each module and the integrated key skills (KS) by DoE lecturers and DTVE student-teachers.

DTVE graduates

DoE lecturers

DN01 Assess Learners Needs

4.6

4.7

DP01 Plan for effective Learning

4.8

5

DP02 Prepare Resources to support Learning

4.8

4.9

DF01 Facilitate Learning

4.8

4.9

DS01 Provide Learners with Support

4.7

4.6

DA01 Assess learning

4.7

5

DA02 Evaluate learning Sessions

4.6

4.8

DE01 Reflect upon Roles of a Teacher

4.4

4.7

DE02 Improve own professional Practice

4.7

4.7

DE03 Undertake Action Research

4.3

4.9

Key Skills

4.8

4.3

The two differences, although not significant, to note are on DE03 and KS. Student-teacher are not strongly convinced about the relevance (and use) of module DE03, the action research module, while lecturers consider it highly relevant. For the integrated key skills it is the other way around. Student teachers find key skill very relevant and mentioned (see section 5.2.1) that more attention and support should be given to key skills.

DTVE programme structure

Identical questions in the DoE lecturers and the DTVE graduates questionnaires allow to compare ratings on programme structures. The ratings and the items are tabulated in table 10.

Table 10: Comparing ratings of related to programme structure by DoE lecturers and DTVE graduates

Item

DTVE graduates

DoE lecturers

Enjoy programme

3.9

4.0

Programme well planned

3.2

3.5

Good sequencing of Units

3.3

4.0

Appropriate Content

3.4

4.4

Time for Completion adequate

2.6

2.0

Workload acceptable

2.2

2.3

High Quality programme

4.2

4.3

Well prepared for TP

4.1

4.1

Integration of TP and college work

4.1

4.4

Programme is flexible

3.5

3.1

On most items DoE lecturers give a slightly higher rating. The problem of too heavy workload and the insufficient time given to complete the programme is shared between lecturers and student-teachers.

DTVE portfolio assessment system

Both DoE lecturers and DTVE student-teachers rated items related to the portfolio assessment. They are tabulated in table 10 with the average agreement rates. Diagram 20 illustrates the data in the table in a double bar chart. The difference in rating are not significant at 95% confidence level as checked by using a t-test (Annex H).

Table 11 Comparing of average ratings on items related to portfolio assessment by DoE lecturers and DTVE graduates

Portfolio assessment related items

DTVE graduates

DoE lecturers

Most appropriate for OBE

4.3

4.6

Ensures evidence for all LOs

4.4

4.5

Encourages self directed learning

4.5

4.8

Demonstrates progress towards identified LOs

4.3

4.3

Encourages reflective practice

4.3

4.1

Provides opportunities to demonstrate competence

4.3

4.1

High validity

4.3

4.4

Fair way to document progress

4.5

4.4

Motivating

4.4

4.3

Reliable method. Lecture independent

3.8

3.5

Ensures authenticity of evidence

3.7

3.9

Transparent method of assessment

4.1

4.1

Encourages critical thinking

4.4

3.3

NOT more time consuming than other modes of assessment

3.2

2.5

DoE lecturers and DTVE student-teachers share the perception that using portfolio assessment is more time consuming than other forms of assessment. They share the view that portfolio assessment is most appropriate for the outcome based DTVE programme, ensures all learning outcomes are covered, gives responsibility to learner for own learning, enhances reflective practice and critical thinking, is motivating, transparent and effective to monitor progress.

DTVE programme support system

In the student-teacher questionnaire a section was focusing on the support system of the DTVE programme. In the DoE lecturer questionnaire only two items (2.17 & 2.18) referred to the support system. The average agreement score on these items is low, 3.0 and 2.7 respectively. Both the support system during the running of the DTVE programme and the support system to support student-teachers that failed (and are moved to the distance learning mode) the programme are perceived as inefficient. The student-teachers share this opinion as outlined in section 6.2.6.

The DTVE programme structure, its content, portfolio assessment system and support system are perceived in the same way by DoE lecturers and student teachers. Any differences in average rating of corresponding items in the two questionnaires are not significant. Strengths and weaknesses are shared ground and hence form a solid basis for implementation of any change in the programme that might be needed. There is no disagreement between DoE lecturers and their former students as to the strong and weak aspect of the DTVE programme.

Structured focus group interview: management and supervisors of DTVE graduates

Structured focus group interviews were conducted with management/supervisors of the DTVE graduates with in 6 TCs and 13 groups in the Brigades. A total of 59 respondents were involved 27 in the TCs and 32 in the brigades. The details are in table 12.

Table 12 Colleges and brigades visited and number of respondents in the focus group interviews with management / supervisors of the DTVE graduates

Institution: College / Brigade

Management / supervisors

ATTC

4

GTC

5

JTeC

4

MTC

7

PaTeCo

2

SPTeCo

5

Total TCs

27

Barolong VTC

2

KRDA

4

Marapong

3

Marobela

2

Naledi

6

Ngethu

3

Okavango

2

Ramatea

1

Shashe

1

Tlokweng

1

Tswelelopele

2

Tutume

5

Total Brigades

32

The purpose of these structured focus group interviews (Annex E) with management / supervisors of the DTVE graduates was to find out: How do managers / supervisors of DTVE graduates rate the quality of their facilitation as compared to facilitators without this teacher training background? The summary of the responses of the 19 structured focus group interviews with management/supervisors of former DTVE students is found in Annex H.

The most frequent mentioned strengths of the DTVE graduates were:

Competent and confident in learner centred methods

Effective use of teaching/learning aids

Effective use of a variety of teaching/learning methods

Effective planning and preparation of lessons

Preparing effective schemes of work

Supporting management/Co operative

Supporting learners in achieving the learning outcomes

Demonstrating professional behaviour/ Good work relationship with learners

Contribute ideas and take initiative in the institution

Professional interaction with learners

The responses are based on classroom / workshop observations by management / supervisor taking place at least once every term and the level of participation in staff meetings observed. Management / supervisors in brigades reported a stronger impact of the DTVE