Evaluation of ESF intervention for individual employability  

Evaluation of Programmes and Policies for Human Resources Development Area

Rome, Corso D’Italia, 33

Evaluation of Programmes and Policies for Human Resources Development Area

Rome, Corso D’Italia, 33

1. National Placement Working Group’s Activities

ContentsIn section 1 we look through the activities of the ESF National Placement Working Group (hereafter NPG), over a period of more than 10 years. The activity of NPG lies on the evalutation of training effects on individual employability.Isfol coordinates the NPG, which members are the regional/national Esf Managing Authorities and their designed referents.

In brief

• In Italy, training policies are under the scope of regional’s competences (jurisdiction).

• Till earlier EU programming period (2008), training projects were mainly co-funded by ESF.

• In this current programming period, 2007-2013, the ESF is still an important financial instrument on training policies but, in particular for regions of Ob.2, it has reduced its financial weight

Evaluation of ESF intervention for individual employability

1. A coordination over regional evalutation activieties is needed to allow getting a consistent national synthesis

2. More than it’s used to be, at present evaluation activity should be “tied” to the policy whatever it is the financial source.

The PNG’s activities: chronology

1. Data on training projects and beneficiaries: the application form for pre-enrolment

3. Models for the impact evaluation (net effects)

2. Harmonizing regional’s surveys: the questionnaire

4. Robustness of the surveys (from a statistical point of view)

5. Exploiting administrative database for evaluation purposes

1. Data on training projects and beneficiaries: the application form for pre-enrolment

“ […]The best solution to the evaluation problem lies in improving the quality of the data on which evaluations are conducted and not in the development of formal econometric methods to circumvent inadequate data.” (Heckman, Lalonde e Smith, 1999)

1. Data on training projects and beneficiaries: the application form for pre-enrolment

Placement National Working Group

(NPG)

National Monitoring Working Group

(NMG)

• Isfol coordinates two national working group that are strongly tied up each other

1. Training projects and beneficiaries database

• 2. Harmonizing regional’s surveys: the questionnaire

• 3. Models for the impact evaluation (net effects)

• 4. Robustness of the surveys (from a statistical point of view)

• 5 ….

2. Harmonizing regional’s surveys: the questionnaire

PNG’s earlier activities went to a shared “core“ questionnaire: a set of common questions that each regional questionnaire should incorporate. This, in turn, entails:

• a national synthesis of the main results;

• the definition of a set of common result indicators such as the “gross employment rate of training interventions”

2. Harmonizing regional’s surveys: the questionnaire

• To reach these objectivies it’s necessary to settle the same “time-lag”, beginnig from the conclusion of the training course (T), which the employability of the beneficiaries will be referred at. Choice had been twelve months after conclusion of training (T+12).

• Once defined the reference time “T+12” one should take care of the interview’s timing. For a trade-off rises up:

• As far as the lag becomes greater, the measurement errors increase for individuals could not have memory of their own condition at time T+12 (lower reliability of answer);

• As far as the lag increases, important data, such as telephone numbers, becomes unusable for a greater number of beneficiaries;

• As far as the lag becomes greater, it’s going to be relevant getting more informations about the currently condition of beneficiary.

3. Models for the impact evaluation (net effects)• Here we distinguish:

– Impact evaluation on job creation (net job) – Impact evaluation for individual employability (net employment rate or

net effects)

• Working document n°6 (CE, 2007) “Measuring structural funds employment effects”

(1) NET JOBS = GROSS JOBS X (1– DEADWEIGHT) X (1 – DISPLACEMENT AND SUBSTITUTION)X (1 + SUPPLIER MULTIPLIER + INCOME MULTIPLIER)

Net jobs represent the overall jobs created as the sum of both direct (deadweight) and indirect

(displacement, substitution, supplier and income multiplier) effects.

• Our analysis on impact evaluation is restricted to the direct effect:

(2) NET EMPLOYMENT RATE = GROSS EMPLOYMENT RATE X (1– DEADWEIGHT) Within econometric models, expression (2) is measured as a difference between the averages

employment rates fitted for people treated (beneficiaries of training intervention) and for people belonged to the control group (not treated) which approximate the counterfactual situation. Clearly these models rely upon both to the consistency of the control group and, from (1), to the assumption that indirect effects are disregarded.

3. Models for the impact evaluation (net effects)

Impact evaluation of training policies (but the same could be said for other active labor market policies) on individual employability deals with demands like this:

What’s the effect (sign and measure) on the probability to get a job produced by the

partecipation to a training intervention? To provide an answer to this/these question(s), the researchers have to adequately

approximate the counterfactual situation (not osservable)

=What if

3. Models for the impact evaluation (net effects)

Remark:• Evaluation on employability effects should not be constrained to the

availability of an impact evaluation

• Surveys on people trained could satisfy several purposes related to the employability scope which don’t need of counterfactual

3. Models for the impact evaluation (net effects)

In the last programming period (2000-2006) Isfol has realised in Objective 3 an impact evaluation (effectiveness of training interventions on the probability to find a job). The study:

– was realised with data come from regional surveys. Those surveys were conducted during the years 2003-2005 on training projects completed between the years 2001-2003

– was restricted to a subset of regions, interventions and beneficiaries, because of the informations available needed for the econometric model specification (we refer to the instrumental variables used to tackle selction bias problem);

– utilised Istat Labor Force Survey panel data (twelve months) to build up an external control group for the counterfactual estimates.

3. Models for the impact evaluation (net effects)

Difficulty to have an internal control group (or consistent/effective external control group) and the lack of effective instrumental variables to face the selection bias problem, are among the main causes that, in the past, precluded to conduct impact evaluation.

Once more, we have to remark the importance of ex-ante data availability (projects and beneficiaries)

4. Robustness of the surveys (from a statistical point of view)

Often in the previous slides we’ve utilized the word “survey”. We’ve punctualized that each region chooses if, when and how realise surveys on training beneficiaries. The NPG guidilines keep attention about the statistical foundations the surveys ought to meet, in each step:

1. Planning.2. Sample design (in case of sample survey).3. Interviews.4. Results analysis.

1. PlanningDrawing the evaluation purposes. This yields effects on both sample design and questionnarie structure. For example, a preliminary study should highlights the informations not available that must be “recovered” through the questionnarie during the interview.

4. Robustness of the surveys (from a statistical point of view)

2. Sample designKeeping attention over all the aspects linked to the sample strategy: sample numerosity, sampling errors, sample estimator, and so on.

3. InterviewsDoing actions able to “prevent” or “minimize” the non-sampling errors (measurement errors due to missing data).

• Take care of beneficiaries list (survey frame), in particular telephone number and postal/home address;

• Send to the beneficiaries (at least the persons sampled) a letter, which gives them information of the survey (goals, assurances as regard the respect of laws on statistical treatment of personal data, and so on)

• Pilot survey to test both the questionnaire and CATI• On going evaluation about the interview process.

4. Robustness of the surveys (from a statistical point of view)

4. Results analysis Inference

Other methods finalized to detect the individual labor market condition (as an example there are the follow-up questionnaires), appears to be not suitable for evaluation scope:• The structure of questionnaires is not adequate to meet evaluation

requirements

• Data doesn’t meet reliability criteria.

5. Exploiting administrative database for evaluation purposes

During the last meeting of NPG, discussion themes have been concerned with the opportunity of exploiting administrative database for evaluation goals. The expression “Administrative Placement” have been coined. Database (Labor informative system) entails, for each individual here registered, to get overall informations on job career.Several points should be stressed:

• Exploiting administrative data sources could indeed reduce survey costs;

• Administrative database could make it possible to get a reliable and consistent external control group for impact evaluation;

• Administrative database could be utilized to update telephone numbers and/or postal addresses.

5. Exploiting administrative database for evaluation purposes

• Administrative Placement spread all these positive effects whenever it’s thought together with, not as a substitute of, placement survey.

• Further, it must be verified the quantitative and qualitative measure of beneficiaries that are not covered by the administrative database.

2. Centralised Placement surveys

ContentsIn section 2 we look through the coexistence of regional and centralised placement surveys. The latter are caracterised by a single sample design and the territorial coverage embeds more than one region, possibly they could involve the whole national territory. Here, the coordination role of NPG is even more necessary to avoid useless and harmful overlaps.

Exploiting administrative database for evaluation purposes

Centralised Placement surveys

ISFOL has conducted two “placement” surveys during the early ESF programming period. Both refer to regions of Objective 1:

• Training interventions for individual employability funded through the ESF Regional Operational Program

• Masters and PhDs funded through the National Operational Program “Ricerca Scientifica, sviluppo tecnologico, alta formazione” (MIUR).

Centralised Placement surveys

• Centralised surveys came from a sharing process between region involved (ESF Regional Managing Authorities) and Isfol.

• Centralised surveys ensure estimates with regional statistical significance

• Each region will have returned the microdata of the survey referring to its training interventions

Centralised Placement surveys

Though the national survey likes covering the whole national territory, some specific evaluation topics (impact evaluation) can be confined to a lower number of regions…

At present, Isfol is conducting a national survey on indivual employability effects of training interventions cofunded by ESF. National survey has been planned within the NPG. Regions provided to the Isfol researcher group the interventions and beneficiaries datasets. The survey concerns training projects completed between the years 2008 and 2009, irrespective of the programming period (2000-2006 or 2007-2013).

Thanks for your attention

Toti Enrico

Isfol – Evaluation of Programmes and Policies for Human Resources Development Area

Tel. +39 06 44590688

e-mail e.toti@isfol.it