final report of the esco skills mapping pilot...european skills, competences, qualifications and...

Final report of the ESCO skills mapping pilot

02 July 2018

European Skills, Competences, Qualifications and Occupations Final report of the ESCO skills mapping pilot

July 2018 2

Contents

1. Introduction .................................................................................................. 5 1.1. Objective and scope of the pilot ................................................................. 5 1.2. The context of the project ......................................................................... 5 1.3. Structure of the document ......................................................................... 6

2. The methodology of the pilot ........................................................................... 6 2.1. Step 1: Select the National Skills Classifications ........................................... 6 2.2. Step 2: Select the skills ............................................................................ 6

2.2.1. Correspondence to transversal skills ..................................................... 6 2.3. Step 3: Create the mappings ..................................................................... 7 2.4. Step 4: Assess the results ......................................................................... 8

2.4.1. Assess the mappings ........................................................................... 8 2.4.2. Assess the level of interoperability ........................................................ 9

3. Results ........................................................................................................14 3.1. Results: Assessment of the mappings ........................................................14 3.2. Results: Assessment of the level of interoperability .....................................20

3.2.1. Use case: Part 1 ................................................................................20 3.2.2. Use case: Part 2 ................................................................................23

3.3. Lessons learnt ........................................................................................25 3.3.1. Lessons learnt about requirements for tools and services ........................25 3.3.2. Lessons learnt about the process .........................................................25 3.3.3. Lessons learnt about resources requirements ........................................26 3.3.4. Lessons learnt about the achieved level of interoperability ......................26


July 2018 3

Executive summary

Objective and scope of the pilot

Through the implementation of the skills mapping pilot, the Commission services aimed

to:

1. Enable the public employment services (PESs) of the MS to test and learn about the

process and the resource requirements for the creation of the correspondence tables

of skills;

2. Understand the requirements for the tools and/or services to support the mapping

process;

3. Understand the level of interoperability that can be achieved between ESCO and

national skill classifications and

4. Understand the level of interoperability that can be achieved among the national

skill classifications during cross-border matching.

The methodology of the pilot

The execution of the pilot consisted of the following steps which encompassed the setup

of the process, execution of the mapping and assessment of the results:

Select the National Skills Classifications

Austria, Romania and Sweden expressed interest to participate. The Commission

services accepted the candidatures of Austria and Sweden and invited Romania to

participate as an observer due to lack of a national skills classification.

Create the mappings

The Commission services transformed the selected data into the required data format

to create correspondences between classifications and import the data into a support

tool for thesaurus alignment.

During the creation of the mappings, the tool provided suggestions for mapping

concepts. PES experts reworked mappings manually, i.e. approve the mapping

relationships suggested by the tool or reject them and select new ones.

Assess the results

Upon the completion of the mappings, the Commission services assessed the mappings

created between the national skills classifications and ESCO.

Assessment of the results

The following numbers of skills from each classification were used as a sample:

(1) ESCO - 610 skill concepts (2) Austrian classification - 419 skill concepts

(3) Swedish classification - 74 skill concepts.

When evaluating the results of the automated transformation of CV/job vacancy info

using the mapping tables, the process provides 60/77% of accuracy.


July 2018 4

The automated transformation using mapping tables introduces a high level of “noise”.

“Noise” represents skills which are introduced during the transformation but are not part

of the correct skills in the target system. Such skills have a negative impact on job

matching since they reduce the accuracy. On the other hand, “noise” maybe be useful

for another purpose, such as career guidance. The degree of “noise” identified in our

testing is 81/92%, meaning that only one/two out of every ten skills represented a

correctly transferred skill. This has to be taken into consideration when using the data

for job matching purposes.

Lessons learnt about requirements for tools and services

With regard to the key functional requirements for an online mapping platform, the

discussions with the participants highlighted what the experts need:

• The software platform has to provide good performant and speedy results to users’

actions. The performance of the current platform was noted to be insufficient.

• The search functionality of the platform has to be improved. The MS noted that they

have to be able to search with only partial strings of words because of the complexity

of their languages.

Lessons learnt about the process

Regarding the mapping process and the organisation of the work it can be noted that:

• More accurate mappings can be achieved with more descriptive skill properties in a

classification (descriptions, alternative labels, scope notes…).

• The workshop participants concluded that the mapping of skills may be perceived as

arbitrary and subjective, since it is affected by:

o sectoral and societal differences among the Member States;

o different understanding of skill concepts;

o differences in applying context to skill concepts.

• The results of the mapping exercise indicate that although the transformation of

documents using the mapping tables introduces a high level of “noise”, the

usability of the documents is retained.

Lessons learnt about resource requirements

The following effort was noted by the participating Member States for mapping a sample

of 610 ESCO skills:

• Austria: 50 Hours

• Sweden: 80 Hours

This effort does not include quality assurance, only an initial creation of the mapping

relations.

Extrapolating the time investment to full mapping (13 485 ESCO skills) would result in

an indicative effort of 170 Days of work. It is worth noting that technical improvements

on the mapping platform could lead to a more streamlined mapping process and to

lowering the effort required.


July 2018 5

1. Introduction

1.1. Objective and scope of the pilot

The Commission is currently testing the ESCO classification in pilot projects. This

includes pilot projects that are launched by third parties, such as private companies or

Member States (MSs) authorities and that want to test ESCO for service delivery. It also

includes pilot projects set up by the Commission, such as a mapping pilot for mapping

skills classifications to ESCO which is a subject of this report.

Through the implementation of the skills mapping pilot, the Commission services aim

to:

1. Enable the public employment services (PESs) of the MS to test and learn about the

process and the resource requirements for the creation of the correspondence tables

of skills;

2. Understand the requirements for the tools and/or services to support the mapping

process;

3. Understand the level of interoperability that can be achieved between ESCO and

national skill classifications and

4. Understand the level of interoperability that can be achieved among the national

skill classifications during cross-border matching.

1.2. The context of the project

The Commission services previously coordinated and carried out two mapping pilots

both with the public and private sector:

1. a pilot in the field of occupations1 between ESCO and national occupational

classifications and

2. a pilot in the field of skills between ESCO and LinkedIn.

The experience and tooling from the past pilots were used during the current skills pilot

between ESCO and national skill classifications. The purpose of this pilot was to support

the implementation of the EURES Regulation 2016/589/EU. As part of this regulation,

the Member States have an option to create mappings between their national skill

classifications and ESCO.

1 https://ec.europa.eu/esco/portal/escopedia/ESCO_occupations_mapping_pilot


July 2018 6

1.3. Structure of the document

In light of the above, this document will be structured as follows:

• The methodology of the pilot: description of the methodology and the work carried

out

• Results analysis divided into:

o Results: Assessment of the mappings

o Results: Assessment of the level of interoperability

• Lessons learnt: lessons learnt for the mapping process and the mapping

environment, as well as an estimate of the workload for a full-scale mapping

between ESCO and a national skills classification.

2. The methodology of the pilot

The execution of the pilot consisted of following steps which encompassed the setup of

the process, execution of the mapping and assessment of the results:

• Step 1: Select the National Skills Classifications

• Step 2: Select the skills

• Step 3: Create the mappings

• Step 4: Assess the results

Each of these steps is further detailed below.

2.1. Step 1: Select the National Skills Classifications

On 31 November 2017 the Commission services invited the Member States to express

their interest in participating in a skills mapping pilot. Austria, Romania and Sweden

expressed interest. The Commission services accepted the candidatures of Austria and

Sweden and invited Romania to participate as an observer due to lack of a national skills

classification. Belgium, also, expressed interest to participate as an observer.

2.2. Step 2: Select the skills

The skills pilot tested the mapping process on a subset of skills. The sample selected to

cover sectors which equally represent blue- as well as white-collar workers. To this end,

the European Commission opted for the following ISCO unit groups:

• 514 Hairdressers, beauticians and related workers and

• 2432 Public relations professionals

Since occupations in ESCO are tagged with an ISCO code, the European Commission

collected those ESCO skills which are listed in the occupational profiles under the

aforementioned ISCO unit groups. The same filtering of skills was performed for the

classifications of Austria and Sweden.

2.2.1. Correspondence to transversal skills


July 2018 7

Job vacancies and CVs capture also information concerning transversal skills. In ESCO

these skills are not part of the occupational profiles (mentioned above for filtering).

These skills were therefore included through a dedicated filtering.

Austria and Sweden also indicated their transversal skills to be included in the testing

sample.

2.3. Step 3: Create the mappings

Within the course of the creation of the mappings, the MS and the Commission services

followed the process described below:

Before the creation of the mappings

1. The Commission services transformed the selected data into the required data

format: Simple Knowledge Organisation System (SKOS). This is the de facto

standard to create correspondences between classifications.

2. The Commission services import the data into a support tool for thesaurus

alignment.

During the creation of the mappings

3. The tool provided suggestions for mapping concepts.

4. PES experts reworked mappings manually, i.e. approve the mapping

relationships suggested by the tool or reject them and select new ones.

5. The Commission services consolidated identified discrepancies stemming from

mapping the same pair of concepts from two different perspectives (mapping and

reverse mapping) for each MS.

6. The PES decided on the resolution of the indicated discrepancies.

The Commission services together with the PES carried out a mapping and a reverse

mapping (bi-directional mapping).

The mapping refers to the creation of correspondences from a subset of skills

(stemming from the selected ISCO unit groups in Step 2: Select the skills) from

national skill classifications against the full ESCO. This was carried out by the

Commission services.

The reverse mapping refers to the creation of correspondences from a subset

of the ESCO skills (stemming from the selected ISCO unit groups in Step 2:

Select the skills) against the full national skill classifications. This was carried out

by the PES.

The main purpose of the bi-directional mapping is to validate the mapping and to make

it more complete, i.e. to find potential mapping relationships that have not been found

during the mapping in only single direction.

https://www.w3.org/2004/02/skos/


July 2018 8

2.4. Step 4: Assess the results

2.4.1. Assess the mappings

Upon the completion of the mappings, the Commission services assessed the mappings

created between the national skills classifications and ESCO by analysing the following:

a) Number and % of concepts of the national skill classification subset which brought

correspondences in full ESCO during the national skill classification-to-ESCO

mapping, i.e. how many concepts could the Commission services map.

Figure 1: National skill classification to ESCO mapping

b) Number and % of concepts of the ESCO subset which brought correspondences in

full national skill classification during the reverse ESCO-to-national skill classification

mapping, i.e. how many concepts could the PES map.

Figure 2: ESCO to national skill classification mapping (reverse mapping)


July 2018 9

c) Number and % of concepts that fell in the two subsets of skills, i.e. how many

mapping relationships go from subset to subset between the two mappings. In other

words, for how many identical skills pairs could the Commission services and PES

create matches.

Figure 3: Correspondences in the subsets of the two mappings

d) Number and % of those mapping relationships that are identical in the two subsets

of skills of the same classification deriving from the mapping and the reverse

mapping.

2.4.2. Assess the level of interoperability

In order to assess the level of interoperability which has been achieved between national

classification systems after mapping to ESCO, the Commission services and the PES

simulated real-life case scenarios.

Preparation for the assessment

1. The Commission services selected anonymous non-structured CVs and job

vacancies related to one of the selected sectors (from desk research);

2. The Commission services translate the selected CVs and job vacancies into the

languages of the participant MSs;

3. PES transform the non-structured CVs and job vacancies into structured

data following the fields and the identifiers that they use in their actual daily work.

This includes annotating structured CVs and job vacancies with the skill concepts in

their national skill classifications;

4. The Commission services transcode structured CVs and job vacancies into

ESCO, based on the mappings established in Step 3: Create the mappings;

5. The Commission services transcode ESCO structured CVs and job vacancies

again into the format of the national system it will be integrated into, based

on the mappings established in Step 3: Create the mappings.

The Commission services will execute an assessment using the following use cases:


July 2018 10

1st use case: exchange of CVs between PES system

Part 1: Receipt and integration of national and cross-border CVs into a national

system

Illustration of the real-life use case:

Francoise is a

hairdresser living in

France seeking

employment in France.

Martin is a hairdresser

living in the Czech

Republic seeking


The CVs of Martin and Francoise are identical, the only difference is that Francoise lives

in France and Martin in the Czech Republic. We will explore the processing of the data

in their CVs, considering they want to submit their CVs to the French system to seek


In this use case, we have:

• a perfect scenario: a French person (in this case Francoise) visits a French PES to

seek employment in France and

• a non-perfect scenario: a non-French person (in this case Martin) visits a PES in

another MS (in this case the Czech Republic) to seek employment in France.

In the perfect scenario: the French PES will transform the non-structured data of

Francoise’ s CV into structured data, according to their usual process.

In the non-perfect scenario: the French PES will receive Martin’s CV from the Czech

PES. In order to achieve the transfer, the Commission services transcode the CV from

the Czech classification to the system of the French PES using ESCO.

The aim of this exercise is to understand how much noise2/information loss the ESCO

transcoding introduces in the transfer. We analysed this by comparing the skill set of

CVs in the perfect scenario against the non-perfect scenario.

2 “Noise” represents skills which are introduced during the transformation but are not

part of the correct skills in the target system. Such skills have a negative impact on job

matching since they reduce the accuracy. The introduction of noise is inherently part of

any mapping exercise where multiple concepts can be mapped to each other in the

related classifications. For example: Concept A can be “more general” than Concept B,

Concept C and Concept D. When a document is transformed using such mapping table,

each concept it contains may be transformed to multiple concepts in the target

classification. Not all of these concepts may be relevant for that specific document. For

example (following the previous), Concept A would be transformed into Concept B,


July 2018 11

Although the skills in the CVs of Francoise and Martin were identical in their original

format, due to multiple transcodes, they may not be the same anymore.

Figure 4: Exchange of CVs between PES

In order to understand how much information is lost when carrying out multiple

transcoding, the Commission services compared:

CV CZ FR PES structured CV FR structured

Figure 5: Comparison of CVs stemming from different PES

Part 2: Matching of national and cross-border CVs to national job vacancies

The French PES are looking for job vacancies for Francoise and Martin. They have

identified four job vacancies which match their skills. For both Francoise and Martin,

the French PES will match the structured format of their CVs to job vacancies in their

system and provide the % of the matches for each job vacancy.

Concept C and Concept D; if only Concept D is relevant to the specific document,

Concepts B and C represent a “noise” for this document.


July 2018 12

In order to understand whether identical CVs from various countries yield the same

matches to job vacancies, the Commission services compared the matches from:

CV FR structured to 4 FR JVs CV CZ FR PES structured to 4 FR JVs

Figure 6: Comparison of matches between identical CVs and job vacancies

Note: The job vacancies are the same for both candidates.

2nd use case: exchange of job vacancies between PES

Part 1: Receipt and integration of national and cross-border job vacancies into

a national system

Illustration of the real-life use case:

L’Oreal is a French

company which is

looking for a hairdresser

for their headquarters in

France.

Everis is a Spanish

company which is

looking for a hairdresser

fluent in French for their

headquarters in Madrid.

The descriptions of their job vacancies are identical, the only difference is that L’Oreal

is established in France and Everis is established in Spain. We will explore the processing

of the data in their job vacancies considering they want to hire professionals from

France.

In this use case, we have:

• a perfect scenario: a French company (in this case L’Oreal) contacts a French PES

to seek an employee in France and

• a non-perfect scenario: a non-French company (in this case Everis) contacts a

PES in another MS (in this case Spain) to seek for an employee from France.

In the perfect scenario: the French PES will transform the non-structured data of the

job vacancy from L’Oreal into structured data, according to their usual process.

In the non-perfect scenario: the French PES will receive the job vacancy from the

Spanish PES. In order to achieve the transfer, the Commission services transcode the

job vacancy from the Spanish classification to the system of the French PES using ESCO.

The aim of this exercise is to understand how much noise/information loss the ESCO

transcoding introduces in the transfer between PES systems. We analysed this by


July 2018 13

comparing the skill set of job vacancies in the perfect scenario against the non-

perfect scenario. Although the skills in the job vacancies of Everis and L’Oreal were

identical in their original format, due to multiple transcodes, they may not be the same

anymore.

Figure 7: Exchange of job vacancies between PES

In order to understand how much information is lost when carrying out multiple

transcoding, the Commission services compared:

JV ES FR PES CV FR structured

Figure 8: Comparison of job vacancies stemming from different PES

Part 2: Matching of national and cross-border job vacancies to national CVs

The French PES are looking for CVs for L’Oreal and Everis. They have identified four CVs

which match their skill requirements. For both L’Oreal and Everis, the French PES will

match the structured format of the job vacancy to the CVs in their system and provide

the % of the matches for each CV.

Structured Non-structured

ES ES

ESCO

ES

Structured

FR

Non-structured

FR

ES

FR PES


July 2018 14

In order to understand whether identical job vacancies from various countries yield the

same matches to CVs, the Commission services compared the matches from:

JV FR structured to 4 FR CVs CV ES FR PES to 4 FR CVs

Figure 9: Comparison of matches between identical job vacancies and CVs

Note: The CVs are the same for both employers.

3. Results

3.1. Results: Assessment of the mappings

The conclusions and charts presented below serve to understand the mapping

characteristics as outlined above in the section Assess the mappings.

Using the method for a sample selection of skills outlined in Step 2: Select the skills we

have gathered the following numbers of skills from each classification respectively:

• ESCO: 610 skill concepts

• Austrian classification: 419 skill concepts

• Swedish classification: 74 skill concepts

Figure 10: Number of concepts in each sample

A correspondence was established from a sample of skills against the full classification

– e.g. correspondences from a subset of skills from national skill classifications against

the full ESCO – (c.f. Step 3: Create the mappings). The chart below (Figure 11: % of

610

419

74

0

200

400

600

800

# Concepts

Number of concepts in each sample

ESCO Sample AUT Sample SWE Sample


July 2018 15

skills mapped in the sample) captures a percentage of the skills mapped in each sample

i.e. what percentage of skills in the sample have at least one mapping relation.

The mappings of the Austrian classification have a high percentage of concepts (over

90%) which indicates that mapping this classification to ESCO would be quite complete.

On the other hand, the mappings involving the Swedish classification have a lower

portion (approximately 70%) of mapping relations due to differences in granularity and

lack of descriptions of concepts.

Mapping from either direction has similar coverage (deviation of only 4% for each

classification), meaning that the perception of “mappable” concepts – concepts which

could be mapped – was aligned between the Commission services and the Member

states.

Figure 11: % of skills mapped in the sample

Please, note that to shorten the legend in the indicated charts we are using the following

abbreviation for each mapping effort performed:

• E(s) > AUT = Mapping from a sample of ESCO to the full Austrian classification o Mapper: Austrian representatives

• E(s) > SWE = Mapping from a sample of ESCO to the full Swedish classification o Mapper: Swedish representatives

• SWE(s) > E = Mapping from a sample of Swedish classification to full ESCO o Mapper: The Commission

• AUT(s) > E = Mapping from a sample of Austrian classification to full ESCO

o Mapper: The Commission

Each skill in the sample was mapped on average to approximately 2,2 skills in the target

classification (c.f. Figure 12: Average number of relations per concept). This average

was maintained by all mappers with the exception of Austria, who mapped each skill

92% 96%

68% 72%

0%

20%

40%

60%

80%

100%

120%

% of sample mapped

% of skills mapped in the sample

E(s)>AUT AUT(s)>E E(s)>SWE SWE(s)>E


July 2018 16

specifically to a single skill in the target classification, preferring (instead of mapping to

multiple skills) to map to a single broader skill.

Figure 12: Average number of relations per concept

1

2.32.1

2.4

0

0.5

1

1.5

2

2.5

3

Average mapping relations per concept

Average number of relations per concept



July 2018 17

The percentages distribution of the different relation types in each mapping are captured

below in Figure 13: % distribution of the mapping types used.

Firstly, it is noted that the percentages of “More specific/general than” relations are

aligned across mappings since they (approximately) mirror each other in the related

mappings (meaning that the percentage of “More specific” in one direction is close to

“More general” in the other direction and vice-versa). On the other hand, there is a

discrepancy between the “Close” and “Exact” mapping relations, since the mappings by

the ESCO team that use “Close” match is significantly higher than the ones by the MS.

This was attributed to (1) miscommunication regarding meaning of the “Exact” / ”Close”

match and (2) unclarity in the user interface of the software platform, where the MS at

first did not notice that they were able to change the mapping type.

Figure 13: % distribution of the mapping types used

39%

13%

69%

5%

9%

38%

15%

7%

48%

16%

13%

12%

4%

32%

2%

77%

0%

20%

40%

60%

80%

100%

120%


% distribution of the mapping types used

Exact More specific than ESCO More general than ESCO Close


July 2018 18

As shown in Figure 14: Portion of skills inside a sample of target classif, the mapping

samples from each classification are only approximately 25% overlapping (an average

among all mappings). For mappings involving the Swedish classification, the

percentages are inconsistent (55% versus 5%) due to the size of the sample. The low

percentage of the sample overlap indicates that the transformation of JVs/CVs

supported by the mapping tables will be less accurate (e.g. higher chance of introduction

of a noise and omission of mapping relations) since only ¼ of the mapping relations

were reviewed by both parties (the Commission and the MSs).

Figure 14: Portion of skills inside a sample of target classification

28%24%

55%

5%

0%

10%

20%

30%

40%

50%

60%

% of skills inside sample of the target classification

Portion of skills inside sample of target classif.



July 2018 19

A low number of common relations between the two mappings was identified (c.f. Figure

15: Number of same/conflicting mappings). This may be caused by the low overlap of

samples mentioned in the previous paragraph (only an average of 25% as indicated in

Figure 14).

The higher number of conflicting mapping relations (compared to same mapping

relations) could be attributed to the misalignment of perception of the Exact match

relation mentioned in Figure 13. The amount of conflicting mapping relations is also

higher for mappings involving the Swedish classification, which may have been caused

by the lack of descriptive properties (descriptions, alternative labels, scope notes…) in

this classification, which made it difficult to establish consistent mapping relations.

Figure 15: Number of same/conflicting mappings

27.00

9.00

39.00

44.00

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

AUT SWE

Number of same/conflicting mappings

# same mapping relations # conflicting mapping relations


July 2018 20

3.2. Results: Assessment of the level of interoperability

The conclusions and charts presented below serve to understand the level of

interoperability provided by the mapping tables. This illustrates to what degree can the

mapping tables support an interoperability of labour market documents (the process of

testing the level of interoperability is described in further detail in the section Assess

the level of interoperability).

In order to assess the level of interoperability, we used 17 documents which each of the

MSs annotated with skills according to the processes of their public employment service.

The documents were distributed as follows:

• 4 CVs of beautician professions

• 4 Job vacancies of beautician professions

• 5 CVs of Public relations professions

• 4 Job vacancies of Public relations professions

Figure 16: Skills identified for each document) presents the number of skills identified

in each of the used documents by Austria and Sweden. Several of the documents

(highlighted in grey) lack skills assignments by one of the MSs, which means that they

cannot be used for analysis in the scope of this chapter.

Figure 16: Skills identified for each document

3.2.1. Use case: Part 1

0

5

10

15

20

25

Skills identified for each document

AUT identified SWE identified


July 2018 21

The aim of this exercise is to understand how much noise/information loss the ESCO

transcoding introduces in the transfer between PES systems. The use cases and the

process used for the testing are described in detail in the section Assess the level of

interoperability.

When transforming the CV and Job vacancy documents using the mapping tables we

identified that on average 2.9 and 1.4 skills are correctly transferred/matched (c.f.

Figure 17: Average number of correct matches).

Figure 17: Average number of correct matches

2.9

1.4

0

0.5

1

1.5

2

2.5

3

3.5

Average matched

Average number of correct matches

SWE > AUT AUT > SWE


July 2018 22

To evaluate further whether the average of 2.9/1.4 skills (c.f. Figure 17) can be

considered as a “successful” matching rate we performed a manual evaluation for

comparison. In this evaluation we inspected each of the original Swedish and Austrian

sets of skills in the related documents and compared them together directly, using a

human evaluation instead of the automated process (i.e. transforming the CV and Job

vacancy documents using the mapping tables).

The resulting set of skills for each document coming from the automated process was

compared against the results of the human evaluation. Cases where the skill sets fully

matched the document were marked as having a 100% “successful” matching rate.

Lower matchings were assigned lower matching rates (down to 0%).

When comparing all of the documents we can conclude that the automated process

provides an average 60/77% of matching rate (c.f. Figure 18: Average % of the success

rate of the matches).

This means that on average the automated transfer gathered 60/77% of the skills

identified using the manual evaluation.

Figure 18: Average % of the success rate of the matches

77.50%

60.83%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Average matching rate

Average % of success rate of the matches

SWE > AUT AUT > SWE


July 2018 23

The automated transformation using mapping tables introduces a high level of “noise”.

On the other hand, it was noted by the MS participants that “noise” maybe be useful for

another purpose, such as career guidance (since it may provide an individual with a

wider range of opportunities based on their skill-set). The degree of “noise” identified in

our testing is 81/92% (c.f. Figure 19: Average % of noise introduced), meaning that

only one/two out of every ten skills represented a correctly transferred skill. This has to

be taken into consideration when using the data for job matching purposes.

Figure 19: Average % of noise introduced

When evaluating the charts above (Figure 17, Figure 18, Figure 19) it may be noticed

that the transformations where the Austrian classification is in the position of the target

classification always give better results. This is most likely due to Austria identifying a

higher number of skills in each document (as can be seen in Figure 16), giving them a

higher chance to match correctly.

3.2.2. Use case: Part 2

In the second part of the use case analysis (following the methodology outlined in the

section Assess the level of interoperability) we evaluated the ranking of each of the

following:

• CV against all Job vacancies

• Job vacancy against all CVs

In order to understand whether identical JVs/CVs from various countries yield the same

matches to CVs/JVs, we compared the matches from:

JV FR structured to 4 FR CVs CV ES FR PES to 4 FR CVs

Figure 20: Comparison of matches between identical job vacancies and CVs

81.29%91.95%

30.00%

50.00%

70.00%

90.00%

110.00%

Average noise ratio

Average % of noise introduced

SWE > AUT AUT > SWE


July 2018 24

The exercise aimed to define whether the order of ranks is identical among the original

ranking and the ranking after automated transformation using the mapping tables. A

ranking represents a position of a specific CV/JV when comparing to a skill set required

by a JV/CV. For example, when searching for the best matching candidate for a JV in a

set of four CVs (CVs A-D), we can state that:

• CV A has rank #1 because it contains 15 of skills required by the JV;

• CV B has rank #4 because it contains 2 of skills required by the JV;

• CV C has rank #2 because it contains 14 of skills required by the JV;

• CV D has rank #3 because it contains 7 of skills required by the JV.

Figure 21: Average accuracy of ranking depicts the results of this exercise. The two

methods of evaluation were:

• Exact: in order to be considered correct in this approach, the rank had to exactly

match the rank in the original data;

• +/-1: here the transfer is considered correct even if the rank is one point lower or

higher compared to the original data (e.g. if a target rank is #3, JVs/CVs ranked #2

and #4 are considered correct).

Taking this into consideration, if we aim for exact reproduction (i.e. the rank being

identical) of the original ranking the transformed data achieve only 27/35% accuracy.

On the other hand, with the “fuzzy” approach (+/-1 rank), the results reach 73/65%

accuracy.

It’s also required to note that the sample usable for this analysis was rather small,

containing only 10 usable documents (as indicated above in Figure 16). This means that

the results indicated here may not be representative of the real-world performance of

such ranking algorithm.

Figure 21: Average accuracy of ranking

73.08%

26.92%

65.38%

34.62%

0.00%

20.00%

40.00%

60.00%

80.00%

+/-1 Exact

Average accuracy of ranking

AUT vs SWE>AUT SWE vs AUT>SWE


July 2018 25

3.3. Lessons learnt

Taking into considerations the results analysed above (c.f. 3.1 and 3.2) and the

discussions with representatives of Austria and Sweden we draw the following lessons.

The lessons learnt refer to the mapping process and the mapping environment, as well

as an estimate of the workload for a full-scale mapping between ESCO and a national

skills classification.

3.3.1. Lessons learnt about requirements for tools and services

With regard to the key functional requirements for an online mapping platform, the

discussions with the participants highlighted that the experts need:

• The software platform has to provide good performant and speedy results to users’

actions. The performance of the current platform was noted to be insufficient.

• The search functionality of the platform has to be improved. The MS noted that they

have to be able to search with only partial strings of words (e.g. use the search

“manag” to find skills mentioning the word “managerial”) because of the complexity

of their languages.

• The different mapping relations should be explained in detail in the descriptions of

the mapping manual, including examples. The scope of the exact and the close

match, i.e. where the exact match starts and where the close relations ends

(boundaries), should be clearly indicated.

Other features of the mapping platform – such as browsing the target classification or

suggestions – were not evaluated by the mappers due to the issues with performance.

3.3.2. Lessons learnt about the process

Regarding the mapping process and the organisation of the work it can be noted that:

• The accuracy of mapping increases with the number of description properties of skills

in a classification (descriptions, alternative labels, scope notes…) (Figure 15).

• It has to be further clarified in the mapping manual how to deal with:

o mapping to a higher level of a classification

o how many concepts should be mapped (for mappings in the scope of the

EURES exchange).

• The participants concluded that the mapping of skills may be perceived as arbitrary

and subjective. It is affected by:

o sectoral and societal differences among the Member States;

o different understanding of skill concepts;

o differences in applying context to skill concepts.

• The ESCO classification and the mapping relations need to reflect the current

situation as well as new developments in the labour market, e.g. new occupations,

new skills, etc. Nevertheless, they should not be updated too frequently since in this

case the maintenance of the mapping tables would be too demanding.

• Guidance is needed on how to deal with skill hierarchies in the mapping process. It

has been noted that it is beneficial to have a skills hierarchy in ESCO in order to

validate whether each sector is fully covered. What should still be addressed is which

level(s) (of ESCO and the National Skill Classifications) the Member States will use

for mapping.

• The results of the mapping exercise indicate that although the transformation of

documents using the mapping tables introduces a high level of “noise”, the usability

of the documents is retained.


July 2018 26

• Mapping tables may be used for different purposes. The ones created during the

pilot served as a simulation of transfer of the documents to EURES. The objective is

not to create mapping tables but to develop a solution that enhances interoperability

in the labour market.

• Establishing multiple mapping relations (on top of what is recommended in the

current version of the mapping manual) will create further “noise”. However,

different stakeholder groups could benefit from different mapping relations, e.g.

unemployed may benefit from close matches rather than exact matches; SMEs may

benefit from broader relations which would subsequently lead to a higher number of

relations. The mapping methodology should be aligned with the function that the

resulting mapping table will serve.

3.3.3. Lessons learnt about resources requirements

The following effort was noted by the participating Member States for mapping a sample

of 610 ESCO skills:

• Austria: 50 Hours

• Sweden: 80 Hours

This effort does not include quality assurance, only an initial creation of the mapping

relations. Extrapolating the time investment to full mapping (13 485 ESCO skills) would

result in an indicative effort of 170 Days of work. It is worth noting that technical

improvements on the mapping platform could lead to a more streamlined mapping

process and to lowering the effort required.

Ideally, the Member States would establish a team of people to create mapping relations

with mapper and reviewer roles. The bidirectional mapping methodology used in this

pilot may be helpful, but the individual mappings have to be quality assured.

Outsourcing the mapping activities may be feasible depending on the national setup.

3.3.4. Lessons learnt about the achieved level of interoperability

Regarding the level of interoperability which can be achieved using the transformation

based on the established mapping tables (c.f. Results: Assessment of the mappings and

Results: Assessment of the level of interoperability), we can note the following:

• When evaluating the results of the automated transformation of CV/job vacancy info

using the mapping tables, the process provides 60/77% of accuracy.

• The automated transformation using mapping tables introduces a high level of

“noise”. On the other hand, “noise” maybe be useful for another purpose, such as

career guidance. The degree of “noise” identified in our testing is 81/92%, meaning

that only one/two out of every ten skills represented a correctly transferred skill.

This has to be taken into consideration when using the data for job matching

purposes.

• The “ranking” of JVs/CVs after their transformation using the mapping tables was:

o if we aim for exact reproduction (i.e. the rank being identical) of the original

ranking, the transformed data achieve only 27/35% accuracy

o with the “fuzzy” approach (+/-1 rank), the results reach 73/65% accuracy.

final report of the esco skills mapping pilot...european skills, competences, qualifications and...

Documents