overview of the mapping pilot process - ec.europa.eu fileobjectives of the skills mapping pilot...

Overview of the mapping pilot process

Final workshop: skills mapping pilot3 May 2018

Objectives of the skills mapping pilot

Process and resource requirements for the creation of the correspondence tables of skills.

Tools and/or services to support the mapping process.

Level of interoperability between ESCO and national skill classifications.

Level of interoperability among the national skill classifications during cross-border matching.

Understand:

Steps Actors

1.Identify the national skill

classifications

Commission services

2. Select the skills subset Commission services

3. Create the mappings Commission services + PES

4. Assess the results Commission services + PES

Overall pilot approach

Step 1: Identify the national skills classifications

Participants Observers

Step 2: Select the skills subset

514 Hairdressers, beauticians and related workers: 285

2432 Public relations professionals: 231

Transversal: 107

Step 3: Create the mappingsPreparatory phase

ISCO unit groups

Occupations Skills

EC transforms the data into

SKOS

EC imports the data into

the tool

PES send data to the

EC

Step 3: Create the mappingsDuring the creation of the mappings

Tool makes suggestions for

mapping concepts

EC and PES create mappings

EC consolidates discrepancies

PES decide on the resolution

Step 4: Assess the results

Assess the level of interoperability which has been achieved between national classification systems after mapping to ESCO

Why?

How? EC and PES will simulate real case scenarios

Step 4: Assess the results

1st use case: exchange of CVs between PES

2nd use case: exchange of job vacancies between PES

Part 1: Receipt and integration of national and cross-border CVs and job vacancies into a national system

Part 2: Matching of national and cross-border CVs to national job vacancies and vice versa

Resources for this pilot

PES Commission services

Human resources to carry out step 3 and 4

Human resources to carry out step 3 and 4

Participation in 2-4 workshops either online or on site

Mapping management platform

Guidelines technical assistance

Organization of workshops

Reimbursement of travel and accommodation expenses of the participants

Documentation and dissemination of results

Summary report

Share with the PES and

Publish on the ESCO portal

Timeline

See concept paper of the skills mapping pilot for the following terms:Mapping: sub-section “During the creation of the mappings”, p.5-6 Annotation: step 3 of the sub-section “Preparation for the assessment”, p.7 Address discrepancies: steps 5-6 of the sub-section “During the creation of the mappings”, p.5Analysis of the results: cases a, b, c, d of the section “Create the mappings”, p.6-7 Compilation of final report: section “Documentation and dissemination of results”, p.12

26/02 Week 1 Week 4 Week 5 Week 8 Week 9Week 2 16/03 06/04 Week 7 04/05 Week 11

Mapping Annotation JV/CV Analysis of the results

Address discrepancies

Compilation of final report

28/0220/03

03/05

Week 12 25/05

Questions?

Creation of mapping relations: results


STEP 3

Note

• Abbreviation:

• E(s) > AUT = Mapping from sample of ESCO to full Austrian classification

Mapper: AUT

• SWE(s) > E = Mapping from sample of Swedish classification to full ESCO

Mapper: EC

610

419

74

0

100

200

300

400

500

600

700

# Concepts

Number of concepts in each sample

ESCO Sample AUT Sample SWE Sample

92%96%

68%72%

0%

20%

40%

60%

80%

100%

120%

% of sample mapped

% of skills mapped in the sample

E(s)>AUT AUT(s)>E E(s)>SWE SWE(s)>E

% of skills mapped in the sample

• SWE efforts have smaller portion of mapping relations due to differences in granularity and lack of descriptions

• Mapping from either direction has similar coverage

1

2,32,1

2,4

0

0,5

1

1,5

2

2,5

3

Average mapping relations per concept

Average number of relations per concept


Average number of relations per concept

• AUT colleagues assigned only a single mapping relation to every concept

• Averages of mapping relations are aligned for the remaining mapping efforts

39%

13%

69%

5%

9%

38%

15%

7%

48%

16%

13%

12%

4%

32%

2%

77%

0%

20%

40%

60%

80%

100%

120%


% distribution of the mapping types used

Exact More specific than ESCO More general than ESCO Close

% distribution of the mapping types used

• Broader/Narrower relations are aligned across mappings (mirroring each other)

• Mappings by ESCO team use close match much more frequently compared to MS. This may be due to miscommunication regarding exact match

Percentage of skills in this intersection

Percentage of skills in this intersection

Portion of skills inside sample of target classification

28%24%

55%

5%

0%

10%

20%

30%

40%

50%

60%

% of skills inside sample of the target classification

Portion of skills inside sample of target classif.


Portion of skills inside sample of target classification

• The samples are not very aligned (AUT)

• Only 1/4 of skills overlap between them

• Transfer of information will be difficult

• SWE percentages are inconsistent due to size of the sample

27,00

9,00

39,00

44,00

0,00

10,00

20,00

30,00

40,00

50,00

60,00

70,00

AUT SWE

Number of same/conflicting mappings

# same mapping relations # conflicting mapping relations

Number of same/conflicting mappings

• Number of relations between common skills is low

• Possibly due to the misalignment of the samples mentioned above

• Higher number of conflicts could be attributed to misalignment of perception of the exact match relation

• Mapping with SWE is less aligned (more conflicts)

• possibly due to lack of descriptions

Questions?

10:45 Coffee break

Annotation process: analysis of the use cases


STEP 4

0

5

10

15

20

25

Skills identified for each document

AUT identified SWE identified

Skills identified for each document

• Part of documents cannot be used for analysis

• Since they are not tagged by one of the MSs

• These are never the same (i.e. there is no document which would lack tagged by both MS)

• AUT on average identified more skills in each document

STEP 4USE CASE: PART 1

Example - CV_PR consultant05

Erstellung von Marketingunterlagen

Creation of marketing documents

Event-Management, Promotionorganisation

Event management, promotion organization

Krisenkommunikation crisis communications

Marktstrategie market strategy

Mediaplanung Media planning

MS Excel Anwendungskenntnisse

MS Excel application knowledge

MS PowerPoing Anwendungskenntnisse

MS PowerPoingapplication knowledge

… …

SMS Excel MS Excel

MS Power Point MS Power Point

Pressmeddelanden Press Releases

Krishantering Crisis management

Marknadsföring/PR Marketing / PR

MS Word MS Word

Korrekturläsning PROOFREADING

S

21 Skills

7 Skills


Starting point: SWE | Skills in sample

MS Excel MS Excel





MS Word MS Word


S

7 Skills



MS Excel MS Excel





MS Word MS Word


S

7 Skills

MS Excel MS Excel





MS Word MS Word


S

5 Skills (2 skills lost)



MS Excel MS Excel





MS Word MS Word


S

7 Skills

MS Excel MS Excel




MS Word MS Word

S



Starting point: SWE | Skills to ESCO

5 Skills

MS Excel MS Excel




MS Word MS Word

S


Starting point: SWE | Skills to ESCO

MS Excel MS Excel




MS Word MS Word

S

5 Skills

use spreadsheets software

public relations

draft press releases

plan marketing strategy

develop public relations strategies

use word processing software

…

S



Starting point: SWE | Skills to AUT


public relations





…

S

24 Skills


Starting point: SWE | Skills to AUT


public relations





…

S

24 Skills


Pressemitteilung

Grundkenntnisse Datenpflege

TextverarbeitungsprogrammeAnwendungskenntnisse

Online-PR

Marktstrategie

…

S

26 Skills (1 skill lost)


Starting point: SWE | Compare to original AUT


Pressemitteilung



Online-PR

Marktstrategie

…

S

26 Skills




Pressemitteilung



Online-PR

Marktstrategie

…

S

26 Skills










… …

S

21 Skills


Side step: Manual comparison











… …

SMS Excel MS Excel





MS Word MS Word


S

21 Skills 7 Skills




• 6 skills can be identified

• Common according to manual comparison

• Even in optimal case the final result cannot transfer much more than 6 skill

• Only these skills are common in the starting tagged documents




Pressemitteilung



Online-PR

Marktstrategie

…

S

26 Skills










… …

S

21 Skills


Results

• Skills matched: 6

• Manually identified matches: 6

• Matching rate: 100%

• Automatically matched skills equal manually identified number

• Skill “noise”: 20 (26 – 6 matches)

• Skill “noise” rate: 77%

2,9

1,4

0

0,5

1

1,5

2

2,5

3

3,5

Average matched

Average number of correct matches

SWE > AUT AUT > SWE

77,50%

60,83%

0,00%

10,00%

20,00%

30,00%

40,00%

50,00%

60,00%

70,00%

80,00%

90,00%

Average matching rate

Average % of success rate of the matches

SWE > AUT AUT > SWE

81,29%

91,95%

30,00%

40,00%

50,00%

60,00%

70,00%

80,00%

90,00%

100,00%

Average noise ratio

Average % of noise introduced

SWE > AUT AUT > SWE

Conclusions

• Transformation provides 60/75% of matching rate between automated transformation and manual comparison

• Fairly accurate for automated process

• SWE > AUT was more accurate on all aspects compared to AUT > SWE

• Likely due to higher amount of tagged skills in the documents by AUT

• Transformation introduces high level of noise, this has to be taken into consideration when using the data for job matching

STEP 4USE CASE: PART 2

S

JV_PR consultant01

S

CV_PR consultant05

S

JV_PR consultant02

S

JV_PR consultant04

Rank 3

S

JV_PR consultant01

S

CV_PR consultant05

S

JV_PR consultant02

S

JV_PR consultant04

Rank 4*

S

S

S

*Rank 4 indicates that thereare no skill common betweenCV and JV

Analysis

• Accuracy

• Exact: 0• No documents have the same rank between the two cases

• +/- 1: 2• JV_PR consultant01 is Rank 2 in direct scenario and Rank 1 in transformed

• JV_PR consultant02 is Rank 3 in direct scenario and Rank 4 in transformed

• Both of these are off by 1 rank, therefore can be partially considered

73,08%

26,92%

65,38%

34,62%

0,00%

10,00%

20,00%

30,00%

40,00%

50,00%

60,00%

70,00%

80,00%

+/-1 Exact

Average accuracy of ranking

AUT vs SWE>AUT SWE vs AUT>SWE

Conclusions

• Sample usable for ranking was rather small

• Results may not be representative

• If we aim for exact reproduction of the source ranking the transformed data achieve only 25/35% accuracy

• With “fuzzy” approach (+/-1 rank) the results achieve 73/65% accuracy

• Ranking using ESCO mapping provides value, but should not be considered 100% accurate

Questions?

Mapping process within the framework of the

EURES Regulation


65

EURES

network and

portal

Cooperation for free

movement of workers

Human network of

1000 advisers

Information, recruitment, placement services

Art. 19 EURES Regulation

2016/589/EU

Better matching people to jobs

Increased IT interoperability

ESCO

Mapping framework

• Implementing acts - summer 2018• Mapping technical support: implementation manual, training

modules and webinars, helpdesk, the mapping platform -summer 2018

• " Support Package" in ESCO website• Mapping can start anytime, advisable for September

Complete framework that allows Member States to start mapping

Questions?

overview of the mapping pilot process - ec.europa.eu fileobjectives of the skills mapping pilot...

Documents