International Survey of Adult Skills (ISAS)

Policy Results

14th October 2013

Victor Dukelow

Principal Economist

Analytical Services

Department for Employment & Learning

International Survey of Adult SkillsNorthern Ireland

October 2013

2012 International Survey of Adult Skills• The 2012 International Survey of Adult Skills was carried out on behalf of the

Northern Ireland Governments by the National Foundation for Educational Research (NFER) in partnership with TNS-BMRB, NatCen Social Research (NatCen) and Northern Ireland Statistics and Research Agency (NISRA).

• It is an OECD survey of adults aged between 16 and 65 in 25 countries (incl. England and NI).

• In Northern Ireland, data was collected between August 11 and April 12 and 3,761 adults participated in the survey. The response rate was 65 per cent.

• The interviews and assessments are carried out in adults’ homes.

• It measures numeracy, literacy and problem solving in technology-rich environments.

• It aims to be able to make comparisons over time, and has some linking questions with previous international surveys.

• It has a particular focus on measuring skills in real-world situations.

How does it work?

• Like most population level surveys it uses a multi-stage sampling design.

• It includes a background questionnaire and a cognitive assessment and both are conducted in the respondent’s home at least in part by Computer-Assisted Personal Interviewing (CAPI).

• Computer based assessments of the three skill areas.

• A paper based assessment of literacy and numeracy is available for those who are unable to carry out the assessment on computer.

How do the assessments work?

• Computer based tests are adaptive in that respondents are presented with items appropriate to their skill level based on responses to previous groups of items.

• All respondents see the most basic questions and are presented with more difficult questions based on their response to these.

• This allows a broad ability range to be effectively assessed.

Skills assessed

• Literacy ‘the ability to understand, evaluate, use and engage with written texts to participate in society, to achieve goals and to develop one’s knowledge and potential.’

• Numeracy ‘the ability to access, use, interpret, and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life.’

• Problem solving in a technology-rich environment ‘the ability to use digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks’

LevelsOECD’s International Survey of Adult Skills Literacy and Numeracy Levels

Rough equivalent National Qualification Framework (NQF) Levels (Literacy)

Rough equivalent National Qualification Framework (NQF) Levels (Numeracy)

Below Level 1

Entry Level 1Entry Level 1

Entry Level 2

Level 1 Entry Level 3 Entry Level 2

Level 2 Level 1 Entry Level 3

Level 3

Level 2 and above

Level 1

Level 4

Level 2 and aboveLevel 5

Findings

Achievement scores

Domain Northern Ireland average

OECD average

Literacy 269* 273

Numeracy 259* 269

Problem solving in technology-rich environments

275* 283

Table 2.1 Average scores for Northern Ireland in each domainSource: PIAAC (2012)* Significant at 5 per cent level compared with OECD averageScores in each of the domains are on a scale from 0 to 500.

Literacy

• Northern Ireland performed similarly to neighbouring English-speaking countries of England and the Republic of Ireland.

• Literacy levels in Northern Ireland have increased significantly since last measured in 1996, but are still below the OECD average.

• Although below the OECD average for literacy skills, Northern Ireland still compares favourably to France, Italy and Spain in this area.

• Northern Ireland had a similar spread of attainment compared with the OECD average, indicating a similar range of skills compared with other countries. However, there was a higher proportion of adults at Level 2 or below compared with the OECD average.

International average literacy scores for IALS and the International Survey of Adult Skills

Country IALS (1996) ISAS (2012)

Average Average

Average of participating countries ** 275 (0.4) 273 (0.2)

Australia* 272 (1.0) 280 (0.9)

Belgium 277 (3.5) 275 (0.8)

Canada** 279 (2.7) 273 (0.6)

Czech Republic** 277 (1.0) 274 (1.0)

Denmark** 289 (0.8) 271 (0.6)

Finland 287 (0.9) 288 (0.7)

Germany** 282 (1.0) 270 (0.9)

Republic of Ireland 264 (3.2) 267 (0.9)

Italy* 243 (1.9) 250 (1.1)

Netherlands 286 (0.9) 284 (0.7)

Norway** 294 (1.1) 278 (0.6)

Poland* 232 (1.1) 267 (0.6)

Sweden** 306 (1.0) 279 (0.7)

United States** 273 (1.4) 270 (1.0)

England (UK)* 267 (2.0) 273 (1.1)

Northern Ireland (UK)* 264 (1.2) 269 (1.9)

Poland

*Ita

ly*

Irelan

d

North

ern

Irelan

d*

Englan

d*

Austra

lia*

United

Sta

tes*

OECD Ave

rage

*

Belgium

Czech

Rep

ublic

*

Canad

a*

Germ

any*

Nethe

rland

s

Finlan

d

Denm

ark*

Norway

*

Sweden

*230

240

250

260

270

280

290

300

310

Figure 1 - Average Literacy Scores by Country and Survey - IALS (1996) vs ISAS (2012)

IALS ISASCountry

Ave

rag

e S

core

* Statistically Significant at 5% levelA non-zero axis is being used for the proficiency score.

Literacy scores in participating countries

Country Average score

Countries significantly above Northern Ireland in literacy

Japan 296 (0.7)Finland 288 (0.7)

The Netherlands 284 (0.7)Australia 280 (0.9)Sweden 279 (0.7)Norway 278 (0.6)Estonia 276 (0.7)

Flanders (Belgium) 275 (0.8)Czech Republic 274 (1.0)Slovak Republic 274 (0.6)

Canada 273 (0.6)OECD Average 273 (0.2)

Countries not significantly different from Northern Ireland in literacy

England 273 (1.1)Korea 273 (0.6)

Denmark 271 (0.6)United States 270 (0.9)

Germany 270 (1.0)Austria 269 (0.7)Cyprus 269 (0.8)

Northern Ireland 269 (1.9)Poland 267 (0.6)

Republic of Ireland 267 (0.9)

Countries significantly below Northern Ireland in literacy

France 262 (0.6)Spain 252 (0.7)Italy 250 (1.1)

Source: PIAAC (2012)Differences are significant at 5 per cent level.() Standard errors appear in parentheses.

Numeracy

• Northern Ireland performed similarly to neighbouring English-speaking countries of England and the Republic of Ireland.

• Although below the OECD average for numeracy skills, Northern Ireland still compares favourably to France, Italy, Spain and the US in this area.

• Northern Ireland had a similar spread of attainment compared with the OECD average, indicating a similar range of skills compared with other countries. However, there was a higher proportion of adults at Level 2 or below compared with the OECD average.

Numeracy scores in participating countries

Country Average score

Countries significantly above Northern Ireland in numeracy

Japan 288 (0.7)Finland 282 (0.7)

Flanders (Belgium) 280 (0.8)The Netherlands 280 (0.7)

Sweden 279 (0.8)Norway 278 (0.8)

Denmark 278 (0.7)Slovak Republic 276 (0.8)Czech Republic 276 (0.9)

Austria 275 (0.9)Estonia 273 (0.5)

Germany 272 (1.0)OECD Average 269 (0.2)

Australia 268 (0.9)Canada 265 (0.7)Cyprus 265 (0.8)

Countries not significantly different from Northern Ireland in numeracy

Korea 263 (0.7)England 262 (1.1)Poland 260 (0.8)

Northern Ireland 259 (1.8)Republic of Ireland 256 (1.0)

Countries significantly below Northern Ireland in numeracy

France 254 (0.6)United States 253 (1.2)

Italy 247 (1.1)Spain 246 (0.6)

Source: PIAAC (2012)Differences are significant at 5 per cent level.() Standard errors appear in parentheses.

Problem solving in technology-rich environments

• Northern Ireland’s average score for problem solving in technology-rich environments was significantly lower than the OECD average and there were no countries that Northern Ireland significantly outperformed. The Republic of Ireland performed similarly, but England performed significantly better.

• A higher proportion of the population in Northern Ireland compared to the OECD average had sufficient experience of computers and basic computer skills to complete the problem solving assessment.

Problem solving scores in participating countriesCountry Average

score

Countries significantly above Northern Ireland in problem

solving

Japan 294 (1.2)

Finland 289 (0.8)

Australia 289 (0.9)

Sweden 288 (0.6)

Norway 286 (0.6)

The Netherlands 286 (0.8)

Austria 284 (0.7)

OECD Average 283 (0.2)

Denmark 283 (0.7)

Czech Republic 283 (1.1)

Korea 283 (0.8)

Germany 283 (1.0)

Canada 282 (0.7)

Slovak Republic 281 (0.8)

Flanders (Belgium) 281 (0.8)

England 281 (1.0)

Countries not significantly different from in problem solving

Estonia 278 (1.0)

United States 277 (1.1)

Republic of Ireland 277 (1.0)

Northern Ireland 275 (2.0)

Poland 275 (1.3)

Countries significantly below in problem solving 

Adult skills compared with International Adult Literacy Survey (IALS) in 1996

• There have been large changes in the makeup of the population between these two surveys and these changes will impact on literacy scores since 1996. The most profound demographic change has been the large increase in those people who have obtained some secondary school qualification.

• The Northern Ireland average literacy score has increased significantly between IALS and the International Survey of Adult Skills. The proportion of adults at the two lowest levels of proficiency (Level 1 and below) has decreased significantly.

• However, the gender gap has widened and is now statistically significant in favour of men.

Spread of proficiency – Percentage of adults at each level

Domain Below Level 1 (below 176)

At Level 1

(176 to below 226)

At Level 2

(226 to below 276)

At Level 3

(276 to below 326)

At Level 4

(326 to below 376)

At Level 5

(at or above 376)

Missing

Literacy

Northern Ireland 2.5 (0.5) 14.9* (0.9) 36.2* (1.5) 34.3* (1.6) 9.4* (0.6) 0.5 (0.2) 2.2 (0.3)

cumulative % 2.5 17.4 53.6 88.0 97.3 97.8 100

OECD Average 3.3 (0.1) 12.2 (0.1) 33.3 (0.2) 38.2 (0.2) 11.1 (0.1) 0.7 (0.0) 1.2 (0.0)

cumulative % 3.3 15.5 48.8 87.0 98.1 98.8 100

Numeracy

Northern Ireland 5.6 (0.8) 18.7* (1.2) 35.9* (1.1) 29.0* (1.1) 7.8* (0.7) 0.7 (0.2) 2.2 (0.3)

cumulative % 5.6 24.4 60.3 89.3 97.1 97.8 100

OECD Average 5.0 (0.1) 14.0 (0.1) 33.0 (0.2) 34.4 (0.2) 11.4 (0.1) 1.1 (0.0) 1.2 (0.0)

cumulative % 5.0 19.0 52.0 86.3 97.7 98.8 100

Spread of proficiency – Percentage of adults at each level

Domain

No computer

experience

Opted out of computer

based assessment

Failed ICT core

Below Level 1

(below 241)

At Level 1

(241 to below 291)

At Level 2

(291 to below 341)

At Level 3

(at or above 341)

Missing

Problem solving

Northern Ireland 10.0 (0.6) 2.3 (0.3) 5.8 (0.4) 16.4* (1.5) 34.5

* (1.2) 25.0* (1.2) 3.7* (0.6) 2.2 (0.3)

cumulative % 10.0 12.3 18.1 16.4 51.0 76.0 79.7 2.2

OECD Average 9.3 (0.1) 10.2 (0.1) 4.9 (0.1) 12.3 (0.1) 29.4 (0.2) 28.2 (0.2) 5.8 (0.1) 1.5 (0.0)

cumulative % 9.3 19.5 24.4 12.3 41.7 69.9 75.7 1.5

Gender differences

• Average scores for all three skills for men were statistically significantly higher than those for women, in common with the majority of other countries. The gender difference in Northern Ireland was higher than the OECD average.

Age differences

• In all three skill areas performance by age in Northern Ireland followed the same general pattern as other OECD countries, with young people in Northern Ireland (16-24) achieving a significantly higher average score than those in the oldest age group (55-65).

Links between adult skills and industry• Adults who worked in professional, scientific and technical

activities had the highest average scores in literacy, numeracy and problem solving compared with other industry sectors.

• Other sectors which had particularly high scores across the three domains include financial and insurance activities; public administration and defence; and compulsory social security.

• At the other end of the spectrum, adults working in construction had the lowest scores in literacy, numeracy and problem solving. Other industry sectors with low scores across the domains of literacy and numeracy were transportation and storage; accommodation and food services; and manufacturing.

Adult skills and salary

• There is a clear relationship between salary and proficiency in the three domains (for the top half of earners).

• However, for the bottom half of earners this relationship levels out.

• The majority of part-time workers and adults still in education are in these groups. It is also likely that these groups include younger workers, who tend to earn less in the early stages of their working lives.

Skills use in the workplace

• For all of these skills, more frequent use was associated with higher average scores in literacy, numeracy and problem solving in technology-rich environments compared with adults who use these skills infrequently, matching the OECD average pattern.

• This provides strong evidence for the importance of developing and assisting workers to extend and utilise their skills in the workplace.

Characteristics of adults with low proficiency

• The characteristics most likely to be associated with low proficiency were having lower levels of education, having parents who have lower levels of education, not having computer experience in everyday life, or working in certain industries and occupations.

Probability of achieving Level 1 or below in literacy, controlling for different characteristics

31%

8%

0%

8%

5%

18%

0%

5%

3%

5%

0% 5% 10% 15% 20% 25% 30% 35%

Less than secondary school (Secondary school)

Above secondary school (Secondary school)

Ethnicity Mixed (White)

Mother secondary school (Less than secondary school)

Mother above secondary school (Less than secondary school)

No computer experience in everyday life (Computer experience)

Industry – Electricity, gas, steam and air conditioning supply (Human health and social work)

Occupation – Legislators, senior of f icials and managers (Services and shop and market sales)

Occupation – Professionals (Services and shop and market sales)

Occupation – Technicians and associate professionals (Services and shop and market sales)

Predicted probability of achieving Level 1 or below

Probability of achieving Level 1 or below in numeracy, controlling for different characteristics

24%

42%

11%

30%

44%

0%

7%

12%

9%

26%

31%

0%

9%

5%

8%

10%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Female (Male)

Less than secondary school (Secondary school)

Above secondary school (Secondary school)

Employment – Not employed and looking for work (Employed or self -employed)

English additional language (First language)

Ethnicity Mixed (White)

Father above secondary school (Less than secondary school)

Mother secondary school (Less than secondary school)

Mother above secondary school (Less than secondary school)

No computer experience in everyday life (Computer experience)

Computer experience – missing response (Computer experience)

Industry – Electricity, gas, steam and air conditioning supply (Human health and social work)

Occupation – Legislators, senior of f icials and managers (Services and shop and market sales)

Occupation – Professionals (Services and shop and market sales)

Occupation – Technicians and associate professionals (Services and shop and market sales)

Occupation – Clerks (Services and shop and market sales)

Predicted probability of achieving Level 1 or below

Probability of achieving Below Level 1 in problem solving, controlling for different characteristics

31%

31%

36%

41%

14%

40%

14%

8%

44%

0%

11%

10%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Female (Male)

Aged 45–54 (35–44)

Aged 55–65 (35–44)

Less than secondary school (Secondary school)

Above secondary school (Secondary school)

Employment – Not employed and looking for work (Employed or self -employed)

Mother secondary school (Less than secondary school)

Mother above secondary school (Less than secondary school)

No computer experience in everyday life (Computer experience)

Industry – Electricity, gas, steam and air conditioning supply (Human health and social work)

Occupation – Legislators, senior of f icials and managers (Services and shop and market sales)

Occupation – Professionals (Services and shop and market sales)

Predicted probability of achieving below Level 1

QUESTIONS?

Plausible values

• Proficiency scores in literacy and numeracy were imputed only for those respondents who had completed at least five questions in the background questionnaire – their skill level was estimated based on the information they had given in the form of ‘plausible values’. Similarly, plausible values were generated for those who did not take any literacy or numeracy items following the core because of the way they were routed through the assessment. For instance, if they were routed through two problem solving components or a numeracy and problem solving component they would have taken only the core literacy items. Those who completed fewer than five of the background questions were not assigned plausible values but were included as part of the weighted population totals.

• Plausible values are a suitable method of imputing scores because we are interested in population estimates not individual scores.

Routes through the assessment