digital reading assessment in macao-china pisa 2009 reports... · - 6 - executive summary macao...

- 1 -

Digital reading assessment in Macao-China PISA 2009:

Findings and policy implications

Pou-seong Sit & Kwok-cheung Cheung

Educational Testing and Assessment Research Centre

University of Macau

Macao, People’s Republic of China

November, 2011

- 2 -

Table of Contents

Table of Contents ……………………………………………………………………………. 2

List of Tables ……………………………………………………………………………....... 4

List of Figures ……………………………………………………………………………...... 5

Executive Summary ………………………………………………………………………..... 6

1. Definition of digital reading literacy ………………………………………………......... 7

2. Sampling design …………………………………………………………………………. 7

3. Test design ……………………………………………………………………................. 8

4. Results and findings

(1) League table of digital reading performance for economies participating in PISA

2009 …………………………………………………………………………………. 10

(2) League table of distribution of digital reading proficiency levels for economies

participating in PISA 2009 …………………………………………………….......... 12

(3) Gender difference in digital reading performance for economies participating in

PISA 2009…………………………………………………………………….…....... 17

(4) Grade level difference in digital reading performance of economies participating in

PISA 2009 …………………………….……………………………………….......... 20

(5) Effect of ESCS on digital reading performance in participating economies in PISA

2009 ………………………………………………………………………………… 22

(6) Macao students’ performance on the released digital reading literacy items for

economies participating in PISA 2009 ……………………………………………… 26

IWANTTOHELP

Question 1: IWANTTOHELP E005Q01

…………………….………………….. 29


…………………………………..……. 30


………….…………………………….. 32

Question 4: IWANTTOHELP E005Q08.1 E005Q08.2 E005Q08.3 E005Q08.4

……...…… 34

SMELL

Question 1: SMELL E006Q02

…………………….………................................. 38


……………………...…………............................ 40


………………...................................................... 42

- 3 -

JOB SEARCH

Question 1: JOB SEARCH E012Q01

…………………………………….…...... 47


……………………………………......…. 48


……………………...………………...…. 50

References ………………………........................................................................................... 54

- 4 -

List of Tables

Table 1 Sampling design of Macao’s print and digital reading assessment in

PISA 2009 ………………………………………………………...………. 7

Table 2

Distribution of items according to the five dimensions of the PISA

framework for the assessment of digital reading …………………………. 9

Table 3 League table of digital reading performance of economies participating

in PISA 2009 ……………………………………………………………… 11

Table 4 League table of distribution of digital reading proficiency levels of

economies participating in PISA 2009 …………………………………… 13

Table 5 Description of digital reading proficiency levels in PISA 2009 ………….. 14

Table 6 Gender difference in digital reading literacy performance for economies

in PISA 2009 …………………………………………………………….... 18

Table 7 Variance decomposition of digital reading performance for economies

participating in PISA 2009 ……………………………………………...... 25

Table 8 Test specification of the released digital reading items in PISA 2009 …… 27

Table 9 Item analysis of E005Q01 ………………………………………………... 30








Table 17 Item analysis of E012Q03T ………………………………………………. 50


- 5 -

List of Figures

Figure 1 Macao’s digital reading performance in comparison with participating

economies in PISA 2009 ………………………………………………...... 10

Figure 2 Distribution of digital reading proficiency levels for economies

participating in PISA 2009 (in descending order of percentage of

students with proficiency level 4 or above) ……………………………..... 15

Figure 3 Distribution of digital reading proficiency levels for economies

participating in PISA 2009 (in descending order of percentage of

students with proficiency below level 2) ………………………………..... 16

Figure 4 Gender difference (female-male) in digital reading performance for

economies in PISA 2009 ………………………………………………...... 19

Figure 5 Breakdown of digital reading performance between males and females

for economies participating in PISA 2009 ………………………………... 20

Figure 6 Distribution of digital reading proficiency levels across the grade levels

of Macao’s sampled students in PISA 2009 ………………………………. 21

Figure 7 Distribution of grade levels of Macao’s sampled students across digital

reading proficiency levels in PISA 2009 ………………………………...... 21

Figure 8 Plot of strength of socio-economic gradient and digital reading

performance ………………………………………………………………. 23

Figure 9 Relationship of print and digital reading performance with ESCS for

Macao sampled students in PISA 2009 ………………………………….... 24

Figure 10 Plot of school digital reading performance with school ESCS ………….... 26

- 6 -

Executive Summary

Macao participated in PISA 2009 (OECD, 2009, 2010a-e, 2011). This report focuses on

the following eight findings of the digital reading assessment making comparisons with

the 19 participating economies.

1. Digital reading performance for 15-year-old students who are studying in secondary

schools in Macao is far from satisfactory. More in-depth analyses are needed for

informed policy-making.

2. The digital literacy performance gaps between the top performing countries (e.g.

Korea, New Zealand, and Australia) and Macao are huge. There is a need to learn

from these countries’ successful classroom practices.

3. There is an acute lack of students who perform at a high level in digital reading

literacy. Curricular provision to facilitate the betterment of education for high-level

digital reading proficiency in Macao schools is needed.

4. Macao females outperform males by 12.4 score points, and this gender gap is roughly

half of that of the OECD average. Student engagement in digital reading with the aim

of narrowing the gender gap deserves examination.

5. Lower digital reading proficiency level is associated with lower grade levels, and vice

versa. Effects of grade repetition on digital reading performance for 15-year-old

students in Macao need investigation.

6. Like the findings of the earlier cycles of PISA assessment, Macao is still one of the

most equitable education systems in the world. Notwithstanding this, there are some

signs of inequity at the between-school level that warrants scrutiny by the researchers.

7. Macao students have not utilized the digital reading assessment time well enough to

answer the assigned tasks in the time available, resulting in a high percentage of “not

reached” responses. More practice in online assessment may be necessary for the

students.

8. Macao students do not feel competent to answer “reflect and evaluate” questions,

especially those which are open-constructed response questions. More practices in

assessment of higher order thinking skills may be required by the students.

- 7 -

1. Definition of digital reading literacy

According to PISA, reading literacy is defined as understanding, using, reflecting on and

engaging with written texts in order to achieve one’s goals, develop one’s knowledge and

potential, and participate in society. This definition applies to both print and digital

reading (OECD, 2009, p.14 & 23; OECD, 2011, p.19 & 40). Digital reading demands that

new emphases and strategies be added to the repertoires of readers. Gathering

information on the Internet requires skimming through and scanning large amounts of

material and immediately evaluating its credibility. Critical thinking becomes more

important in digital reading literacy. Overcoming the “digital divide” is not only a matter

of achieving online access, but also of enhancing people’s abilities to integrate, evaluate

and communicate information (OECD, 2009, p.22; OECD, 2011, p.37-38). In the

information age, digital reading literacy is envisaged as an essential component of

reading literacy. Both print and digital reading literacy are important for 21st century

learners who wish to access higher education, find and succeed in a well-paid job, and

participate fully in society (OECD, 2011, p. 209).

2. Sampling design

Because of the complicated logistics of assessing digital reading literacy, not all sampled

students in PISA 2009 took the digital reading assessment. The sampling design for the

digital reading assessment is presented below, together with that of the print reading

assessment.

Table 1

Sampling design of Macao’s print and digital reading assessment in PISA 2009

Macao Sample

Print Reading Assessment Digital Reading Assessment

Number of 15-year-old

students sampled 5,996

(3,036 Males, 2,960 Females) 2,565

(1,255 Males, 1,310 Females)

Number of 15-year-old

students tested 5,952

(3,011 Males, 2,941 Females) 2,519

(1,223 Males, 1,296 Females)

Response rate (%) 99.3 98.2

- 8 -

3. Test design

There are altogether six computer-administered test forms, and each sampled student is

assigned one form in the digital reading assessment. Each form includes test materials

and examinees are allowed a total of 40 minutes assessment time. It is a requirement that

students who take the digital reading assessment must also take the print reading

assessment. Students who take the print assessment, but not the digital assessment, will

have the digital reading literacy score imputed statistically, after equating the scores for

digital reading with those for print reading, using item response theory. This test design

allows for a sound estimation of population statistics for digital reading performance that

can be compared on the same measurement scale with those for the print reading

performance. Table 2 presents the distribution of items according to the five dimensions

of the PISA framework for the assessment of digital reading: (1) Environment; (2) Text

format; (3) Text type; (4) Aspect; and (5) Situation. Section 4 of this report presents item

statistics of the released items of Macao’s sampled students in comparison with

participating economies in PISA 2009.

- 9 -

Table 2

Distribution of items according to the five dimensions of the PISA framework for the

assessment of digital reading

Number of

items

Number of

multiple-choice

items

Number of

complex

multiple-choice

items

Number of

open-constructed

response items

By environment Authored 19 14 0 5 Message-based 8 4 3 1 Mixed 2 0 0 2

Total 29 18 3 8

By text format

Continuous 2 2 0 0 Non-continuous 3 2 0 1 Mixed 2 1 0 1 Multiple 22 13 3 6

Total 29 18 3 8

By text type

Argumentation 6 4 0 2 Description 9 6 1 2 Exposition 9 7 0 2 Transaction 4 1 2 1 Not specified 1 0 0 1

Total 29 18 3 8

By aspect

Access and retrieve

7 7 0 0

Integrate and interpret

10 9 1 0

Reflect and evaluate

6 2 0 4

Complex 6 0 2 4

Total 29 18 3 8

By situation

Personal 6 2 2 2 Public 13 10 0 3 Occupational 7 4 1 2 Educational 3 2 0 1

Total 29 18 3 8 Source: OECD, 2011, p.252.

- 10 -

4. Results and findings

(1) League table of digital reading performance for economies participating in PISA

2009

Figure 1 presents a visual display of the digital reading performance of the 19 economies

participating in PISA 2009. Korea, New Zealand and Australia are the three

top-performing countries. After equating the digital reading performance scores with

those of the print reading scores, the OECD mean of digital reading performance is

calibrated at 498.9. On this measurement scale Macao’s mean (491.9) is 7 score points

lower than that of the mean of the OECD countries. On this scale, 39 score points are

envisaged to be equivalent to one grade level of progression in digital reading

performance in the OECD countries (OECD, 2010, p.13).

Figure 1

Macao’s digital reading performance in comparison with participating economies in

PISA 2009

Table 3 presents the league table of digital reading performance of the 19 economies

participating in PISA 2009. Sixteen economies are OECD countries, and the remaining

368.5

434.5

458.6

463.5

468.3

475.4

488.9

491.9

494.3

498.9

499.9

507.4

508.9

510.3

511.8

514.8

519.2

536.6

537.4

567.6

300 350 400 450 500 550 600

Colombia

Chile

Austria

Poland

Hungary

Spain

Denmark

Macao

France

OECD average

Norway

Belgium

Ireland

Sweden

Iceland

Hong Kong

Japan

Australia

New Zealand

Korea

- 11 -

three are partner economies. Macao ranks 12 in the league table, and Macao’s mean

(491.9) is statistically significantly below the OECD average (498.9). Macao’s

15-year-olds are very homogeneous in digital reading performance, and the standard

deviation of digital reading performance is the lowest (66.4) amongst the 19 participating

economies. Comparing Macao’s performance with that of Korea, which is the

top-performing country in PISA 2009, there is a 75.7 score point difference, amounting to

a 1.94 grade level difference in progression in digital reading performance. The first

important finding is that Macao’s digital reading performance for 15-year-old students in

PISA 2009 is far from satisfactory. More in-depth analyses are needed for informed

policy-making.

Table 3

League table of digital reading performance of economies participating in PISA 2009

Country/Economy Mean

(S.E.)

S.D.

(S.E.) Rank

Upper

Rank

Lower

Rank

Korea+ 567.6 (3.0) 68.4(1.9) 1 1 1

New Zealand+ 537.4(2.3) 98.7(1.8) 2 2 3

Australia+ 536.6(2.8) 97.2(1.7) 3 2 3

Japan+ 519.2(2.4) 75.9(2.8) 4 4 5

Hong Kong 514.8(2.6) 81.8(2.3) 5 4 7

Iceland+ 511.8(1.4) 90.9(1.1) 6 5 8

Sweden+ 510.3(3.3) 88.8(1.8) 7 5 9

Ireland+ 508.9(2.8) 87.1(1.6) 8 6 9

Belgium+ 507.4(2.1) 94.1(1.7) 9 7 9

Norway+ 499.9(2.8) 82.7(1.5) 10 10 11

OECD average* 498.9(0.8) 90.2(0.7) - - -

France+ 494.3(5.2) 95.6(7.1) 11 10 13

Macao 491.9(0.7) 66.4(0.8) 12 11 13

Denmark+ 488.9(2.6) 83.9(1.3) 13 11 13

Spain+ 475.4(3.8) 94.6(2.3) 14 14 15

Hungary+ 468.3(4.2) 102.5(2.7) 15 14 16

Poland+ 463.5(3.1) 90.7(1.5) 16 15 17

Austria+ 458.6(3.9) 102.6(3.9) 17 16 17

Chile+ 434.5(3.6) 89.3(1.9) 18 18 18

Colombia 368.5(3.4) 83.1(1.9) 19 19 19 *

Average is calculated on the basis of the data of 16 OECD countries participating in digital reading

assessment in PISA 2009. + OECD countries.

- 12 -

(2) League table of distribution of digital reading proficiency levels for economies

participating in PISA 2009

Table 4 presents the league table of distribution of digital reading proficiency levels for

economies participating in PISA 2009. It reveals that the digital reading proficiency level

of Macao’s 15-year-olds is mainly concentrated at levels 2 and 3. A total of 71.7% of

sampled students are located within these two middle proficiency levels. In addition,

there are 10.5% of sampled students whose digital reading proficiency is below level 2,

and 2.0% whose digital reading proficiency is level 5 or above. Table 5 details the

description of the digital reading proficiency levels so that one can understand what

students can typically do at each proficiency level. In examining the top-performing

country in PISA 2009, it can be seen that for Korea only 1.8% of sampled students have a

digital reading proficiency below level 2, and there are 19.2% whose digital reading

proficiency is level 5 or above. The second important finding is that the digital literacy

performance gap between Macao and the top-performing countries in PISA 2009 is huge.

There is a need to learn from the successful classroom practices of the top-performing

countries.

- 13 -

Table 4

League table of distribution of digital reading proficiency levels of economies


Country/Economy

Proficiency Level (% of sampled students)

Below level 2 Level 2 Level 3 Level 4 Level 5 or

above

Korea 1.8 8.3 28.7 42.0 19.2

New Zealand 10.2 16.1 27.2 27.8 18.6

Australia 9.6 16.5 28.2 28.5 17.3

Japan 6.7 20.5 38.9 28.2 5.7

Hong Kong 9.8 20.3 36.8 26.8 6.3

Iceland 12.9 21.1 32.2 24.1 9.7

Sweden 13.0 21.2 32.4 24.7 8.6

Ireland 12.1 23.4 32.7 24.0 7.8

Belgium 15.9 20.2 28.8 26.3 8.8

Norway 13.3 25.5 34.4 21.4 5.4

OECD Average* 16.9 22.3 30.4 22.6 7.8

France 16.7 22.4 32.3 23.6 5.1

Macao 10.5 31.8 39.9 15.8 2.0

Denmark 16.4 26.8 33.9 19.2 3.7

Spain 23.1 25.4 30.2 17.3 3.9

Hungary 26.8 25.0 27.1 16.3 4.8

Poland 26.3 28.4 28.6 14.7 2.0

Austria 28.5 25.7 28.3 14.9 2.6

Chile 37.7 30.6 22.5 8.0 1.1

Colombia 68.4 22.4 7.7 1.4 0.1 *


assessment in PISA 2009.

- 14 -

Table 5

Description of digital reading proficiency levels in PISA 2009

Note: There is an additional “below 2” level in the digital reading literacy scale for those students who

cannot attain at the baseline level (i.e. level 2).

Level What students can typically do at each level?

Level 5

and above

(626)

Tasks at this level typically require the reader to locate, analyze and critically

evaluate information, related to an unfamiliar context, in the presence of ambiguity.

They require the generation of criteria to evaluate the text. Tasks may require

navigation across multiple sites without explicit direction, and detailed

interrogation of texts in a variety of formats.

Level 4

(553)

Tasks at this level may require the reader to evaluate information from several

sources, navigating across several sites comprising texts in a variety of formats, and

generating criteria for evaluation in relation to a familiar, personal or practical

context. Other tasks at this level demand that the reader construe complex

information according to well-defined criteria in a scientific or technical context.

Level 3

(480)

Tasks at this level require that the reader integrate information, either by navigating

across several sites to find well-defined target information, or by generating simple

categories when the task is not explicitly stated. Where evaluation is called for, only

the information that is most directly accessible or only part of the available

information is required.

Level 2

(407)

Tasks at this level typically require the reader to locate and interpret information

that is well-defined, usually relating to familiar contexts. They may require

navigation across a limited number of sites and the application of web-based tools

such as dropdown menus, where explicit directions are provided or only low-level

inference is called for. Tasks may require integrating information presented in

different formats, recognizing examples that fit clearly defined categories.

- 15 -

Figures 2 and 3 present the distribution of digital reading proficiency levels respectively

for economies participating in PISA 2009. In Figure 2, the economies are arranged in

descending order of percentage of students with proficiency level 4 or above, whereas in

Figure 3, the economies are arranged in descending order of percentage of students with

proficiency below level 2. The third important finding is that for Macao there is an acute

lack of students who perform at a high level in digital reading literacy. There is a need to

examine curricular provision to facilitate the betterment of education in terms of

high-level digital reading proficiency in Macao schools.

Figure 2

Distribution of digital reading proficiency levels for economies participating in PISA

2009 (in descending order of percentage of students with proficiency level 4 or above)

100 80 60 40 20 0 20 40 60 80

Colombia

Chile

Poland

Austria

Macao

Hungary

Spain

Denmark

Norway

France

OECD Average

Ireland

Hong Kong

Sweden

Iceland

Japan

Belgium

Australia

New Zealand

Korea

Level 4 Level 3 Level 2 Below level 2 Level 5 or above

- 16 -

Figure 3

Distribution of digital reading proficiency levels for economies participating in PISA

2009 (in descending order of percentage of students with proficiency below level 2)

80 60 40 20 0 20 40 60 80 100

Korea

Japan

Australia

Hong Kong

New Zealand

Macao

Ireland

Iceland

Sweden

Norway

Belgium

Denmark

France

OECD Average

Spain

Poland

Hungary

Austria

Chile

Colombia

Below level 2 Level 2 Level 3 Level 4 Level 5 or above

- 17 -

(3) Gender difference in digital reading performance for economies participating in

PISA 2009

Table 6 presents gender difference in digital reading performance for the 19 economies

participating in PISA 2009. Figure 4 is a visual display of this gender difference, showing

that in all 19 economies females outperform males in digital reading literacy to different

extents. The OECD average for gender difference is 24.5 score points, equivalent to 0.63

grade level difference in progression of digital reading performance. The fourth important

finding is that Macao females outperform males by 12.4 score points, and the gender

difference is roughly half of that of the OECD average. There is a need to examine student

engagement in digital reading to see whether there is a possibility of narrowing the

gender gap further.

Figure 5 shows not only the extent of the gender difference, but also how each gender

scores in digital reading literacy. It can be seen that Korea’s males and females

outperform all males and females in other economies participating in PISA 2009. Their

achievements are truly extraordinary. Korea sets the highest standard for each gender to

achieve in PISA 2009. Inspired by the Hong Kong results, it is reckoned that Macao’s

overall digital reading literacy can be enhanced by raising the proficiency levels of the

males, resulting in the narrowing of the gender gap in digital reading performance.

- 18 -

Table 6

Gender difference in digital reading literacy performance for economies in PISA 2009

Country/Economy Mean

(Female)

S.D.

(Female)

Mean

(Male)

S.D.

(Male)

Diff.(F-M)

(Mean)

Korea 576.8 63.7 559.3 71.4 17.5

New Zealand 558.0 86.9 517.5 105.1 40.5

Australia 550.3 90.5 522.3 101.9 27.9

Japan 531.1 72.6 508.1 77.1 23.0

Iceland 526.8 86.0 496.7 93.1 30.2

Ireland 524.6 80.4 493.6 90.6 31.1

Sweden 523.8 83.9 497.3 91.5 26.5

Belgium 519.7 89.9 495.6 96.6 24.0

Hong Kong 518.9 78.4 511.1 84.6 7.8

Norway 517.6 78.3 483.1 83.3 34.5

OECD average* 511.3 85.6 486.8 92.6 24.5

France 504.2 95.7 483.9 94.3 20.3

Macao 498.2 64.5 485.8 67.7 12.4

Denmark 492.0 83.5 485.8 84.2 6.2

Spain 485.1 91.8 466.0 96.2 19.0

Hungary 479.0 98.5 457.8 105.3 21.2

Poland 478.2 82.2 448.9 96.3 29.3

Austria 469.4 100.4 447.4 103.7 22.0

Chile 444.0 86.1 425.4 91.4 18.6

Colombia 370.2 81.9 366.7 84.2 3.5 *



- 19 -

Figure 4

Gender difference (female-male) in digital reading performance for economies in PISA

2009

3.5

6.2

7.8

12.4

17.5

18.6

19.0

20.3

21.2

22.0

23.0

24.0

24.5

26.5

27.9

29.3

30.2

31.1

34.5

40.5

0 10 20 30 40 50

Colombia

Denmark

Hong Kong

Macao

Korea

Chile

Spain

France

Hungary

Austria

Japan

Belgium

OECD average

Sweden

Australia

Poland

Iceland

Ireland

Norway

New Zealand

- 20 -

Figure 5

Breakdown of digital reading performance between males and females for economies


(4) Grade level difference in digital reading performance of economies participating

in PISA 2009

Figures 6 and 7 present two perspectives regarding the reciprocal relationships between

digital reading proficiency level and grade level of Macao’s 15-year-olds in PISA 2009.

The fifth important finding is that lower digital reading proficiency level is associated

with lower grade levels, and vice versa. There is a need to examine the effects of grade

repetition on digital reading performance for 15-year-old students in Macao.

370.2

444.0

469.4

478.2

479.0

485.1

492.0

498.2

504.2

511.3

517.6

518.9

519.7

523.8

524.6

526.8

531.1

550.3

558.0

576.8

366.7

425.4

447.4

448.9

457.8

466.0

485.8

485.8

483.9

486.8

483.1

511.1

495.6

497.3

493.6

496.7

508.1

522.3

517.5

559.3

300 350 400 450 500 550 600

Colombia

Chile

Austria

Poland

Hungary

Spain

Denmark

Macao

France

OECD average

Norway

Hong Kong

Belgium

Sweden

Ireland

Iceland

Japan

Australia

New Zealand

Korea

Mean (Male) Mean (Female)

- 21 -

Figure 6

Distribution of digital reading proficiency levels across the grade levels of Macao’s

sampled students in PISA 2009

Figure 7

Distribution of grade levels of Macao’s sampled students across digital reading

proficiency levels in PISA 2009

0

5

10

15

20

25

30

35

40

45

50

Grade 7 Grade 8 Grade 9 Grade10 Grade 11

Below level 2

Level 2

Level 3

Level 4

Level 5 or

above

0

5

10

15

20

25

30

35

40

45

50

Below level 2 Level 2 Level 3 Level 4 Level 5 or

above

Grade 7

Grade 8

Grade 9

Grade10

Grade 11

- 22 -

(5) Effect of ESCS on digital reading performance in participating economies in

PISA 2009

Figure 8 shows that amongst the 19 economies participating in PISA 2009, Macao is

revealed to be the most educationally equitable from an examination of strength of

socio-economic gradient and digital reading performance. This is because the percentage

of variance in digital reading performance explained by the Economic, Social and

Cultural Status (ESCS) of the home is the lowest amongst all participating economies.

Figure 9 shows further that for both print and digital reading the slope of the

socio-economic gradient is not steep. This is another important sign of the educational

equity of schools attended by 15-year-olds in Macao. The sixth important finding is that

Macao is one of the most equitable education systems in the world. This finding is same

as those found in the earlier cycles of PISA assessment conducted since 2003.

In spite of this favorable finding, variance decomposition of digital reading performance

into between-school and within-school components reveals that between-school factors

account for about a quarter of the digital reading performance variance (see Table 7). This

indicates that there is still a small degree of educational inequity in Macao’s basic

education system. Figure 10 attempts to clarify this by plotting for each sampled school in

PISA 2009 the scatter diagram depicting the relationship between school ESCS and

school digital reading performance. In this figure one can see that there is a higher

concentration of low-performing schools at the low end of the ESCS continuum. There is

still some room for improvement in educational equity in Macao schools, and this is

especially so when the digital reading performance cannot be considered as satisfactory.

- 23 -

Figure 8

Plot of strength of socio-economic gradient and digital reading performance

Percentage of variance in digital reading performance explained by ESCS

Australia

Austria

Belgium

Chile

Colombia

Denmark Spain

France

Hong Kong

Hungary

Ireland

Iceland Japan

Korea

Macao Norway

New Zealand

Sweden

Poland

350

400

450

500

550

600

5 10 15 20 25 30

OECD average

R square = 0.141

OECD average

- 24 -

Figure 9

Relationship of print and digital reading performance with ESCS for Macao sampled

students in PISA 2009

350

400

450

500

550

600

650

700

-2.5 -1.9 -1.3 -0.7 -0.1 0.5 1.1

Print Reading Performance Digital Reading Performance

ESCS

- 25 -

Table 7

Variance decomposition of digital reading performance for economies participating in

PISA 2009

Country/Economy Within-school

variance

Between-school

variance Total variance

% of

between-school

variance within

economy

Australia 6877.1 2768.4 9645.5 28.7%

Austria 4121.3 8248.8 12370.1 66.7%

Belgium 4167.3 5900.5 10067.8 58.6%

Chile 4228.2 6107.2 10335.4 59.1%

Colombia 3887.1 3482.5 7369.6 47.3%

Denmark 5383.9 1747.7 7131.6 24.5%

Spain 6090.7 2628.0 8718.8 30.1%

France 3147.8 4052.2 7200.0 56.3%

Hong Kong 3992.7 3327.2 7319.9 45.5%

Hungary 3799.6 7248.1 11047.7 65.6%

Ireland 6123.4 1706.4 7829.7 21.8%

Iceland 6703.5 1675.6 8379.1 20.0%

Japan 4496.3 1809.4 6305.7 28.7%

Korea 3873.6 2302.7 6176.3 37.3%

Macao 3483.5 1152.5 4636.0 24.9%

Norway 5701.8 1349.9 7051.7 19.1%

New Zealand 7627.0 2473.6 10100.6 24.5%

Poland 6489.8 2177.3 8667.1 25.1%

Sweden 6155.8 2048.5 8204.3 25.0%

OECD average* 5311.7 3390.3 8702.0 36.9%

* Average is calculated on the basis of the data of 16 OECD countries participating in digital reading


- 26 -

Figure 10

Plot of school digital reading performance with school ESCS

(6) Macao students’ performance on the released digital reading literacy items for

economies participating in PISA 2009

There were three test units released after the PISA 2009 main survey, namely: (1)

IWANTTO HELP (Task 1-4); (2) SMELL (Task 1-3), and (3) JOB SEARCH (Task 1-3).

Table 8 details the blueprint used in the design of digital reading literacy assessment tasks,

and where the released test items of the test units fit into it.

20 31 29 30 32

12 16 36 13 1 19 2 43 14 35

4 24 42

44 5 23 40 7

11 34 37 21 22 15 27

10 17

25

26 9 18 41 6

33 8 39 28 38

3

400

450

500

550

600

-1.6 -1.1 -0.6 -0.1 0.4 0.9 1.4

Dig

ita

l R

ead

ing

Per

form

an

ce

ESCS

Average-Performing

School

Low-Performing

School

High-Performing

School

- 27 -

Table 8

Test specification of the released digital reading items in PISA 2009

Competencies

Access

&

Retrieve

Integrate

&

Interpret

Reflect

&

Evaluate

Complex

Reading Text Format

Continuous

Public

Occupational IWANTTOHELP

Task 1

Non-

Continuous

Public

Occupational

JOB

SEARCH

Task 1

Mixed

Public

Occupational

JOB

SEARCH

Task 3

Multiple

Public

SMELL

Task 1 and 3

SMELL

Task 2

Occupational IWANTTOHELP

Task 2

IWANTTOHELP

Task 3

JOB SEARCH

Task 2

IWANTTOHELP

Task 4

The three released test units are exemplary of the various categories of reading text

format (i.e. Continuous versus Non-continuous, Public versus Occupational) and

competencies (i.e. Access and Retrieve, Integrate and Interpret, Reflect and Evaluate,

Complex) in the digital reading literacy assessment. Interested readers can access the

website (http://www.umac.mo/fed/pisa/samples(eng).htm) to examine the contents of the

digital test units. Documented in this report are the coding guides of the items, and based

on the released items, item analyses are conducted to examine Macao 15-year-olds’

performance in comparison with that of 15-year-olds from economies in PISA 2009 (see

Table 9-18 for details).

http://www.umac.mo/fed/pisa/samples(eng).htm)

- 28 -

Two more findings, namely, the seventh and eighth findings, may be suggested as a

consequence of interpreting Macao’s item analysis results in comparison to other

participating economies in PISA 2009:

1. There is an abundance of responses registered as “not reached” by the computer

system, indicating that some students have not utilized the assessment time well

enough to answer the assigned tasks in the time available. One plausible explanation

noted by the test administrators is that some students are not conversant with the use

of Chinese input methods for answering the open-constructed test items online.

Another explanation is that some students like to browse irrelevant pages and spend a

lot of time surfing these pages before accessing a page which is relevant to answering

the item correctly.

2. Macao students do not feel competent to answer “reflect and evaluate” questions.

This is especially so when the answer is not a multiple-choice but an

open-constructed response question. Some students can gain partial credits but fail to

gain full credits because of their less-than-adequate reflection and evaluation of the

materials presented to them. One plausible explanation for this incompetence may be

due to their lack of classroom practice in reading of the kind of digital reading

materials assessed in PISA.

- 29 -

IWANTTOHELP


Read Maika’s blog entry for January 1. What does the entry say about Maika’s experience of

volunteering?

A. She has been a volunteer for many years.

B. She only volunteers in order to be with her friends.

C. She has done a little volunteering but would like to do more.

D. She has tried volunteering but does not think it is worthwhile.

IWANTTOHELP SCORING Q01

QUESTION INTENT:

Access and retrieve: Retrieve information

Locate explicitly stated information

Full Credit

Code 1: C. She has done a little volunteering but would like to do more.

No Credit

Code 0: Other answers.

Code 9: Missing.

- 30 -

Table 9

Item analysis of E005Q01

Country/Economy

%

A

(no credit)

B

(no credit)

C

(full credit)

D

(no credit) Missing

Not

Reached

Australia 5.8 2.9 88.5 1.5 0.6 0.7

Austria 9.4 11.5 73.3 3.2 2.0 0.7

Belgium 5.3 3.6 86.6 2.3 1.5 0.7

Chile 9.8 4.9 68.3 2.1 4.0 11.0

Colombia 20.3 7.7 52.9 3.3 4.0 11.7

Denmark 8.7 5.0 82.4 2.4 0.4 1.1

Spain 7.7 3.6 84.4 1.3 1.6 1.4

France 8.6 2.5 82.8 0.8 2.0 3.3

Hong Kong 4.8 2.5 85.5 2.9 1.5 2.8

Hungary 11.8 8.2 70.4 6.0 2.4 1.2

Ireland 5.4 2.9 86.9 1.3 1.6 1.9

Iceland 5.2 4.1 85.9 2.1 2.0 0.7

Japan 5.0 3.7 85.9 2.3 1.1 2.0

Korea 2.9 1.4 93.1 1.3 0.6 0.6

Macao 5.6 3.0 80.5 3.2 2.0 5.6

Norway 6.8 3.4 85.5 1.6 1.7 1.0

New Zealand 4.2 2.4 89.2 1.9 0.8 1.5

Poland 10.1 6.4 79.4 2.4 1.0 0.7

Sweden 3.9 4.4 86.0 2.7 1.3 1.6

OECD Average* 6.9 4.4 83.0 2.2 1.5 1.9




Go to Maika’s “About” page.

What kind of work does Maika want to do when she leaves school?

A. Photography.

B. Web design.

C. Banking.

D. Social work.


QUESTION INTENT:


Locate explicitly stated personal information on correctly located web page

- 31 -

Full Credit

Code 1: B. Web design.

No Credit


Code 9: Missing.

Table 10


Country/Economy

%

A

(no credit)

B

(full credit)

C

(no credit)

D

(no credit) Missing

Not

Reached

Australia 7.1 82.8 0.6 8.2 0.5 0.8

Austria 18.4 66.7 3.7 7.9 2.2 1.1

Belgium 9.9 78.8 1.3 6.7 2.3 1.0

Chile 5.3 53.9 1.8 21.5 4.7 12.9

Colombia 9.8 37.8 5.6 29.3 3.9 13.7

Denmark 9.5 78.7 0.7 8.4 1.6 1.2

Spain 8.4 66.0 2.6 18.6 2.7 1.9

France 7.5 81.5 0.8 4.4 1.1 4.8

Hong Kong 8.4 71.2 0.5 14.5 1.7 3.7

Hungary 13.6 73.9 3.7 5.7 1.6 1.5

Ireland 13.1 74.3 1.3 7.6 1.6 2.1

Iceland 8.4 82.3 1.0 5.0 1.8 1.5

Japan 4.8 71.6 1.1 17.5 2.4 2.5

Korea 4.1 92.6 0.2 1.8 0.1 1.2

Macao 5.0 70.9 0.6 14.1 2.2 7.1

Norway 5.5 82.8 0.8 7.8 1.3 1.9

New Zealand 7.9 82.5 0.9 5.6 1.3 1.8

Poland 11.8 76.2 2.3 7.6 1.3 0.8

Sweden 7.1 81.0 1.5 5.9 2.2 2.3

OECD Average* 8.9 76.6 1.5 8.8 1.8 2.4



- 32 -


Open the link that Maika refers to in her January 1 post. What is the main function of this website?

A. To encourage people to buy iwanttohelp products.

B. To encourage people to give money to people in need.

C. To explain how you can make money by volunteering.

D. To provide people with information about ways to volunteer.

E. To tell people in need where they can find help.


QUESTION INTENT:

Integrate and interpret: Form a broad understanding

Recognise the main purpose of a website

Full Credit

Code 1: D. To provide people with information about ways to volunteer.

No Credit


Code 9: Missing.

- 33 -

Table 11


Country/Economy

%

A

(no

credit)

B

(no

credit)

C

(no

credit)

D

(full

credit)

E

(no

credit)

Missing Not

Reached

Australia 3.2 8.2 3.5 80.6 2.7 0.8 0.9

Austria 9.6 17.1 10.9 53.3 5.2 2.2 1.7

Belgium 5.0 6.5 5.7 74.4 5.7 1.1 1.6

Chile 8.4 7.4 7.6 51.0 7.6 3.3 14.7

Colombia 13.7 10.0 9.8 34.3 13.2 4.0 15.1

Denmark 5.5 9.4 9.8 68.7 3.6 1.1 2.0

Spain 6.3 11.7 6.8 63.1 8.3 1.7 2.3

France 2.8 7.3 5.4 68.8 9.0 1.6 5.1

Hong Kong 1.4 6.7 2.7 76.9 6.7 1.5 4.1

Hungary 12.5 12.3 14.4 50.1 6.8 2.4 1.5

Ireland 6.4 10.3 4.2 70.8 4.2 2.1 2.1

Iceland 3.8 6.0 4.1 78.8 3.8 1.6 2.0

Japan 5.4 8.7 0.9 62.3 17.6 2.0 3.2

Korea -- 2.4 0.6 82.6 13.0 0.2 1.2

Macao 2.4 6.1 2.1 71.5 7.7 2.2 8.1

Norway 4.2 12.8 6.7 67.0 5.3 1.5 2.4

New Zealand 3.3 7.2 4.5 76.9 4.9 1.1 2.1

Poland 6.1 12.6 6.6 64.7 7.4 1.4 1.2

Sweden 4.9 11.1 8.4 62.8 8.9 1.2 2.8

OECD Average* 5.8 9.4 6.3 67.2 7.1 1.6 2.9



- 34 -

Question 4: IWANTTOHELP Q08.1, Q08.2, Q08.3, Q08.4 E005Q08.1 – 0 1 2 9

E005Q08.2 – 0 9

E005Q08.3 – 0 9

E005Q08.4 – 0 1 2 9

Read Maika's blog for January 1. Go to the iwanttohelp site and find an opportunity for Maika.

Use the e-mail button on the “Opportunity Details” page for this opportunity to tell Maika about it.

Explain in the e-mail why the opportunity is suitable for her. Then send your e-mail by clicking on

the "Send" button.


QUESTION INTENT:

Complex

Construct and send an e-mail message integrating and reflecting upon information accessed

from several texts


Locate a series of related pages by following a series of links

Integrate and interpret: Develop an interpretation

Compare information in multiple places on one website with criteria stated on another

website

Reflect and evaluate: Reflect on and evaluate content of text

Explain a choice based on a comparison

Responses to Q08.1 to Q08.4 are coded separately but only one code will be used for scoring. The

highest code is treated as the total score for the item. Note that any response to Q08.2 or Q08.3

receives Code 0.

IWANTTOHELP SCORING Q08.1 (E-mail message for Graphic Artist)

Full Credit

Code 2: Refers to ongoing position or future or web design or art. You’re a great artist and it is ongoing - you said you wanted a longer type of work right?

It’s ongoing and it would help you get experience for your future.

You are obviously interested in graphic design, and want to pursue this when you finish

school, and you would also love to volunteer. This would be a great opportunity to do both

these things, and will look great on your CV too!

Partial Credit

Code 1: Gives insufficient or vague answer. You’d like it.

Shows inaccurate comprehension of the opportunity or gives an implausible or

irrelevant answer. You’d be working with kids a lot. [Irrelevant, not one of Maika’s criteria.]

It gives you a chance to get out and about.

No Credit

Code 0: Gives unintelligible or incomplete answer. You could

- 35 -

you

Code 9: Missing.

IWANTTOHELP SCORING Q08.2 (E-mail message for Vegfest)

No Credit

Code 0: Any response.

Code 9: Missing.

IWANTTOHELP SCORING Q08.3 (E-mail message for Help fix up Twin Falls Track!)

No Credit

Code 0: Any response.

Code 9: Missing.

IWANTTOHELP SCORING Q08.4 (E-mail message for Upway Primary School)

Full Credit

Code 2: Refers to ongoing position or making a difference. This would be a good job - ongoing and you get to help some kids.

Here’s a job where you’ll really make a difference.

Partial Credit

Code 1: Gives insufficient or vague answer. You need an hour a week but it sounds like this could be what you’re looking for. [Lacks

reference to job criteria, repeats part of stem.]

You’d like it.

Shows inaccurate comprehension of the opportunity or gives an implausible or

irrelevant answer. It gives you a chance to get out and about.

No Credit

Code 0: Gives unintelligible or incomplete answer. The students

Code 9: Missing.

- 36 -

Table 12


Country/Economy %

No credit Partial credit Full credit Missing Not reached

Australia 1.7 13.2 61.5 22.3 1.3

Austria 3.1 11.6 26.0 56.3 3.0

Belgium 5.7 10.2 45.5 36.4 2.2

Chile 4.6 12.9 16.3 49.5 16.7

Colombia 2.9 8.1 6.9 64.6 17.6

Denmark 6.7 17.7 39.9 33.0 2.7

Spain 5.4 18.8 34.8 37.9 3.1

France 5.3 11.0 35.3 42.0 6.4

Hong Kong 3.5 15.8 43.3 32.1 5.3

Hungary 8.0 14.6 25.4 49.3 2.7

Ireland 3.4 14.5 47.5 31.6 3.1

Iceland 5.7 14.4 55.9 21.3 2.7

Japan 3.7 14.4 52.2 25.1 4.6

Korea 2.2 12.2 67.6 16.5 1.4

Macao 2.9 19.8 35.4 32.4 9.5

Norway 5.3 14.2 51.5 25.8 3.1

New Zealand 1.9 9.9 63.5 22.3 2.3

Poland 1.8 7.3 17.7 71.3 1.9

Sweden 7.3 14.6 42.5 32.2 3.4

OECD Average* 4.5 13.2 42.7 35.8 3.8



- 37 -

OECD PISA acknowledges the following sources for the use of their material in this assessment.

Every effort has been made to trace the original source material contained in the assessment.

Where the attempt has been unsuccessful, the PISA project consortium would be pleased to hear

from the author/publisher to rectify any omission.

Adapted from:

“Hand” by Mike-Kevin Maul

http://www.sxc.hu/pic/l/a/ap/apollo777/730953_39615320.jpg

Access date: January 2011

Adapted from:

bizior photography - www.bizior.com

http://www.sxc.hu/pic/l/b/bi/bizior/698452_43583333.jpg


Adapted from:

“On Duty” by Miles Pfefferle

http://www.sxc.hu/pic/l/t/th/theusher/645220_43082727.jpg


Adapted from:

“Smile” by Jose Rivera

http://www.sxc.hu/pic/l/a/ac/acambaro77/778208_49349093.jpg


http://www.sxc.hu/pic/l/a/ap/apollo777/730953_39615320.jpg

http://www.bizior.com/

http://www.sxc.hu/pic/l/b/bi/bizior/698452_43583333.jpg

http://www.sxc.hu/pic/l/t/th/theusher/645220_43082727.jpg

http://www.sxc.hu/pic/l/a/ac/acambaro77/778208_49349093.jpg

http://www.sxc.hu/browse.phtml?f=download&id=7782

- 38 -

SMELL


Go to the “Smell: A Guide” web page. Which of these statements best expresses the main idea

on this page?

A. Smell can interfere with normal patterns of behaviour.

B. Smell warns humans and animals of danger.

C. The primary purpose of smell is to help animals to find food.

D. The development of smell takes place early in life.

E. The basic function of smell is recognition.

SMELL SCORING Q02

QUESTION INTENT:

Integrate and interpret: Form a broad understanding

Identify the main idea of a web page

Full Credit

Code 1: E. The basic function of smell is recognition.

- 39 -

No Credit

Code 0: Other responses.

Code 9: Missing.

Table 13


Country/

Economy

%

A

(no

credit)

B

(no

credit)

C

(no

credit)

D

(no

credit)

E

(full

credit)

Missing Not

Reached

Australia 10.3 18.1 5.4 10.5 53.3 2.1 0.2

Austria 10.7 34.9 13.8 10.7 26.0 3.6 0.3

Belgium 5.4 28.7 9.5 12.0 40.7 3.5 0.2

Chile 8.5 16.0 8.3 10.3 46.3 8.0 2.7

Colombia 10.5 20.4 13.2 4.9 39.6 7.2 4.2

Denmark 10.9 25.7 8.8 17.8 34.2 2.2 0.4

Spain 10.8 18.1 8.3 10.2 48.5 3.3 0.8

France 3.1 27.1 8.6 16.9 40.6 2.7 1.1

Hong Kong 7.6 31.4 4.5 5.0 49.1 1.4 0.9

Hungary 4.7 27.7 11.0 8.5 42.8 5.0 0.3

Ireland 8.1 31.6 6.8 11.8 37.3 4.1 0.3

Iceland 8.1 14.6 6.4 14.6 52.6 3.4 0.1

Japan 1.3 40.5 5.2 18.7 32.6 1.4 0.3

Korea 2.7 19.6 2.3 12.1 62.8 0.6 --

Macao 11.0 27.5 5.8 9.0 42.4 3.2 1.1

Norway 7.3 26.0 4.5 12.6 47.4 2.2 --

New Zealand 9.3 18.1 5.9 10.5 53.0 2.8 0.4

Poland 11.2 40.2 16.7 14.4 11.8 5.3 0.4

Sweden 3.3 29.4 6.9 15.0 43.2 1.8 0.5

OECD Average* 7.2 26.0 8.0 12.9 42.1 3.2 0.6



- 40 -

Question 2: SMELL E006Q05 – 0 1 9

Go to the “Food in the news” web page. Would this web page be a suitable source for you to

refer to in a school science assignment about smell? Answer Yes or No and refer to the content

of the “Food in the news” web page to give a reason for your answer.

SMELL SCORING Q05

QUESTION INTENT:

Reflect and evaluate: Reflect on and evaluate content or form of text

Evaluate web page in terms of credibility/trustworthiness of information

Full Credit

Code 1: Answers (or implies) No and gives a plausible supporting explanation, referring to

the trivial or sensational nature of the website content, or the popularisation of the

issues by journalists.

No, it’s just trying to popularise science and has almost certainly oversimplified the

original research.

No, it just offers sensational news. Look at the superficial issues covered in this site.

No, it is obviously from a popular news magazine not a scholarly source.

No, it has loads of silly links that show it’s not a serious site.

No, not suitable because it is just written by journalists not scientists.

Answers (or implies) Yes and indicates that the site would be helpful as a secondary

source, leading to more reputable sources.

Yes, it would help me to find the original research.

Yes, I would use it to look and see if more serious publications said the same thing.

Answers (or implies) Yes and gives a plausible supporting explanation, referring to

the article’s sources of information or the level of detail provided.

Yes, because it is a review of real research.

Yes, because it talks about several real studies.

Yes, they’re talking about a study that won a Nobel prize, so it must be true.

Yes, the study is described in detail so I don’t think they would make it up.

No Credit

Code 0: Gives insufficient or vague answer.

Yes, The Food in the News page was convincing because the results that they were

showing did not seem opinionated and sounded reliable. [vague]

- 41 -

I don’t think it’s reliable because it’s about the power behind our sense of smell. [vague]

Yes, it’s a long article. Why would they make all that up?

No, my teacher would not be impressed.

Shows inaccurate comprehension of the material or gives an implausible or

irrelevant answer.

Yes, because it’s by a motoring organisation, which really matters. [irrelevant]

I think it would be reliable because it describes how smell can affect your mood.

[irrelevant]

Code 9: Missing.

Table 14


Country/Economy %

No credit Full credit Missing Not Reached

Australia 52.6 42.5 4.7 0.2

Austria 71.8 11.1 16.8 0.3

Belgium 59.4 32.6 7.6 0.4

Chile 51.9 26.8 16.3 4.9

Colombia 66.4 11.5 16.1 6.0

Denmark 71.4 20.2 7.9 0.4

Spain 59.6 32.2 7.3 0.9

France 57.4 33.7 7.3 1.6

Hong Kong 76.4 13.7 8.7 1.3

Hungary 72.2 10.1 17.1 0.5

Ireland 68.8 22.7 8.1 0.3

Iceland 72.4 19.6 7.7 0.3

Japan 47.3 39.0 13.3 0.4

Korea 69.4 26.6 3.8 0.2

Macao 68.8 21.7 7.3 2.2

Norway 66.5 25.0 8.3 0.2

New Zealand 53.3 39.9 6.4 0.4

Poland 70.8 17.8 11.0 0.4

Sweden 65.7 25.2 8.4 0.7

OECD Average* 63.2 26.6 9.5 0.8



- 42 -


There is information about the smell of lemon on the pages “Food in the news” and

“Psychology Now”. Which statement summarises the conclusions of the two studies about the

smell of lemon?

A. Both studies suggested that the smell of lemon helps you work quickly.

B. Both studies suggested that most people like the smell of lemon.

C. Both studies suggested that the smell of lemon helps you to concentrate.

D. Both studies suggested that females are better at detecting the smell of lemon than males.

SMELL SCORING Q06

QUESTION INTENT:


Synthesise information from two web pages

Full Credit

Code 1: C. Both studies suggested that the smell of lemon helps you to concentrate.

No Credit


Code 9: Missing.

- 43 -

Table 15


Country/Economy

%

A

(no credit)

B

(no credit)

C

(full

credit)

D

(no credit) Missing

Not

Reached

Australia 7.0 8.4 71.1 11.2 2.0 0.4

Austria 9.3 15.0 55.1 16.5 3.6 0.6

Belgium 7.9 9.0 66.0 14.3 2.3 0.4

Chile 13.0 13.8 45.0 11.5 9.7 7.1

Colombia 15.3 23.2 30.9 15.2 7.1 8.3

Denmark 10.2 9.1 65.5 13.0 1.6 0.6

Spain 10.2 9.8 62.6 12.5 3.9 0.9

France 6.4 12.2 62.4 13.8 2.8 2.4

Hong Kong 13.9 6.3 66.9 9.2 2.3 1.4

Hungary 11.6 16.8 50.2 16.3 4.2 0.9

Ireland 8.8 9.9 60.5 15.4 4.9 0.5

Iceland 8.8 8.3 67.9 10.4 4.1 0.4

Japan 6.2 6.4 77.9 6.3 2.4 0.8

Korea 9.1 2.1 78.4 9.5 0.7 0.2

Macao 9.4 5.0 67.5 11.3 3.8 2.9

Norway 11.6 10.9 59.4 15.2 2.3 0.6

New Zealand 8.0 8.1 66.6 13.5 3.1 0.7

Poland 12.3 15.9 54.4 13.5 3.4 0.5

Sweden 6.9 10.9 69.3 9.4 2.4 1.0

OECD Average* 9.2 10.4 63.3 12.6 3.4 1.1



- 44 -

OECD PISA acknowledges the following sources for the use of their material in this

assessment. Every effort has been made to trace the original source material contained

in the assessment. Where the attempt has been unsuccessful, the PISA project

consortium would be pleased to hear from the author/publisher to rectify any omission.

Adapted from:

“Wall 2” by Dmitry Poliansky

http://www.sxc.hu/pic/l/p/ph/phre3a/830642_11324173.jpg


Adapted from:

“Fisheye analogue” by Dennis Dude

http://www.sxc.hu/pic/l/d/de/dennis/856824_38252752.jpg


Adapted from:

“Find Your Way” by Ann- Kathrin Rehse

http://www.sxc.hu/pic/l/k/ka/kalilo/490615_79640989.jpg


Adapted from:

http://www.sxc.hu/photo/860707

Access date: September 2007

Adapted from:

“Pyxis 2012” by Griszka Niewiadomski

http://www.sxc.hu/pic/l/d/da/datarec/505245_35319552.jpg


Adapted from:



Adapted from:

“Rose” by Ayfer Uys

http://www.sxc.hu/pic/l/g/ge/gezegen/804566_51154921.jpg


http://www.sxc.hu/pic/l/p/ph/phre3a/830642_11324173.jpg

http://www.sxc.hu/pic/l/d/de/dennis/856824_38252752.jpg

http://www.sxc.hu/pic/l/k/ka/kalilo/490615_79640989.jpg


http://www.sxc.hu/pic/l/d/da/datarec/505245_35319552.jpg


http://www.sxc.hu/pic/l/g/ge/gezegen/804566_51154921.jpg

- 45 -

Adapted from:

“Lolipop” by Claudia Meyer, Paris

http://www.sxc.hu/pic/l/c/cl/claudmey/784354_69676614.jpg


Adapted from:

“Dragon-Fly Head 3” by Andrzej Pobiedziński

http://www.sxc.hu/pic/l/s/sa/sarej/594785_99878886.jpg


Adapted from:



Adapted from:

“Leaves and Life” by Luisa Migon

http://www.sxc.hu/pic/l/l/lu/luisamigon/864662_16808575.jpg


Adapted from:

“Body Cuts 3” by Matteo Canessa

http://www.sxc.hu/pic/l/s/sc/sciucaness/515952_34724779.jpg


Adapted from:

“Baby face 1” by Samuel Rosa

http://www.sxc.hu/pic/l/k/ko/kodakgold/817352_58786767.jpg


Adapted from:



“Cow's nose” by Michael Grunow

http://www.sxc.hu/pic/l/g/gr/grunow/837212_56528949.jpg


“Horse Nose” by Dorthe Bjerg

http://www.sxc.hu/pic/l/w/wu/wusse/782838_58727973.jpg


“Peeping over the edge” by Bas van de Wiel

http://www.sxc.hu/pic/l/c/cl/claudmey/784354_69676614.jpg

http://www.sxc.hu/pic/l/s/sa/sarej/594785_99878886.jpg


http://www.sxc.hu/pic/l/l/lu/luisamigon/864662_16808575.jpg

http://www.sxc.hu/pic/l/s/sc/sciucaness/515952_34724779.jpg

http://www.sxc.hu/pic/l/k/ko/kodakgold/817352_58786767.jpg


http://www.sxc.hu/pic/l/g/gr/grunow/837212_56528949.jpg

http://www.sxc.hu/pic/l/w/wu/wusse/782838_58727973.jpg

- 46 -

http://www.sxc.hu/pic/l/b/bv/bvdwiel/757753_92465570.jpg


“Face Melt” by Kenson Lai

http://www.sxc.hu/pic/l/a/ad/advocate_x/631325_48075691.jpg


“Bird from Ecuador” by M Nota

http://www.sxc.hu/pic/l/n/no/nota/581178_68748771.jpg


“Stipey” by Kristin Smith

http://www.sxc.hu/pic/l/k/kl/klsmith77/550258_36058030.jpg


“Camel” by Leila Haj-Hassan

http://www.sxc.hu/pic/l/l/le/leilahh/765393_22111757.jpg


Adapted from:



Adapted from:

“.| Future |.” by Clix

http://www.sxc.hu/pic/l/c/cl/clix/815359_90510088.jpg


http://www.sxc.hu/pic/l/b/bv/bvdwiel/757753_92465570.jpg

http://www.sxc.hu/pic/l/a/ad/advocate_x/631325_48075691.jpg

http://www.sxc.hu/pic/l/n/no/nota/581178_68748771.jpg

http://www.sxc.hu/pic/l/k/kl/klsmith77/550258_36058030.jpg

http://www.sxc.hu/pic/l/l/le/leilahh/765393_22111757.jpg


http://www.sxc.hu/pic/l/c/cl/clix/815359_90510088.jpg

- 47 -

JOB SEARCH


This is a page from a job search website. Which job in this list is most suitable for school

students?

Click on the button next to the job.

JOB SEARCH SCORING Q01

QUESTION INTENT:


Use criteria in a set of short descriptions and prior knowledge to select a job suitable for

a student

Full Credit

Code 1: B. Juice Bar Team Members.

No Credit


Code 9: Missing.

- 48 -

Table 16


Country/Economy

%

A

(no credit)

B

(full

credit)

C

(no credit)

D

(no credit) Missing

Not

Reached

Australia 13.0 79.7 2.1 2.4 2.6 0.2

Austria 26.8 42.3 10.5 14.8 5.4 0.2

Belgium 17.2 62.2 6.8 11.3 2.4 0.1

Chile 17.5 55.9 11.8 4.8 8.9 1.2

Colombia 16.1 40.3 28.0 3.1 11.1 1.3

Denmark 23.2 64.4 2.9 8.8 0.8 --

Spain 16.7 55.0 15.3 9.5 3.2 0.3

France 12.0 69.0 6.4 10.6 1.1 0.8

Hong Kong 18.5 72.1 4.2 4.3 0.8 --

Hungary 19.6 58.6 7.9 7.6 6.2 0.1

Ireland 11.5 77.0 4.1 4.9 2.4 0.1

Iceland 7.9 72.1 8.0 10.2 1.7 0.1

Japan 13.8 75.0 3.1 5.5 2.5 0.1

Korea 6.9 84.8 4.5 3.6 0.2 --

Macao 18.9 68.2 7.6 4.4 0.9 --

Norway 13.6 63.7 3.1 18.0 1.5 --

New Zealand 10.9 81.9 3.0 1.5 2.5 0.2

Poland 17.8 60.4 8.7 8.3 4.5 0.3

Sweden 23.4 67.2 2.1 6.0 1.2 0.1

OECD Average* 15.7 66.8 6.3 8.0 2.9 0.3




You have decided to apply for the Juice Bar job. Click on the link and read the requirements for

this job. Click on “Apply Now” at the bottom of the Juice Bar job details to open your résumé

page. Complete the “Relevant Skills and Experience” section of the “My Résumé” page by

choosing four experiences from the drop down lists that match the requirements of the Juice

Bar job.


QUESTION INTENT:


- 49 -

Analyse a list of options using predefined criteria

Scoring Comment: Initially each part is coded separately. Final scoring combines codes as

shown below.

Full Credit

Code 2: Selects the following four experiences (in any order):

Efficient at cleaning dishes: working at Corner Restaurant.

Good at following instructions: followed kitchen safety regulations daily.

Knowledge of food handling and preparation experience: work at Corner Restaurant.

Work well with team: won the 2007 sports team player award.

Partial Credit

Code 1: Selects any three of the following four experiences (in any order):

Efficient at cleaning dishes: working at Corner Restaurant.

Good at following instructions: followed kitchen safety regulations daily.

Knowledge of food handling and preparation experience: work at Corner Restaurant.

Work well with team: won the 2007 sports team player award.

No Credit


Code 9: Missing.

- 50 -

Table 17

Item analysis of E012Q03T

Country/Economy

%

No credit Partial

credit

Partial

credit Full credit Missing

Not

Reached

Australia 9.4 35.1 7.9 40.4 6.9 0.2

Austria 19.0 24.3 13.7 24.2 18.5 0.3

Belgium 16.7 30.4 13.6 27.4 11.8 0.1

Chile 21.8 30.8 6.9 8.5 30.7 1.3

Colombia 18.9 21.3 3.7 5.4 48.8 2.0

Denmark 16.4 28.9 10.9 33.5 10.3 0.0

Spain 24.6 30.2 12.3 15.6 16.8 0.3

France 13.9 30.8 17.1 26.3 11.1 0.8

Hong Kong 13.5 36.7 13.2 27.7 8.5 0.4

Hungary 20.0 19.5 12.0 20.6 27.8 0.1

Ireland 11.5 33.6 9.2 33.5 12.1 0.1

Iceland 13.2 28.1 8.2 36.0 14.3 0.1

Japan 10.4 35.7 10.1 35.3 8.4 0.1

Korea 7.2 18.5 11.4 60.1 2.8 --

Macao 21.0 33.3 11.6 27.0 6.7 0.4

Norway 15.9 32.6 13.6 25.2 12.6 --

New Zealand 8.0 36.5 6.6 41.6 7.0 0.3

Poland 21.1 19.7 14.2 25.4 19.3 0.3

Sweden 10.3 22.2 12.0 43.9 11.5 0.1

OECD Average* 15.0 28.6 11.3 31.1 13.9 0.3



Question 3: JOB SEARCH E012Q05 – 0 1 9

“Note: Successful applicants can work a maximum of two shifts per week.”

Why do you think the employer has made this rule?


QUESTION INTENT:


Hypothesise about the reason for including a condition in a job advertisement using prior

knowledge and information from the text

- 51 -

Full Credit

Code 1: Refers (explicitly or implicitly) to a benefit or protection for the employer OR

employee. Must be consistent with the stipulation of not working more than two

shifts and with working a fixed two shifts. May refer to the flexibility, reliability or

effectiveness of the (pool of) employees or to the employer’s concerns about

employee welfare.

It is safer that way because the business can still operate OK if someone is off work for a

few weeks.

Students often have other priorities at those times. [“Those times” refers to the shifts in

the advertisement. Implies benefit to employee.]

It is unlikely most students can do more than 2 shifts a week.

They don’t want to rely on any one person. [implied protection from risk]

They say that at the start in case you’re not very good.

They want lots of different people working there.

They want lots of happy faces.

They don’t want you to get tired.

Because it's a tough job, and they don't want you to get tired and quit.

Because they want a big staff in case someone quits or gets sick.

Because the chaos at the Juicebar is too much for anyone more often than twice a week.

Because the best workers are people with other interests/hobbies than the job, and they

want you to keep doing what you like.

So students and other people who may be studying or holding down other jobs can still

work casually but don't have the restrictions of working all day every day.

No Credit

Code 0: Refers to two shifts as a minimum rather than a maximum requirement.

The employer has made this rule because if the employee was only working one shift, they

may not get used to the work as easy and be less efficient at the job. [Does not deal with

idea of maximum of two shifts.]

Gives insufficient or vague answer.

It is safer that way. [vague]

Shows inaccurate comprehension of the material or gives an implausible or

irrelevant answer.

The employer may have made this rule so employers can have equal shifts. [inaccurate

comprehension. The test does not imply that all employees should have an equal number

of shifts.]

- 52 -

So you don’t make too much money. [implausible]

So they don’t work overtime. [implausible]

So you can work different days each week, which is more flexible. [inaccurate

comprehension]

Code 9: Missing.

Table 18


Country/Economy %

No credit Full credit Missing Not Reached

Australia 31.5 61.1 7.1 0.2

Austria 33.8 42.2 23.7 0.3

Belgium 36.6 49.9 13.4 0.1

Chile 42.2 38.4 17.2 2.2

Colombia 46.8 31.4 18.6 3.2

Denmark 59.2 29.2 11.3 0.2

Spain 35.5 50.4 13.4 0.7

France 38.8 43.2 17.1 0.9

Hong Kong 45.6 37.4 16.5 0.6

Hungary 21.3 57.4 21.0 0.2

Ireland 41.7 46.2 11.9 0.2

Iceland 32.6 54.4 12.8 0.1

Japan 32.6 43.4 24.0 0.1

Korea 44.7 46.5 8.8 --

Macao 58.1 25.3 15.8 0.7

Norway 23.6 63.1 13.3 --

New Zealand 27.8 62.9 9.0 0.3

Poland 28.2 46.6 24.9 0.3

Sweden 34.8 54.1 11.0 0.2

OECD Average* 35.3 49.3 15.0 0.4



- 53 -

OECD PISA acknowledges the following sources for the use of their material in this assessment.

Every effort has been made to trace the original source material contained in the assessment.

Where the attempt has been unsuccessful, the PISA project consortium would be pleased to

hear from the author/publisher to rectify any omission.

Adapted from:

http://www.aclimages.net/images/thumbnails.php?album=39

Access date: July 2007

Adapted from:

http://www.aclimages.net/images/thumbnails.php?album=39&page=3


Adapted from:



http://www.aclimages.net/images/thumbnails.php?album=39



- 54 -

References

OECD (2009). PISA 2009 assessment framework: Key competencies in reading,

mathematics and science literacy, OECD Publishing.

OECD (2010a). PISA 2009 results: What students know and can do? Student

performance in reading, mathematics and science (Volume I), OECD

Publishing.

OECD (2010b). PISA 2009 results: Overcoming social background -- Equity in

learning opportunities and outcomes (Volume II), OECD Publishing.

OECD (2010c). PISA 2009 results: Learning to learn -- Student engagement, strategies

and practices (Volume III), OECD Publishing.

OECD (2010d). PISA 2009 results: What makes a school successful? Resources,

policies and practices, (Volume IV), OECD Publishing.

OECD (2010e). PISA 2009 results: Learning trends -- Changes in student performance

since 2000 (Volume V), OECD Publishing.

OECD (2011). PISA 2009 results: Students on line -- Digital technologies and

performance (Volume VI), OECD Publishing.

digital reading assessment in macao-china pisa 2009 reports... · - 6 - executive summary macao...

Documents