english adaptive assessment in the uruguayan educational system, 2017 … · 2021. 4. 14. · a2...

English Adaptive Assessment in the Uruguayan

Educational System, 2017

Report results

Departamento de Segundas Lenguas y Lenguas Extranjeras – CEIP Inspección de Inglés – CES Inspección de Inglés - CETP

Programa Ceibal en Inglés - Plan Ceibal Departamento Monitoreo y Evaluación - Plan Ceibal

División de Investigación, Evaluación y Estadística - CODICEN

British Council Consultor: Mario Luzardo

Contents

2017 English Adaptive Assessment 2

Introduction 2

Assessment Objectives 2

Coverage of English Teaching 3

Coverage of the Assessment 3

Assessment Design 4

Students’ Results in Primary Education 5

Results according to the Sociocultural Context 6

Results according to the Program 8

Results in Sixth Grade 10

Results in the 2017 edition 10

Results in compared years (2014-2017) 14

Students’ Results in Middle School Education 16

Conclusions 17

Annex 1 – Descriptors 18

English Adaptive Assessment

Introduction

The fourth edition of the English adaptive test was carried out during November 2017.

The evaluation was directed to all students from 4th to 6th grade of Primary Education who

learn English in face-to-face mode (Program of the Second and Foreign Languages Department)

or by videoconference (Ceibal en Inglés Program), students of 1st grade of Middle School, and

for the first time, it was extended to students in the 2nd grade of Middle School.

The design, elaboration and administration of the test is the result of the joint effort of Ceibal

en Inglés program of Plan Ceibal; the Second and Foreign Languages Department of CEIP; the

English Supervisors from CES and CETP; the Research, Evaluation and Statistics Division of

CODICEN; Plan Ceibal’s Supervision and Evaluation Department and The British Council,

together with technical experts

The test is adaptive in nature, i.e. it adjusts to the level of English proficiency of each student

who takes it. This kind of assessment uses the activities from an items bank in order to assess

students’ performance. It has a dynamic modality, which means that the activities that each

student answers are graded according to their answers. In this way, each student goes through

a set of activities according to the performance that he or she is demonstrating. Depending on

whether they answer a question correctly, the test offers a question of greater or lesser

difficulty. It is a very appropriate tool to evaluate learning from a very diverse population in

their sociocultural contexts, years of studying English, etc.

As in previous editions, the test was administered online through the SEA platform and it

comprised three components: a reading comprehension component (vocabulary, grammar and

reading comprehension as such), a listening comprehension component and finally, a writing

component. The first two components of the test are of adaptive modality and automatic

correction, while the third component of the test is not adaptive and its correction is

performed by a team of teachers after the implemention.

Assessment Objectives

The assessment objectives are as follow:

• To assess the English levels reached by students in 4th, 5th and 6th Grade of Primary

Education (both in face-to-face and videoconference lessons) and in 1st and 2nd Grade of

Middle School Education.

• To generate an ongoing assessment plan that allows to see the inter-annual progress in

the English learning process.

• To contribute to the construction of the ongoing English learning throughout the

educational journey.

• To provide students who sit for the test with a certificate that attests the results

obtained.

Coverage of English Teaching

In the previous edition of the test, the coverage of English teaching in 4th, 5th and 6th grade

students of Primary Education reached 93% of the country's urban schools. In 2017, this

coverage amounts to 94%. Of the 106,881 students that make up the universe of urban school

students who study English, 74,842 (70%) do so by videoconference (Ceibal en Inglés Program),

and 32,039 (30%) through Segundas Lenguas program.

In Middle School Education, the coverage of English teaching is total due to the compulsory

nature of the subject in the curricula. However, there may be groups without English, because

they haven’t been allotted since no teachers have chosen them.

Coverage of Assessment

58,082 students of Primary Education participated in the adaptive test, which represents a

response rate of 54% in reference to the 106,881 students that constituted the potential

universe. In the case of Middle Education, 12,183 students participated, which represented a

response rate of 12%

In the following graphs, it can be seen that, as in the previous edition, the response in Primary

Education was relatively balanced according to sociocultural context or school year.

Graph 1. Distribution according to the school’s sociocultural context – 2015

(% of students from 4th

to 6th

grade of Primary School

who took the test vs. target population)

Level 1 Level 2 Level 3 Level 4 Level 5

More critical

Target Pop. Assessment

19 % 17 % 19 % 15 % 18 % 20% 20% 23% 25% 20%

Source: Data of Primary students of ANEP and Base of adaptive assessment 2017. Graph 2. Distribution according to school year

(% of students from 4th to 6th grade of Primary School)

33% 35% 33% 33% 33% 32 %

Source: Data of Primary students of ANEP and Base of adaptive assessment 2017.

Assessment Design

The activities in the test were designed by a specialized unit from The British Council and

gauged in a representative sample of education centers. The same item bank (1765 in the 2017

edition) is used in Primary and Middle School Education, adjusting to the performance level of

each student.

The test was administered in a way that it allowed students to begin with the reading

comprehension component (Vocabulary, Reading and Grammar), then move on to the listening

comprehension one and finally do the written production task (Writing).

In the first component students answered multiple-choice questions and they could score

between 0 and 1.500 points. In the second component, students listened to brief audio files

and then answered a multiple-choice question. The score scale ranged from 0 to 1.500 points.

Both scoring scales were rated in six levels (A0, A1-, A1 +, A2-, A2 + and B1) associated with the

performance levels of the Common European Framework of Reference for Languages (CEFR)

with their respective achievement descriptors. The third and final part of the test consisted of a

written production task. From an image and a series of prompts, the student had to produce a

written production that was subsequently corrected from a six-level rubric (A0, A1-, A1 +, A2-,

A2 + and B1)1.

In the case of the adaptive tests, the estimation of students’ level of English was reported in

real time. Once the test was finished, the teacher could access the results of his students in the

first two modules of the test through the SEA platform. Written productions were corrected

and marked by a team of teachers between December 2017 and February 2018.

1 See Annex 1

4 º 5 º 6 º Target Pop. Assessment

Students’ Results in Primary Education

Graph.3 Vocabulary-Reading-Grammar according to school year

(% of the students who took the test)

Source: Base of adaptive assessment 2017.

Graph.4 Listening- according to school year



4 5 6 3 % % 4 % 8

% 18 % 20 % 24

% 43 47 %

48 %

% 18 17 %

% 12 17 % 11 % % 8 1 % 1 % 0 %

B1 A2+ A2 A1+ A1 A0

4 5 6

% 8 % 8 % 11

% 26 % 30 % 31

% 20 25 % % 24 10 % 10 % 9 %

% 26 % 21 % 17

% 11 7 % 8 % B1 A2+ A2 A1+ A1 A0

Graph.5 Writing- according to school year (% of the students who took the test)

Note: The students who took the test did not reach B1 level and only 0.1% reached A2 + level. In order to

better visualize the data, the A2 + level was not included in the graph.


When looking at the results by school grade, we see that the levels obtained by the primary

school students who took the test are better at the higher school level in the two adaptive

components. The same happens with the results of the writing test. It is worth clarifying that

the increase that had been generated in the A0 level in the listening test in the 2016 edition

was corrected. On that occasion, audio problems in some devices had been detected during the

administration of the test (mainly concentrated in 4th and 5th grade).

Results according to the Sociocultural Context

When analyzing the results by sociocultural context of the educational center, it is observed

that in all the contexts the students reach the six levels of the scale, and that the distribution in

the students that do not demonstrate knowledge of English (A0), is similar in all contexts. In any

case, the less critical the context of the school is, the better the students' performance.

4 5 6 % 1 % 2 3 % 10 % % 15 20 %

24 % % 27

% 29

% 65 57 % 48 %

(%of the students who took the test

A2 A1+ A1 A0

Graph 6. Vocabulary-Reading-Grammar according to context (Primary)


critical


Graph 7. Listening according to context (Primary)


critical


Level 1 more Level 2 Level 3 Level 4 Level 5 less critical % 4 4 % 5 % 5 % 6 % % 19 % 19 19 % % 21 22 %

% 40 % 42 % 45 % 47 % 53

19 % % 19 % 17 % 15 % 12 % 17 % 15 13 % 10 % 7 %

% 1 % 1 1 % 1 % 0 %

B1 A2+ A2 A1+ A1 A0

Level 1 more Level 2 Level 3 Level 4 Level 5 less critical

7 % % 8 % 8 % 9 % 11

% 26 27 % % 28 % 29 33 %

20 % % 22 % 22 24 % 24 % 9 % % 9 10 % 9 %

% 9 25 % 24 % % 23 % 21 % 17 % 13 % 9 9 % % 8 % 7

B1 A2+ A2 A1+ A1 A0

Graph 8. Writing according to context (Primary)

critical




Results according to the Program

Graph 9. Vocabulary-Reading-Grammar according to the Program (Primary)



Level 1 more Level 2 Level 3 Level 4 Level 5 less critical % 1 1 % 1 % % 2 3 %

9 % 12 % % 13 % 16 21 %

% 20 % 24 % 26 % 28

% 31

% 70 % 62 % 59 % 55 % 45


A2 A1+ A1 A0

Ceibal en inglés Segundas Lenguas

5 % % 5

% 21 % 19

% 46 46 %

16 % 17 %

% 11 % 13 1 % % 1

B1 A2+ A2 A1+ A1 A0

Graph 10. Listening according to the Program (Primary) (% of the students who took the test)


Graph 11. Writing according to the Program (Primary)





.

The previous results show that there are no differences in learning between the students who

have class by videoconference and those who learn by face-to-face mode in both the adaptive

components and in the written production test.

Ceibal en inglés Segundas Lenguas

% 8 % 9

% 29 29 %

22 % % 23

9 % 9 %

% 22 % 21

9 % % 9

B1 A2+ A2 A1+ A1 A0

Ceibal en Inglés Segundas Lenguas 1 % 2 %

14 % 16 %

27 % 26 %

% 58 55 % A2 A1+ A1 A0

Results in Sixth Grade

Results in the 2017 edition

It is especially interesting to analyze the results of the 6th grade of primary education. As it is

known, both programs differ in their teaching modality and exposure time. Ceibal en Inglés

Program has a maximum of three years beginning in the 4th grade of Primary Education, while

Segundas Lenguas Program works up to a maximum of seven years with the students, starting

at the Preschool Level (5 years old) and until 6th grade.

The graduation goal for both programs is A2 level according to the Common European

Framework. 2 This framework provides a detailed description of the competences, as well as

the knowledge and necessary skills to communicate in a foreign language.

In the following graphs it can be seen that 79.5% of the 6th grade students who took the test

reached A2 or higher in the first part of the test (Vocabulary Reading-Grammar). The results in

the listening test worked in the same direction; 65.9% of the students reached A2 or higher.

Graph 12. Vocabulary-Reading-Grammar – students of 6th grade of Primary School

(% of the students of 6th grade who took the test)


2 The Common European Framework indicates that at an A2 level the student is able to understand phrases and

expressions of frequent use related to areas of expertise that are especially relevant. The student knows how to

communicate when carrying out simple and daily tasks that do not require more than simple and direct exchanges

of information on issues that are known or usual. The student can describe in simple terms aspects of their past and

their environment as well as issues related to their immediate needs.

A0 A1- A1+ A2- A2+ B1

0.4 7 , 7 12.3

48.2

23.8

7.5

Graph 13. Listening - students of 6th grade of Primary School

(% of the students of 6th

grade who took the test)

31.5

A0 A1- A1+ A2- A2+ B1


Graph 14. Writing – students of 6th grade of Primary School



Note: The students who took the test did not reach B1 level


Following the situation of previous editions of the test, the results achieved by the students

of both modalities were very similar in the two adaptive components as well as in the

written production test.

8.2 17.3

8.6

23.5

10.9

A0 A1- A1+ A2- A2+

48.0

29.1 20.0

2.7 0.2

Graph 15. Vocabulary-Reading-Grammar according to the Program




Graph 16. Listening according to the Program

(% of the students of 6th grade who took the test)


Ceibal en Inglés Segundas Lenguas

% 8 % 7

25 % 21 %

47 % 50 %

% 12 % 14

% 7 % 9 0 % 0 %

B1 A2+ A2 A1+ A1 A0

Ceibal en Inglés Segundas Lenguas

11 % % 11

% 32 % 31

% 23 24 %

8 % 9 %

% 18 % 17

8 % % 8

B1 A2+ A2 A1+ A1 A0

Graph 17. Writing according to the Program (% of the students of 6th grade who took the test)

Note: The students who took the test did not reach B1 level and only 0.2% reached A2 + level. In order to better visualize

the data, the A2 + level was not included in the graph.


The percentage of students who fail to demonstrate English knowledge is similar in all contexts.

However, it is possible to observe that the less critical the context of the school is, the better

the student performance (88% A2 or more in level 5 vs. 72% in level 1) in Vocabulary-Reading-

Grammar.

Ceibal en inglés Segundas lenguas

2 % % 3 % 20 % 20

% 29 29 %

% 49 % 47 A2 A1+ A1 A0

Graph 18. Vocabulary-Reading-Grammar according to the context



critical


Results in compared years (2014-2017) It is interesting to take a look at the results achieved by the 6th grade students who took the test

in the four 2014-2017 editions. As it is possible to appreciate in the graph, the learning curve has

a great jump in 2015 year in which 65% of the 6th grade students that took the test achieved an

A2 level or higher in the reading comprehension test; that is, 9% more than the results obtained

by the 6th grade in 2014. In 2016 it is also possible to observe another jump, in this case of 11%.

Graph 19. Vocabulary-Reading-Grammar in students of 6th grade who took the test in 2014-2017 editions

(% of the students who got in the test A2 or +) 85 %

Source: Base of adaptive assessment (2014-2015-2016 and 2017).

Level 1 more Level 2 Level 3 Level 4 Level 5 less critical

% 6 6 % 6 % 8 % 10 %

% 22 % 22 22 % % 26 26 %

% 44 % 46 % 48 % 49

% 52

16 % % 15 % 14 % 11

% 8 10 % % 10 % 9 7 % 4 % % 1 % 1 1 % 0 % 0 %

B1 A2+ A2 A1+ A1 A0

2014 2015 2016 2017 50 %

55 %

60 %

65 %

70 %

75 %

80 %

56 %

65 %

76 % 79 %

Graph 20. Vocabulary-Reading-Grammar in students of 6th grade who took the test in 2014-2017 editions according to the school context

(% of the students who got in the test A2 or +)

100.0 %

2014 2015 2016 2017

Source: Base of adaptive assessment (2014-2015-2016 and 2017).

As it happens in the general level test, in the previous graph it can be observed that when we

look at the results by the socioeconomic context of the school, the 6th grade students who get

higher levels of learning in the reading comprehension test, are students from less vulnerable

contexts. Anyway, one of the most important learning outcomes can be seen at the most

critical economic level. When comparing the results of the 6th grade students who took the test

in 2014 and 2017, an increase of 27 percentage is observed in A2 level or higher.

Graph 19. Vocabulary-Reading-Grammar in students of 6th grade who took the test in 2014-2017 editions according to the teaching modality

Ceibal en Inglés Segundas Lenguas Source: Base of adaptive assessment (2014-2015-2016 and 2017).

In the same way it happens when looking at the results distinguishing the teaching modality. In

each new edition, the 6th grade students who took the test reached better performance in both

modalities.

Level 1 more critical Level 2 Level 3 Level 4 Level 5 less critical % 40.0

% 50.0

% 60.0

% 70.0

80.0 %

90.0 %

44.0 % % 47.0

52.2 % 55.7 %

70.4 %

% 52.7 58.5 %

63.1 % % 69.8

% 77.8 % 69.2 71.1 % % 72.8

% 77.3 % 83.2

% 71.6 % 72.9 % 78.2

% 82.2 87.8 %

2014 2015 2016 2017 % 50.0

55.0 %

% 60.0

% 65.0

% 70.0

% 75.0

80.0 %

% 85.0

% 90.0

56.6 %

% 68.2

% 78.0 % 80.7

54.6 %

% 60.5

% 70.2

% 77.4

(% of the students who got in the test A2 or +)

Students’ Results in Middle School Education

For middle school students, it is the second year that the adaptive test is implemented. In this

edition it is extended for the first time to the 2nd grade students. As in the 2016 edition, it is

necessary to emphasize that the number of students who took the test is not representative of

the total number of students in 1st and 2nd year of Middle School education.

As it can be seen in the following graphs, 82% of the students in 1st grade of Middle School that

took the test reached A2 or higher in reading comprehension, while 75% achieved this level in

the listening test. In the case of the 2nd grade of Middle School, 89% of them reached A2 level

or more and 80% achieved it in the second part of the test. With respect to the writing

component, we observe that the 2nd grade students got results significantly better than the 1st

grade students.

Graph.20 Vocabulary-Reading-Grammar in students of Middle School (% of the students who took the test)


Graph.21 Listening in students of Middle School



A0 A1- A1+ A2- A2+ B1

0 5

13

54

22

6 0 3

9

55

24

10

1 º 2 º

A0 A1- A1+ A2- A2+ B1

4

13 8

26

36

13

4 9 7

26

38

16

1 º 2 º

Graph.22 Writing in students of Middle School



Conclusions

The fourth edition of the adaptive English assessment took place between November and

December 2017. The test was available to students in 4th, 5th and 6th grade of Primary

Education; 1st grade of Middle School and for the first time it was extended to 2nd grade

students of Middle School Education.

As in previous editions, the results show achievements in learning English in Primary Education

in the two teaching-learning modalities. When looking at the results by school grade, the

results are better at the higher school grade in the two adaptive components. The same

happens with the results of the writing test. In the same way, the less critical the context of the

school is, the better the performance of the students is.

With the aim of contributing to the construction of ongoing English learning throughout the

educational journey, it is especially interesting to analyze the results of the 6th grade students

of Primary Education. 79.5% reached A2 or higher in the reading comprehension test. The

results in the listening test worked in the same direction; 65.9% of the students reached A2 or

higher. With regard to written production, only 3% reached A2 or higher.

In this edition, we incorporate a look at the results achieved by the 6th grade students who took

the test in the four 2014-2017 editions. The learning curve has a great leap in 2015, when 65%

of the 6th grade students who took the test scored A2 or higher on the reading comprehension

test; that is, 9% more than the results obtained by the 6th graders in 2014.

Regarding Middle School Education, the students who took the test showed a higher

performance than the elementary students. In any case, they do not constitute a representative

sample of the 1st and 2nd grade students of Middle Education due to the low participation rate.

82% of the students of 1st grade of Middle School reached an A2 level or higher in reading

comprehension, while 75% reached this level in the listening test. In the case of the 2nd grade

students of Middle School, 89% of them reached an A2 level or higher and 80% achieved it in

the listening test. With respect to the writing component, we observe that the 2nd grade

students got results significantly better than the 1st grade students.

A0 A1- A1+ A2- A2+ B1

49

30

18

3

43

29 22

5 1

1 º 2 º

ANNEX

Annex 1 - Descriptors

Table 1. Descriptors and Levels. Test: Vocabulary - Reading – Grammar

Level Category Descriptor Min Max

A0 IN PROGRESS

The student is in the process of learning English but

requires more time to achieve progress in the

language. 0 322

A1- BEGINNER

LOW

The student demonstrates initial understanding of

basic vocabulary and very simple sentences and

grammar structures in English related to him or

herself and his/her family and surroundings.

322 364

A1+ BEGINNER

HIGH

The student demonstrates a clear understanding of

basic vocabulary and simple sentences and grammar

structures in English related to him or herself and

his/her family and surroundings.

364 384

A2- ELEMENTARY

LOW

The student differentiates and demonstrates

understanding of various common expressions in

English and recognizes some of the most frequent

and essential grammar structures related to him or

herself and his/her activities, preferences and

surroundings.

384 481

A2+ ELEMENTARY

HIGH

The student differentiates and demonstrates a

consolidated understanding of various common

expressions in English and recognizes the most

frequent and essential grammar structures related to

him or herself and his/her activities, preferences and

surroundings.

481 625

B1 INTERMEDIATE

The student differentiates and demonstrates a broad

understanding of expressions and brief texts about

different topics familiar to him/her (studies, free time

activities, his/her areas of interest and everyday

surroundings). He/she recognizes the most frequent

grammar structures in the different verb tenses.

625 1500

*The parameters are taken as follows: the level is greater or equal to the Initial and lower than the final.

Table 2. Descriptors and Levels. Test: Listening

Level Category Descriptor Min Max

A0 IN PROGRESS

The student requires more exposure to the

language to develop his/her listening

comprehension skills in English. 0 338

A1- BEGINNER LOW

The student demonstrates initial listening

comprehension ability in English. He/she

understands greetings and very basic

dialogues about topics familiar to children

his/her age. He/she understands basic

descriptions worded in a clear and slow

manner and which contain specific

references to color, size and shape.

338 373

A1+ BEGINNER

HIGH

The student demonstrates initial listening

comprehension ability in English. He/she

understands greetings and simple

dialogues about topics familiar to children

his/her age. He/she understands basic

descriptions of physical surroundings and

people, worded in a clear and slow manner.

373 386

A2- ELEMENTARY

LOW

The student demonstrates listening

comprehension ability in English of simple

messages and exchanges which contain

expressions and topics frequent to children

his/her age, such as the weather, clothes

and preferences. He/she understands

simple descriptions of physical

surroundings and people.

386 428

A2+ ELEMENTARY

HIGH


comprehension ability in English of

messages and exchanges related to

situations and topics familiar to children

his/her age. He/she understands simple

descriptions of situations, physical

surroundings and people.

428 558

B1 INTERMEDIATE


comprehension ability in English of the main

ideas in long texts and dialogues about

everyday topics related to his/her

surroundings, activities, studies and free

time, as well as explanations, opinions and

plans.

558 1500

*The parameters are taken as follows: the level is greater or equal to the Initial and lower than the final.

Table 3. Descriptors and Levels. Test: Writing

Level Category Descriptor Score

A0 IN PROGRESS

The student is in the process of learning how to

write in English but requires more time to achieve

progress in written production. 0-1

A1- BEGINNER LOW The student can write isolated words in English

and is in the process of starting writing sentences

about very familiar topics. 2

A1+ BEGINNER

HIGH

The student can write very basic sentences in

English about very familiar topics and can also give

very basic information about him or herself and

somebody else.

3

A2- ELEMENTARY

LOW

The student can write sentences in English about

familiar topics, including some detail and making

proper use of syntax and vocabulary. 4

A2+ ELEMENTARY

HIGH

The student can write different sentences in

English about a range of familiar topics, including

more detail and making a correct use of syntax

and vocabulary. He/she can add cohesion devices

to his/her writing.

5

B1 INTERMEDIATE

The student can write brief texts in English about a

range of familiar topics, including more detail,

offering explanations and opinions and making a

correct use of syntax, a wide vocabulary and

cohesive devices.

6

english adaptive assessment in the uruguayan educational system, 2017 … · 2021. 4. 14. · a2...

Documents