Young Lives School Surveys

Research Update, August 2016

Padmini Iyer, Rhiannon Moore & Jack Rossiter

Young Lives, University of Oxford

PRESENTATION OVERVIEW

Overview of YL school surveys School surveys 2016-17: research

questions School effectiveness research design Survey timeline Sampling approach School survey instruments

• Background questionnaires• Cognitive tests

YOUNG LIVES SCHOOL SURVEYS, 2010 – 2017

SCHOOL SURVEYS 2016-17: RESEARCH QUESTIONSBenchmarking levels of attainment & progress in key domains What are students’ levels of

performance in core curricular and transferable skills domains (Maths, Functional English and Problem Solving), and how do levels compare across and between contexts?

How do students progress over time (in one academic year & since previous surveys)?

What is the nature of inequalities in attainment & progress, and how are these shaped over time?

School & teacher quality and effectiveness What is the role of key

dimensions of school quality (incl. inputs, leadership, organisation & management) in shaping educational outcomes over time?

How is access to educational quality distributed (in terms of equity) and which policies explain or may be used to improve this position?

Which teacher characteristics & practices are associated with improved learning outcomes?

SCHOOL SURVEYS 2016-17: RESEARCH QUESTIONS

Transitions Which factors and characteristics at

student and school levels explain timely progression through educational careers, and specifically transition into secondary schooling?

SCHOOL SURVEYS 2016-17: RESEARCH QUESTIONS

SCHOOL EFFECTIVENESS RESEARCH DESIGNStudent outcome measures Background instruments

Maths test Repeated measures; testing mathematical knowledge, application and reasoning skills

Principal questionnaire Background data on the principal; school-level information

Functional English test Repeated measures; testing reading and comprehension skills

Teacher questionnaire Background data on Maths & English teachers; class-level information

Transferable Skills testCross-sectional measure; testing problem-solving and critical thinking skills

Student questionnaire Background data on students; psycho-social measurements

 School facilities observationData on school infrastructure

 

School rosters School, teacher, class and student rosters to ensure identification of students and linking of data across levels

SCHOOL SURVEYS: TIMELINEINDIA VIETNAM ETHIOPIA

2016

JuneItem selection: Maths and English

Fieldworker training

July Wave 1 data collectionFieldworker trainingAugust

September Wave 1 data collection

Fieldworker training

OctoberWave 1 data collectionNovembe

r Wave 2 pilotDecember

2017

January Fieldworker training Wave 2 pilot

FebruaryWave 2 data collection

Wave 2 pilot

March Fieldworker training & Wave 2 data collectionApril Fieldworker training &

Wave 2 data collectionMay

SAMPLING: ETHIOPIA

30 sites• 20 YL • 10 more from Round 2

school survey School census in each

site Includes 62 primary

schools offering G7/G8 Approx. 12,000

students Around 3,000 students

linked from Round 2 school survey in 2012/13

100-200 YL children

Sample design to explore school choice available in each of the 20 Young Lives sites

Sample stratified by 4 school types: • State government schools• Tribal/Social Welfare schools• Private Aided schools• Private Unaided schools

Number of schools sampled in each site proportional to the total number of schools in that site:Total number of schools in a site Proportion sampled

> 80 schools 10% sampled

21-80 20% sampled

8-20 schools 50% sampled

<8 schools 100% sampled

(Exception: less prevalent school types are oversampled)

SAMPLING: INDIA

SAMPLING: INDIA 20 Young Lives sites 212 secondary schools

• 83 State Government• 41 Tribal/Welfare• 31 Private aided• 57 Private unaided

All Class 9 students in sampled schools: around 12,000 students• 7,856 in Telugu medium

schools• 4,164 in English medium

schools• 200 in Urdu medium schools

119 Young Lives children expected to be in Class 9 at our sampled schools• Attending 60 schools• 18 sites (none in Site 3 or

16)

SAMPLING: VIETNAM

20 Young Lives districts55 upper secondary schoolsMaximum of five Grade 10 classes

per school• Random selection process for schools

with six or more Grade 10 classes

~9,000 Grade 10 students~1,000 Young Lives children

expected to be in Grade 10 in 2016-17 • …but maximum number of classes

means we may not include all of them

• Any primary school survey partcipants (non-YL) to be identified retrospectively

BACKGROUND QUESTIONNAIRESWave 1 background questionnaires

1. Head Teacher Questionnaire: HT background: experience, qualifications & training School background: School type, management, fees &

funding Information on teachers: numbers; absence; means of

rewarding/disciplining teachers

2. Student: Student background: ethnic background/language; meals per

day; health problems Home environment: asset index; family size; parents’ literacy

and education; student work; parental support School: travel to school & residence during school term; use

of toilets at school; use of computer at school; school meals; school choice; education history and repetition of grades; educational aspirations; exposure to learning English

COGNITIVE TESTS: MATHSAssessing Maths knowledge and skills Country-specific content domains,

led by curriculum E.g. Ethiopia’s Minimum Learning

Competencies for Grades 5-8

Cross-country cognitive domains TIMSS 2015 framework

• Knowledge• Application• Reasoning

basic number competency

integers, rational numbers, powers and bases

fractions, decimals, ratios and percentages

area, perimeter, volume and surface area

geometry and shapes algebra

measurement charts and graphs

reasoning and problem solving

COGNITIVE TESTS: MATHSPriority Finely graded test scores in each

countryOption Common scale – taking advantage of

substantial content-domain overlap between countries

Bethlehem has balanced some bags of marbles. All the marbles are of the same weight. The number of marbles in each bag is written on it.

How many marbles are there in the bag marked M?

A. 13B. 17C. 33D. 49

COGNITIVE TESTS: ENGLISHThe English test is intended to… Measure the construct of ‘functional’

English, based on the tasks which learners need to complete or may need to complete in their future;

Focus on the types of English language knowledge and skills which students may be exposed to:• in school;• at home and in their wider

community;• in the future.

Be suitable for children studying in different mediums of instruction

Be suitable for use across different contexts

Link to the CEFR international framework for language learning, with clear ‘can-do’ indicators

Developed in partnership with Educational Initiatives Items benchmarked against other Indian studies CEFR grades A1 (basic user) – C2 (proficient user)

COGNITIVE TESTS: ENGLISH

Areas covered by the English test:

word identification and vocabulary

word meaning and contextual vocabulary

sentence comprehension and construction

reading comprehension

Identifying the meaning of words relating to a range of topics, for example school and employment

For example, questions requiring children to identify the meaning of an unfamiliar word based on the context in which it is used

For example completing a sentence using an appropriate word

Different types of text, for example posters, stories, non-fiction texts

CEFR level: A2Sentence construction and comprehension

COGNITIVE TESTS: ENGLISHCommon items across Ethiopia, India and Vietnam Around 30 common items across the three countries for Wave 1

• More common items between Ethiopia and Vietnam • Fewer ‘easy’ items and additional ‘harder’ items in India

For example:

India pilot: 81% correct

Ethiopia pilot: 62% correct

Vietnam pilot: 57% correct

India pilot: 41% correct

Ethiopia pilot: 31% correct

Vietnam pilot: 39% correct

CEFR level: A2Sentence construction and comprehension

COGNITIVE TESTS: TRANSFERABLE SKILLS

Assessing Transferable Skills Problem solving, critical thinking,

communication, teamwork (World Bank 2014)

Transferable skills test at Wave 2

Cross-curricular problem solving: • ‘An individual’s capacity to use

cognitive processes to resolve real, cross-disciplinary situations where the solution path is not immediately obvious’ (PISA 2003)

PISA (2003) problem-solving assessments• Suitable for 15-year-olds• Paper & pen tests

Pre-piloted in Ethiopia, India, Vietnam

STUDENT TESTS: TRANSFERABLE SKILLSLevel 2 Problem Solving (‘Reasoning, decision-making problem solvers’)

VN pre-pilot results (N = 57): 51%ET pre-pilot results (N = 24): 31%

PISA 2003 results: 52%

COGNITIVE TESTS: TRANSFERABLE SKILLS

Next steps Exploring critical thinking

assessments Alternative problem-solving

assessments Full-scale pilot of problem-solving &

critical thinking items

Any questions / comments?