operational research on literacy boost and multilingual · pdf fileliteracy boost and...

Literacy Boost and Multilingual Education! Kapilvastu, Nepal

!!Operational Research on

Literacy Boost and Multilingual Education!

!Kapilvastu District, Nepal!!

July 2014!!

Miranda Weinberg!!Save the Children - University Partnership for Educational Research Fellow!!

[email protected]!!!Acknowledgments:!!I gratefully acknowledge and thank the support of many people who supported the research and writing that went into this report:!Amy Jo Dowd, Save the Children USA!Colin Alfred, Save the Children International!Gopini Pandey, Tara Bajracharya, Narayan Kafle, Suman Rai, Seema Baral and others, Nepal Country Office!Subas Yadav Nar Maya Thapa and others, Nepal Western Regional Office!Ajay Chaudhary and others at KSSC!Shiv Lal Yadav, Saraswati Paudyal and others at SSDO!Bhupendra Kumar Badhai, Bikram Gurung, Ghanashyam Bhar, Habib Nau, Laxmi Narayan Yadav, Manisha Khanal, Nisha Puri, Ram Narayan Yadav, Shiv Kumar Badhai, Sita Baral, Sita Belbase, Umesh Kewat, Enumerators!Ameena Ghaffar-Kucher, University of Pennsylvania!The many teachers, parents, and children who graciously took the time to answer my questions, allowed me to watch their classes, and completed challenging reading assessments

�1

mailto:[email protected]


Executive Summary!!Save the Children began sponsorship-funded programming in the Kapilvastu District in Western Nepal in 2010. This report focuses on two elements of the Basic Education component of the Common Approach to Sponsorship Programming: Literacy Boost (LB) and Multilingual Education (MLE). Both of these approaches are focused on improving the quality of education in early grades.!!In 2014, SC Nepal, with the support of SUPER Fellow Miranda Weinberg and in collaboration with two partner NGOs (PNGOs), conducted a mixed methods study of these programs. This study was part of the newly developed operational research program of SC Nepal, which has a goal of conducting research in ongoing programs to identify gaps and best practices. The quantitative strand of the research used the Literacy Boost reading assessment to assess early grades literacy and numeracy in Class 2 students in their third month of the school year. The qualitative strand of the research incorporated classroom observation, teacher interviews and parent focus groups at a subset of the schools where reading assessments were conducted.!!The twenty schools in the study were chosen in collaboration with partner NGO (PNGO) staff. They comprised 5 MLE schools, 5 LB schools, and 10 comparison schools. Comparison schools were selected by the two PNGOs to be similar to the intervention schools from VDCs that neighbor intervention schools but where there has been no Save the Children support.!!A major finding of this study is that both models (LB and MLE) improve instruction and learning outcomes, but could be improved by a model that draws from the strengths of each approach. In MLE schools, Awadhi-medium instruction improves some aspects of quality teaching but does not address all elements included in LB. LB has begun to be introduced in all MLE schools starting this school year, which should help address this issue. LB schools in largely Awadhi-speaking communities could benefit from Awadhi instruction, as observations and interviews show that students do not understand instruction in Nepali.!!The four groups of schools had significant differences in background characteristics, making direct comparisons across groups impossible. However, in general, intervention groups seemed to outperform comparison groups. Literacy Boost schools performed especially well on reading of connected text, and in particular answering comprehension questions. Reading assessments demonstrate that many students, especially at MLE schools but also at LB schools, had not gained fundamental literacy skills, such as recognizing letters or commonly used words. However, comparing these results to the 2014 Nepal EGRA study , both MLE and LB schools 1

had much lower proportions of students unable to read a single word than the national or regional averages.!!An additional finding from the reading assessments was the need for immediate technical support for future reading assessments. There were issues at many levels of the assessment process, from the tool itself to sampling strategies and delivery of the assessment. As tools with the same set of tasks, though slightly different items, is used twice a year for planning and

�2

RTI International (2014). Nepal education sector early grade reading assessment report. 1

Research Triangle Park, NC: RTI International. Online at https://www.eddataglobal.org/countries/index.cfm?fuseaction=pubDetail&ID=601

https://www.eddataglobal.org/countries/index.cfm?fuseaction=pubDetail&ID=601


assessment purposes in the Literacy Boost program, the capacity of Save the Children and Partner NGOs to conduct these assessments is crucial.!!The qualitative side of the project had several key findings regarding these program models. At Nepali-medium schools in this study, teaching Awadhi L1 students in Nepali was a challenge for teachers, who needed to translate into Awadhi for students who had not yet learned to speak Nepali. The MLE pilot was therefore well received by teachers and parents, and allowed teachers to increase time on task by removing the need to translate in class. A lack of monitoring and documentation reduces the ability of SC and partners to base future programs or advocacy on this pilot, though. Going forward, documenting activities and monitoring learning achievement at these schools should be a priority.!!While using pedagogical materials other than textbooks is a key element in Literacy Boost teacher training, few teachers were observed using such materials in instruction. This was one of several topics that had been covered in training workshops but had not yet been implemented in classrooms. This finding calls into question the effectiveness of current models of teacher training and the possibility of developing alternative models that would help teachers transfer knowledge from training workshops into classroom practice. Alternatively, it may point to the need for increased on-site support to teachers after they participate in training workshops.!!Teaching practice and classroom environments were noticeably different in intervention as opposed to comparison schools, though. In particular, LB classes had a greater emphasis on asking comprehension questions and providing students with opportunities to contribute individually, rather than in chorus with the whole class. The LB early grades reading classes followed a clear sequence but every class observed had a lot of wasted time and room for improvement in classroom management. In all classrooms, there were significant challenges with classroom management leading to some students remaining uninvolved in class activities, and limited time on task.!!Key findings of this study are:!!• Literacy Boost and Multilingual Education programs are having positive effects on classroom

practices and learning achievement in schools in Kapilvastu!• Students at Literacy Boost schools performed well on reading assessments, most notably in

reading and comprehending connected text!• Using Awadhi as the medium of instruction at MLE schools improved student understanding

and increased time on task by reducing the need for teachers to translate during class, but did not address other elements of quality instruction, especially for early literacy, included in LB interventions!

• Parents and teachers in Awadhi-speaking communities have positive opinions of Awadhi-medium instruction!

• There is an immediate need for technical support to SC Nepal and PNGOs to improve future reading assessments in order to document and strengthen programs!

• Information that teachers learn in training workshops is not being transferred into their teaching in the classroom. Future teacher professional development may need to experiment with different models or approaches to facilitate the transfer of skills to practice, including greater on-site support after training workshops.!

�3


• Documentation and assessment of the Multilingual Education pilot program should be a high priority in order to contribute to policy and research, as this is a key area for Nepali and global education!

• Developing a model for teaching Nepali as a Second Language will be an important next step to strengthen Multilingual Education, both in this pilot and in other contexts.!!!

�4


Contents!!Executive Summary! 2! Introduction! 6! Research questions! 7! Methodology! 7!

Observation, interview, and focus group methodology! 7! Reading assessment methodology! 8!

Findings! 11! Research Question 1: Children’s Reading Skills! 11! Research Question 2: Language use! 16! Research Question 3: Perceptions of language use! 19! Research Question 4: Pedagogical materials! 21! Research Question 5: Classroom Practices! 25! Conclusions! 26

�5


Introduction!!Save the Children began sponsorship-funded programming in Kapilvastu District in Western Nepal in 2010. Two elements of the Basic Education component of the Common Approach to Sponsorship Programming are Literacy Boost (LB) and Multilingual Education (MLE). Both of these approaches focus on improving the quality of education in early grades.!!In 2014, SC Nepal, with the support of SUPER Fellow Miranda Weinberg and in collaboration with two partner NGOs (PNGOs), conducted a mixed methods study of these programs. This study was part of the newly developed operational research program of SC, which has a goal of conducting research in ongoing programs to identify gaps and best practices. The quantitative strand of the research used the Literacy Boost reading assessment tool to assess multiple components of literacy and numeracy, and was conducted at 20 schools representing LB, MLE and comparison schools. The qualitative strand of the research incorporated classroom observation, teacher interviews and parent focus groups at a subset of the schools where reading assessments were conducted.!!The Literacy Boost program, begun in Kapilvastu in 2012, focuses on basic literacy and numeracy skills for children in the earliest years of school. These foundational skills are crucial for success later in school. The Literacy Boost approach incorporates three strands: measuring kids’ reading skills to understand how well they recognize letters, can sound out words, and can read and understand sentences; training teachers to help students learn to read successfully; and involving communities through weekend reading camps and forming reading buddies. In Kapilvastu, these activities have been implemented by two partner NGOs, Sunshine Development Organization (SSDO) and Kalika Self-Reliance Social Centre (KSSC), beginning in 20 schools through each PNGO and recently expanding into several others. Inputs from SC and the PNGOs have included several rounds of teacher training, the introduction of a 90-minute Early Grades Reading class (twice as long as a normal class) to provide sufficient time for focus on basic literacy skills, learning materials such as storybooks, big books and workbooks, and support for reading camps.!!Literacy Boost training is contextualized to align with the national curriculum and textbook prescribed by Ministry of Education. Six language competencies are reviewed. For each grade, objectives are set and teaching is divided into phases. A stepwise teaching sequence is developed integrating six language sub-skills with writing as an added component. Every day each of the six language sub-skills is covered with children, followed by work in the practice book developed to fill gaps in the textbook and providing more opportunities for practice. Each of the language arts teachers in grades one and two is provided a total of nine days of training (four-day first phase; three-day second phase; two-day refresher). One day of teaching practice in the classroom is included as part of teacher training.!!The Multilingual Education pilot was also started in 2012, and is aligned with Nepali education policies such as the School Sector Reform Plan and Multilingual Education Implementation Guidelines. This pilot program, implemented in five schools, recognizes that children learn basic literacy and academic skills best through the language they speak best. In the case of students at these five schools, that means teaching through the medium of Awadhi, which is the first language of all the students and nearly all teachers. This pilot has been implemented by SSDO with technical support from INGO United Mission to Nepal and Jan Kalyan Higher Secondary School, the focal school for UMN’s work in multilingual education. Inputs from SC and partner

�6


NGOs have included teacher training, support in the development of Awadhi-language textbooks, and community awareness-raising. In addition, beginning in the 2014-2015 school year, these five schools are part of the Literacy Boost expansion. At the time of this study, early grades reading teachers had received an initial training workshop in Literacy Boost, but other components of the Literacy Boost package, including materials and a baseline assessment, had not been conducted.!!Research questions!!The research questions focused on understanding what model of intervention is most effective for promoting early grades literacy, and where future efforts should be focused. The five research questions were:!1. What are differences in learning achievement in different intervention and control groups?!2. How much and in what contexts are different languages (Nepali, local languages) used in

different program models? Outside of school?!3. What are teachers’, parents’, and NGO staff perceptions of the use of Nepali and Awadhi in

school and other domains? What are understandings, practices, and concerns about MLE?!4. How are pedagogical materials used in schools? What languages are they in? How are

students interacting with various kinds of texts in school?!5. How do classroom practices and understandings of schooling differ between two intervention

groups and comparison groups?!!Methodology!!The twenty schools in the study were chosen in collaboration with PNGO staff. They included 5 MLE schools, 5 LB schools, and 10 comparison schools. The 5 MLE schools represent all the schools in the MLE pilot program. They were chosen for the MLE intervention in 2012 on the basis of having entirely Awadhi-speaking student populations. The 5 LB schools were chosen from the 20 schools where KSSC has supported Literacy Boost programming since 2012. They were chosen largely on the basis of convenience for enumerators, though NGO staff mentioned after the conclusion of the study that they had not chosen any of the lowest-performing LB schools; this study may therefore represent an overly positive picture of LB school performance. Comparison schools were selected by the two PNGOs to be similar to the intervention schools, from VDCs that neighbor the locations of intervention schools but where there has been no Save the Children support.!!For reading assessments, following Percent Proportionate to Size sampling, 16% of students on the roll for Class 2 were chosen randomly following Percent Proportionate to Size sampling. The sample size of 16% of students reflects the percentage of all Class 1-10 students in Kapilvastu district who are in Class 2. This sampling strategy ensures that schools are represented proportionately to the size of their population.!!! Observation, interview, and focus group methodology!!Classroom observations, teacher interviews, and parent focus group discussions were conducted at nine schools across the two intervention models and comparison groups (see Appendix A). All observations and interviews were conducted by Miranda Weinberg, usually in collaboration with PNGO staff, with the exception of one parent focus group discussion

�7


conducted by Subas Yadav, Education Project Officer from SC’s Western Regional Office. Teacher and parent interviews in Awadhi were conducted by PNGO staff or Subas Yadav,.!!Classroom observations were conducted largely in Class 2 to correspond with the classes who were the focus of the reading assessment. However, this was supplemented with observation in ECD, Class 1, and Class 3 and the general environment of the school. During classroom observations, field notes were supplemented with a snapshot tool (Appendix B) filled out at intervals of five minutes.!!Teacher interviews were conducted after classroom observation, usually in the head teacher’s office. In some schools this was conducted with only one teacher at a time, or a few early grades teachers, while in others the entire teaching staff of the school participated. The interviews were semi-structured, following the protocol in Appendix C but pursuing emergent themes. In most of these, PNGO staff took the lead in asking questions, while Miranda Weinberg took notes and asked supplementary questions. At other schools, Miranda Weinberg took the lead in asking questions, with assistance from PNGO staff. Most of these interviews were conducted in Nepali, with some Awadhi depending on the preference of teachers.!!Parent focus group discussions were usually led by PNGO staff, and in majority Awadhi-speaking areas were conducted in Awadhi. In many schools focus group discussions switched between Awadhi and Nepali depending on speaker and addressee. Parent focus groups generally included a mix of members of the School Management Committee, Parent Teacher Association, and other parents. While we attempted to talk with male and female parents/guardians, at some schools only male guardians attended the focus group discussion; in general, men were much more talkative in discussions than women. Parent focus group discussions were semi-structured, covering the questions in the protocol in Appendix D, but also allowing for emergent themes.!!! Reading assessment methodology!!The quantitative reading assessment strand of the study used the Nepali Literacy Boost Assessement tool (described in Appendix E), which was developed by SC and previously field tested in Kapilvastu. The assessment was delivered by enumerators who already work with PNGOs as reading camp volunteers. The enumerators visited schools in pairs during regular school hours and pulled students from Class 2 to conduct the evaluation individually. Enumerators used Awadhi to interact with Awadhi-speaking students, including asking questions and giving directions during the assessment, and Nepali with other students.!!While many attempts were made to ensure the quality of the data, there were significant issues in data collection. The following sections detail attempts to ensure data quality, and limitations of the data.!!Attempts to Ensure Data Quality!!• At the initial planning meeting, program officers and program coordinators from KSSC and

SSDO were introduced to the study and sampling methods !• SSDO enumerators and Community Mobilizers attended a half-day training session on using

the LB reading assessment tool. Several enumerators were familiar with the tool from using it

�8


in previous endlines and baselines. Subas Yadav and Miranda Weinberg led the training, which covered sampling and how to conduct each task in the assessment.!

• Tools had been previously field tested and employed in baselines. Note that while they had been used previously, the present sample of Class 2 students were not among those who took the reading assessment in prior years, so they had no experience with this style of assessment.!

• In debriefing meetings with enumerators and community mobilizers from both PNGOs, SC staff recorded challenges and discrepancies from data collection. Where there were multiple ways of encoding answers to questions, enumerators were instructed to correct their forms to conform to a consistent standard. Where it was not possible to easily correct such errors, this provided a record of issues in data collection.!

• To ensure consistent data entry, 10% of the questionnaires in hard copies were cross-checked against data entered in the database. This showed no data entry errors.!

• To check for systematic differences in scores across enumerators, Amy Jo Dowd looked for significant differences in scores across enumerators. There was no significant difference found across enumerators.!!

Data Limitations!!As part of the operational research goal of identifying concrete corrective measures (if any) to improve the quality of the program within the project cycle and/or during scale-up, challenges in the assessment process were recorded with a goal of improving further LB data collection and training for enumerators. These issues were documented both through observing data collection by one team each from KSSC and SSDO, and through discussion with enumerators at debriefing meetings. Specific recommendations for future reading assessments are provided in the recommendations related to research question 1.!!• In general, the small sample size (total of 159 students across 20 schools) makes it hard to

detect statistically significant differences.!• All enumerators chose a sample of 16% of the students listed on the Class 2 roster, half boys

and half girls. However, while SSDO enumerators picked randomly using a lottery method, KSSC teams employed different approaches, some lining up girls and boys and simply selecting the ones closest to them, and others selecting every third student from the roster. Students may not have been chosen randomly.!

• SSDO and KSSC teams employed slightly different tools, an error that was not recognized until after the completion of data collection. The differences included a typo in the letter recognition task on the KSSC form, which had a half pa, which would never appear on its own in Nepali writing, instead of a full pa.!

• The Awadhi passage for the reading fluency/comprehension task was significantly more challenging than the Nepali passage. While the stories were nearly the same length (Awadhi 98 words; Nepali 99 words), the average sentence length in the Awadhi story was nearly twice as long as in the Nepali story (Awadhi 12 words/sentence; Nepali 6 words/sentence).!

• SSDO enumerators used laminated, large font printouts for all tasks. KSSC enumerators did not have these pages; some wrote out copies in larger print while others used a spare copy of the assessment tool, which is in small print.!

• Some enumerators reported that children were very scared during the assessment. One male enumerator said students were ready to cry during the assessment, which others agreed with. A female enumerator said that she was careful to make students feel comfortable, talk to them like a friend, and they were not afraid. In my observations, enumerators did not explain to

�9


children what they were doing, and that this assessment would not affect their grades. It seems likely that children were scared and therefore potentially unable to complete tasks to the best of their ability. Worryingly, the forms did not include a consent passage.!

• Several enumerators conducted the assessment inside the 2nd grade classroom, where other students could hear. It is possible that students heard their peers completing the assessment.!

• For the Concepts About Print task, most enumerators used either the Nepali or Awadhi language textbook, but one used a storybook from reading camp. Also, most handed the book to students upside down and backwards, while a few handed it to children with the spine up.!

• Enumerators differed in the length of time and amount of coaching they were willing to give to students before recording an answer. This especially seemed to be an issue in the Concepts about Print task, in which some enumerators would immediately mark a wrong answer, while others would continue to prompt and give multiple chances for students to answer.!

• Enumerators from both organizations seemed confused about how to conduct the reading fluency/comprehension task. Some enumerators did not give students time to finish the whole story, even if they had correctly read several words in the first minute. Others did complete the story.!

• The Nepali passage for the reading fluency/comprehension task was taken directly from the Class 2 textbook with minor changes in the vocabulary. One school had already begun studying this lesson by the time the assessment was conducted.!

• Some enumerators read the comprehension passage to students after they attempted to read it themselves, while others may not have. Comprehension questions may be more accurately considered listening comprehension rather than reading comprehension.!!

Comparability of children in school types on background characteristics!!An additional limitation of these data is the lack of comparability across groups on background characteristics. The Literacy Boost and MLE groups vary significantly, not only on the language background as expected (57% of LB students speak Nepali at home, compared to 0% of MLE students). The significant differences on other variables include that LB students were more likely to have attended ECD, reported reading more days of the week, and were more likely to be able to name a favorite story. MLE students were significantly more likely than LB students to work at home, to have repeated Class 1, and to have missed at least one day of school in the previous week. These different background characteristics, which are indicators of differing school experiences, make direct comparison across these groups impossible.!!Across the MLE and MLE comparison groups, background characteristics are largely comparable, differing significantly only in that MLE students were more likely to be able to name a favorite story. In both MLE and MLE comparison groups, all children speak Awadhi as their home language. This means that all students in the MLE group are learning to read in their home language, while none of the students in the MLE comparison group are learning to read in their home language.!!A major difference between the LB and LB comparison groups is that a higher proportion of LB students speak Nepali as a first language than in the LB comparison. This means that only 12% of students in the LB comparison group are learning to read in their home language, as compared to 55% of students in the LB group who are learning to read in their home language. On other characteristics, these groups are largely comparable, though Literacy Boost students

�10


report studying significantly on average significantly more than LB comparison students (1.52 hours for LB as opposed to 0.68 hours for LB comparison).!!With the small sample size in these groups, issues in data collection, and differences in background characteristics across groups, it is important to view data from this reading assessment study as preliminary and formative only. These data point to the importance of carefully selecting groups for future studies that will allow for rigorous comparison, and to the urgent need for technical assistance for successful completion of future reading assessments.!!Findings!Research Question 1: Children’s Reading Skills!!The first research question asks: What are differences in learning achievement in different intervention and control groups? This question is answered largely using the reading assessment data. However, due to the limitations of the data discussed above, these data should be taken as only broadly representative of differences between groups.!!Figure 1 shows a summary of the average performance of students at LB and MLE schools on basic literacy tasks. This skill profile chart is not intended to provide a comparison at the moment, but rather to provide a snapshot of the current performance of the two treatment groups. We hope to see these scores improve over time with further intervention and monitoring.!!

�11

0%

23%

45%

68%

90%

Literacy Boost MLE

0.5380.6753

0.1544

0.6275

0.08880.1462

0.4057

0.826

lettersNepali fluency (wcpm)Nepali listening comprehensionMath

Figure 1: Skill Profiles by treatment group


Concepts About Print!!Students correctly answered on average between five and eight out of 11 concepts about print. In past LB assessments, the most difficult tasks were pointing to words as they are read and singling out words/letters and reading/naming them. In this assessment those were the hardest questions at all schools except at the LB schools. That is, these were the hardest areas at MLE and both groups of comparison schools. Within LB schools, the lowest proportion of students answered correctly on showing the title and page number, and performed better on pointing to words as they were read and singling out words/letters and reading/naming them. This seems to point to success in LB schools at teaching specific skills including identifying words/letters and following along while reading.!!

Letter Recognition!!Students in both intervention groups outperformed comparison groups on identification of letters. LB students on average recognized nine more letters than comparison students, with seven students recognizing all letters and no students who were unable to recognize a single letter; this was a statistically significant difference. MLE students on average recognized seven more letters than comparison students, though none recognized all letters, and seven were unable to name a single letter, which was equal to the number of comparison students unable to name a single letter. More students were able to name common letters and letters that appear earlier in the alphabet order than less common letters. The most challenging letters were vowels, which rarely appear in isolation in written Nepali, particularly those that represent diphthongs (e.g., ai, au).!

�12

Figure 2: % of letters identified (MLE and MLE Comparison)

0.2

.4.6

.81

% o

f let

ters

cor

rect

ly id

entif

ied

3 4 6 10 12 13 14 15 17 20


However, averages across groups hide variation among schools. Figure 2 shows the percentage of letters correctly identified by students at MLE and MLE comparison schools. This chart shows that in two of the MLE comparison schools (10 and 20), almost all students are struggling with the foundational skill of identifying letters. However, among the MLE schools, students at three schools (3, 4, and 6) are achieving this basic skill much more consistently than at two of the other schools (15 and 17). This demonstrates the importance of looking at individual schools’ characteristics as well as at the models as a whole.!!

Reading Commonly Used Words!!Students in both intervention groups outperformed comparison groups on reading commonly used words. LB students on average were able to read five more words than comparison students (an average of 63% for LB students compared to an average of 40% for comparison students; this is a statistically significant difference). MLE students on average were able to read four more words than comparison students (an average of 31% for MLE students compared to 11% for comparison students).!!While length of word or number of syllables was not always correlated to the number of students who could read it, words with consonant clusters were particularly challenging. In addition, words that included letters that were challenging in isolation proved harder for the students. For example, among the LB and LB comparison groups, four words were read correctly by fewer than 1/3 of the students: aujaa, bhaisi (water buffalo), trishul (trident), and ambaa (guava). These may have been less familiar words than some others included in the task, but also include diphthongs (aujaa, bhaisi), little used letters that were read by few students in the letter identification task (trishul) or a consonant cluster (ambaa). For the MLE students, the most challenging words were those with three syllables or diphthongs. The only consonant cluster on the Awadhi language form had three syllables so it is impossible to determine whether difficulty stemmed from the length of the word, the consonant cluster, or both factors.!!As with the letter identification task, there was major variation across schools in completion of this task. Figure 3 shows that the same schools where students fared poorly on recognizing letters also had low performance on decoding words. One of the MLE schools did nearly as poorly on this decoding task as the low-performing comparison schools, while three of the MLE schools performed better than the rest (3, 4, and 6), with higher averages and no students unable to read a single word. Again, this points to the importance of attending to variation among schools, which may need different kinds of support and inputs in order to successfully support student learning.!!

Reading Nepali and Awadhi Text!!Students were also asked to read a passage of connected text. LB and LB comparison students only read a short story in Nepali while MLE and MLE comparison students read stories in both Awadhi and Nepali. Variation in the ways enumerators administered and scored this task make it difficult to draw many conclusions from this task. However, comparing it against reading comprehension task data from RTI’s 2014 Early Grades Reading Assessment report offers some insight into reading skill levels across Nepali, the Terai in particular, and these intervention groups. !

�13


�14

Figure 4: Group skill levels alongside EGRA results

0%

18%

35%

53%

70%

% scoring zero fluency fluency listening comprehension

0.15440.0888

0.2353

0.6275

0.1462

0.054

0.5667

0.1147

0.42

0.6

0.14

0.37

National averageTerai averageLiteracy Boost averageMLE average

0.2

.4.6

.81

% M

UW

tota

l

3 4 6 10 12 13 14 15 17 20

Figure 3: % of common words identified (MLE and MLE Comparison)


While the passages used by EGRA and SC are not identical, the skill levels among the national and Terai samples and those in LB and MLE schools are fairly comparable with two interesting exceptions. The first exception is the percentage of nonreaders: among both LB and MLE intervention groups, the percentage of nonreaders is lower than the EGRA national and Terai averages. This is especially striking for the LB schools.!!The second exception is in listening comprehension, where LB students are comparable to national and Terai averages, but MLE students are well behind. As noted above, it is impossible to know which students completed the comprehension task on the basis of their own reading and which had the passage read to them. Leaving this as a matter for future improvement, we call all of these scores listening comprehension, even knowing that for some children the score represents a higher order reading comprehension score. Even so, given the prevalence of L2 children in the Terai, it is unclear why these scores are so different, and possibly worth investigating further.!!We can categorize students as emergent readers, beginners, and reading with comprehension. Figure Five shows that all the students in MLE and MLE comparison schools are at the emergent reader level, still struggling with reading connected text. Over 80% of the LB comparison group are also at this level. On the other hand over two thirds of the LB students have begun connected words, and 19% are at the level of reading with comprehension. This seems to point to success in the LB program at teaching students to master basic skills to a level sufficient to read for understanding. It is worth noting as well that this assessment was conducted only three months into the school year, making it likely that even more students will reach higher levels of reading proficiency by the end of the school year.!

!Interestingly, students in Awadhi medium schools did not perform significantly higher on reading Awadhi than reading Nepali. In a positive light, this may demonstrate that their learning of Nepali reading is developing apace with Awadhi reading. However, as no students were reading with comprehension in MLE schools at the time of the study, this seems rather to be a matter of concern. While LB and MLE groups differ sufficiently in background that it is impossible to

�15

MLE comp

MLE

LB comp

LB

0% 25% 50% 75% 100%

19%

8%

49%

11%

32%

81%

100%

100%

emergentbeginnerreading with comprehension

Figure 5: Reading levels across groups


compare directly across the groups, it seems that LB groups are far outperforming the rest. This encouraging result makes the implementation of LB in MLE schools seem like a positive step.!!

Math!!Students in all groups correctly answered on average between 7 and 11 out of 15 numeracy questions. There were no significant differences between the groups.!!! Recommendations related to Research Question 1!!Conducting Reading Assessments!• There were significant issues with the reading assessment process, from the tool to the

training of enumerators and communication among all parties. The team should request technical support from Save the Children International headquarters for future assessments.!

• Conduct intensive training for enumerators including orientation, piloting period, and refreshers!• Train a pool of enumerators (maybe university students?) who can be employed regularly!• Involve enumerators in data entry to improve data collection!

• Conduct data entry concurrently with data collection, in order to catch issues or inconsistencies in data collection promptly and ensure timely data analysis!

• Include detailed, scripted instructions on enumerators’ assessment form, including consent passage!

• Ensure that enumerators have all necessary materials for conducting assessments, such as tasks written in large font!

• Check for comparability of tools used for Nepali L1 and Awadhi L1 students in length and difficulty of tasks!

• Consider whether the length of the assessment (30-60 minutes!) is contributing to student fatigue!

• For common words task, use words that are common to the two languages (e.g., daal) as much as possible to ensure comparability!

• Investigate feasibility of using tablets for data collection!• Carefully select comparison schools in future assessments; in the present study, LB and LB

comparison schools were very different in terms of language background, making it impossible to compare these groups.!!

Recommendations based on Assessment Findings!• Consider devoting extra resources to the lowest-performing schools. NGO staff was aware of

which schools were stronger or weaker, with impressions that correlate closely to these results. Significant variation across schools shows that some schools may need greater inputs than others.!

• Include LB intervention components in MLE intervention.!!Research Question 2: Language use!!Research question 2 asks: How much and in what contexts are different languages (Nepali, local languages) used in different program models? Outside of school? This was answered primarily through classroom observation, in particular the snapshot tool used to

�16


record what language was being used at regular intervals during the observation. Interviews and background information on the reading assessments also contribute to answering this question.!!At MLE schools, oral and written language was almost entirely in Awadhi; across observations at four schools in multiple classrooms, there were only two moments when I observed spoken Nepali. This matches with community language practices, as described in interviews and students’ self reports of home languages; nearly all teachers at MLE schools reported speaking Awadhi as a first language and that their students speak Awadhi at home, in the community, and with each other. The few teachers who do not speak Awadhi as a first language consider themselves fluent in Awadhi and use it to speak with their students. For students at MLE schools, this means that they may not get much exposure to Nepali language outside of school, which should be taken into account in planning a curriculum to teach Nepali as a second language.!!Teachers at MLE schools noted that reading and especially writing is harder in Awadhi than Nepali. They provided two explanations for this. The first is that they learned to read and write in Nepali and therefore have little experience writing in Awadhi. Second, some thought that the phonological form of Awadhi makes writing the language harder. This points to a need to provide teachers with more opportunities to read and write in Awadhi.!!At Nepali medium schools with predominantly Awadhi L1 students, instruction was largely in Awadhi, while written language was entirely in Nepali. While teachers said that they used Awadhi only with young children until they learned to speak Nepali, I observed a health class in Class 8 being taught entirely in Awadhi, and many early primary classes where Awadhi was spoken. Teachers at a comparison school reported that they speak entirely in Awadhi, but have to use Nepali because that’s the language that their books are in. At another comparison school that claimed to be using English as the medium of instruction, observed classes were almost entirely in Nepali, with one observation of a teacher using Awadhi to speak to a student.!!Reading assessment data provide some interesting additional insights into students learning in their own language. Figure Six divides students from the four intervention and comparison groups according to whether they are studying in a school where their first language is the language of instruction, or in a school where the language of instruction is their second language. The Nepali-Nepali group is therefore the students from LB and LB comparison schools whose first language is Nepali. The Awadhi-Awadhi group comprises all of the students in MLE schools, whose first language is Awadhi and language of instruction is Awadhi. The “Learning in L2” group is those students in Nepali-medium schools (LB, LB comparison and MLE comparison) whose first language is not Nepali. This sample includes students who

�17

The Class Two reading lesson starts with looking at the picture in the textbook. The teacher asks in Nepali what they see. They all yell out animals in Nepali, then he starts going turn by turn. One of the first picked stands and doesn’t say anything; after he translates the question to Awadhi she says there’s a chhagadi (goat in Awadhi). He says, “Yes, but in Nepali we say bakhra. Chhagadi means bakhra.” There’s another one where he checks whether they’re saying syaa or syaal for jackal, because “in our own language we say syaa but in Nepali it’s syaal.” All the students in the class speak Awadhi as a first language, as does the teacher.!! !! - Observation notes, Class 2 Early !! Grades Reading, Literacy Boost school


reported their first language as Awadhi, Tharu, and other. As shown in Figure Six, scores for students whose home language matches the language of instruction do not vary significantly based on the age of the student. However, for students whose home language does not match the language of instruction, the older the child, the higher the predicted performance. It seems that these children may be more gradually learning the language of instruction, perhaps repeating early grades or else making it through without fully understanding lessons due to the language of instruction. This finding corresponds to similar findings in Nepal and elsewhere 2 3

that students who are not given the opportunity to attend school in a language they speak frequently repeat early grades, or are unable to acquire basic literacy skills until much later than their peers.!! !Recommendations related to Research Question 2!• Create opportunities for teachers to read and write in Awadhi!

• e.g., literary magazine, materials development, writing contests!• Develop an approach to teaching Nepali as a Second Language, perhaps including an ECD or

break time component!• should build from students’ existing Awadhi-language competency!• teacher education is most important, though additional bridging materials may be helpful!• bridging courses developed in other countries may serve as helpful models!

�18

Awasthi, L. D. (2004). Exploring monolingual school practices in multilingual Nepal. 2

Unpublished PhD thesis, Copenhagen University.

Ball, J. (2010). Educational equity for children from diverse language backgrounds: Mother 3

tongue-based bilingual or multilingual education in the early years. Paris: UNESCO.

Figure 6:


• International research shows that it takes five to seven years for children to develop academic competence in a second language, so Nepali language support should last several years!

• Consider whether other SC-supported schools would benefit from MLE intervention!!Research Question 3: Perceptions of language use!!Research question 3 asks: What are teachers’, parents’, and NGO staff perceptions of the use of Nepali and Awadhi in school and other domains? What are understandings, practices, and concerns about MLE? This question was primarily answered through teacher and parent interviews and focus groups. Additional data for answering this question comes from classroom observations of language practices.!!The key finding related to this question is that parents and teachers are happy with the MLE pilot. While at other schools, teachers and parents label Awadhi language as a problem, at MLE schools, past policies are seen as a problem which has now been fixed.!

Perceptions at MLE schools At several MLE schools, teachers reported that parents, students and teachers are all happy with the MLE program. While parents were skeptical of the MLE program initially, wondering why students needed to be taught Awadhi when they already learn it at home, they are now convinced that their children are learning well. Teachers and parents reported that their children were more interested in school, participated more in class, had higher attendance, understood concepts more quickly, and are learning more. In addition, some parents and teachers reported that they were proud that students could learn and they could teach in their own language. In addition to the ease of using a familiar language, teachers appreciated that the textbook reflected local culture, and said that it was much easier to teach from a book using local examples. Previously, they had to take time to both translate language and interpret unfamiliar pictures or information from the textbooks; now, they find it is easier and faster to teach the same material. As discussed below, this was supported by observation in schools in which teachers in Nepali-medium classes regularly translated from Nepali into Awadhi.!!Most teachers said that students studying in Awadhi-medium schools were doing at least as well as students in previous years who had studied in Nepali-medium classes. A few teachers thought that students in Awadhi-medium classes are weaker in the Nepali subject, but those teachers were confident that they would catch up quickly. The only negative aspect that they reported, which was common across several MLE schools, was the worry that this program would not be able to continue after the conclusion of Save the Children support. Teachers said that the District Education Office does not provide support

�19

“When teaching in Nepali, we had to take a long time to explain things, we would have to show examples in order to get students to understand. For example, if you said a cooking pot in Nepali, you would have to explain to them what it was in Awadhi and describe it. Now you can just say it in Awadhi and they know what you mean. We feel for them because we spoke Awadhi until we went to school, and so we had to learn Nepali in school. They used to not understand, and they were afraid to ask. Now they pay attention, they are much more interested, and they ask if they have questions.”!! - Teacher, MLE school


for continual improvement, and that all the improvement at their school had come from Save the Children support. While there is not sufficient data from this study to understand the cause of this attitude, it could point to a lack of coordination with the DEO, leading SC activities to appear separate from government activities, or could stem from a lack of community ownership of these programs. This is a problem if SC-initiated improvements are expected to persist after the end of direct SC support to these schools.!

Perceptions at Nepali-medium schools At other schools, teachers report translating from Nepali to Awadhi in order for students to understand. They also described Awadhi speakers as having trouble learning, particularly in Nepali. At one comparison school, the head teacher pointed out to me as I observed a class the mistakes that he saw as characteristic of “mother tongue speakers.” At these schools, teachers noted language as a problem for L2 speakers of Nepali, and said that they addressed it by using Awadhi language orally with those students. Teachers at schools other than MLE schools were not aware that learning materials in Awadhi exist for early grades and are available through the Ministry of Education’s Curriculum Development Centre.!!At all schools where there were students learning Nepali as a second language, teachers and parents reported that students are only able to understand and speak Nepali fully by around Class 5, with a few saying as late as Class 7. This observation matches international research findings that it takes 5-7 years of instruction in a second language in order to become academically proficient in that language. This points to the need to extend Awadhi-medium 4

instruction through at least Class 5, at least in schools with 100% Awadhi L1 students.!

Perceptions at NGOs The MLE pilot has been implemented in collaboration with partner NGO Sunshine Development Organization (SSDO). I visited classrooms with SSDO’s Education Project Coordinator and with two different Community Mobilizers. All three described ways that the MLE program is easing teachers’ work by reducing the need to translate from Awadhi and Nepali, and noted the pride expressed by community members at having the opportunity to speak Awadhi in school. As with the teachers, this program seems to make intuitive sense to PNGO members. At the Save the Children Western Regional Office (SC WRO), though, I was asked questions about why this program was being implemented when parent demands are for English and Nepali. This seems to demonstrate a lack of awareness at the SC regional office about the reasons for introducing the MLE program, which could impede successful implementation of the project.!!An area of concern is the lack of monitoring of the MLE program. While initial documents from 2011 provided an outline for an approach to monitoring and evaluation of the MLE program, there is very limited documentation of activities and outcomes in the MLE program. The program has been implemented directly by the Western Regional Office, with limited information reaching the Country Office in Kathmandu. In addition, institutional memory about the program

�20

Ball 2004; Pinnock, H. (2009). Language and education: the missing link. London, UK: Save 4

the Children UK; Thomas, W. P., & Collier, V. P. (2001). A National Study of School Effectiveness for Language Minority Students’ Long-Term Academic Achievement. Washington D.C: NCBE Resource Collection Series.


is limited, as SC staff who were key to developing and implementing the program have either left the organization or transferred to other offices. It appears that there was no baseline conducted to measure subsequent years against, and there has been no data collection or monitoring regarding learning outcomes. There is a national conversation in Nepal around the feasibility and desirability of introducing mother tongue-based multilingual education. Save the Children could contribute to this conversation using experiences from this pilot project in Kapilvastu, but is hampered by a lack of data about the effectiveness of this pilot.!!Recommendations related to Research Question 3!• Carefully document inputs and learning outcomes of the MLE pilot!

• Develop written record of training and materials inputs, including who conducted training and what it covered!

• Introduce regular monitoring at all grade levels included in the MLE pilot!• This should follow students after they exit the MLE program to see whether they

succeed in Nepali-medium instruction in higher grades!• Extend Awadhi-medium instruction to Class 5 over the next two school years in MLE schools!• Further investigate perceptions of program ownership among teachers and community

members to develop action plan to promote community ownership and alignment with government programs!

• Ensure all NGO staff understand theoretical underpinning and reasons for MLE program!! !

Research Question 4: Pedagogical materials!!Research question 4 asks: How are pedagogical materials used in schools? What languages are they in? How are students interacting with various kinds of texts in school? This question was primarily answered through classroom observation. Additional information is drawn from interviews with teachers and discussions with reading camp volunteers.!!In most classrooms observed in this study across all groups, teachers relied solely on textbooks, chalkboards, and students’ notebooks (copies). Teachers also reported using primarily textbooks. In almost all classroom observations, teachers started the class using the textbook, moved to using the textbook and blackboard, and finally switched to having students copy something over in their own notebooks.!!A few schools did break this format. In Class 2 Nepali period, a teacher used vocabulary cards to introduce key words in the story that the class was about to read. The teacher led students in several activities using the vocabulary cards, such as reading and defining the key words, passing the cards around the class to look at the words individually, and having students note when they encountered one of the words from the vocabulary cards as they read it in the context of the story. (These students, it should be noted, performed very well on the reading assessment; see inset box “An Exceptional School”). This was the only school where teachers were observed using materials other than the textbook and blackboard for teaching in a language class. Non-textbook materials were observed more often in math classes, where manipulatives were employed in several classes to teach arithmetic. Teachers also mentioned using locally-produced materials in science classes, none of which were observed, since they could use materials from the natural world in their teaching.!!

�21


In general, more materials were displayed than were observed being used in classrooms. For example, in observations of one Literacy Boost school’s Early Grades Reading class, the class was working on a lesson based on the story of Lion and Rabbit. Strings hung across the class held story cards with lines from the story. In the observed class, though, these were not used, and the teacher instead relied solely on the textbook. All of the Literacy Boost schools had print-rich classrooms with written materials displayed on the walls and hung across the classroom on strings. However, the MLE schools had not yet received Literacy Boost materials and ranged from somewhat decorated, with paintings of the alphabet, body parts, and national symbols, to completely bare.!

�22

An Exceptional School!!On entering the grounds of Saraswati Lower Secondary Schools, it is immediately clear that this is a well managed school. The grounds are clean and well-maintained, with facilities for drinking water and separate boys’ and girls’ toilets. The classroom buildings are painted with students’, guardians’, and teachers’ bills of rights and responsibilities, and also labeled with information about the funding source for each building. This spirit of transparency continues inside the head teacher’s office, where financial contributions and spending are displayed in charts on the wall. The head teacher’s desk is located near an accountant’s desk; they pass financial information between the two desks, keeping close track of the school’s money.!!Other charts in the head teacher’s office point to the high student achievement at this school. In charts of the previous year’s pass percentages on exams, the high percentage of students passing exams, 100% in several of the early grades and still in the range of 80% in later grades stand out. Inside the second grade classroom, story books and cards are hung on strings across the classroom. Shelves hold big books, a box with letters on it for playing word games, and additional word cards. Students update a calendar on the wall every day.!!A conversation with parents is noticeably upbeat: they know that things are going well at school and they are happy about it. They have achieved 100% enrollment of students in the area, the SMC and PTA are both functioning smoothly, and they have observed improvements in the school due to their partnership with Save the Children. Parents ask for improvements only to do with extracurricular opportunities for students; they are happy with classes.!!As part of Literacy Boost, students attend Saturday reading camps. Students unanimously say that going to Saturday class and going to school is more fun than staying at home. One mother told us that one of her sons used to have trouble learning, and would rarely talk. After attending reading camp, he opened up and started telling her stories that he’d learned in school or at reading camp. Now he sings songs, tells stories, and learns more successfully. As the chair of the PTA noted, the reading camp has reversed roles in families: before, children would ask their parents to tell them a story. Now, if parents feel lazy they can ask the children to tell them a story. “Now we’re learning from the children,” he summarized.!!In teaching, too, care for students and concern for their achievement is clear. The second grade Early Grade Reading focal teacher, a 26-year-old man from the local area, recalls that he used to find teaching early primary grades boring. He used to think: “how do you spend 45 minutes with kids too young to be taught?” Through a training in Active Teaching and Learning, he learned how to use games to teach, and through Literacy Boost training he learned new ways to teach rather than just teaching the way he learned 20 years ago. Now he notices big differences between students who have gotten a strong base through Literacy Boost and students who do not have the same strong foundational skills.!


�23

In the classroom, these lessons are evident. In order to get students’ attention, he begins with a game in which they clap their one, two, three, four and then all five fingers together to make the sound of a rainstorm. As he described, he sneakily works with numbers and talks about weather and building materials, while students think they are playing a game. Next, they turn to their books, each student holding their book open to the correct page so he can check quickly that all 60 students are on the same page. Next, he takes out a set of vocabulary cards that correspond to the story, and passes them around the class, asking them to try reading individually. After everyone has gotten a chance to see the cards, he tapes them to the chalkboard, and asks students to read them quietly before they join in repeating the words together. Individual students get the chance to come to the front and read the words; some, though not all, of the students, correctly read most of the words.!!Turning to the textbook, he asks students to follow along in the book with their fingers and notice when they reach their vocabulary words as he reads the story aloud. Before starting to read, though, he has them all stretch out their arms and then fold them, allowing them a chance to move around before sitting still for the story. After each sentence, the teacher rephrases it in his own words, and then asks comprehension questions. After a few sentences, he stops and goes over that portion of the story again. As he discusses the story, he draws illustrations on the chalkboard to clarify vocabulary and concepts. The next time he reads the story portion aloud, he points to the vocabulary words on the board to draw attention to them. After that, he gives a few students the chance to read aloud to the class, which a girl and then boy both do confidently and comfortably. The next step in the lesson is to hide the vocabulary words, and dictate them to the students, to see if they have learned how to write them. Then he’ll check each of their attempts to write the key words.!!Reading assessment data confirmed that the exceptional management and classroom practices at this school go along with high student achievement. Figure 7 below shows the skill profiles of the 5 Literacy boost schools, with Saraswati on the far right. While the small sample size makes it impossible to make claims about statistical significance, this chart shows two interesting things: first, the exceptional Saraswati school is achieving impressive results on this assessment, especially in reading and understanding connected text. However, another interesting finding is that these scores are nearly matched by a second school, indicating that the success reached at Saraswati is not out of reach of other schools.!

Figure 7: Literacy Boost schools skills profile


In interviews, teachers often requested more materials, or else more training in how to develop materials. Responding to followup questions, most teachers had received some training in using local materials or developing their own learning materials. When interviewers, often NGO staff, pointed out that teachers had received training on how to make their own materials but were not using them in classes, teachers continued to insist that they needed more training. This raises a question of what kinds of training or other intervention would help teachers incorporate different kinds of materials into their teaching practice. Teachers were also disappointed that Save the Children had not delivered Literacy Boost materials by the time of this study, which was conducted in the third month of the school year, due to holdups in printing and distribution at the Country Office.!!In relation to the question of language of materials, a particular concern was whether there are sufficient pedagogical materials to support quality Awadhi-medium instruction. Teachers at MLE schools consistently mentioned that they appreciated the Awadhi-medium textbooks, which they found much easier to teach from both because of the ease of using Awadhi language and the cultural and geographical relevance of the topics and illustrations. They noted that students understood lessons much more quickly when they were based on familiar festivals and had illustrations that helped students understand quickly. National Nepali-language textbooks had required multiple levels of cultural and linguistic translation. Teachers did mention that they were still working through some small typos in the books, and also wished that Awadhi-language textbooks had clearer, color illustrations. However, other resources do not yet exist in Awadhi language; as these schools are incorporated into Literacy Boost it will be important to develop sufficient learning materials in Awadhi language.!!The final portion of this research question asks how students interact with different kinds of texts. In general, students only interacted with texts through the mediation of their teachers; they did not read independently or demonstrate that they chose to read. In many classes, the focus was reading accurately, rather than for meaning. As discussed below, teachers in Literacy Boost classrooms were more likely to focus on meaning, through comprehension questions asked throughout lessons, rather than just decoding or sounding out words.!!While the reading camps that are part of the Literacy Boost package were not the focus of this study, there were several opportunities to talk to reading camp volunteers. Several reading camp volunteers mentioned that while they have many materials for the camps, books were not always at an appropriate level. The major concern was that the books were too hard for young students, with too much writing and not enough pictures. !!Recommendations related to Research Question 4!• Focus in teacher training on how to use non-textbook materials in the classroom!

• Before training, determine what would best help teachers use knowledge from training in the classroom, e.g., develop specific lesson plans, form teacher groups to work together, conduct whole-school training rather than just focal teacher!

• Encourage Community Mobilizers and other NGO staff who visit schools to follow up on use of materials regularly and suggest improvements to lesson plans!

• Develop sufficient non-textbook materials in Awadhi language!• These could be used at both MLE schools and other schools with Awadhi-speaking students!

• Ensure delivery of Literacy Boost materials by the beginning of the school year!• Pick reading camp materials that are age and level appropriate!

�24


Research Question 5: Classroom Practices!!The final research question asks: How do classroom practices and understandings of schooling differ between two intervention groups and comparison groups? This question is answered through classroom observation, teacher interviews, and parent focus group discussions.!!As discussed in the findings related to Research Question 3, there are major differences in perceptions of appropriate use of different languages in Awadhi-medium and Nepali-medium classrooms. This also has an effect on classroom practices. There were no classrooms observed that had entirely monolingual Nepali speakers, and in almost all Nepali-medium classes the teacher spent some time translating between Nepali and Awadhi. Even in the classes where such translation was not observed, teachers brought up the need to translate into a language that students speak during interviews. In Nepali-medium classrooms where most students spoke Awadhi, teachers spent significant time translating between these languages. Awadhi-medium teachers did not need to do such translation, allowing for greater time on task in Awadhi-medium classrooms. Teachers’ comments in interviews that Awadhi L1 students learn faster in Awadhi medium classes therefore seem to be confirmed in classroom observation.!!Teachers who had been part of Literacy Boost since the beginning of implementation taught in ways that were obviously different from teachers at other schools. LB teachers asked comprehension questions at more steps during reading lessons (before, during, and after reading a story), and solicited individual responses rather than choral responses more often than teachers at other schools. The focus on comprehension may be related to the much stronger performance by LB students on the comprehension task in the reading assessments, as discussed in the findings for Research Question 1. LB teachers provided more individual feedback than other teachers, though it was still unclear that they provided sufficient feedback to help students understand how to correct their own mistakes. When reading aloud, LB teachers were more likely to read whole sentences at a time and have students read connected text, rather than reading word by word as was common in other classrooms. In conversations and interviews, LB teachers mentioned that they have learned to teach in new ways through LB training, and have noticed improvements in their students as a result of their changed teaching, for instance that students now learn all their letters in Class 1 because teachers introduce letters more slowly rather than drilling the whole alphabet from the beginning of the year.!!Despite these comments and differences in teaching, some elements of the LB program were not apparent in classroom observations. While teachers did ask individual students to answer questions, providing an opportunity for individual assessment, there was no obvious record keeping. Some teachers mentioned that they found it impossible to keep records of student achievement in such large classes. In the classes observed, there was no explicit teaching of phonemic awareness, or how to sound out words from individual letters. There was also basically no differentiated instruction, whether for students who needed extra help or advanced exercises for students who had mastered basics.!!In conversations with parents, most parents articulated limited ways that they were involved with school. They reported their involvement as sending children to school and buying them materials as necessary; some parents mentioned that they make sure children complete homework, but many felt that they were unable to help their children learn, especially after early grades. Teachers expressed dissatisfaction with parents, who did not always send their children

�25


to school regularly, on time, or with necessary materials. Parents expressed eagerness to help their children improve in school, though, so information for parents about how they can support children’s learning might be helpful.!!At all observed schools, students participated orally throughout lessons, answering questions or repeating what teachers had said. In many classrooms, a few students completed exercises or read aloud at the front of the class. Teachers were also familiar with the names of multiple teaching methods. However, in almost all classes students remained seated for almost the entire time, and worked either as a whole class or else individually. Only one teacher was observed working with a small group, in an early grades reading class at an LB school, where he gathered a group of students without pencils or notebooks to go over answers together while other students copied vocabulary words in their notebooks. This seems to show that teachers have heard or been exposed to multiple methods of teaching in training but are not using different techniques in the classroom. Future professional development with teachers may need to use different approaches to encourage teachers to transfer skills from trainings to the classroom.!!A major challenge for all teachers was classroom management, especially since many of the classes observed had fifty or more students. Poor classroom management meant that there was limited time on task. In particular, transitions during which teachers wrote something on the board or asked students to take out materials often took 5-10 minutes of class time, with no direct instruction taking place during that time. Another source of wasted time occurred when students completed individual writing tasks, and were given no new task until the teacher was ready to move on with the whole class to the next activity. Some teachers had established classroom routines that addressed some of these issues, such as asking students to hold up their books open to the appropriate page in order to quickly check that they all had the proper materials, or brief physical activities to restore order. However, wasted time was a pervasive feature of all classrooms observed, and should be addressed in future teacher professional development.!!Recommendations related to Research Question 5!• Focus in teacher training on: classroom management strategies to increase time on task;

giving detailed feedback; support for struggling students; use of group or pair work!• Use teacher professional development models beyond NGO training for focal

teachers!• e.g., whole school training, exemplary teachers as trainers, observing other teachers!

• Provide parents/guardians with specific strategies for supporting student learning !• e.g., having children read out loud even if parents do not read, checking homework!

• Work with head teachers on monitoring and supporting teachers !• Introduce recognition for excellent teachers!

• e.g., committee of head teachers, teachers, and parents to select exemplary teachers and honor in an awards ceremony!!

Conclusions!!Key recommendations and findings of the study have been provided throughout this report, especially in the recommendations sections that conclude the discussion of each research question. A major finding of this study is that both models (LB and MLE) improve instruction and

�26


learning outcomes, but could be improved by a model that draws from the strengths of each approach. In MLE schools, Awadhi-medium instruction improves some aspects of quality teaching but does not address all elements included in LB. LB has begun to be introduced in all MLE schools starting this school year, which should help address this issue. LB schools in largely Awadhi-speaking communities could benefit from Awadhi instruction, as observations and interviews show that students do not understand instruction in Nepali.!!To summarize, the central findings of this study are:!!• Literacy Boost and Multilingual Education programs are having positive effects on classroom

practices and learning achievement in schools in Kapilvastu!• Students at Literacy Boost schools performed well on reading assessments, most notably in

reading and comprehending connected text!• Using Awadhi as the medium of instruction at MLE schools improved student understanding

and increased time on task by reducing the need for teachers to translate during class, but did not address other elements of quality instruction, especially for early literacy, included in LB interventions!

• Parents and teachers in Awadhi-speaking communities have positive opinions of Awadhi-medium instruction!

• There is an immediate need for technical support to SC Nepal and PNGOs to improve future reading assessments in order to document and strengthen programs!

• Information that teachers learn in training workshops is not being transferred into their teaching in the classroom. Future teacher professional development may need to experiment with different models or approaches to facilitate the transfer of skills to practice, including greater on-site support after training workshops.!

• Documentation and assessment of the Multilingual Education pilot program should be a high priority in order to contribute to policy and research, as this is a key area for Nepali and global education!

• Developing a model for teaching Nepali as a Second Language will be an important next step to strengthen Multilingual Education, both in this pilot and in other contexts.

�27


�28

Appendix A: Data collected at LB, MLE & Comparison Schools

School Name PNGO Group VDC # of reading assessments

Qualitative data

Dev Pratap PS SSDO MLE Udaypur 6

Nepal Rastriya PS SSDO MLE Manpur 8 classroom observation, teacher interview

Phulkali PS SSDO MLE Manpur 8 classroom observation, teacher interview, parent FGD

Brahmdev PS SSDO MLE Thuniya 6 classroom observation, teacher interview

Durgesh Nandini PS

SSDO MLE Thuniya 14 classroom observation, teacher interview

Kapilbastu Rastriya SS

SSDO LB Dhankauli 0 classroom observation, teacher interview

Saraswati LSS KSSC LB Shivgadi 10 classroom observation, teacher interview, parent FGD

Prathamik PS KSSC LB Gugauli 8

Shivgadi PS KSSC LB Shivgadi 6 teacher interview, parent interview

Janjyoti PS KSSC LB Barkulpur 10

Haribodh PS KSSC LB Khurhuriya 6

Nepal Rastriya PS MLE comparison

Maharajgunj 7

Nageshwor PS MLE comparison

Maharajgunj 3

Nepal Rastriya LSS

MLE comparison

Maharajgunj 6

Sundar PS MLE comparison

Kapilvastu Municipality

8 teacher interview, parent FGD

Kriti PS MLE comparison

Kapilvastu Municipality

12

Bal Jyoti SS LB comparison

Bisanpur 10

Bhanu PS LB comparison

Bisanpur 8

Durga PS LB comparison

Bisanpur 5

Laxmanpur PS LB comparison

Bisanpur 10

Janchetana SS LB comparison

Bisanpur 8 classroom observation, teacher interview, parent FGD


Appendix B: Observation Snapshot Tool!!Class and Teacher information:Date: Scheduled start time: Scheduled finish time:

Actual start time: Actual finish time:

Observer : School Type:

Class level and subject: School:

Name of teacher: VDC:

Total # of students Girls: Boys:

Students present: Present girls: Present boys:

Lesson topics covered & approximate time:

CLASSROOM OBSERVATION SNAPSHOT

DESCRIPTIONSNAPSHOTS

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

Time (please note time of observation)

1 Teacher's Activities & language use

1.1 Oral language use by teacher. Specify: MT, N, E

1.2Visual language use by teacher (Reading/writing). Specify: MT, N, E (If reading aloud tick oral & visual)

1.3Interacts with children in a positive and respectful way, creating a child-friendly atmosphere

1.4 Uses familiar or culture-based examples, activities or materials

1.5 Lectures at front of classroom (no child response solicited)

1.6 Interacts with children, solicits responses

1.7Teaching focuses on issues of accuracy (eg. spelling, memorization, repetition)

1.8Teaching focuses on issues of meaning (eg. stories, comprehension, applied concepts)

1.9 Not actively involved in teaching/learning

Other:

�29


!!*A Nepali translation of this tool is also available.

2 Childrens' activities & language use 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

2.1 Estimated percentage of students engaged in lesson

2.2 Oral language use by children. Specify MT, N, E

2.3 Visual language (Reading/writing). Specify MT, N, E

2.4 Children actively reply to questions

2.5Children are silent but demonstrate engagement (listening, working on task)

2.6Children talk to teacher spontaneously; pose questions, state opinions. etc.

2.7 Children do drill/exercise/repeat after teacher

2.8 Children do tasks or games in groups (not whole class)

2.9 Majority of children are seated in rows

2.1Majority of children are out of rows (eg. group work, learning corners, games, etc.)

2.11 No organized task or teaching

Other:

3Materials use; indicate teacher and/or children: T, C; and language: MT, N, E

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

3.1 Blackboard

3.2 Textbook

3.3 Alphabet Primer

3.4 Notebook/paper/workbook

3.5 Big Book

3.6 Small Books

3.7 Pictures

3.8 Cards (alphabet, number, word)

3.9 Manipulatives

Other:

�30


Appendix C: Teacher Interview Protocol!!Demographic & background questions*!1. Teacher’s name (optional):!2. Age:!3. Gender:!4. What is your home district?!5. What is your teaching experience (how many years, at what schools, what levels)?!6. What language(s) do you speak at home?!7. What other languages do you know besides your home language(s)?!8. How would you rate your ability in your students’ home languages in:!

1. speaking!2. writing!3. listening and understanding!4. reading!

9. What percent of teachers at this school can understand students’ home language?!10. What class (or classes) do you teach at present?!11. What subject(s) do you teach at present?!!Opening questions!1. What are you most proud of in your class right now? What is going well?!2. What is the biggest challenge in your classroom?!!Language-specific questions!3. What percent of students in your class speak Awadhi? Nepali? Others?!4. What language(s) do you use to speak to students in class?!5. What language(s) do your students use to speak to you in class?!6. What language(s) do you use to speak to your students outside class?!7. What language(s) do your students use in peer groups?!8. What language(s) do you think your students should be taught in? Why?!9. According to what you’ve seen, at which grade level do children feel comfortable and

understand most things in Nepali?!10. Does the language background of students influence their success in school? If so, how?!11. Can you describe how different languages are used in the classroom?!

1. What language(s) do you use for teaching?!2. Is it different for different subjects?!

12. Does the language of instruction cause any difficulty with teaching and learning? If so, how? !13. What have been the solution(s) you use for overcoming the difficulty?!14. How do communities and parents support their children’s education here? (to what degree

are they supportive?)!15. Do you think language and literacy plays a role in parents’ level of participation in school?!

1. What language(s) do they speak?!2. What % of parents in this community can read and/or write?!

16. What methods do you mainly use in the class while teaching different subjects (MT, Nepali, math, social studies, science, English)?!!

Training and intervention!17. Have you received instructions or training in literacy instruction? If so, what kind of

instructions and from whom?!

�31


18. Have you received instructions or training in using students’ home language(s) in the classroom? If so, what kind of instructions and from whom?!

19. What kinds of materials do you use in the classroom? In what languages? What methods do you use when using these materials?!

20. What type of technical support would help you improve your teaching most (additional materials, training, changes in school structure)?!!

In intervention schools only:!21. What is your perception towards MLE/Literacy Boost? Is this an appropriate programme in

your school? Why or why not? !22. What challenges do you face while teaching in MLE/LB?!23. Have you received sufficient training for implementing MLE/LB? If no, what additional

support do you need?!24. What is the perception of children, and parents towards MLE/LB? !25. Have you noticed in children’s performance after the implementation of MLE/LB? If so,

what?!26. Do you think the MLE policy and programme is appropriate in Nepal? Why or why not? !!Final question:!27. Do you have other observations about your teaching, school, [in intervention schools only:

and experience working with Save the Children]?!!* A Nepali translation of these questions was also used.

�32


Appendix D: Parent Focus Group Discussion Protocol!!Background Questions!1. How many children do you have?!2. How many attend school?!3. What grades are they in?!4. Which of your children attend this school? What grades are they in?!5. What language(s) do you speak at home?!6. What language(s) do your children speak at home?!!Opening questions!7. What are you happy with about your child’s education? What is going well?!8. What needs to be improved about your child’s education?!!School-related questions!9. Do you know what happens in your child’s classroom?!10. Does your child bring his textbook home?!11. How often does your child have homework? How much?!12. How often do you help your child with his/her homework?!!Literacy-related questions!13. How many people in your family know how to read?!14. How many people in your family know how to write?!15. By what age do you think a child should be able to read? to write?!16. Do you think that you can help your child learn how to read and write?!17. What do you do to help your child learn how to read and write?!![for LB schools only: !

18. Do you know about Literacy Boost? !19. Does your child participate in activities through Literacy Boost? !20. Have you noticed any change in your child’s school since Literacy Boost started?!21. What is your opinion of this program?]!!

Language-related questions!22. What language does your child’s school use in the classroom?!23. What language do you think your child’s school should use in the classroom?!![for MLE schools only:!

24. Do you know about multilingual education in your child’s school?!25. Have you noticed any change in your child’s school since the multilingual education

program started?!26. What is your opinion of this program?]!!

Closing questions!27. Do you have any other observations about your child’s school [in intervention schools: and

Literacy Boost/multilingual education/Save the Children]?!!*A Nepali translation of these questions was also used.

�33


Appendix E: Summary of Literacy Boost Reading Assessment Components!!1. Student Background Information!

This component gathers information about the child, such as the student’s sex, age, language spoken in the home, and socioeconomic status indicators.!!

2. Concepts About Print!This component includes 11 questions to determine whether the child understands basic ideas about literacy, such as what direction to read in, and can identify letters and words that the assessor points to.!!

3. Letter test!This component measures children’s knowledge of the alphabet. They were asked to identify 49 letters displayed on a chart.!!

4. Most used words!This component measures children’s ability to read single words in isolation. Students were asked to read 20 words that appear frequently in their language arts textbook.!!

5. Oral reading fluency!This component measures children’s ability to read connected text. Students were asked to read a short passage aloud, and assessors recorded both how many words they read and how many were read correctly.!!

6. Comprehension!This component follows the reading passage and asks children to respond to four questions about the text they have just read. !!

7. Numeracy!This component measures children’s understanding of basic concepts about math and numeracy. Students were asked to answer fifteen questions involving mathematical concepts such as counting, comparisons of bigger and smaller numbers, and word problems.

�34

operational research on literacy boost and multilingual · pdf fileliteracy boost and...

Documents