CONCEPT NOTE

IMPACT EVALUATION OF

EARLY CHILDHOOD DEVELOPMENT IN MOZAMBIQUE

Prepared by:

Sebastian MartinezBarbara Bruns

Sophie NaudeauVitor Pereira

1

1- SummaryObjective: This impact evaluation aims to measure the effect of Early Childhood Development (ECD) on the cognitive and physical development, education and welfare of beneficiary children and their families. By evaluating the Save the Children ECD program in Mozambique, the evaluation seeks to accomplish a number of objectives, including (1) measure the impact of the program on key outcomes (children’s health, nutrition, cognitive development, social development and subsequent primary school performance) (2) identify ways to improve the program’s design and effectiveness over time (3) increase the evidence base for ECD in the region, and inform policy makers and development practitioners on the effectiveness and cost-effectiveness of such interventions, and (4) build capacity for the use of impact evaluation for managing by results through collaboration of the World Bank, Save the Children, Government of Mozambique and Donor partners. Impact Evaluation Strategy: The impact evaluation will use random assignment of the intervention to treatment and comparison communities to test the effectiveness of the ECD model. Save the Children will expand its ECD operations in Gaza Province to include an additional 30 ECD community based centers (Escolinhas) into the program. With hundreds of eligible communities in the province, assignment of treatment through a lottery system provides a clear and transparent mechanism for selecting treatment areas. In addition to the 30 treatment communities, data will be collected on a sample of 46 comparison communities, for a total of 76 communities in the sample. The evaluation will collect data on a cohort of children aged 36 to 59 months at baseline, to be followed over the life of the evaluation to determine the long-term effects of participation in the ECD program. In addition, a cross section of 1st graders in treatment

2

and comparison communities may be collected to assess performance and cohort effects, amongst others. Implementation and Timing: The impact evaluation will be implemented as a partnership between Save the Children and the World Bank. The World Bank will provide technical and financial resources for the design of the evaluation and baseline data collection. Save the Children will manage the program operations. Baseline data collection will take place in early 2008, with the program beginning in treatment areas shortly thereafter. A projected start date for the ECD centers is April 21, 2008. A follow-up survey would be planned within approximately one year of commencement of program operations, with initial program impact results available by late 2009.

2-Background

More than half a million children in Mozambique have been orphaned by HIV/AIDS. The epidemic and its effects are a serious threat to children’s development in their early years and can prevent them from reaching their full potential in life. The Mozambique Early Childhood Development project provides community mobilization and resources for the construction of “Escolinhas,” or ECD centers that target orphans and vulnerable children (OVC) in selected communities. The Escolinhas are led by a team of community volunteers, including two “animadores” per classroom who are trained through experiential learning techniques to facilitate children’s learning at the centers, with a curriculum focused on cognitive stimulation through games, art and music, as well as elementary math and reading (including introducing children to Portuguese, in preparation for elementary school).

3

In addition, ECD centers encourage access to critical health care and psychological support. The intervention is currently run as a pilot program by Save the Children in 12 communities in five districts of the Gaza province in Mozambique, and has a target of scaling up to approximately 5,000 3-6 year old children in 42 rural communities over the next year, with the primary objective of improving the likelihood that orphans and vulnerable children in the community are able to reach their full learning potential and have a successful transition to primary school. This innovative model of ECD in the African context provides an important opportunity for learning the effectiveness and cost-effectiveness of ECD centers in improving the cognitive development, health and welfare of beneficiary children and their families.

4

3-Intervention summary-Program description

Skilled and engaged communities

The great first step into the implementation of the program is the initial contact with the community. Only the 30 communities selected to receive the program were visited. After some initial contacts with the leaders and the communities, all the 30 selected communities agreed to establish the partnership with Save the Children.

Following the initial contacts, a series of meetings between Save’s representatives, community leaders, caregivers and other community members have been taken, at an attempt to present the program to the community ; learn about the culture and idiosyncrasies of each community; and discuss the best format of the program. At this phase, the communities committed to arrange a space to the construct the escolinhas, to mobilize parents and caregivers to enroll their children, and to form a committee that will be responsible for managing and supervising the escolinhas.

The committee is composed by 8-12 members appointed by the community. Each committee must have at least one member specifically responsible for education and another one responsible for the health of children. Other tasks include managing contributions and materials of the escolinhas, as well as mobilizing local laborers to construct the escolinhas classes.

After the beginning of the class, meetings with the community are held every month. The goal of those meetings is to dialog with parents and caregivers about positive and stimulating parenting practices,

5

thus mobilizing and effectively engaging the community with the activities of the escolinhas.

Parenting program

Parents and caregivers are those who spent most of the time with children, and are the main responsible for providing love, care, nutrition, and a healthy home environment in which children can fully reach their potential in life. Caregivers play a fundamental role in transmitting social values and building the morale and self esteem of children.

Children spent 3 hours per day at the escolinhas. All the remaining time is spent at home. The quality and amount of care and stimulation provided at home can influence on how well children benefit from their schooling. Rearing sef-confident, persistent, healthy and happy children is a task that can only be fully accomplished if caregivers have the correct knowledge and capacity to do so.

Save the Children facilitates meetings in which positive parenting practices are discussed between parents and caregivers of participating children. Parents are motivated to participate in monthly meetings, creating an opportunity for them to discuss childrearing and to learn from one another.

The meetings are facilitated by the animadores, the community development agents and a parent of the day, who is chosen by the

6

community. Some of the topics discussed at parent meetings include breastfeeding, de worming, nutrition, child development, literacy, playing with children, attendance and demand driven components to be defined by the community. Topics are discussed using an appreciative inquiry approach in which knowledge is built from existing positive parenting practices and harmful practices are brought to light with strategies to change those practices (such as the use of positive deviants to lead discussions and model new behaviors).

Parents must also commit to arrange for bringing children to school and taking them home and contribute with water and in-kind counterpart for animadores.

Qualified Community Teachers

Well trained and qualified pre-school teachers are a key component of Save the Children’s ECD program in Gaza. Each class counts with a team of two animadores (as pre-school teachers are known in Mozambique), resulting in a pupil-teacher ratio of around 15 pupils per teacher. The animadores are selected by the community with the community development agents’ help, and they are required to have completed at least the primary school, 5 years of education. Each animador receives 10 USD per month from Save the Children and each community is motivated to provide in-kind counterparts as feasible.

Animadores are prepared to have a positive and warm interaction with children, creating a welcome environment for learning. The first

7

long term training and coaching of the animadores was held in May 2008. Another long term training is planned for 2009.

The animadores training is facilitated by two education specialists and the CDA’s. The foundational training takes one week and is designed using experiential learning methodologies where new knowledge is introduced, modeled, practiced, and reflected upon. In this training, animadores learn about children’s holistic development and activities to support children’s development. A daily routine is introduced, modeled, and practiced mimicking to the extent possible an escolinhas environment.

After the foundational training, animadores receive support on the escolinha’s opening day and mentoring visits by the CDA at least once a month. Based on observations and monitoring tools, the CDA gives constant feedback to the animadores on how to improve their practices. Animadores participate in learning circles initially facilitated by the CDA to consolidate foundational skills and also learn additional activities to promote children’s development. Study circles will have an emphasis on strengthening the instruction of math, reasoning and literacy activities, plus reflection on practice.

Learning Program and Materials

The classes take 3-4 hours per day, but specific time of operation is left up to community. Usually classes are taken by the morning. Each

8

class has up to 35 children. Children are not clustered by age; instead assignment to classes is made at random.

The 30 new escolinhas don’t provide food to children. It’s been realized from the experience in the previous 12 escolinhas that food supplementation may cause parents to view the program as a feeding service rather than learning program.

The daily routine and the classroom structure are prepared to support positive social and emotional development. Classes are taken in the local language, Changana, but children are introduced to Portuguese in order to make a successful transition to primary school.

Children begin class by greeting each other, showing his/her attendance, washing hands and singing a song. During class, they are continuously stimulated to listen to and discuss stories, tell about personal experiences, draw pictures, play games with alphabet letters, label, align and group objects, among other activities.

The physical development is enhanced by outdoor play and activities to develop hands and fingers muscles. When playing outside, children can dance, walk, hop or jump, thus enhancing gross motor, social and emotional skills.

Thinking skills are the foundation upon which children learn to make decisions, regulate their own behavior, meet complex challenges and take responsibility for their actions. Thinking is developed in the escolinhas by planning and reflecting about news or, at closing, about what they did learn during the class.

Activities like story telling, news sharing and alphabet learning are also practiced. Those activities are powerful tools to develop not only

9

language and communicating skills, but also the capacity of thinking and reasoning.

Hand washing with soap, safe drinking water and access to latrines are absolute and non-negotiable components of the program.

Toys are adapted to the communities’ environments using local materials. The toys are preferentially made at the communities. At the parents meetings the CDA’s will also advice parents how to make simple toys at home with cheap materials. They can also show parents some games that can stimulate their children cognitive skills, like putting together pieces of wood and asking their children to reorder them from the smallest to the bigger one.

Children are daily exposed to math through everyday play activities. They have materials to count, sort, compare, match, put together and take apart. Rhythm and sequence games, calendar activities, and measuring sand and water with cups also provide informal exposure to math concepts.

The next three pages contain, as a illustration, a series of math activities that are played with children to enhance their capacity of logical thinking. After the description of math activities, the following page presents the daily routine of a typical Save the Children escolinha in Gaza.

10

Save the Children-American Idol Early Childhood Development Project in Mozambique

Sixteen Preschool Math Lessons-Month 1 Time: 20 minutes

1.MATH BAG – SODA BOTTLE CAPS

TAKE OUT THE SMALL WORK MAT. EMPTY THE SODA BOTTLE CAPS ON THE MAT AND SORT THEM INTO GROUPS SO THAT ALL THE ONES OF THE SAME COLOR ARE TOGETHER. ASK THE CHILDREN TO TALK ABOUT HOW THEY SORTED THE CAPS. CAN WE NAME THE COLORS OF THE GROUPS?

SORTING – NOTICING COLOR DIFFERENCES

2. Math bag- seeds

Children use 10 seeds to make a design anyway they like. Next they work in pairs. Child # 1 makes a design with the seeds. Then Child #2 tries to make the same design. Next Child #2 makes a design for Child # 1 to copy.

Noticing color and design differences

3. Math bag –soda bottle caps

Pick two colors of tops. Lay them in a straight row. Make a pattern such as orange green orange green…

Patterns & colors

4. Math bag – seeds

Children make a pattern using only seeds of 1 color. Now they can turn add 5 seeds of another color to with two-colors. Finally they can play with the seeds anyway they like.

Following directions; noticing colors; making patterns of two colors

5.Cubes from Block corner

Give each child 5 cubes. Let them play with them for a few minutes, building anything they like. Now ask them to work with a partner to make two towers that are equal or the same size. Now ask them can they make two towers one that has more and one that has less. They should be able to point out which is less and which is more.

Concepts: same/equal; more, and less

6. Cubes from block corner

Each child has 5 cubes. Count them together. Let them play as they like. Notice if any make a pattern as they did with the caps. Point this out to the others. Then give them a challenge. The teacher uses a child and an object to show above and below. Work with a partner and find some other ways to show above and below.

Review patterns, Concept: Above and below

11

7. Math bag- Sodabottle caps

We sorted the bottle caps into groups by colors. Can you find another way that the caps are alike or different and put them in groups to show ones that are the same? Teacher gives time to explore. Then she asks student to explain what they did. She encourages others to find more ways to sort the caps. The students explain. Examples might be caps with words; caps that are bent; caps that are new and caps that are old.

Sorting & concepts large and small, more or less

8. Cubes from blocks corner

Children work in pairs with 3 blocks each of 3 shapes. One child stacks blocks to make a pattern, and then asks his friend what comes next? The other child makes a pattern stacking blocks and asks which comes next? They continue playing this game until time is up. Teacher then asks them how many blocks of each shape do they have.

Patterns; and understanding the number 3

9. Math bag-soda bottle caps

The children again work in pairs to solve a problem using 3 colors of 3 soda bottle caps. The teacher writes the number 3 on the board. The children draw number 3 in the air. The teacher poses a problem to solve. Show all the different ways you can combine the caps to make 3. Example 1 red cap and 2 blue caps. After they are finished, the teacher asks for someone to tell one way they solved the problem. The student shows the others. The teacher asks the students to raise their hands if they made the same combination. The teacher then asks another student to different way. Again she asks all students to look at their caps and see if they solved the problem the same way. The teacher says something like, “Oh look we have 1+1+1 or we have 1 + 2 = 3. She uses the language of addition but she does not write it as such at this early stage.

First experience with the meaning of addition and how things add up to make 3. The teacher does not call it addition at this early stage. Children also notice things that are same and different.

10. Math bag – Toothpicks

Today take out your toothpicks. Yesterday we worked with 3 cubes. Today let’s try something different with 3. Show all the different ways you can make a design with 3 toothpicks. The sticks have to touch each other in the design.

Solving problems with the number 3. Creative design,

12

When the students are finished, asked them to work in pairs. They are given this problem: Look at your friend’s toothpicks. Make sure that every design is different and no two are the same. If they are the same, the friend has to put the toothpicks back in the box. The second student does the same thing, looking carefully at his friend’s toothpicks. When everyone is finished the teacher asks. How many had more than one design? How many had more than two? Three? Four? Five?

developing concept of “3ness”, noticing things that are different; practicing identifying “more than”; practice counting together sets of things up to five.

11. Math bag-string + soda bottle caps

The teacher shows the circle shape. The children say the name and make a circle with the string. Next they take out the caps and lay the caps on the string to make a cap circle.

Shape - circle

12. Math bag- Soda bottle caps

The children remember the shape called circle. They now use two colors of caps and make a pattern with the two colors and form the pattern in the shape of a circle. They can make other cap pattern circles with other colors.

Patterns and circle concept

13. Math bag – stirng, small rocks and toothpicks

Teacher draws a straight line on the board. She asks students to make a straight line with the string. Next she shows them a circle shape and asks them to make a circle. Another way to make a curved line (waves). You can make a curve line that looks like ocean waves. Try it with a string. Students now take out seeds and toothpicks. Can you make a long straight-line pattern with seeds and toothpicks? Now can you make the same pattern in a circle shape? Now make a different straight-line pattern. Now make the same pattern in a circle shape.

Straight-line curves and circle.

14. Walk Today we are going on a circle hunt. I will take a piece of paper. Every time we find a circle I will make a tally mark. The teacher asks the children to hold hands on the walk and follow her. They are to stay in a line. If they see a circle they are to call out to her. If everyone agrees it is a circle, the teacher will make a tally mark.

Noticing shapes (circle) in the environment. Extending math out of the

13

When they return to the room, the teacher draws a large circle on the blackboard. She asks the children to use their journals and draw one circle they remember finding during the walk. Each child gets to show the circle. If this activity is done on paper rather than a slate, it can be made into a book about “Circles”.

classroom. Remembering things they have seen.

Circle poster on Wall labeled with name circle. Teacher writes number of objects seen on the walk.

15. Math bagToothpicks

Now take out your toothpicks. Teacher draws a triangle on the board. Asks students if they know the name of the shape. How many sides does the triangle have? Next she poses a problem. See how many triangles you can make using your toothpicks. When everyone is finished, the teacher says, “Let’s all count together.” Now see if you can make triangles that connect to each other so that a new triangle shares one side of the first triangle. Use your toothpicks and keep making more and more triangles that connect to each other. When everyone is finished the teacher admires the different ways students solved the problem. The students look at each other’s work.

Review of 3, introduce triangle, solving a problem with triangles, there are different ways to solve a problem.

16. Math bags –Toothpicks

Independent. What is the smallest triangle you can make with your toothpicks? What is the largest triangle you can make with your toothpicks? Pairs. Together with all your toothpicks, what is the largest triangle you can make? Now separate the toothpicks so that you each have the same number. How can you be sure that each has the same number? Can anyone suggest a method? Children listen to each other’s suggestions and then divide the toothpicks into two equal groups. After the teacher checks each one, they put them back in the box.

“3ness”, different sizes of triangles, different ways to solve a problem. Creative exploration

14

Save the Children-American IdolEarly Childhood Development Project in Mozambique

Daily Routine

1) Greetings (15 minutes).At the beginning of the day, each child must turn on a card

with his/her name to their own drawing to show her attendance. Children wash hands before entering the classroom. Greet each child. Review the attendance chart. Identify the Child of the Day. Invite the Child of the Day to help lead a song or game.

2) Literacy Circle (45 minutes).After seeing a picture, the child must tell to the other what’s

going on. News Sharing (Monday/Wednesday);Journals

(Tuesday/Thursday);ThemeJournal (Friday). Storybook Reading (Monday/Tuesday) or Storytelling

(Wed/Thursday) .or Big Book (Friday) .

3)Rhymes or Song (5 minutes) .

4) Alphabet Activity in Portuguese (10 minutes).One letter per day to all children, independent of their age.

5) Break (10 minutes).Time to go to the latrine or to drink water.

15

6) Corner Play (1 hour)Each escolinha will have five “corners” with specific activities

(one “corner” outside the classroom). The corners are: Language (Books and Pictures); Mathematics (playing with cubes, puzzles); Arts and culture ( Construction, paintings, musical

activities); Imagination (storytelling); Sand and Water Play (outside of the classroom).

7) Outdoor Play (30 minutes)Children play outside freely or with a game organized by the

animador. They must wash their hands before re-entering the classroom.

8) Math Circle (4 days)/Cultural Day (1 day) (25 minutes). Calendar activity, Days of the Week (5 minutes). Lesson using Math bags (20 minutes). Counting Song/Rhyme (as time allows). Mathematics, playing with numbers (10 minutes).

9) Closing/Review (15 minutes)

16

Clean-up (about 10 minutes). Reflection on day, Review what they have done during the

day, song or game (5 minutes).

Health and Nutrition Services

Better health outcomes in childhood can have long lasting effects. Communities with healthy children can have lower levels of absenteeism. Healthier children may also have better capacity of learning and developing its brain, boosting the effects of the program on cognitive and language development.

One member from escolinha committee is specifically charged with coordinating health interventions in each community. The health condition of each child is monitored each month by this health activist volunteer and may be reported to the health authorities if necessary. As a monitoring tool aimed to constantly access the quality of the program, information on weight and immunization are collected periodically by the CDAs and the health activist volunteer by checking each child’s health card.

Each escolinhas has a washing station with clean water, for hand-washing and drinking. Before the beginning of class, every child must clean hands, learning the importance of hand washing. All built classes have cement floor, which can help interrupting the transmission of parasite infections.

Vitamin A is supplied twice a year , as well as de-worming. Parents and caregivers are taught at the monthly meetings with the CDAs about the importance of vitamin A supplementation. They are also

17

explained about the role of de-worming and the possible side-effects that children may feel. De-worming is done with Mebendazol, useful for hookworms, pinworms, roundworms, tapeworms, and whipworms. No de-worming for schitossoma is planned. The administration of the drugs is done by the health authorities, not directly by the animadores.

Safe and Welcoming Learning Space

Wooden classrooms are planned to be built in all the communities within the two years of ‘Save the Children’ support of the program. In some communities, while classrooms are not ready yet, classes can be taken in open-air. The chosen place needs to be clean, dry, safe, have adequate ventilation and light; and allow 1.5 meters per child. Shades of cash nut trees usually satisfy those conditions.

Classrooms are build with by Save the Children, jointly with the community, who can provide labor and local materials. Classrooms have a thatched or tin roof depending on each community’s context, with cement floor, and shall have a good ventilation, with windows. The space for each child is set by 1.2 to 1.5 squared meters per child. This is important to enable adequate contact with play materials and facilitators. Each escolinha will have latrines and hand washing stations. The average estimated building cost for each classroom is about 1000 US$ (3.000 US$ for a complete escolinha with 3 classrooms). Each community has up to 3 classes, with a maximum capacity of 105 children. Only children in community may be enrolled, and they may be aged 3 to 6 years. Playgrounds are built next to the classrooms, using local materials.

18

Support for Transition to Primary School

One of the main short outcomes of the escolinhas is making a successful transition to primary school. Many children who begin schooling without going to pre-school tend to fear going teachers, cry at class and have lower performance in the first years of school. The language spoken at primary school imposes an additional complication to children in Gaza. As it occurs in many other African countries, the language in which classes are taken is not the mother tongue of children. Although children in Gaza learn Changana or Chope at home, according to the law, they must be instructed in Portuguese in primary school. The lack of bilingual instruction in primary school can delay gains in verbal and communicating skills in the first schooling years.

Although classes at escolinhas are taken in Changana, children are early introduced to Portuguese. They learn reading and writing simple words in Portuguese. School welcome days are scheduled for children from the escolinhas to visit their future primary school. By the other hand, primary teachers are invited to visit the escolinhas and early get in contact with children and their families.

Counterfactual Analysis – experimental design

In order to estimate the causal relationship between the ECD intervention (treatment) and the outcomes of interest, impact evaluation requires the construction of a counterfactual – that is, what would have happened to the target group in the absence of the intervention. In the case of ECD interventions, it is possible that factors such as weather, macro-economic shocks, or other new and

19

ongoing education, public health, and nutrition programs could influence the same set of outcomes that are targeted by the ECD program (e.g., children’s cognitive or physical development). To account for factors external to the intervention, counterfactuals are estimated using control or comparison groups that are equivalent to the treatment group on every dimension (observed and unobserved) except for the treatment, and thus account for time varying factors that may affect the target population. Since a good counterfactual approximates what would have happened to treatments in the absence of the treatment, any measurable differences in the average outcomes of treatment and control groups following the program implementation can then be attributed as the causal effect of the intervention.

This evaluation uses a randomized experiment to estimate the causal impacts of the ECD program on the outcomes of interest. Random assignment of treatment to a sub-set of communities guarantees that ex-ante, the treatment and comparison groups are equal, and thus that an appropriate counterfactual can be measured following the treatment. A randomized experimental evaluation reduces the possibility that the observed before-to-after changes in the intervention group are due to factors external to the intervention, by controlling for time-varying factors. The use of a random control group also helps to prevent selection bias. For example, if a program targets communities that have favorable local conditions for implementation, such as strong leadership, strong community mobilization, a higher educated or wealthier population, etc, then areas that receive the intervention may be systematically different from areas that are considered less desirable for implementation, on both observable and unobservable dimensions. If a non-random

20

control group is used for the evaluation, a comparison of treated and untreated areas would confuse the program impact with these observable and unobservable differences. A random control group helps avoid this bias by ensuring that the communities that receive the program are no different than those that do not.

For the Mozambique ECD impact evaluation, the following randomization process is proposed:

Selection of sitesi. Province and district level selection

The intervention will be implemented in communities from three districts of the Gaza province, selected by Save the Children from a list of Districts that were targeted for previous health interventions. The districts are: Manjacaze, Xai Xai and Bilene. Given operational constraints of the program within selected districts, the evaluation design will not attempt to provide external validity to the district, province or national level by selecting a representative sample. Instead, the evaluation will concentrate on obtaining internal validity on a sample that meets the operational requirements of the program.

Mozambique

Gaza Province

21

Source: National Institute of Statistics

ii. Community Level Selection

22

Step 1: Community Eligibility.Within the three target districts, a subset of eligible communities is identified that meets two key operational requirements for implementation of the program:

1. Population size: To qualify for the intervention, communities must have a population no less than 500 and no more than 8000 people. This range was determined as operationally feasible given the community mobilization process that accompanies the establishment of each ECD center.

2. Clusters: Management of the intervention requires that the intervention be clustered in groups of 6 treatment communities that can be served by a SC program staff. The definition of cluster will be set by SC, based on minimum criteria of operational feasibility (distance or time traveled between sites).

The complete universe had 252 villages in three intervention districts. After applying eligibility criteria of population size and clustering, the sample was reduced to 167 villages in 11 clusters.

23

Cluster Num Bilene Manjacaze XaiXai Total

Cluster 1 15 0 0 15Cluster 2 9 0 0 9Cluster 9 0 12 0 12Cluster 10 0 22 0 22Cluster 11 0 14 0 14Cluster 13 0 11 0 11Cluster 14 0 16 0 16Cluster 15 0 14 0 14Cluster 16 0 0 21 21Cluster 17 0 0 24 24Cluster 18 0 0 9 9Total 24 89 54 167

Number of communities per cluster

Step 2: Community level randomization

The largest clusters in each district were selected for inclusion in the sample, resulting in total of 98 villages. To achieve coverage in all three distracts, it was further agreed with SC that the sample would include 2 clusters each in Manjacaze (cluster 10, 14) and Xai Xai (cluster 16, 17), and one cluster in Bilene (cluster 1).

RandomizationWithin clusters of communities that meet the two requirements outlined in step 1, communities form triplets based on population size, and from each triplet a treatment community is selected at random. The two smallest villages which did not form part of a triplet were dropped. The process of clustering communities into areas of geographical proximity, and matching on population size is expected to improve balance and comparability amongst treatment and comparison groups. The final sample is composed of 37 treatment (7

24

for replacement) and 59 control villages (11 replacement), for a total sample of 96 villages. A total of 30 new intervention communities were then selected for this round of implementation through random assignment.

ReplacementGiven the small sample sizes, there is very little room for replacement. Clusters 1 (Bilene), 10 (Manjacaze) and 16 ( Xai Xai) have 1 replacement set each, while 14 (Manjacaze) and 17 ( Xai Xai) have two replacement sets each. The replacement treatment communities will be included in treatment only if one of the originally selected treatment villages in the same cluster has dropped out of treatment.

BalanceBased on population size, the sample appears well balanced:

25

Group Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]

Treatment 37 1782.7 263.2 1600.8 1249.0 2316.4Control 59 1827.3 205.8 1580.8 1415.4 2239.3

Combined 96 1810.1 161.3 1580.3 1489.9 2130.3

Difference -44.6 333.1 -706.0 616.8

Difference = mean(1) - mean(2) t = -0.1339Ho: diff = 0 degrees of freedom = 94

Ha: diff < 0 Ha: diff > 0Pr(T < t) = 0.4469 Pr(|T| > | t| ) = 0.8937 Pr(T > t) = 0.5531

Two-sample t test with equal variances

Ha: diff != 0

iii. Child Level Selection

In addition to randomization at the community level, there is exogenous variation in treatment within communities, based on rules of eligibility for OVC. ECD centers will have a maximum of 3 class rooms with 35 students per class, for a maximum of 105 students per Escolinha. In the case of over-subscription of children to the ECD centers, Save the Children and the communities agreed to select the children through a lottery system described in the Appendix XXX.

The feasibility of the lottery system is still needs to be verified as the program rolls out. Drop out rates in some communities with lotteries have been very high, forcing the replacement of children. Community development agents are instructed to follow the waiting list to replace children that drop out.

26

Evaluation Samples and Data

The evaluation will use primary data collected through household and school level surveys, as well as any existing or planned monitoring data. The universe of treatment sites is composed of 76 communities (30 treatment and 46 control). Within each community, a sample of 23 households with children between the ages 36 and 59 months old will be included. At the 5 communities where the lottery took place, 86 households will be included, for a total sample of 2000 households.

A pre-intervention baseline, mid-line and end-line are proposed, for a total of three rounds of data. Original data will be collected on a representative sample of households and primary schools in the evaluation communities. The household sample will form a longitudinal sample of children to be followed over time, while the school level sample will be repeated cross sections.

Household sample: A representative sample of 23 households per community. The sample will be concentrated on households with children who are eligible to participate from the program. Thus, the sample will include children ages 36 to 59 months of age, who would be eligible to enroll in the Escolinha once the facilities are in place, and would receive between 1 and 3 years of treatment on average. The interview includes information on all household members, with a special emphasis on cognitive development indicators for children within the target age range. Households and children in this sample will form a longitudinal cohort or panel to be followed over time.

School sample: In addition to the household sample, a sample of information on 1st graders will be collected on primary schools in treatment and control communities in order to assess the effect of pre-school on learning indicators (including test scores) and class

27

composition. Cross sections of data on 1st graders will be collected during each round of data collection.

Leader sample: A sample from the communities leaders will be collected to access information of prices, distances and social norms inside each community.

Other Data Sources: Other administrative data routinely collected in treatment and comparison areas may also be used for the evaluation. This data include information on enrollment, attendance, vaccination and records of weight. A monitoring database in Access will be continuously updated recording this data. This database will be integrated with data from other primary educational programs of Save the Children in Gaza, allowing tracking children through all their primary studies. Other data sources can also be used to evaluate the program. As an illustration, labor and wage information gathered through other household surveys in Mozambique can be a valuable resource to estimate gains from increased grade attainment and future earnings.

Measurement/Outcome Indicators: The primary research questions relate to the effectiveness of ECD centers in improving the cognitive, psycho-social and physical development of children ages 3-5 years old in Mozambique, and the consequent implications for attendance and performance in primary school. Outcome measurements will include school enrolment, school progress (grade promotion, repetition, dropout) and educational performance (test scores). Additionally, a rich set of cognitive and social development indicators appropriate for the target age range would be collected, as well as self-reported and biometric health outcomes including anemia and anthropometric measurements. In addition, the impact evaluation

28

will seek to test a number of hypotheses related to potential effects on family welfare, including time use, primary care-giver labor supply and income, household income and consumption, schooling outcomes for siblings, etc. Finally, the evaluation will consider community level outcomes including measures of social capital and social interactions.

Specific questions to be addressed by the evaluation include: Do Escolinhas improve child development in terms of the following outcomes: Care and protection, as measured in child height, weight, and family hygiene and child care practices? Behavioral development, including personal and social outcomes, Language development, including the use of gestures, sounds and movements to communicate? Cognitive development, including working memory and executive function? Environmental exploration, measured using a test of gross and fine motor skills? In addition, do Escolinhas improve parental care giving practices and child and family decision making? Do Escolinhas improve levels of social capital or social interactions in the community?

Outcome indicators are proposed in the following areas:i. Primary Schooling

a. Enrollmentb. Dropoutc. Completiond. Academic performance (grades, tests)

ii. Child Developmenta. Cognitive Development: analytical skills, mental problem-

solving, memory, and early mathematical abilities (EDI)b. Language and communication (TVIP- Peabody Picture

Vocabulary test)c. Socio-emotional development (Ages and Stages)d. Child-Parents interaction

29

e. Health and Nutrition:i. Height and weight (wasting and stunting)

ii. Nutritional intake (food recall/food frequency)iii. Preventive health seeking (vaccinations, growth

monitoring, etc)iii. Time use and habits

a. Schooling and enrollment of siblingsb. Labor supply and child care of older siblingsc. Practices and customs (hygiene)d. Social capital, trust in neighbors

iv. Economicsa. Maternal and paternal labor supply and earningsb. Child labor supplyc. Consumption of school materials

In addition to the specialized outcome indicators proposed here, a standard socio-economic survey will be collected on each household including demographic data, assets/consumption, activities of daily living, time discounting, etc.

Evaluation Team and Products

The impact evaluation will be implemented by the World Bank, in consultation with a joint team of World Bank and Save the Children staff and Government, donor and academic partners in country. To our knowledge, this evaluation is the first prospective randomized impact evaluation of ECD to be conducted in Mozambique, and as such constitutes an opportunity for developing capacity in country for the use of impact evaluation methods. The evaluation is co-led by

30

Barbara Bruns and Sebastian Martinez of HDNVP, in collaboration with Sophie Naudeau (HDNCY), Paul Gertler (UC Berkeley), Elizabeth Pearce (Save the Children), Jodie Fonseca (Save the Children), Ana Tenorio (Save the Children) and Damião Mungoi (Save the Children). An international consultant, Mr Vitor Periera, has been identified to help develop and manage the impact evaluation in-country, working closely with the principal investigators to draft TORs, design the questionnaire, adapt test to the local context, supervise the survey institution and coordinate all logistics related to the evaluation. In addition, a survey firm, Austral-Cowi has been contracted to collect the baseline data.

Final impact analysis would be conducted by the principal investigators with the assistance of a research assistant, and with input from the impact evaluation team. Key products from the impact evaluation include:

- Baseline database- Baseline descriptive statistics report- Follow-up database (first follow-up)- Final impact evaluation report

31

Timeline

Table. Mozambique ECD Impact Evaluation Timeline (preliminary)

S O N D J F M A M J J A S O N D J F M A M J J A S O N D

Evaluation Design X X X X

MOU X X

Site Selection X X X

REOI X

RFP X

Contracting of Survey Firm X X

Human Subjects X

Survey Instrument pre-pilot & translation X X

Field Training and Pilot Testing X

Baseline Survey - phase 1 treatment communities (15) X

Intervention roll out phase 1 treatment communities (15) X X X X X X X X X X X X

Baseline Survey - phase 2 treatment communities (15) X

Baseline Survey - control communities (60) X X

Intervention roll out phase 2 treatment communities (15) X X X X X X X X X

Data capturing/ processing X X

Final baseline data and report from Survey firm X

Baseline Data analysis X X

Preliminary baseline report X

Baseline report review and revision X

Final baseline report X

Contracting of Survey Firm X X

Survey Instrument pre-pilot X

Field Training and Pilot Testing X

End-line survey X X

Data capturing/ processing X X

Data analysis X X

Preliminary End-line evaluation report X X

End-line evaluation report review X

Final evaluation report X

2008 20092007

32

APPENDIX 1:

Power Estimates

Power has been estimated trough Optimal Design software. Averages, standard deviations and intra-cluster correlations were calculated using the Mozambique’s 2003 DHS data. We restricted those estimates for the rural region of the country. The number of clusters used at the estimation is 76. We considered 23 observations per cluster, not accounting for the 5 communities with oversubscription. This gave us a lower bound for power once the number of observations was kept fixed.

Variables from Mozambique DHS 2003:

Height for age (months) in standard deviations from the American populationWeight for age (months) in standard deviations from the American populationChild has been received BCG vaccinationChild had fever in the last two previous weeksChild had cough in the last two previous weeks

Summary statistics for rural region of Mozambique

33

Weight 5234 -124.354 123.8948 -593 556 0.02405Height 5234 -183.2639 144.4584 -600 519 0.02088Cough 770 0.1967071 0.3975435 0 1 0.01113Fever 5780 0.266955 0.4424069 0 1 0.00000*BCG 5874 0.7933265 0.4049537 0 1 0.10124

Min Max Intraclass Correlation

Variable Obs Mean Std. Dev .

Graphics of power against effect size

Weight for ageIntra-cluster corr. = 0.02405. Power of 90% is reached with effect of 0.19 SD

Height for ageIntra-cluster corr.= 0.02088. Power of 90% is reached with effect of 0.19 SD

34

Probability of having cough in the last two weeksIntraclass corr=0.01113. Power of 90% is reached with effect of 0.177SD

Probability of having fever in the last two weeksIntraclass corr=0.00000. Power of 90% is reached with effect of 0.158 SD

35

Probability of having received BCG vaccinationIntraclass corr= 0.1. Power of 90% is reached with effect of 0.28 SD

36

Appendix 3:

Protocol for Child level Randomization When Program is Oversubscribed

The new escolinhas have only 3 classes, and a maximum capacity of 35 children per class. According to pedagogical studies, this number is the ideal one to allow the teacher keeping a personalized treatment to all its children. That means that each escolinha can only have up to 105 children.

In some communities, it may happen that the number of children applying for a place exceeds 105. In this case, we intend to implement a process that can be, simultaneously fair and simple, that can be accepted by the community and that fulfill the goals of the Escolinhas, including the most vulnerable children.

37

For this reason, the following procedure is proposed to select the enrolled children:

1- The CDA will schedule with the community one specific day to register all the children whose parents are interested to enroll. At this day, the CDA will fill the enrollment template for the children that are applying. That’s very important that the caregiver is present by this moment to correctly fill the enrollment template.

2- Although each escolinha has a maximum capacity of 105 children, we want that all children have the opportunity to get enrolled. This must be emphasized to the community. It can happen that, even giving priority to orphaned children, some places are left to be filled. Consequently, that’s very important that the CDA explain and reiterate to the community that every caregiver who wants to enroll its children at the escolinha has to be present to fill the enrollment template. Even if the child is not selected at the first attempt, its name will be kept in a waiting list and she/he can be selected later.

Once the enrollment template is filled, children will be selected through the following way:

A. Orphaned children are automatically eligible. B. Children whose parents are members of the escolinha’s

committee are also automatically eligible for the program, recognizing the time and the effort spent with the program. Grandchildren of the committee members are also automatically enrolled provided that they live in the same

38

household of its grandparents (This should be checked by the CDA).

If the number of applicants is greater than 105, every child shall be given the same opportunity. The best way of doing so in a transparent and fair process is by making a lottery. Only through a lottery we can guarantee that every child is given the same chance of being selected. In order to participate of the lottery it is essential that the caregiver fills the enrollment template.

Lottery procedure for the first year:

1. All applying children that fulfill the requirements A and B are automatically enrolled.

2. If the number of applicants is greater than 105 all children (except those automatically enrolled) will participate from a lottery. They will be randomly selected, regardless the order of their application.

3. All the children that were not selected at this moment will have their names registered in a waiting list, which may also be produced by the lottery (the ordering of the waiting list is also produced randomly by the lottery). Once a child drops out, the place will be replaced according to the order set at the waiting list.

The lottery must be done in a transparent way, preferably by he CDA or by someone highly respected at the community. A very simple way of doing so is by writing the names in pieces of paper, cutting those small stripes, mixing in a hat, thus picking each stripe of paper at each time and announcing the name of the child to the community.Lottery Procedure for the second year:

39

i. All the children enrolled in the past year are automatically eligible for enrollment in the following year.

ii. If there is an empty place, children fulfilling the criteria A and B will be given priority.

iii. If, after all, there are still places to be filled, those places will be filled with other applicants from the community. If the number of lottery and a waiting list will be produced.

Eligibility Criteria

Only children more than 36 months (3 years) or less than 66 months (5 years and 6 months) are eligible for the program. Children can stay at the program until 72 months. Children must live at the community.

Requirement and Replacement

Children must attend classes. Children who miss 12 classes within a period of one month will be replaced. The attendance list will be checked each month by the CDA. Replacement will follow the waiting list. Parents may also commit to attend the monthly meetings with the CDAs.

40