Wendy Walker and Claude Bodart

EASS/EARD

Both nationwide programs designed and supported by ADB Both programs target 5% of the population The food stamp program has designed and implemented a national Proxy Means

Test (PMT) which is the basis for targeting in both programs.

Foodstamps

◦ Response to impact of high inflation for the poor during food and oil crisis in 2008. The objectives of the Food Stamp program are to support the consumption of basic foods by extreme poor families. The Food Stamp program distributes benefits via foodstamps or electronic card to selected poor households so that they can purchase a specified number of high protein foods in the market place.

Medicard ◦ Response to impacts of financial crisis (2010). The specific objectives of the

Medicard program are to ensure access to health services of the poor by reducing the amount of out-of-pocket expenses paid by the poor, so that they seek medical care when needed and have access to discounted medicines.

When fully rolled out nationally the programs are

intending to reach about 18,000 households or over

100,000 people.

The transfer in terms of food stamp is about USD 33

per household per month so that the program is a

large percentage of food expenditure of eligible

households.

The Medicard program is open to the target population

irrespective on their health insurance status. ◦ For not insured poor the program will pay for outpatient,

laboratory and diagnostic tests, impatient care at the

secondary level of medical care and discounted medicines,

and for insured poor it will pay for co-payments for most of

medical care and for the cost of discounted medicines.

The purpose of the evaluation is to provide

evidence to the government on the impact of the

programs. It will provide the government with good

indications of whether or not the program had

achieved the expected outcomes or results

In turn the evaluation results could also be used to

decide whether to expand the coverage of the

programs.

The selection of a control group is always a challenging task: ◦ we need a control group because it is not enough to

observe an improvement in the treatment group, ◦ but we also want to single out the specific improvement

due to the programs

At the same time control and treatment groups must be comparable

Control and treatment are chosen based on the targeting criteria: the proxy means test score

The cut-off point for programme eligibility is exogenous and so households below and above the threshold should be ideal treatment and control groups

Non-equivalent control group design (Regression discontinuity design)

Period of one year Baseline Endline Treatment group 1 O1 FS and Medicard O2 (HHs in 5 % PMT score) Comparison group O3 No treatment O4 (HHs in 5-10 % PMT score) Both treatment (beneficiary) and comparison groups received an

initial observation (the pretests O1 and O3). ◦ treatment group was drawn from the eligible households in the

bottom 5 % PMT score ◦ comparison group from the following 5 % PMT score (who is not

eligible for food stamps or Medicard). The treatment group then receives the Medicard and FS programs,

but the comparison group does not receive either of these programs.

After the implementation period of 1 year is completed, a second set of measurement observations is made (O2 and O4).

We are assuming that the lowest 5% and the next 5% of households are similar in living standard. So we are expecting that O1 would be approximately equivalent to O3. This equivalence can be established by comparing O1 and O3.

After both program’s implementation, we are expecting O2 will be greater than O4 if the programs had an impact (looking at absolute and proportional change and also controlling for other factors).

Because both groups were approximately equivalent at the beginning of both programs, any difference between the treatment and comparison groups will be attributed to the effect of implementation of both programs.

This allows us to assess the impact of the two programs.

The Food Security analysis will use specially designed FANTA modules to measure food security. These modules should enable us to make cross country and even international comparisons of food security, ie. how the households in our sample compare to other households in food security issues.

Validating the targeting. Under the food stamp program, we were able to get a Food Stamps module into the Household Socio-Economic Survey (a nationally representative survey, conducted independently by the National Statistical Office). Through these data we will be able to have national data on who is receiving or not receiving the food stamps and see to what extent they are poor based on the national distribution.

This will enable us to determine whether the targeting by PMT and its implementation was effective or not.

Not a lot of targeted programmes can assess the targeting method in this way using the national poverty survey. In the case of Mongolia, it was particularly good to achieve such cooperation between statistical office and Ministry of Social Welfare.

The food stamps program has two rounds of qualitative impact monitoring built into the program. This will help to further contextualize the impact evaluation findings.

Status of impact evaluation work Baseline: 1,000 hh surveyed. Completed in

December 2011, exploiting the roll-out of the PMT data collection (which in October 2011

covered Ulaanbaatar, Orkhon and Selenge aimags)

Analysis: Ongoing

Endline survey: Planned for October-November 2012.

Analysis: Planned for January-February 2013. This is also when the HSES data will be available.

Not just a pilot and experimental – real time and real programs

◦ Implementation of Food Stamps and Medicard in the sampled areas

was delayed: households started to receive support only at the end of

April/May and therefore they will be re-interviewed only after few months

of treatment

◦ Threshold changes: The baseline sampled households on both sides of

the PMT threshold. After the baseline data was collected, the

Government decided to increase the threshold for foodstamps. This

effectively meant that most of our ‘control’ group was then targeted to

receive the food stamps. These are the challenges of ‘real-world’ quasi-

experimental impact evaluations.

◦ New benefits with same target population : In June 2012 Government

announced that part of a large cash distribution of mining revenue would

be targeted to the same 5% households.

◦ Typical impact evaluations (that are done by JPAL and Innovations for

Poverty Action) do not face such problems because in general, they

control the intervention and the evaluation.

◦ At follow-up we will need to re-sample a control group (adding an

extra 500 hh) because the initial control group was also partly covered by

the programs. Treatment and control group will be compared using a

regression discontinuity design.