evaluation policy and guidelines for evaluations · 5 evaluation policy and guidelines for...
TRANSCRIPT
IOB
Evaluation policy and guidelines for evaluations
IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | P
Evaluation policy and guidelines for evaluations
tyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüssstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTeneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP
vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP
vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
Evaluation policy and guidelinesfor evaluations
3
Evaluation policy and guidelines for evaluations
Contents
1 Introduction 4
2 Short history 6
3 Basic principles of IOB’s evaluation policy 10
4 Definitions and criteria for evaluations 14
4.1 Whatisevaluation? 15
4.2 Typesofevaluation 16
4.3 Evaluationcriteria 17
4.3.1 Efficiency 17
4.3.2 Effectiveness 18
4.3.3 Impact 19
4.3.4 Relevance 19
4.3.5 Sustainability 20
4.3.6 Policyconsistency 21
5 Guiding principles and ethical standards 22
6 The evaluation process 26
6.1 Programming 27
6.2 Preliminarystudy 29
6.3 TermsofReference 30
6.4 Deployingresearchers 32
6.5 Cooperatingwithresearchersinpartnercountries 32
6.6 Methodology 33
6.7 Reporting 37
7 Quality control 40
7.1 Internalpeerreview 41
7.2 Referencegroups 42
7.3 Departmentalmeeting 42
7.4 Advisorypanel 43
7.5 Self-evaluation 43
8 Communication and feedback 44
1Introduction
5
Evaluation policy and guidelines for evaluations
ThisdocumentdescribestheevaluationpolicyandguidelinesforevaluationsofthePolicyandOperationsEvaluationDepartment(IOB)oftheDutchMinistryofForeignAffairs.ItplacestheevaluationpolicyinthecontextoftheOrder on PeriodicEvaluation and Policy Information(knownasRPE2006)1,thegovernment-wideframeworkfortheevaluationofgovernmentpolicyingeneral.ItalsoreflectsrecentdecisionsregardingthefunctionofevaluationattheMinistryofForeignAffairs,andtheroleIOBplayswithinit.
IOB’sevaluationpolicyandtheguidelinesmustbeseeninthecontextofachangedenvironment.AnincreasinglycriticaloutsideworlddemandsaclearerinsightintotheresultsoftheNetherlands’foreignpolicy.Thereisgrowingpressureforministerstoaccountnotonlyfortheuseofpublicfunds,butevenmoresofortheresultsachieved.Thisalsoappliestothevariousareasofforeignpolicy.Ifevaluationsaretocontributetothis,theymustbeindependent,public,transparentandaccessible.
Inpublishingtheseguidelines,IOBwishestoprovidetransparencytothirdparties:colleaguesattheMinistryofForeignAffairs,theDutchparliament,organisationsandinstitutionsaffectedbyIOBevaluations,usersoftheevaluationresults,andinterestedcitizens.Ineachevaluationreport,IOBaccountsforthespecificevaluation;thisdocumentclearlypresentsthegeneralprinciplesandguidelinesIOBappliesinternallyinconductingevaluations.
TheguidelinesoutlinedinthisdocumentapplyinprincipletoallevaluationsconductedbyIOB,unlessexplicitlystatedotherwise.ItshouldbeborneinmindthattheNetherlandshasalongtraditionofevaluationinthefieldofdevelopmentcooperation,whichisexpressedthroughinternationalagreements,fora,cooperation,etc.andwhichimpactsonIOB’swork.Thatisnotthecaseforotherareasofforeignpolicy.Becauseitsmandatealsoextendstotheseotherareasofpolicy,IOBisinanexceptionalposition,especiallyinternationally.
TheevaluationpolicyandguidelinesreflecttheopinionsandperspectivesthatunderpinIOB’swork.Inthatsense,thisdocumentdescribestheidealsituationforIOB.Realitywillsometimesdeviatefromthisidealinsomeways.Therewillalwaysbeacertaintensionbetweentheoryandpractice.However,IOBaimstoworkasmuchaspossibleaccordingtotheseguidelines.Itisimportanttorememberthatneitherpolicynorguidelinesarecarvedinstone.Thismustbeseenasagrowthdocument,inthemostliteralsense.Newinsightsmayleadtoitbeingadaptedinthefuture.IOBwillthusperiodicallyreviewitspolicyandtheguidelinesand,ifnecessary,revisethem.Thelatestupdatedversionofthisdocumentwillbeavailableathttp://www.minbuza.nl.
1 Order on Periodic Evaluation and Policy Information 2006, Government Gazette, 26 April 2006, no. 83.
2Short history
7
Evaluation policy and guidelines for evaluations
Inthefirsthalfofthe1970sthebudgetfordevelopmentcooperationincreasedrapidly.Sorapidlythatconcernswerevoicedfromvariousquarters,andespeciallyinthemedia,thatthiswashavinganadverseeffectonthequalityofaid.Asaconsequenceofthisnegativepublicity,theMinisterforDevelopmentCooperationfelttheneedforanindependentopinionondevelopmentpolicy,autonomousofthepolicydepartmentsinvolved.On1July1977theOperationsReviewUnit(IOV)wassetupbyministerialorder.ItwasmadeclearfromthebeginningthatIOVwouldfocusonbilateralcooperation,bothgovernmentalandnon-governmental(insofarasthelatterwasgovernment-funded),butwouldinprinciplerestrictitsactivitiestotheNetherlands’contributiontojointinterventions.Onthebasisofthisrestriction,theministertookthepositionthatIOVcouldconductinspectionswithoutthepermissionoftherecipientcountryororganisation.IOV’sworkprogrammewasdrawnuptogetherwiththepolicydepartmentsandsubmittedtotheministerforapproval.
OutsidetheworkprogrammetheministerandtheministrymanagementwerealsoauthorisedtocommissionIOVtoconductinspectionsonanadhocbasis.However,thisoccurredonlysporadically.FromthestartIOVfocusedonindividualprojects,submittingconfidentialreportstotheminister.Thereportswereconfidentialbecausetheyreferredtoidentifiableeventsandindividuals.
From1979theHouseofRepresentativesoftheDutchparliamentcalledeachyearforIOV’sreportstobemadeaspublicaspossible.Eventually,itwasagreedthatgeneralreportswouldbesenttoparliament,togetherwithsummariesofprojectinspections.ThefirstsummarisedreportthatIOVsenttotheHousewasthe1984GeneralEvaluationofDutchBilateralDevelopmentCooperation.
Inthe1980s,IOV’sareaofworkshiftedfromtheinspectionofindividualprojectstoevaluationsofsectors,policythemesandcountryprogrammes.DuringverbalconsultationswiththeHouseofRepresentativesin1987theministeragreedthatallIOVreportswouldbefullypublicandwouldbesenttoparliament,togetherwithapolicyletterfromtheministercontaininghisorherresponsetothereport.ThereportswereusedactivelyindebatesbymembersofboththeHouseofRepresentativesandtheSenate.Universitiesandotherinstitutionsinthepublicandprivatesectorsalsoshowedalivelyinterestinthereportsoncetheybecameaccessible.
In1996afar-reachingreassessmentofDutchforeignpolicytookplace.Itwaswidelyacknowledgedthatdevelopmentcooperation,foreignpolicyandinternationaltradecouldnolongerbeseenasindependentpolicyareas.Foreignpolicywas‘decompart-mentalised’andthevariousareasofinternationalactivitywereintegrated.Forthepurposeofthereview,IOVpublishedanewsynthesisstudy,summarisingtheresultsoftheevaluationreportsoftheprecedingdecade.Duringthereorganisationthataccompaniedthepolicyreassessment,itwaslogicaltomodifyIOV’smandate.Whereas,beforethereview,IOVwasonlyresponsibleforevaluationsinthefieldofdevelopmentcooperation,itsareaofoperationswasnowextendedtoincludethe
8
2 Short history
wholeofforeignpolicy.Thisimpliedthat,inthefuture,theUnitcouldaddressitselftoallsubjectsthatfellundertheHomogenousBudgetforInternationalCooperation(HGIS).ThechangeinIOV’smandateledtoarevisionofitspositionintheorganisationalstructureoftheministry.ItwouldnolongerbeaccountabletotheDirector-GeneralforInternationalCooperation(DGIS)butdirectlytotheSecretary-General.TheUnitwasalsogivenanewname:inthefutureitwouldbeknownasthePolicyandOperationsEvaluationDepartment(IOB).
InrecentyearsIOB’spositionandrolehasonceagainbeenthesubjectofpoliticaldebateanddiscussionwithintheministry.Manyoftheissuesdiscussedinthe1980swereonceagainontheagenda,includingtheindependenceofevaluation,thepublicnatureofthereports,therelationshipbetweenexternalaccountabilityandinternallearning.Thisledin2008totheMinisterforDevelopmentCooperationreconfirmingtoparliamenttheindependenceofIOBandthepublicnatureofitsreports.However,IOB’sprogrammingauthoritywasrestrictedtodevelopmentcooperation.2Intheotherfieldsofforeignpolicy,primaryresponsibilitynowlieswiththeDirector-Generalconcerned,whocanrequestIOBtoconductevaluations.
2 Letter from the Ministers of Foreign Affairs and for Development Cooperation of 16 October 2007 on implementation of the Boekestijn/Gill’ard motion regarding the Policy and Operations Evaluation Department (IOB) (31200V, no. 9). Report from the Permanent Committee on Foreign Affairs on IOB of 29 May 2008, approved on 25 June 2008.
9
Evaluation policy and guidelines for evaluations
3Basic principles of IOB’s evaluation policy
11
Evaluation policy and guidelines for evaluations
ThePolicyandOperationsEvaluationDepartmentistheevaluationdepartmentoftheMinistryofForeignAffairs.Institutionally,IOBisdirectlyaccountabletotheSecretary-General.Itformspartoftheministry’sAuditCommittee.TogetherwiththeAuditDepartmentandtheInspectionandEvaluationUnit(ISB),IOBformsthecluster‘InControl’.
Internationally,IOBisamemberoftheOECD/DACnetworkofevaluationservices.ItalsotakespartinconsultationsbetweentheevaluationservicesoftheEuropeanUnion,andofthecountriesunitedintheNordic+group.IOBcontributestotheactivitiesofanumberofotherformsofinternationalcooperation,includingtheconsultationcommitteeofGerman-speakingevaluationservices(DACH),theNetworkofNetworksforImpactEvaluation(NONIE)andtheInternationalInitiativeonImpactEvaluation(3IE).
IOB’smandateisasfollows:
The Policy and Operations Evaluation Department meets the need for evaluation of policy and its implementation with regard to all policy areas that are financed by the homogenous budget for international cooperation (HGIS). IOB evaluates the implementation of the Netherlands’ foreign policy, giving particular attention to its relevance, efficiency and effectiveness, the sustainability of the results achieved and the consistency and coherence of the policy.
Inthelightofthismandate,IOBhasthefollowingtasks:
1)toconducthighqualityevaluationsforlearningandaccountabilitypurposes;2)toensurebalancedevaluationprogramminginthefieldofdevelopmentcooperation
andtoadviseonprogramminginotherareasofforeignpolicy;3)tocontributetoimprovementofthequalityofevaluationsconductedby,oronbehalf
of,policydepartments,institutionssubsidisedbytheministry,andmultilateralinstitutionsbydrawingupgenerallyapplicablequalityrequirementsandguidelinesforconductingevaluations,advisingonthedesignandimplementationofindividualevaluations,andconductingsystematicassessmentsoftheseevaluationsorpeerreviewsofotherevaluationservices;
4)tostrengthencapacitytoconductevaluationsinpartnercountriesbysupportingeducationandtrainingandparticipatinginjointevaluations;
5)topromotethedistributionandcommunicationofevaluationresults.
Re1:IOB’smaintaskistoconducthigh-qualityevaluationswithintheframeworkoftheMinistryofForeignAffairs’evaluationprogramme.TheseevaluationscanrelatetoallareasofforeignpolicythatfallundertheHGIS.Inthiscontext,evaluationcomprisescriticalandthoroughinvestigationofapolicyorprogrammeandexplainingtheresults.IOBevaluationsdonotstandinisolationbutareinprinciplepartofthepolicycycle,inwhichplanning,monitoringandevaluationareseenasanintegratedwhole.
12
3 Basic principles of IOB’s evaluation policy
Evaluationisthereforeanessentialcomponentofthe‘knowledgefunction’withintheministry.Knowledgeontheimplementationandimpactofpolicysupportstwoconcretefunctions:
• itprovidesthebasisforaccountingtoparliamentandsocietyatlargeforpolicyandexpenditure(accountabilityfunction);
• itgeneratesinformationforthepurposesofimprovingpolicyformulationandimplementation(learningfunction).
Re2:AsstatedintheRPE2006systematicevaluationmustbeafixedcomponentofeverypolicyprocess.Eachpolicyareamustbeperiodicallyevaluatedatthelevelofoperationalobjectives.TheRPE2006thusformsthebasisofevaluationprogramming.Theprogrammingisalsodeterminedbyspecificrequestsfromparliament,ministersandstatesecretaries,andbudgetholders.WithintheMinistryofForeignAffairs,IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperationandadvisesonprogramminginotherareasofforeignpolicy.
Re3:IOBisnottheonlyunitattheMinistryofForeignAffairsconcernedwithevaluation.Policyadvisorsalsohavetodealwithevaluations,asthecontactforevaluationprogrammingwithintheirpolicydepartment,asa‘client’commissioninganevaluation,sometimesas(joint)implementer,orasauseroftheresults.Inadditiontoconductingevaluations,IOBthereforehasanadvisoryfunction.ThisentailsformulatingqualitycriteriaandguidelinesforimplementingevaluationsandadvisingontheTermsofReferenceforevaluationsthatIOBdoesnotconductitself.Duringtheannualsettlementoftheprogramme,agreementsaremaderegardingtheextenttowhichIOBwillplayanadvisoryroleinevaluationsconductedbyoronbehalfofministrydepartments.IOBalsoadvisesonthedesignoftheministry’sevaluationfunction,theevaluationsystemandinstruments,andcoordinationwithplanningandmonitoring.InadditionIOBperformsqualitytestsonprogrammeevaluationsconductedbycofinancednon-governmentalorganisations(MFS),withtheaimofstrengtheningtheirevaluationcapacity.
ForthemultilateralchannelIOBisinvolvedininternationalreviewsoftheevaluationservicesofrelevantinstitutions.
Re4:Strengtheningevaluationcapacityinpartnercountriesisanimportantthemeindevelop-mentcooperation.Independentevaluationcapacityinpartnercountriesisindispensableiftheaimofestablishingasystembasedonmutualaccountabilityistobeachievedinreality.IOBseesitasitstasktocontributetothisbysupportinginternationalinitiativesaimedatstrengtheningevaluationcapacity.Inaddition,wherepossibleandsafeguardingitsowncriteriaforindependenceandquality,IOBworkstogetherwithresearchersfrompartnercountriesinconductingevaluations.
13
Evaluation policy and guidelines for evaluations
Ad5:Evaluationsgenerateknowledgeonthebasisofexperiencefromthepastregardingwhatworksandwhatdoesnot,andespeciallywhy.IOBconsidersitimportantthattheresultsofevaluationsaredistributedaswidelyaspossible.Toachievethis,itdiscussesthefindingsinstaffmeetings,withministersandstatesecretaries,andwithothernationalandinternationalpartiesinvolvedintheevaluations.Inaddition,wherepossibleandwhereitdoesnotconflictwithitsownresponsibilities,IOBwilloffersupportinpreparingpolicyresponsesanddescriptionsofevaluationfindingsintheMinistryofForeignAffairs’AnnualReports.
4Definitions and criteria for evaluations
15
Evaluation policy and guidelines for evaluations
4.1 What is evaluation?
Thereisawidevarietyofdefinitionsoftheconceptofevaluation.Inthe1980sand1990s,theOECD/DACworkedonasharedframeworkofconceptsforevaluationpracticesinthefieldofdevelopmentcooperation.TheresultingdefinitionofevaluationservesasabasisforIOB’swork.
‘Anevaluationisanassessment,assystematicandobjectiveaspossible,ofanongoingorcompletedproject,programmeorpolicy,itsdesign,implementationandresults.Theaimistodeterminetherelevanceandfulfilmentofobjectives,developmentalefficiency,effectiveness,impactandsustainability.Anevaluationshouldprovideinformationthatiscredibleanduseful,enablingtheincorporationoflessonslearnedintothedecision-makingprocessofbothrecipientsanddonors.’3
TheOECD/DACdefinitionreflectsanunderstandingofevaluationthatisbroadlysupportedintheinternationalcommunityofevaluators.Thedefinitionisintendedforevaluationsinthefieldofdevelopmentcooperation.SinceIOB’sworkextendsfurtherthandevelopmentcooperation,conceptslike‘developmental’and‘recipientsanddonors’willnotapplytoallIOBevaluations.
Evaluationisdifferenttoreviewsandmonitoring.TheOECD/DACGlossarydescribesareviewasanassessmentoftheperformanceofanintervention,periodicallyoronanadhocbasis.ForIOBthemaindifferencebetweenareviewandanevaluationisthatareviewisconsideredamanagementinstrumentforoperationalmonitoringoftheimplementationofanactivity,andthereforefocusesonoperationalaspectsandrecordingresults,whileevaluationisamorein-depthassessmentoftheaims,implementationandresultsofanintervention,withthequestionofattributionasitsmainfocus.
Monitoringisaninstrumentofinternalmanagement.Itaims,onthebasisofafixedsetofindicators,toprovidearegularinsightintotheprogressmadeinimplementingactivities.Monitoringallowsabudgetholderortheimplementerofaninterventiontostayuptodateoninputssuppliedandactivitiesperformed(outputs).Monitoringshowswhetherintendedactivitieshaveactuallybeencarriedout.Good,systematicmonitoringisofgreatimportanceforevaluations,asitprovidesasignificantpartofthedataonwhichevaluationisbased.
3 DAC principles for Aid Evaluation, OECD/DAC, 1991 (the definition was also included in the DAC Glossary
of Key Terms in Evaluation and Results-based Management in 2002).
16
4 Definitions and criteria for evaluations
4.2 Types of evaluation
TheOrderonPeriodicEvaluationandPolicyInformation2006(RPE2006)prescribesasystemofevaluationbasedontheso-calledoperationalobjectives,OD.Thesystementailsperiodicalevaluationoftheseoperationalobjectives.
TheRPE2006distinguishesbetweenpolicyreviewsandex-postimpactstudies.Bothconceptsdeviatefromtheevaluationterminologyusedinternationally,butaretheequivalentofthemorecommonterms‘policyevaluation’and‘impactevaluation’.Thestartingpointisthatpolicyreviews,whicharelinkedtotheoperationalobjectives,areasfaraspossiblefedbyimpactstudies.Inprincipletheyresemblesynthesisstudies.IOBgivesprioritytothesetypesofevaluation,where–especiallyinthefieldofdevelopmentcooperation–policyreviewsareprecededbyaseriesofex-postimpactstudies(impactevaluations).
OnthebasisofthesystemprescribedintheRPE2006,IOBimpactevaluationswillgenerallyfocusoninterventions(rangingfromprogrammestovariousformsofbudgetsupport).Policyevaluations(orreviews)addressthemesorsectors.Inaddition,integratedforeignpolicycanalsogiveriseto‘decompartmentalised’evaluationsofcountryorregionalpolicies.
Besidesthese–usuallyextensive–evaluations,IOBcanalsoconductshortinternalevaluationsthatarelimitedinnatureandnecessarytomonitortheprogressofprogrammes.
PartoftheevaluationsinIOB’sprogrammingfocusessolelyonDutchprogrammesandactivities.ItislogicalfortheseevaluationstobeconductedbyIOB.However,itisbecomingincreasinglycommontoworktogetherwithinternationalpartnersinthefieldofdevelopmentcooperation.Projectsbyindividualdonorsaregraduallybeingreplacedbyjointfinancingofgovernmentpolicyinpartnercountries.Itisthenalsologicalthatdonorsgivegreaterpreferencetothejointprogrammingandimplementationofevaluations.Incaseswheredonorsworktogether,forexampleinprovidingjointsupportforsectorpolicyorotherfar-reachingformsofjointfinancing(suchasbasket-funding),IOBevaluationswillinprinciplebeconductedincooperationwithotherdonors,aslongasIOB’scriteriaforindependenceandqualityaremet.Thenatureofthecooperationcanvaryfromactualparticipationbyseveraldonorsinanevaluation,agreeingonadivisionoftaskswithinasharedTermsofReference,toagreementonthedivisionoflabourbetweendifferentthemes,sectorsorcountries.
17
Evaluation policy and guidelines for evaluations
4.3 Evaluation criteria
Evaluationcallsforatailor-madeapproach.Thatdoesnotmean,however,ignoringqualitycriteriaorabandoningconsistencyinapproach.Inallevaluations,theevaluationcriteriamustbeappliedinanunambiguousand,aboveall,transparentway.Here,inparticular,standardisationisparamounttoenableevaluationsandtheirresultstobecompared.TheevaluationcriteriaappliedbyIOBarelargelyidenticaltothosedrawnupbytheOECD/DAC.TheOECD/DACdefinitionofevaluationspecifiesfivecriteriaforevaluationofdevelopmentinterventions:efficiency,effectiveness,impact,relevanceandsustainability.Theevaluationcriteriarefertodifferentresultlevelsfoundedonthelogicalframeworkorresultchainsbasedonit.Together,thesecriteriaprovideaframeworkwithinwhichresearcherscanpresenttheirfindingsinanorderedandcoherentway.Theextenttowhichequalattentioncanorwillbepaidtoallcriteriawillvaryperevaluation.Furthermore,additionalcriteriacanbeapplied:evaluationof‘decompartmentalised’policy,forexample,mayrequirethatIOBdevotesexplicitattentiontopolicyconsistency.
4.3.1 Efficiency Efficiencymeasureshoweconomicallyresources(inputs)andthewaytheyareappliedareconvertedtodirectresults.Theconceptthereforeillustratestherelationshipbetweeninputandoutputandreferstoalevelofresultsthattheimplementercancheck.
Inevaluatingefficiency,boththequantityandqualityofinputsandoutputsareassessed.Themosteconomicalorcost-effectiveinputisnotalwaysthemostappropriateandtherecanbesignificanttradeoffsbetweenthequantityandqualityofoutputs.Assessingefficiencyalsocallsforcomparisonwithalternativeapproaches,withwhichthesameoutputscanbeachieved.Thisisbynomeansalwayseasybecausethereareoftennopreviouslyformulatedbenchmarksandtheymighthavetobedevelopedonthebasisofpracticalexperience.Thatismorelikelyinthecaseof,forexample,interventionsinvolvingconstructionactivitiesthanofthoseaimingtobringaboutsocialorinstitutionalchange.
Thefollowingelementsshouldbetakenintoaccountwhenevaluatingefficiency:
• Haveappropriateinputsbeendeployedatthelowestpossiblecost?• Haveactivitiesbeenconductedinanuncomplicatedmanner?• Havedecisionsbeenmadeattherightlevelandhasbureaucracybeenavoided
asfaraspossible?• Haveoverheadcostsbeenkepttoaminimum?• Hasduplicationbeenavoided?• Haveconflictsduringimplementationbeenpreventedorsolved?• Haveoutputsbeenachievedwithintheplannedperiodandbudget?
18
4 Definitions and criteria for evaluations
4.3.2 Effectiveness Effectivenessrelatestotheextenttowhichthedirectresultsofinterventions(output)contributetothesustainableachievementofpolicyobjectives(outcome).Policyobjectivesaretheobjectivesthatinterventionsareintendedtoachieveandshould,inprinciple,beformulatedintheprojectdocument.Insomecasespolicyobjectivescanbeidenticaltotheoperationalobjectivesspecifiedinthebudgetorspecificcomponentsofthem.
Aninterventionisconsideredeffectiveifitsoutputshavemadeademonstrablecontributiontoachievementoftheintervention’sintendedobjectives.Efficiencyandeffectivenessrefertotwosuccessivelevelsintheresultschain.Apositiveassessmentatoneleveldoesnotautomaticallyimplyapositiveassessmentattheother.Aninterventioncanbeefficientwithouthavinganeffect,andaneffectiveinterventioncanbeinefficientlyimplemented.
Theeffectsofaninterventionarebeyondthedirectcontroloftheimplementer.Interventionsgenerateoutputswhichareassumedtogeneratethedesiredeffects.Evaluatingeffectivenessaimstoestablishthiscausality.Thecoreproblemisthatitwouldbeidealtoknowwhatwouldhavehappenediftheinterventionhadnotoccurred(thecounterfactual).Onlythencanobservedchangesbeattributedtotheinterventionandisitpossibletospeakoftheneteffects.Inotherwords,thereisaproblemofattribution.Aslongasotherfactorshaveavisibleimpactontheobservedchanges,theproblemisrelativelysimpletosolve.Itismoredifficultwheninfluentialfactorsarenotimmediatelyvisible(the‘unobservables’).Researcherscannotknowwhattheydonotknow.Evaluationstrytofindasolutiontothisproblem(see6.6).
Inpractice–especiallyinthecaseoflargeandcomplexinterventions,butalsowithinterventionsthataimtobringabouteffectsthataredifficultorimpossibletoquantify–evaluationsoftenmakeuseofplausibilityargumentation.Itisthenimportantthattheargumentistransparentandcredible.
Inessence,threestepscanbedistinguishedintheevaluationofeffectiveness:
• Themeasurementofchangesintheeffectvariablesincomparisontothesituationatthestart(baseline);
• Attributionoftheobservedchangestotheintervention;• Assessmentofchangesobservedandattributedtotheinterventionintermsofthe
objectives.
19
Evaluation policy and guidelines for evaluations
4.3.3 Impact Thebroadconsensusinthecommunityofevaluatorsabouttheconceptsofefficiencyandeffectivenessislessevidentwhenitcomestoimpactandrelevance.
TheOECD/DACdefinesimpactasallsignificanteffectsproducedbyadevelopmentintervention,directlyorindirectly,intendedorunintended,ontheultimatestakeholdersandthirdparties.Whileeffectivenessrefersonlytotheintendedeffectsofanactivity,inthisterminologyimpactreferstoallpossibleeconomic,social,political,technicalandecologicaleffectsatlocal,regionalornationallevelthathaveadirectorindirecteffectonthetargetgrouporotherparties.
Inthestrictercontextofthelogicalframework,theconceptofimpactusuallyreferstothehighestlevelofresults.Thequestionisthentowhatextent,withtheaidofoutputs,theeffectsachieved(outcomes)havecontributedtobroader,morefar-reachingobjectives.Thesecanbeoperationalobjectivesorobjectivesatahigherlevel,suchasmoregeneralpovertyreduction.
Intheinternationalliteratureonimpactevaluations,theconceptofimpactisusedfortheneteffectsattributedtotheintervention,i.e.atthelevelofeffectiveness.
Oneinterpretationofimpactisnotbydefinitionbetterthananother.Itisimportantthatconceptsaremadeexplicitinevaluationreports.Intheinterestofmutualcomparisonitisdesirableinex-postimpactstudiestousetheinterpretationofimpactevaluationsmostcommonlyappliedintheinternationalliterature.InthisrespectIOBfollowstheinstructionsintheRPE2006.
4.3.4 Relevance Theconceptofrelevanceisalsosubjecttodifferentinterpretations.TheOECD/DACdefinesrelevanceastheextenttowhichtheobjectivesofadevelopmentinterventionareconsistentwithbeneficiaries’requirements,countryneeds,globalprioritiesandpartners’anddonors’policies.Thatmakesrelevanceacontainerconcept,withinwhichnochoicesaremade.Thisdefinitionisunsatisfactoryforseveralreasons.First,theassessmentofwhetheraninterventionisalogicalconsequenceoforiscompatiblewithpolicyisinessenceanexantequestionthatshouldbepartoftheapprovalprocedure.Second,donorpolicy,thepartner’spolicyandtheneedsofthetargetgroupdonotnecessarilyallperfectlycoincide.Thatwouldimplythataninterventioncanberelevantandirrelevantatthesametime.Third,policyprioritiescanchangeovertime,suchthataspecificinterventionisrelevantatonemomentandnotthenext,orviceversa.Lastly,thisdefinitionimpliesthattherelevanceofthepolicyitselfisnotupfordiscussion.Inotherwords,itexcludesproblemswithpolicyitself.
Inthelogicalframework,relevancereferstotheextenttowhichtheeffectsofinterventionsmakeasustainablecontributiontoachievingtheultimateobjective(theimpact).Aninterventionhasbeenvaluable,orrelevant,totheextentthatithas
20
4 Definitions and criteria for evaluations
generatedeffectsthatbringachievementoftheultimatedevelopmentobjectivecloser.IOBwillapplytheconceptofrelevanceinitsevaluationsinthisrespect.
4.3.5 Sustainability Sinceanactivitycanhardlybeconsideredeffectiveiftheeffectithasachievedisnotlasting,sustainabilityisinfactanaspectofeffectiveness.4Atthesametimetheconceptofsustainabilitycomprisesavarietyofdimensionsthatarerelevantforassessinganintervention.Inthedonorcommunitythesustainabilityofinterventionsislinkedtosevenfactors,whichcanbeusedtoanalyseinterventions.Whichfactorsorcombinationsoffactorsareusefulforassessingsustainabilitywilldependonthenatureoftheintervention.
• Theinvolvementoftherecipientisusuallyrecognisedasafactorthataffectsthesuccessofanintervention.Involvementisexpressedthroughagreementonobjectivesandtheprovisionofhumanandmaterialresources;
• Whendeployingtechnologyitisimportanttotakeaccountofthefinancialandinstitutionalcapacitiesoftherecipientandthedegreetowhichtheconditionsformaintenanceandrenewalareguaranteed;
• Interventionscanhaveaneffectontheenvironmentintheshortorlongterm.Whileshort-termeffectsareusuallyenvisagedandarepreventedormitigated,itisespeciallyimportanttoincorporatelong-termenvironmentaleffectsintheassessment,astheycanneutralisepositivechangesorhaveotherharmfulconsequences;
• Thedegreetowhichchangesaresustainabledependsverystronglyonsocialandculturalfactors,whichareoftennotdirectlyvisible.Studyingperceptionscanprovidebetterinsightintothisdimensionofsustainability;
• Institutionalandcapacitydevelopmentplayanimportantroleindeterminingsustainability.Whenassessinginterventionstheattentiondevotedtotheseaspectswillhelpdeterminetheextenttowhichsustainabilityisachieved;
• Financialandeconomicsustainabilityreferstothedegreetowhichmeasureshavebeentakentoguaranteethatactivitiescancontinueandcompletedworkscanbemaintainedinthefuture;
• Interventionsalwaystakeplaceinawidernatural,political,economicandsocio-culturalenvironment,whichcannotbeinfluencedinanywayatallbutwhichitselfhasastronginfluenceoneffectivenessandsustainability.Changesinpoliticalstability,economiccrises,ornaturaldisasterscanhaveadecisiveeffectonthesustainabilityofinterventions.
4 In the context of humanitarian aid this is, of course, different. It usually entails short-term,
punctual interventions, the effects of which it is hoped will only be necessary for a brief period.
21
Evaluation policy and guidelines for evaluations
4.3.6 Policy consistency Thisconceptreferstotheabsenceofinherentcontradictionsbetweendifferentpolicyareas.ConsistencyisparticularlyimportantforIOBintheevaluationofdecompartmentalisedpolicy.Developmentobjectivescanconflictwithinternationaltradeinterestsorforeignpolicyconsiderations.Thatiswhyitisimportant,particularlyincaseswhereobjectiveshavetobeachievedinthecontextofdecompartmentalisedpolicy,todeterminetheextenttowhichinterventionsperformedfromdifferentperspectivesarecoherentwithanddonotobstructeachother.Theaimistohelpachieveintendedobjectiveswithouttheeffectsofoneinterventionneutralisingthoseofanother.
5Guiding principles and ethical standards
23
Evaluation policy and guidelines for evaluations
IOB’sguidingprinciplesreflectthecorevaluesonwhichitsevaluationsarefounded.Theyareinterrelatedandmutuallyreinforcingand,assuch,theymustbeseenasprovidinganoverallframework,withwhichallaspectsofIOB’sevaluationworkmustbeconsistent.Theprinciplesdeterminewhatcanbeexpectedfromaninspectorandotherresearchersinvolvedinanevaluation,fromevaluationprocessesandproducts,andfromthemethodology.TheyareconsistentwiththeOECD/DAC’sPrinciplesforEvaluationofDevelopmentAssistance(1991)andEvaluationStandards(draft2009).
AllIOBevaluationsmustbeconsistentwiththefollowingguidingprinciples:
Independence Thejudgementsoftheresearchermustnotbeinfluencedbyexternalpressureoraconflictofinterests.Organisationsorinstitutionsthatarecommissionedtoperformanevaluationandmembersofevaluationteamsmaynothavebeenpreviouslyinvolvedintheinterventionsthataretobeevaluatedorinformulationofthepolicyonwhichtheyarebased.
Objectivity Theevaluationmustbebasedonverifiablefacts.Theresearchermustmakeeveryefforttoensurethatthedataonwhichtheevaluationisbaseddoesnotcontainanyinaccura-cies.Thepresentationoffactsshouldbeclearlyandrecognisablydistinguishedfromjudgments.
Transparency Theevaluationmustbeconductedinawaythatcanbefollowedclearlybystakeholdersandthirdparties.Thequestionstobeaddressed,thedatabase,theresearchmethods,findingsandconclusionsmustbepresentedinthereportinaclearandaccessibleway.
Validity TheevaluationmustmeasurewhattheTermsofReferencespecifiesasrequiringmeasurement,andmustdothisinawaythatthereadercanunderstand.
Reliability Theresultsoftheevaluationshouldbestable,i.e.thatarepeatoftheevaluationwillproducethesameresultsandconclusions.
PartnershipInthecaseofevaluationsinthefieldofdevelopmentcooperation,wherepossibleandinsofarasitdoesnotconflictwithotherprinciples,partnersfromtherecipientcountrieswillbeinvolvedintheimplementationofevaluations.Everyeffortwillbemadetocooperatewithindependentresearchersfromthepartnercountries.
24
5 Guiding principles and ethical standards
Feasibility Evaluationsareonlyconductedifthedatabaseisaccessibleandtheassociatedmethodologyisavailable.
Correctness Theevaluationmustnotcauseharmtoindividuals.
Cost effectiveness Theevaluationmustbeconductedatthelowestpossibleprice,takingaccountoftheotherguidingprinciples.
Usability Theevaluationmustresultinconclusionsandpointsforattentionthatarevaluablefortheuseroftheevaluationbysuggestingactionsthatliewithintheuser’scapacity.
Evaluationisaformofresearchthatentailsajudgment.Itdiffersfrompureacademicresearchinthatitalwaysresultsinanopinionontheresultsofconcretehumanactions.Researchersthereforehaveagreatresponsibilitytowardsstakeholdersinvolvedintheevaluation.ThisappliesequallytoIOBinspectorsandtoexternalresearcherscontractedbythem.Researcherscanthereforebeexpectedtoadheretointernationallyacceptedethicalstandardsforevaluations.
ThefollowingethicalstandardsapplytoIOB’swork5:
Respect and understanding for other cultures Manyevaluationsareconductedinadifferentculturalenvironmenttothatoftheresearchers.Theycanthereforebeexpectedtocomplywithlocalculturalcodesduringinterviewsandotherformsofdatacollection.
Universal values Someculturalcustomsandpracticesmaybeincompatiblewithfundamentalrightsandfreedoms.Althoughresearchersshouldinprincipleshowrespectforothercultures,theymustnotignoretheeffectofcertainculturalvaluesongenderrelationsorminoritiesandotherspecificgroups.InsuchcasestheyshouldactinaccordancewiththeUniversalDeclarationofHumanRightsandtheassociatedinternationalagreements.
5 The standards are largely based on the evaluation policies of the Danish International
Development Agency (DANIDA) and the Canadian International Development Agency (CIDA).
25
Evaluation policy and guidelines for evaluations
Gender Evaluationsshouldbecarriedouttakingaccountofthedifferentrolesperformedbymenandwomenandtheaimofachievinggenderequality.Inpolicyareaswheregenderdifferencesareadimensionoreffectofthepolicypursued,theyshouldbetakenintoaccountinthedesignandimplementationoftheevaluation.
Anonymity and confidentiality Researchersareexpectedtorespectrequestsfrominformerstotreattheinformationtheyprovidewithconfidentialityandtotakestepstoensurethatconfidentialinforma-tioncannotbetracedbacktothesource.
Responsibility Inmanycountriesacriticalstanceregardingthegovernmentcanhavesevereconse-quencesforresearchersfromthecountryitself.Inevaluationsinvolvinglocalresear-chersitisimportantthatsuchrisksberecognisedintimeandstepstakentoavoidthem,forexample,bymakingclearagreementsaboutwhoisresponsibleforthecontentofthereport.
Omissions Researchersmayobtaininformationthatisnotdirectlyimportantinansweringthequestionsaddressedintheevaluation,butisneverthelessrelevantinthelightoftheaccountabilityandlearningfunctionsoftheevaluation.Insuchcases,theclient(inthelastinstance,theIOBDirector)willbeconsultedtodeterminewhetherthisinformationshallbeincludedinthereportandinwhatform.
Assessment of individuals Evaluationsarenotassessmentsofindividuals.Yetevaluationreportscanincludefindingsonleadershipandthemanagementorotherqualitiesofimplementersthatcomeclosetopersonalassessments.Researchersshouldthenmakeeveryefforttopresentthefactsofeasilytraceablecasesinanabstractform.Ifindoubt,theyshouldconsultwiththeIOBDirector.
Evidence of malpractice Evaluationsarenotinvestigationsintolegitimacy.Neverthelessitispossiblethatmalpracticescometolightduringevaluations.InthatcasetheresearchershouldsubmitthecasetotheIOBDirector.Ifthereissufficientreasontodoso,theAuditDepartment(ACD)willbeinformed.
6The evaluation process
27
Evaluation policy and guidelines for evaluations
6.1 Programming
Thequalityofevaluationbenefitsfromacarefulandtransparentdecision-makingprocess.Thisprocessstartswiththeprogrammingofevaluations.IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperation.ThatmeansthattheIOBDirectorapprovestheprogrammingafterithasbeendiscussedwiththeFinancialandEconomicAffairsDepartment(FEZ)andtheotherpolicydepartmentsinvolved.Withrespecttotheotherareasofforeignpolicy,IOBadvisesontheprogramming,whichisapprovedinthemeetingoftheSecretary-GeneralandtheDirectors-General.IOBappliesthebasicprincipleslaiddownintheRPE2006forsystematisationoftheprogramming.Theevaluationprogrammingdistinguishesbetweenthreetypesofevaluation:policyreviews(policyevaluations),ex-postimpactstudies(impactevaluations)andotherevaluations.Policyreviewsandex-postimpactstudiesfallundertheRPE2006.TheRPEprescribesthateachpolicyareamustbeperiodicallysubjectedtoapolicyrevieworpolicyevaluationatthelevelofoperationalobjectives.Apolicyreviewshouldmakeasmuchuseaspossibleofex-postimpactstudies.PolicyreviewsandimpactstudieslargelydeterminetheIOB’sresearchagenda.
InadditiontotheRPE2006therecanbeotherreasonsforincludingevaluationsintheprogramming.
First,therearepoliticalconsiderations.Atcertaintimespolicyissuescanbethesubjectofpoliticalorsocialdebate.Itisthenimportanttofeedthedebatewithinsightsobtainedthroughthoroughandindependentevaluation.
Policyrelevancecanalsoplayarole.Thisisdeterminedbytheprioritygiventoanissueandthemomentinwhichtheevaluationiscarriedout.Ifitisclearwhenresultsarerequiredfortheformulationofnewprogrammesorpoliciesandanevaluationcanbeconductedwithinthatperiodwithoutlossofquality,IOBensuresthattheevaluationisprogrammedandimplementedintime,irrespectiveofthescaleorcomplexityofthesubject.
Inthefieldofdevelopmentcooperationthereisinternationalpressuretoimprovecoordinationbetweentheevaluationprogrammingofindividualdonorsandtoconductjointevaluationsasfaraspossible.Jointevaluationsincludethosecarriedouttogetherwithevaluationservicesintherecipientcountries,multidonorevaluations,andvariouscombinationsofthetwo.Workingtogetherwithevaluationservicesinrecipientcountrieshelpsstrengthentheirevaluationcapacityandimprovemutualaccountability.ThechoiceofwhichjointevaluationsIOBparticipatesinwilldependoninitiativesdevelopedduringinternationalconsultations,opportunitiestoenterintostrategicallianceswithlike-mindedevaluationservices,theextenttowhichIOB’scriteriaforindependenceandqualityareguaranteed,andwhethertheycanbefittedintoIOB’sexistingprogramming.
28
6 The evaluation process
IOBalsoconsidersitimportanttoallowspaceinitsprogrammingforadhocrequests.ThesemayberequestsfromwithintheministryforIOBtoconductshortevaluationsofalimitednaturetodeterminetheprogressofprogrammesortosupportthedecision-makingprocess.Inaddition,ministers,statesecretariesorparliamentcanrequestIOBtoconductspecificevaluations.IOBmakeseveryefforttocomplywiththeserequests.
Theevaluationprogramminginthefieldofdevelopmentcooperationrelatesnotonlytobilateralcooperationbutalsotothenon-governmentalandmultilateralchannels.Inthecontextofthecofinancingsystem(MFS)IOBconductssystematicassessmentsofprogrammeevaluationscarriedoutundertheresponsibilityoftheco-financednon-governmentalorganisationsthemselves.Inaddition,IOBselectsanumberofthemesorcountriestoconductitsownevaluations.Todate,IOBhasbeensomewhatreservedinprogrammingevaluationsofmultilateralcooperation.Togetherwithotherdonors,IOBhasencouragedmultilateralinstitutionstoimprovethequalityoftheirownevaluationsthroughpeerreviews.Inthefuture,wherepossible,IOBwillparticipateinjointdonorinitiativesaimedatamoresystematicevaluationofmultilateraldevelopmentcooperation.
ThedecompartmentalisationofforeignpolicyandthesubsequentbroadeningofIOB’smandatemeanthatIOBevaluatesnotonlyinterventionsinthefieldofdevelopmentcooperation,butalsoactivitiesinotherareasofforeignpolicy.Itisnotunusualforotherforeignpolicyinstrumentstobedeployedtoaddressaspecifictheme,alongsidedevelopmentcooperation.Thiscanbegoodreasontoconducta‘decompartmentalised’evaluation.Inmanycases,onlytheMinistryofForeignAffairsisinvolvedinsuchanevaluation,butnotalways.Therearealsoissuesofforeignpolicyinwhichotherministriesplayaprominentrole.ForIOBthatisreasontoexplorethepossibilitiesforconductingevaluationsjointlywithotherministries.Insuchcases,andifIOB’sconditionsregardingindependenceandqualityaremet,theseevaluationswillbeincludedintheprogramming.
29
Evaluation policy and guidelines for evaluations
6.2 Preliminary study
Onceasubjectforevaluationhasbeenincludedintheprogrammingapreliminarystudycanbeinitiated.Especiallyinthisinitialphase,anumberofimportantdecisionshavetobemadethatwillhavefar-reachingsignificanceforthefurtherprogressoftheevaluation.Theinspectorwillhavetobecomefamiliarwiththepolicyareatobeevaluatedandtheavailablesourcesofinformation,andformsomeideaofthequestionstobeaddressedintheevaluationandthemostadequateapproach.Withlarge,complexevaluationstheinspectorcandrawupastart-upreport.Thereportdescribesthereasonfortheevaluationandtheobjective.Itcanalsogiveabriefdescriptionofthepolicyareatobeevaluatedandformulatecorequestionsforthepreliminarystudy.Itcansuggestwhichtypeofevaluationwouldbemostappropriateandhowitmightbeorganised(theexpectedduration,whetherfieldmissionsareconsiderednecessary,andwhenandhowexternalconsultantsmightbeneeded).Lastly,thestart-upreportcanofferaninsightintotheexpectedproductsoftheevaluationandthebudget.
Start-upreportsarediscussedwiththeIOBmanagementandtheinternalpeerreviewersdesignatedfortheevaluation(see7.1).Afterdecisionshavebeenmadeontheabovementionedmainpoints,thepreliminarystudywillmoveontodraftingdefinitiveTermsofReferencefortheactualevaluation.
Thepreliminarystudycomprisesanumberofactivities:
Reconstruction of policy theory and intervention logic Theextenttowhichasubjectcanbeevaluatedisimprovedbyclearlyformulatedpolicyprinciplesandprogrammeobjectives,adescriptionoftheactivitiesundertaken,andalogicalrelationshipbetweenpolicy,objectivesandactivities.Practiceshowsthatthesecriteriaarebynomeansalwaysmet.Generallyspeaking,thisappliestobothdevelopmentcooperationandotherareasofforeignpolicy.Especiallyatthelevelofoperationalobjectives,policyisoftenbroad-rangingandnotveryspecificallyformulated,aimsandeffectsarefrequentlyunclearandthelogicofinterventionsisnotsufficientlyexplicit.Insuchcases,onthebasisofthepreliminarystudy,inspectorsthemselveswilldevelopapolicytheoryandareconstructedinterventionlogicbasedonit.Theywillthenpresentthetheoryandthelogictothepolicymakertodeterminewhethertheyareinlinewithwhatthelatterhadinmind.
Defining the evaluation objectPolicythemesandsectorsareoftenbroadlyformulatedandcoverawidevarietyofactivities.Thisappliesinparticularattheleveloftheoperationalobjectivesspecifiedintheministry‘sbudgetdocument.Thepreliminarystudyshouldthereforeestablishthelimitsoftheevaluationobject–alsointhecaseofwide-rangingevaluations–suchthatithasaclearfocusandthusremainsfeasible.Thisoffersabetterguaranteethatthefocusoftheevaluationremainsclearandthatonlythosequestionsareformulatedthat
30
6 The evaluation process
canreasonablybeexpectedtobegivenavalidanswer.Establishingthelimitsofanevaluationrelatestothetimedimension,thegeographicaldimensionandthenatureoftheactivities.Itisbasedonconversationswithrelevantinformants(stakeholders,externalexperts)andonliteratureandfilestudies.
Identifying result indicators Animportantcomponentofreconstructingpolicytheoryisformulatingaresultschain,distinguishingvariouslevelsofresults.Measurableresultindicatorsareidentifiedforeachoftheresultlevels,toenableperformancetobeassessedatthatlevel.
Identifying data sources Identifyingthescaleandqualityofexistingsourcesofinformation(files,literature,informants,statisticaldatabases)andtheopportunitiesforgatheringsupplementaryinformationdeterminetheextenttowhichthesubjectcanbeevaluatedandhowtheevaluationquestionsareformulated.
Formulating the research questions TheresearchquestionsincludedintheTermsofReferencearedeterminedbytheultimateaimoftheevaluation.Theyrelatetotheself-evidentneedforinformationonresultsachievedatthelevelofoutputsandeffects.Thisinformationisofcoursevaluableforboththeaccountabilityandlearningfunctionofanevaluation.Inaddition,supplementaryquestionsrelatingtoaspecificlearningneedcanbeformulated.
Determining the method of evaluation Thechoiceofresearchmethodsdependsonalargenumberoffactors.Itwillbeinfluencedbythenatureofthesubject,thetimeandbudgetavailablefortheevaluation,datathatisalreadyavailableandneedstobecollected,andthenatureoftheresearchquestions.
6.3 Terms of Reference
TheTermsofReferenceestablishtheconditionsunderwhichtheevaluationwillbeconductedandformtheresultofthepreliminarystudy.TheToRmustcontainatleastthefollowingcomponents:
• thereasonandjustificationfortheevaluation:ashortexplanationoftherelevanceoftheproposedevaluation;
• the(external)aimoftheevaluation;• thecorequestions:formulationofthecentralresearchquestionandthesecondary
questionsderivedfromit;• thedemarcationandscopeoftheevaluation:explanationofhowthefieldof
31
Evaluation policy and guidelines for evaluations
researchwillbedemarcatedintermsoftheme,geographicallocationandtime;• approachandchoiceofmethodology,includingfocusonsubsidiarystudies:
descriptionofthedesignoftheevaluation,theindicators,informationsourcesandresearchtechniques;
• representativeness:explanationofhowandtheextenttowhichrepresentativenessplaysaroleintheevaluation;
• organisation:explanationoftheorganisationoftheevaluation;deploymentofexperts,organisationofworkshops,missions,etc.;
• products:indicationoftheexpectedsubsidiaryproducts,includingsubsidiarystudiesoradatabase.Alsoanindicationofwhetherthereportwillbepublishedorwhethertheevaluationisinternal;
• timepath:thetimepathfortheevaluation;• budget:aspecifiedoutlineofthecosts.
IftheTermsofReferenceareforapolicyreviewthequestionsprescribedbytheRPE2006mustalsobetakenintoaccount.Policyreviewsmustaddressthefollowingquestions:
• Whatproblemgaverisetothepolicy?Doestheproblemstillexist?Whatisthecauseoftheproblem?
• Whydoesthegovernmentconsidertacklingtheproblemasitsresponsibility?Whyistheresponsibilityatcentralgovernmentlevel(andnotatlocalorEUlevel)?Whatformdoesthatresponsibilitytakeandwhy?
• Whatobjectivehasthegovernmentformulatedtosolvetheproblem?• Whatinstrumentsarebeingdeployed?Howcoherentarethey?Isthereoverlap?
Whatisknownabouttheimplementationofthepolicyandtheefficiencyofoperationalmanagement?
• Whatistheeffectoftheinstrumentsontheformulatedobjectives(solutiontotheproblem)?Doinstrumentsinotherpolicyareasalsohavesignificanteffectontheformulatedobjectives?Whatarethemainpositiveandnegativeside-effects?
• Howarethebudgetsdetermined?Onwhatbasis?
ThedraftTermsofReferencearenotonlyputontheagendaoftheIOBdepartmentalmeetingbutshouldalsobediscussedinadvancewithdirectstakeholdersandexternalexperts(preferablyintheformofareferencegroup,see7.2),andwiththeinternalpeerreviewers.Thedegreetowhichandthewayinwhichcommentsaredealtwithmustbecleartoallparties.TheTermsofReferenceforanevaluationareapprovedbytheIOBDirector;ifthesubjectoftheevaluationliesoutsidedevelopmentcooperation,theToRwillbeapprovedbytheDirector-Generalconcerned.
TheTermsofReferenceformthebasisonwhichabudgetwillbereservedfortheevaluation.Italsoprovidesaframeworkforcontractingexternalresearchers.ByapprovingtheTermsofReference,theIOBDirectoralsoapprovesthebudgetincludedintheToR.
32
6 The evaluation process
6.4 Deploying researchers
TheIOBinspectorbearsfullresponsibilityforsettingup,conductingandcompletingtheevaluation.Inmostcases,theinspectorwilldeployresearcherstoconducttheevaluation.Themomentatwhichtheywillbedeployedwillvary,aswilltheextenttowhichtheyoperateautonomously.
Researcherscanberesearchassistantsorconsultantsfromresearchinstitutions,universitiesorconsultantagencies.
Researchassistantsareusuallyrecentgraduateswhosupporttheinspectorinconductingtheevaluation.Theactivitiestheyperformvaryfromfilestudies,assistancewithfieldmissionsorinterviewinginformantstodraftingpartsofthefinalreport.Thewayinwhichtheyaredeployedwilldependontheirexperienceandcanchangeovertime.Researchassistantsarefullmembersoftheevaluationteam.
WiththemajorityofevaluationsIOBcontractsexternalconsultants.6Twocriteriaaredecisive:theexpertiseofexternalexpertsmustbecomplementarytothatoftheinspector,andtheymusthaveampleresearchexperience,preferablyinthecontextofevaluations.Variousmodalitiesarepossiblefordeployingexperiencedexperts.Oneormoreexpertscanbedeployedasadirectcounterpartfortheinspectorandadivisionoftasksagreedbetweenthem.Anotheralternativeisforwholeorpartoftheevaluationtobecontractedouttoaresearcheroraresearchbureau.Thewayinwhichexternalexpertiseisdeployeddependsonthenatureoftheservicesrequiredandassociatedcosts.Inthecaseofjointevaluationswithotherdonors,thewholeevaluationisgenerallycontractedout.
6.5 Cooperation with researchers in partner countries
InconductingevaluationsIOBaims,wherepossible,toworktogetherwithresearchersfrompartnercountries.Thathelpstobuildindependentresearchcapacityinthesecountries.Furthermore,researchersfrompartnercountriescanbeinvaluableastheyareoftenwellinformedonlocaldevelopmentsandrelations.However,notallevaluationsareofcoursesuitedtosuchcooperation.Thepossibilityofmakinguseofresearchersfrompartnercountriesdependsinthefirstplaceonthesubjectoftheevaluation.Somesubjectsdonotwarrantsuchachoice,forexample,becausetheevaluationwilladdressonlyDutchactivitiesorinterventionsofaglobalnature.Wherethedeploymentofexpertsisalogicaloption(forexample,evaluationofsectorpolicyinaspecificcountry)IOBactivelyexploresthepossibilities.
6 Contracting out and entering into contractual relationships are determined by the procedures laid
down in the Operational Procedures Manual (HBBZ). They will not be considered further in this document.
33
Evaluation policy and guidelines for evaluations
6.6 Methodology
Evaluationisaformofresearchthatentailsajudgment.Thecustomarymethodologyislargelyderivedfromthesocialandeconomicsciences.TheTermsofReferencegiveageneraloutlineofthemethodologytobeappliedintheevaluation.Whentheevaluationistobecontractedouttoexternalresearchers,theyareusuallyaskedtoelaboratethemethodologytheywilluseinaninceptionreport.
InaccordancewiththeOECD/DACEvaluationQualityStandards,IOBrequiresthatasoundmethodologyisappliedandthatitisaccountedforinthefinalreport.Themethodologicalchoicesmadeinanevaluationaredeterminedbytheextenttowhichtheyleadtothecollectionofreliabledataandprovideabasisforreachingvalidandreliablejudgments.Themethodsandtechniquesusedcanvaryperevaluation.
Thevalidityofanevaluationisdeterminedbytheextenttowhichthedesignoftheevaluation,themethodofdatacollectionandtheanalysisofthedataleadtoreplicableanswerstothequestionsposedintheTermsofReference.Validityrelatesto:
• thestatementoftheproblem(theclaritywithwhichtheproblemisstated,theformulationoftheresearchquestionsandthedefinitionoftheevaluationcriteria);
• theevaluationobject(thedefinition,operationandlimitationoftheevaluationobjectandsituatingitinthepolicyandinstitutionalcontext);
• thepolicytheory(thepresentationorreconstructionoftheinterventionlogicandresultlevels,andtheoperationalisationofresultmeasurementonthebasisofindicators);
• theanalysis(datasources,datacollectionandtheprocessingandargumentationofconclusionswithfindings).
Thereliabilityofanevaluationisdeterminedbytheextenttowhichtheresultsarestable,i.e.thatrepeatingtheevaluationwillproducethesameresults.Reliabilityisrelatedto:
• theresearchmethods(specificationofandaccountingfortheresearchmethodsused,verificationofdataandtriangulation);
• thescope(representativenessofthesampleandselectionofcasestudiesandspecificationofthelimitsoftheevaluation);
• independence(ofthesourcematerialandoftheresearchersinrelationtostakeholders);
• theprogressoftheevaluationandqualitycontrol(reportingontheprogressoftheevaluationandorganisedqualitycontrolthroughinternalandexternalmonitoring).
ToguaranteethevalidityandreliabilityofevaluationsasmuchaspossibleIOBchoosesacombinationofresearchmethods(‘mixedmethods’).Thiscombinationrelatestothedesignoftheevaluationandtoqualitativeandquantitativemethodsandtechniquesofdatacollectionandanalysis.
34
6 The evaluation process
The design of the evaluationThemajorityofIOBevaluationsaredesignedaccordingtotheguidelinesfor‘theory-based’evaluation.Thatislogicalbecausetheaimistoassesstheefficiencyandeffectivenessofpolicyimplementation.Thatrequirestakingpolicytheoryasastartingpoint.Themodeloftheory-basedevaluationisverysuitedtopolicyreviews.Theory-basedevaluationsfocusonadetaileddescriptionofwhathashappenedinthepastandwhy,andanalysingthedemonstrablecausesofthesedevelopments.Todothatinaresponsiblemanner,itisimportantinthefirstinstancetoreconstructthepolicytheory.Theevaluationthenassessesthevalidityofthepolicytheory.Themosttriedandtestedmethodofidentifyingaimsandinterventionstrategiesisthroughthelogicalframeworkortheoftenmoredetailedresultchain.Thismodeloffersaframeworkforsystematicdatacollection,identifyingindicatorsforsystematicassessment,anddeterminingtherelationshipsbetweenthedifferentresultlevels.
Analternativeforthisevaluationdesignistotakechangesinrealityasastartingpointratherthanthepolicytheory.Supportersof‘realistevaluation’advocatethiskindofapproach,whichfirstdevotesattentiontothecontextinwhichchangesoccurandthemechanismsinvolved.Onlyinthesecondinstancedoesitattempttoattributechangestospecificactivities.
Inrecentyearstherehasbeenincreasingemphasisinternationallyontheneedtoprovidestrongerevidenceoftheeffectivenessofgovernmentpolicy.Commonevaluationpractice,whichgenerallyprovidesplausibleexplanationsoftherelationshipbetweeninterventionsandtheireffects,isnolongerconsideredadequate.Thereisaneedtoidentifytheneteffectsofinterventions,ifpossibleinquantifiableterms.Thatmakesitnecessarytoconductimpactevaluations.ThedesireforharderimpactstudiesisalsoexpressedintheRPE2006,whichexplicitlydistinguishesex-postimpactstudiesasaseparatecategoryinadditiontopolicyreviews.InitsevaluationprogrammingIOBthereforeincludestheimplementationofaseriesofimpactevaluationsinappropriatepolicysectorsandthemes.Inpractice,however,theoptionsarenotunlimited.‘Randomisedcontroltrials’(RCTs),inwhicha‘treatmentgroup’iscomparedwitha‘controlgroup’andwhichineffectreproducealaboratorysituation,arenotapplicableinthegreatmajorityofcases.ThemostcommonalternativeforRCTsisaquasi-experimentalresearchdesign.AlthoughthiscloselyresemblesafullyexperimentalRCT,itrecognisesthattheresearcherdoesnothavefullcontroloverthecompositionofthetreatmentgroupandthecontextinwhichthemeasurementsaremade.Theselimitationsareaccountedforintheevaluationreport.
ForthesakeofcompletenessitshouldbemadeclearthatIOB’simpactevaluationsarealsodesignedonthebasisofatheoryofhowtheinterventionstobeevaluatedwork.Themostimportantdifferenceisthatthistheoryisnotexclusivelybasedonareconstructionofpolicy,butislargelyfuelledbyexistingscientificinsights.
35
Evaluation policy and guidelines for evaluations
Qualitative and quantitative methods of data collection Variousmethodshavebeendevelopedtomeasuretheeffectivenessofinterventions.Generallyspeakingadistinctioncanbemadebetweenqualitativeandquantitativeresearchmethods.Qualitativeresearchmethodsfocusmoreonprocessesandemphasisethebehaviourofactors.Theyareflexible,oftenbecausetheyhaveaneclecticapproach,andcanplacefindingsintheirculturalandpoliticalcontext.Aqualitativeapproachcangiveaplausibleexplanationofwhyacertaininterventionhasorhasnothadaneffectinagivencontext.Qualitativemethodsplayanimportantroleindetermininginternalvalidity.Itishowevernotalwayspossibletogeneralisethefindingsofqualitativeresearch.Quantitativeresearch,ontheotherhand,explicitlyaimstomakegeneralstatements(externalvalidity).Statisticaltechniquesmakeitpossibletoisolatetheeffectsofaninterventioninacomplexenvironment.Thismethodis,however,verydata-sensitive.Itrequiresalargequantityofreliableinformationtoallowtheformationofsamplepopulations.
Quantitativeandqualitativetechniquesarenotinterchangeable,butcomplementary.Quantitativetechniquescanprovideastatisticalbasisforamorequalitativeapproach,whileaqualitativeanalysecanexplainrelationshipsestablishedbyquantitativeresearch.Withoutatheoreticalframeworkandqualitativeresearchintothemechanismsbehindassumedrelationships,statisticalanalysishasnosubstance.InitsevaluationsIOBappliesbothquantitativeandqualitativetechniques.
Methods of data collection Thevalidityandreliabilityofanevaluationstandsorfallswiththereliabilityofthedatacollection.Wherethedataiscollecteddependsstronglyonthesubjectoftheevaluation.Whendevelopmentinterventionsarebeingevaluatedsomeofthedatawillusuallybecollectedinthepartnercountriesconcerned.Iftheyareavailable,existingdatabasescanbeused.
Themostcommonformsofdatacollectionare:
• File and literature studiesToobtainaclearpictureofthefieldofstudyitisnecessarytobecomefamiliarwiththecompleterangeofwrittensourcesavailable.Thisincludesinformationrelatingtointerventions,morepolicy-relateddocumentationandacademicliterature.Itrequiresworkingthroughthearchivesingreatdetail,preferablyusingasystematiclistofpointsforattention.Oneimportantpointforattentioninafilestudyistoassessthereliabilityoftheinformation.
• Interviews with key informants Generallyadistinctionismadebetweenstructured,semi-structuredandnon-
structuredinterviewmethods.Whichmethodisapplieddependsontheroleoftheinterviewintheresearchandthenatureoftheinformationthatisbeingsought.Especiallyintheinitialorientationphaseinterviewswillgenerallybeopentoallowasmanydifferentopinionsandviewpointstobecollectedfromtheinterviewees.
36
6 The evaluation process
• Group interviews Variouskindsofgroupinterviewsareusedinevaluations.Oneofthemost
commonformsisthefocusgroupdiscussion,inwhichtheinteractionbetweentheintervieweescanalsobeavaluablesourceofinformation.
• Direct measurement Alotofquantifiableinformationcanbeacquiredbysimplerecordingand
counting.Especiallyatthelevelofoutputs,thisisthemostlogicalapproach.Directmeasurementisparticularlyusefulifdataisavailablefromabaselinestudy.
• Direct observation Observationmethodscanbelimitedtothephysicaloutputsofanactivity,butcan
alsorelatetocertainprocesses.Observationtechniques(withorwithoutthedeploymentofaudiovisualaids)havealwaysplayedanimportantroleinfieldresearch.Itshouldbeaddedthattheycanbeverytime-intensive.Evaluationsgenerallymakeuseofaquick(moreorlessstructured)tourofphysicaloutputs,a‘transect’,and/orattendingoneormoremeetingsofthetargetgroupsconcerned.
• Surveys Surveysaimtocollectalargequantityofstandardiseddatafromalargepopulation.
Suchdataisverysuitableforstatisticalanalysis.Surveysmakeuseofpreviouslydrawnupquestionnaireswhicharepreferablytestedfirstandwhichcanbeinwrittenorverbalform.Theyareessentialindeterminingthestartingsituation(baseline)andareindispensableinimpactevaluations.Becausesurveyscanbetime-consumingandcostly,impactevaluationsoftenusetheresultsofpreviousquestionnairesormodulesaddedtopendingsurveys,sothattheycanservemultiplepurposes.
• Case studies Casestudiesarein-depthstudiesofasmallnumberofspecificinterventions.They
offerthepossibilityofconsideringtheperceptionsofstakeholdersorreconstructinghistories.Casestudiesareavaluablesupplementtobroadersurveys,becausetheyprovideanopportunitytoexamineandexplainrelationshipsthatbecomeapparentfromstatisticalanalysisingreaterdetail.Ifanevaluationisbasedexclusivelyonaseriesofcasestudies,itisessentialtodeterminetowhatextenttheselectedcasesarerepresentative.
37
Evaluation policy and guidelines for evaluations
6.7 Reporting
ThefinalreportofanevaluationisapprovedbytheIOBDirector,aftershe/heissatisfiedthatthecommentsofthereferencegroup,internalpeerreviewersandtheIOBDepartmentalMeetinghavebeenincorporatedadequately.IOBthensendsthereporttotheministerorministersconcernedtobeforwardedtoparliament.Onrequest,IOBcanalsoprovidetheministerwithadditionalinformationonthefindingsoftheevaluation.Ifanevaluationisaddressingapoliticallyverysensitiveissue,IOBwillinformtheministerconcernedingoodtimeontheprogressoftheevaluationand,asfaraspossible,theexpectedoutcomes.
Inprinciple,allIOBevaluationreportsareaccessibletothepublic.Thisrulecanonlybedeviatedfrominexceptionalcases.Anevaluationmight,forexample,beveryconfidentialinnatureormayonlyserveaninternallearningfunction.Thedecisionnottopubliciseanevaluationreportmustalwaysbetakenbeforetheevaluationstarts.
Thereportshouldtakeaccountofthefactthatthefindingsoftheevaluationaretobepresentedtodifferentstakeholderswithvariousinterests.ThebasicprincipleisthatIOBreportsaredesignedtoaccounttobothparliamentandsociety,andtofulfilalearningfunction.Thatsetshighstandardsforthereporting.Parliamentbenefitsfromashortandconcisepresentationofthemainfindingsandconclusions.Accountingfortheresultsandthedeploymentofthepublicfundsrequiredtoachievethemhaspriority.Policydepartmentswillbeinterestedintheextenttowhichthereportprovidesinformationthatoffersopportunitiestoadjustpolicyformulationorimplementation.Inallcasesitisimportanttoaimforatransparentpresentationoffindings,analysisandassessment.
IOBevaluationsidentifyanumberofburningissuesthat,inthelightoftheevaluationfindings,requireattention.IOBthereforeavoidsmakingconcreterecommendations,puttingitselfinthepositionofthepolicymakerandtherebyofabutcherinspectingthequalityofhisownmeat.Furthermore,IOBworksaccordingtotheOECD/DACdefinitionofevaluation,whichrefersonlyto‘lessonslearnt’,andtheguidelinesforpolicyreviewintheRPE2006.
IntheinterestoftransparencytheTermsofReferenceshouldalwaysbeattachedtotheevaluationreportasanannex.Theresearchquestionsshouldbesystematicallyaddressedinthereport,alsoifsomeofthemcannotbeanswered.Ifnecessary,additionalinformationshouldbeprovidedintheannexesontheresearchmethodsemployedandanyproblemsthatariseduringimplementationoftheevaluation.
IftherearefundamentalandsubstantiateddifferencesofopinionbetweenIOBandthestakeholdersregardingthemainconclusionsofthereport,theseshouldbeexplicitlymentionedinthereport.ThedecisiontothiswillbetakenbytheIOBDirector.
38
6 The evaluation process
Manyevaluationreportsareasynthesisofsubsidiarystudies.Especiallywherethesesubsidiarystudiescanalsobereadindependentlyitisimportanttomakethemaccessibleforpolicydepartmentsorotherstakeholders.Theyoftencontaindetailedinformationthatisnotincludedinthemainreport.TheevaluationreportsarepublishedbothinprintformandontheMinistryofForeignAffairswebsite.Subsidiarystudiesarealwaysavailableonthewebsiteandareinsomecasesalsopublishedinprintform.Inthecaseofveryextensiveevaluations,intheinterestsofaccessibility,IOBproducesasummarisedversionofthereport.BoththesummaryandthefulltextaresubmittedforinclusioninOECD/DACnetworkforevaluationservices’evaluationinventory.Completedimpactevaluationsarealsoenteredinthedatabasesetupby3IEandNONIEforevaluationsofthiskind.
39
Evaluation policy and guidelines for evaluations
7Quality control
41
Evaluation policy and guidelines for evaluations
TheIOBinspectorisresponsibleforconductingindependentevaluations.Thequalityoftheworkcarriedoutis,however,aconcernforIOBasawhole.TheIOBmanagementhasitsownresponsibilityinthiscontext,asitapprovestheevaluationreports.Toprepareforthisdecision-makingprocess,IOBhasaqualitycontrolsystemcomprisingfiveinstruments:internalpeerreviewers,referencegroups,thedepartmentalmeeting,theadvisorypanelandtheself-evaluation.
7.1 Internal peer review
Qualitycontrolbenefitsfromintensivecooperationandasystematicintercollegialmonitoring.IOBthereforehasasystemthatensuresthatinspectorscanrelyonsubstantivesupportfromcolleagueswhenconductingevaluations.Start-upreports,TermsofReference,subsidiarystudiesandthedraftfinalreportarediscussedwithtwointernalpeerreviewers,fellowinspectors,beforebeingsubmittedtothedepartmentalmeeting.Theinspectorchooses,inconsultationwiththeIOBDirector,whichcolleaguesaretoactasinternalpeerreviewers.TheDirectorappointsinternalpeerreviewersperevaluation.Theyareselectedandappointedonthebasisofthefollowingcriteria:
• coachingoftheinspector;• internalreadershipasalearningfunctionforrecentlyappointedinspectors;• evaluationexperience;• affinitywiththeevaluationsubjectandexpertiseinthethemeconcerned;• avoidingconflictsofinterest;• anequaldistributionoftheworkloadwithinIOB.
Notallcriteriaareequallyimportantandaspecificcriterioncanbedecisive.
Thetaskoftheinternalpeerrevieweristogiveadvicetotheinspectors.Thisadviceisnotbindingbutinspectorsdohavetoindicatewhattheydowithit.LastingandfundamentaldifferencesofopinionmustbesolvedinconsultationwiththeIOBDirector.
Inspectorsmustincludethecommentsandadviceofinternalpeerreviewersandashortreportoftheirdiscussionswiththereadersinthefileontheevaluation.
Theinternalpeerreviewersarenotpartofthereferencegroupfortheevaluation.
AstheIOBDirectorisresponsibleforapprovingevaluationproductshe/shecannotbeaninternalpeerreviewer.
ThedecisionwhethertomakeuseofinternalpeerreviewintheeventofjointevaluationsbetweenIOBandtheevaluationservicesofotherdonorsislefttotheIOBinspectorresponsiblefortheevaluation,inconsultationwiththeIOBDirector.
42
7 Quality control
7.2 Reference groups
AsecondformofqualitycontroloverIOBevaluationsisreferencegroups.Thesegroups,whicharecreatedforeachevaluation,arepartlyexternal,consistingofrepresentativesofthepolicydepartmentsinvolvedintheevaluation,organisationsand/orlocalbodies,andexternalexperts.AreferencegroupischairedbytheIOBmanagementandmeetsatkeymomentsduringtheprocess(e.g.todiscussthedraftToR,partialreportsandthedraftfinalreport).Theinspectorisresponsibleforreportingonreferencegroupdiscussionsand/orprovidingsummariesofwrittencommentsbyreferencegroupmembers.
Forevaluationsthatentailintensivecooperationwithresearchersinaspecificpartnercountry,asoundingboardgroupissetupinthecountryconcerned.Theformthegrouptakeswillvaryfromcasetocase.
ForjointevaluationswithotherdonorsaDutchreferencegroupisusuallynotsetup.Thereisajointsteeringgroupcomprisingrepresentativesoftheparticipatingdonors.Inthecaseoflarge-scale,complexevaluationsasmallermanagementteamissetupfromwithinthesteeringgrouptoruntheevaluationonadailybasis.Thesteeringgroupisthenprimarilyconcernedwithqualitycontrol.Inaddition,withtheseevaluations,theresearchbureauconductingtheevaluationisexpectedtoarrangeitsownqualitycontrol.Thisisoneoftheaspectstakenintoaccountwhenassessingtenders.
7.3 Departmental meeting
TheIOBdepartmentalmeetingplaysapartinqualitycontrolattwomomentsduringtheimplementationofanevaluation:whendiscussingtheTermsofReference(orsometimestheinceptionreport)andduringdiscussionofthedraftfinalreport.TheToRarealwaysdiscussedatafulldepartmentalmeeting.BeforethismeetingitmustbecleartotheinspectorwhichcolleagueshavesubmittedcommentsontheTermsofReferenceandthesubstanceofthecomments,ingeneralterms,sothathe/shecanprepareforthediscussion.Draftfinalreportsarethendiscussedonceagaininaseparatefinalmeetingbetweentheinspectorandtheinternalpeerreviewers,chairedbytheIOBDirector,beforetheDirectorapprovesthereport.Thisfinalmeeting,whichisopentoallIOBstaffmembers,hastwopurposes:itofferstheopportunityforafinalassessmentofthequalityoftheproduct,andhelpscolleaguestolearnfromeachothers’experiences.ItalsoensuresthatthefinalproductofanindividualinspectorbecomesaproductofIOBasawhole.
43
Evaluation policy and guidelines for evaluations
7.4 Advisory panel
ToincreasetheusabilityanduseofevaluationsinthefieldofdevelopmentcooperationtheMinisterforDevelopmentCooperationmadeapledgetoparliamenttoestablishanAdvisoryPanelofprominentexternalexperts.Thepanel,withitssecretariatatIOB,willplayaroleinqualitycontrolbymonitoringtheusabilityofevaluationsandofferingadviceonrequestoronitsowninitiative.Itwillbecomeoperationalinthecourseof2009.
7.5 Self-evaluation
Everyevaluationconcludeswithacriticalreviewofhowitproceeded,withtheaimofimprovingtheapproachtofutureevaluations.Theseself-evaluationsarewrittenbytheinspector,sometimestogetherwitharesearchassistantandtheinternalpeerreviewers.TheTermsofReferencefortheevaluationdeterminetheformatoftheself-evaluation.BysystematicallygoingthroughthevariouscomponentsoftheToR,theself-evaluationprovidesaclearinsightintothesuccessfulandlesssuccessfulelementsoftheevaluation.Itshouldinanycaseaddress:(1)theextenttowhichtheresearchassignmentcouldbeimplemented,(2)anydifferencesbetweentheplannedtimepathandtheactualdurationoftheevaluation,and(3)differencesbetweenestimatedandactualcosts.Self-evaluationsarediscussedinternallyandperiodicallyatthedepartmentalmeeting.
8Communication andfeedback
45
Evaluation policy and guidelines for evaluations
Communicationandfeedbackrelatetothefeedbackofevaluationresults,withaviewtothetransparencyoftheevaluationprocess,theaccountabilityfunctionandtheintegrationoflessonslearnedinfuturepolicyformulationandimplementation.Feedbackisnotlimitedtothephasefollowingcompletionoftheevaluation.Consultationwithandfeedbacktostakeholderstakeplacethroughoutthewholeperiodofimplementation:duringthepreliminarystudy,inrelationtotheTermsofReference,andduringdiscussionofsubsidiarystudiesandthedraftfinalreport.Wherenecessary,consultationandfeedbackisalsoprovidedtothepartnercountries.Whilefeedbackduringtheevaluationisimportantatimplementationlevel,aftertheevaluationhasbeencompleteditismoreimportantatpoliticalandmoregenerallevel.
Theextenttowhichcompletedevaluationsfulfilthelearningfunctiondependsonanumberoffactors.Inthefirstplace,itisamatteroftheusabilityoftheevaluationitself.Thisisdeterminedby:
• theclaritywithwhichtheexternalaim(i.e.outsidetheevaluationitself ),for whichtheevaluationresultscanbeused,isspecified;• theclarityandcompletenesswithwhichtheessenceoftheevaluationandespecially
itsmainfindingsarepresentedintheevaluationreportandthesummary;• thecompletenesswithwhichallresearchquestionsareansweredbythe
conclusions;• theextenttowhichtheresultsoftheevaluationsuggestactionsthatliewithin thecapacityoftherecipientofthereport.
Inthesecondplacelearningfromevaluationsdependsontheextenttowhichtherecipientsandusersarereceptivetothelessonsdrawnfromthestudies.Thisisrelatedtotheinstitutionalcontextandtheorganisationalculture.IOBhasnoinfluenceonthisanditisnotIOB’sresponsibility.Itisthedirectresponsibilityofthemanagementoftheorganisation(theministryorotherorganisationsthatreceiveIOBreports).
IOBdoeshoweverhavearesponsibilitytocommunicateonevaluationreports.Ascommunicationwithpolicydepartmentsiscrucialtoincreasethelearningeffect,itiscrucialthatevaluationsarediscussedintherelevantstaffmeetings.Withregardtoexternalcommunication,IOBdrawsupacommunicationplanforeachevaluation,inconsultationwiththeministry’sInformationandCommunicationDepartment.IOBincreasestheaccessibilityofevaluationresultsandensurestheirwiderdistributionbypreparingmediapresentations,givinglecturesandinterviews,writingarticles,andorganisingconferencesandseminars.
IOBevaluationsarealsoaninstrumentforrenderingaccountabilitytoparliament.WithinthreemonthsofthereportbeingapprovedbytheIOBDirector,theministersendsittoparliament,usuallytogetherwithapolicyreaction.Thetextofthereaction,whichisusuallydraftedbythepolicydepartmentconcerned,containsindicationson
46
8 Communication and feedback
howtheresultsoftheevaluationaretobeacteduponinthelightoffuturepolicy.Inthisway,accountingtoparliamentalsocontributestothelearningprocess.
Inthecaseofevaluationsinthefieldofdevelopmentcooperationthereportsarealsopublishedinthepartnercountries,offeredtothegovernmentandmadewidelyknownthroughtheorganisationofmeetings.WithaviewtomutualaccountabilityIOBinvestssystematicallyinsuchfeedbacktothepartnercountries.
47
Evaluation policy and guidelines for evaluations
297 2004 Over solidariteit en professionalisering
Evaluatie van Gemeentelijke Internationale Samenwerking (1997-2001).
isbn 90-5328-341-2297 2004 On solidarity and
professionalisation Evaluation of Municipal International Co-operation (1997-2001). isbn 90-5328-342-0 298 2004 Onderzoek naar de kwaliteit van in
2002 afgeronde decentrale evaluaties
Eindrapport. isbn 90-5328-344-7299 2005 Een uitgebreid Europabeleid Evaluatie van het Nederlands beleid
inzake de toetreding van Midden-Europese landen tot de Europese Unie 1997-2003
isbn 90-5328-347-1300 2005 Aid for Trade? An Evaluation of Trade-Related
Technical Assistance isbn 90-5328-349-8301 2006 Van Projecthulp naar Sectorsteun Evaluatie van de sectorale
benadering 1998-2005 isbn 90-5328-351-x301 2006 From Project Aid towards Sector
Support An evaluation of the sector-wide
approach in Dutch bilateral aid 1998–2005.
isbn 90-5146-000-7302 2006 Evaluatie van het Nederlandse
mensenrechtenbeleid in de externe betrekkingen
isbn 90-5328-352-8
303 2006 Dutch Humanitarian Assistance An Evaluation isbn 90-5328-352-8304 2007 Evaluatie van de vernieuwing van
het Nederlandse onderzoeksbeleid 1992-2005
isbn 978-90-5328-353-0304 2007 Evaluation of the Netherlands'
Research Policy 1992-2005 (Summary)
isbn 978-90-5328-353-0305 2007 Impact Evaluation: Water Supply
and Sanitation Programmes Shinyanga Region, Tanzania 1990-2006
isbn 978-90-5328-354-7306 2007 Chatting and Playing Chess with
Policymakers Influencing policy via the Dutch
Co-Financing Programme isbn 978-90-5328-355-4307 2008 Beleidsdoorlichting seksuele en
reproductieve gezondheid en rechten en hiv/aids 2004-2006
isbn 978-90-5328-358-5 308 2008 Het Nederlandse Afrikabeleid
1998-2006 Evaluatie van de bilaterale
samenwerking isbn 978-90-5328-359-2308 2008 Het Nederlandse Afrikabeleid
1998-2006 Evaluatie van de bilaterale
samenwerking (Samenvatting) isbn 978-90-5328-359-2308 2008 The Netherlands’ Africa Policy
1998-2006 Evaluation of the bilateral cooperation (Summary) ISBN 978-90-5328-359-2
Evaluation studies published by the Policy and Operations Evaluation Department (IOB) 2004-2009 Evaluation studies published before 2004 can be found on the IOB website at www.minbuza.nl/iob
48
Evaluation studies published by the Policy and Operations Evaluation Department (iob) 2003-2008
309 2008 Het Vakbondsmede-financieringsprogramma
Een evaluatie van steun gericht op versterking van vakbonden en vakbonds- en arbeidsrechten
isbn 978-90-5328-357-8309 2008 The Netherlands Trade Union
Co-Financing Programme An evaluation of support for trade
unions and trade union and labour rights (Summary)
isbn 978-90-5328-357-8309 2008 El Programa de Cofinanciamiento
Sindical Una evaluación del apoyo orientado
al fortalecimiento de sindicatos y de derechos sindicales y laborales (Resumen)
isbn 978-90-5328-357-8310 2008 Clean and sustainable? An evaluation of the contribution of
the Clean Development Mechanism to sustainable development in host countries
isbn 978-90-5328-356-1311 2008 Impact Evaluation: Primary
Education in Uganda isbn 978-90-5328-361-5312 2008 Impact Evaluation: Primary
Education in Zambia isbn 978-90-5328-360-8313 2008 Xplore-programma isbn 978-90-5328-362-2314 2008 Primus inter pares Een evaluatie van het Nederlands
EU-voorzitterschap 2004 isbn 978-90-5328-364-6315 2008 Impact Evaluation: Support to Rural
Water Supply and Sanitation in Dhamar and Hodeidah Governora-tes, Republic of Yemen
isbn 978-90-5328-364-6316 2008 Be our guests Beleidsdoorlichting Nederland als
gastland van internationale organisaties en Samenvatting
isbn 978-90-5328-370-7316 2008 Be our guests Policy review on hosting international
organisations in the Netherlands and Summary
isbn 978-90-5328-371-4
316 2008 Be our guests Examen de la politique meneé par les
Pays-Bas en tant qu'État hôte des organisations internationales (Sommaire)
isbn 978-90-5328-372-1317 2008 Sectorsteun in milieu en water isbn 978-90-5328-369-1318 2008 Samenwerking met Clingendael Evaluatie van de subsidieovereen-
komst tussen de ministeries van Buitenlandse Zaken en Defensie en
Instituut Clingendael isbn 978-90-5328-365-7319 2008 Meer dan een dak Evaluatie van het Nederlands beleid
voor stedelijke armoedebestrijding isbn 978-90-5328-365-3320 2008 Het tropisch regenwoud in het
OS-beleid 1999-2005 isbn 978-90-5328-374-8321 2009 Maatgesneden Monotoring 'Het
verhaal achter de cijfers' Beperkte beleidsdoorlichting
Medefinancieringsstelsel 2007-2010 isbn 978-90-5328-374-5322 2009 Draagvlakonderzoek Evalueerbaarheid en resultaten isbn 978-90-5328-375-2323 2009 Preparing the ground for a safer
world Evaluation of the Dutch efforts to control landmines and explosive remnants of war 1996-2006
isbn 978-90-5328-377-6
Evaluation policy and guidelines for evaluations
49
50
51
Evaluatiebeleid en richtlijnen voor evaluaties
52 Publishedby:
MinistryofForeignAffairsPOBox20061|2500EBTheHague|TheNetherlandswww.minbuza.nl
Layout:Eindeloos,TheHaguePrinter:OBT,TheHague
©MinistryofForeignAffairs|October2009|Ordercode:BZDR6662/E
ratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe
nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa
aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum
houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMia
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono
dIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MpalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisVelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgH
openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauW
restBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/K
etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntweenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKa
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBa
asChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKiazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSan
alle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
furt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaMan
ekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono
adHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhou
openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusW
BernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWtislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIst
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthe
iroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeR
okioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroenouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
aDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
enenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaP
obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo
elsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiaortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonodIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTjnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavanna
aoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisV
elgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWrestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStad
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwe
enosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsP
eheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBaasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKi
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSanalle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
alaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
ashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitK
obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCa
HanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideo
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono
JubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHel
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonW
tislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheiroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe
nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa
aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum
houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo
IOB
Evaluation policy and guidelines for evaluations
Published by:
Ministry of Foreign Affairs PO Box 20061 | 2500 EB The Hague | The Netherlandswww.minbuza.nl© Ministry of Foreign Affairs| October 2009 | Order code: BZDR6662/E
olicy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department
This document describes the evaluation policy and guidelines for evaluations of the Policy and Operations Evaluation Department (IOB) of the Dutch Ministry of Foreign Affairs. In publishing these guidelines, IOB wishes to provide transparency to third parties: colleagues at the Ministry of Foreign Affairs, the Dutch parliament, organisations and institutions affected by IOB evaluations, users of the evaluation results, and interested citizens. IOB will periodically review its policy and the guidelines and, if necessary, revise them.