evaluation policy and guidelines for evaluations · 5 evaluation policy and guidelines for...

56
IOB Evaluation policy and guidelines for evaluations IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and O Evaluation policy and guidelines for evaluations

Upload: others

Post on 25-May-2020

10 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

IOB

Evaluation policy and guidelines for evaluations

IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | P

Evaluation policy and guidelines for evaluations

Page 2: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

tyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüssstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTeneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP

vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP

vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

Page 3: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

Evaluation policy and guidelinesfor evaluations

Page 4: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations
Page 5: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

3

Evaluation policy and guidelines for evaluations

Contents

1 Introduction 4

2 Short history 6

3 Basic principles of IOB’s evaluation policy 10

4 Definitions and criteria for evaluations 14

4.1 Whatisevaluation? 15

4.2 Typesofevaluation 16

4.3 Evaluationcriteria 17

4.3.1 Efficiency 17

4.3.2 Effectiveness 18

4.3.3 Impact 19

4.3.4 Relevance 19

4.3.5 Sustainability 20

4.3.6 Policyconsistency 21

5 Guiding principles and ethical standards 22

6 The evaluation process 26

6.1 Programming 27

6.2 Preliminarystudy 29

6.3 TermsofReference 30

6.4 Deployingresearchers 32

6.5 Cooperatingwithresearchersinpartnercountries 32

6.6 Methodology 33

6.7 Reporting 37

7 Quality control 40

7.1 Internalpeerreview 41

7.2 Referencegroups 42

7.3 Departmentalmeeting 42

7.4 Advisorypanel 43

7.5 Self-evaluation 43

8 Communication and feedback 44

Page 6: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

1Introduction

Page 7: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

5

Evaluation policy and guidelines for evaluations

ThisdocumentdescribestheevaluationpolicyandguidelinesforevaluationsofthePolicyandOperationsEvaluationDepartment(IOB)oftheDutchMinistryofForeignAffairs.ItplacestheevaluationpolicyinthecontextoftheOrder on PeriodicEvaluation and Policy Information(knownasRPE2006)1,thegovernment-wideframeworkfortheevaluationofgovernmentpolicyingeneral.ItalsoreflectsrecentdecisionsregardingthefunctionofevaluationattheMinistryofForeignAffairs,andtheroleIOBplayswithinit.

IOB’sevaluationpolicyandtheguidelinesmustbeseeninthecontextofachangedenvironment.AnincreasinglycriticaloutsideworlddemandsaclearerinsightintotheresultsoftheNetherlands’foreignpolicy.Thereisgrowingpressureforministerstoaccountnotonlyfortheuseofpublicfunds,butevenmoresofortheresultsachieved.Thisalsoappliestothevariousareasofforeignpolicy.Ifevaluationsaretocontributetothis,theymustbeindependent,public,transparentandaccessible.

Inpublishingtheseguidelines,IOBwishestoprovidetransparencytothirdparties:colleaguesattheMinistryofForeignAffairs,theDutchparliament,organisationsandinstitutionsaffectedbyIOBevaluations,usersoftheevaluationresults,andinterestedcitizens.Ineachevaluationreport,IOBaccountsforthespecificevaluation;thisdocumentclearlypresentsthegeneralprinciplesandguidelinesIOBappliesinternallyinconductingevaluations.

TheguidelinesoutlinedinthisdocumentapplyinprincipletoallevaluationsconductedbyIOB,unlessexplicitlystatedotherwise.ItshouldbeborneinmindthattheNetherlandshasalongtraditionofevaluationinthefieldofdevelopmentcooperation,whichisexpressedthroughinternationalagreements,fora,cooperation,etc.andwhichimpactsonIOB’swork.Thatisnotthecaseforotherareasofforeignpolicy.Becauseitsmandatealsoextendstotheseotherareasofpolicy,IOBisinanexceptionalposition,especiallyinternationally.

TheevaluationpolicyandguidelinesreflecttheopinionsandperspectivesthatunderpinIOB’swork.Inthatsense,thisdocumentdescribestheidealsituationforIOB.Realitywillsometimesdeviatefromthisidealinsomeways.Therewillalwaysbeacertaintensionbetweentheoryandpractice.However,IOBaimstoworkasmuchaspossibleaccordingtotheseguidelines.Itisimportanttorememberthatneitherpolicynorguidelinesarecarvedinstone.Thismustbeseenasagrowthdocument,inthemostliteralsense.Newinsightsmayleadtoitbeingadaptedinthefuture.IOBwillthusperiodicallyreviewitspolicyandtheguidelinesand,ifnecessary,revisethem.Thelatestupdatedversionofthisdocumentwillbeavailableathttp://www.minbuza.nl.

1 Order on Periodic Evaluation and Policy Information 2006, Government Gazette, 26 April 2006, no. 83.

Page 8: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

2Short history

Page 9: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

7

Evaluation policy and guidelines for evaluations

Inthefirsthalfofthe1970sthebudgetfordevelopmentcooperationincreasedrapidly.Sorapidlythatconcernswerevoicedfromvariousquarters,andespeciallyinthemedia,thatthiswashavinganadverseeffectonthequalityofaid.Asaconsequenceofthisnegativepublicity,theMinisterforDevelopmentCooperationfelttheneedforanindependentopinionondevelopmentpolicy,autonomousofthepolicydepartmentsinvolved.On1July1977theOperationsReviewUnit(IOV)wassetupbyministerialorder.ItwasmadeclearfromthebeginningthatIOVwouldfocusonbilateralcooperation,bothgovernmentalandnon-governmental(insofarasthelatterwasgovernment-funded),butwouldinprinciplerestrictitsactivitiestotheNetherlands’contributiontojointinterventions.Onthebasisofthisrestriction,theministertookthepositionthatIOVcouldconductinspectionswithoutthepermissionoftherecipientcountryororganisation.IOV’sworkprogrammewasdrawnuptogetherwiththepolicydepartmentsandsubmittedtotheministerforapproval.

OutsidetheworkprogrammetheministerandtheministrymanagementwerealsoauthorisedtocommissionIOVtoconductinspectionsonanadhocbasis.However,thisoccurredonlysporadically.FromthestartIOVfocusedonindividualprojects,submittingconfidentialreportstotheminister.Thereportswereconfidentialbecausetheyreferredtoidentifiableeventsandindividuals.

From1979theHouseofRepresentativesoftheDutchparliamentcalledeachyearforIOV’sreportstobemadeaspublicaspossible.Eventually,itwasagreedthatgeneralreportswouldbesenttoparliament,togetherwithsummariesofprojectinspections.ThefirstsummarisedreportthatIOVsenttotheHousewasthe1984GeneralEvaluationofDutchBilateralDevelopmentCooperation.

Inthe1980s,IOV’sareaofworkshiftedfromtheinspectionofindividualprojectstoevaluationsofsectors,policythemesandcountryprogrammes.DuringverbalconsultationswiththeHouseofRepresentativesin1987theministeragreedthatallIOVreportswouldbefullypublicandwouldbesenttoparliament,togetherwithapolicyletterfromtheministercontaininghisorherresponsetothereport.ThereportswereusedactivelyindebatesbymembersofboththeHouseofRepresentativesandtheSenate.Universitiesandotherinstitutionsinthepublicandprivatesectorsalsoshowedalivelyinterestinthereportsoncetheybecameaccessible.

In1996afar-reachingreassessmentofDutchforeignpolicytookplace.Itwaswidelyacknowledgedthatdevelopmentcooperation,foreignpolicyandinternationaltradecouldnolongerbeseenasindependentpolicyareas.Foreignpolicywas‘decompart-mentalised’andthevariousareasofinternationalactivitywereintegrated.Forthepurposeofthereview,IOVpublishedanewsynthesisstudy,summarisingtheresultsoftheevaluationreportsoftheprecedingdecade.Duringthereorganisationthataccompaniedthepolicyreassessment,itwaslogicaltomodifyIOV’smandate.Whereas,beforethereview,IOVwasonlyresponsibleforevaluationsinthefieldofdevelopmentcooperation,itsareaofoperationswasnowextendedtoincludethe

Page 10: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

8

2 Short history

wholeofforeignpolicy.Thisimpliedthat,inthefuture,theUnitcouldaddressitselftoallsubjectsthatfellundertheHomogenousBudgetforInternationalCooperation(HGIS).ThechangeinIOV’smandateledtoarevisionofitspositionintheorganisationalstructureoftheministry.ItwouldnolongerbeaccountabletotheDirector-GeneralforInternationalCooperation(DGIS)butdirectlytotheSecretary-General.TheUnitwasalsogivenanewname:inthefutureitwouldbeknownasthePolicyandOperationsEvaluationDepartment(IOB).

InrecentyearsIOB’spositionandrolehasonceagainbeenthesubjectofpoliticaldebateanddiscussionwithintheministry.Manyoftheissuesdiscussedinthe1980swereonceagainontheagenda,includingtheindependenceofevaluation,thepublicnatureofthereports,therelationshipbetweenexternalaccountabilityandinternallearning.Thisledin2008totheMinisterforDevelopmentCooperationreconfirmingtoparliamenttheindependenceofIOBandthepublicnatureofitsreports.However,IOB’sprogrammingauthoritywasrestrictedtodevelopmentcooperation.2Intheotherfieldsofforeignpolicy,primaryresponsibilitynowlieswiththeDirector-Generalconcerned,whocanrequestIOBtoconductevaluations.

2 Letter from the Ministers of Foreign Affairs and for Development Cooperation of 16 October 2007 on implementation of the Boekestijn/Gill’ard motion regarding the Policy and Operations Evaluation Department (IOB) (31200V, no. 9). Report from the Permanent Committee on Foreign Affairs on IOB of 29 May 2008, approved on 25 June 2008.

Page 11: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

9

Evaluation policy and guidelines for evaluations

Page 12: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

3Basic principles of IOB’s evaluation policy

Page 13: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

11

Evaluation policy and guidelines for evaluations

ThePolicyandOperationsEvaluationDepartmentistheevaluationdepartmentoftheMinistryofForeignAffairs.Institutionally,IOBisdirectlyaccountabletotheSecretary-General.Itformspartoftheministry’sAuditCommittee.TogetherwiththeAuditDepartmentandtheInspectionandEvaluationUnit(ISB),IOBformsthecluster‘InControl’.

Internationally,IOBisamemberoftheOECD/DACnetworkofevaluationservices.ItalsotakespartinconsultationsbetweentheevaluationservicesoftheEuropeanUnion,andofthecountriesunitedintheNordic+group.IOBcontributestotheactivitiesofanumberofotherformsofinternationalcooperation,includingtheconsultationcommitteeofGerman-speakingevaluationservices(DACH),theNetworkofNetworksforImpactEvaluation(NONIE)andtheInternationalInitiativeonImpactEvaluation(3IE).

IOB’smandateisasfollows:

The Policy and Operations Evaluation Department meets the need for evaluation of policy and its implementation with regard to all policy areas that are financed by the homogenous budget for international cooperation (HGIS). IOB evaluates the implementation of the Netherlands’ foreign policy, giving particular attention to its relevance, efficiency and effectiveness, the sustainability of the results achieved and the consistency and coherence of the policy.

Inthelightofthismandate,IOBhasthefollowingtasks:

1)toconducthighqualityevaluationsforlearningandaccountabilitypurposes;2)toensurebalancedevaluationprogramminginthefieldofdevelopmentcooperation

andtoadviseonprogramminginotherareasofforeignpolicy;3)tocontributetoimprovementofthequalityofevaluationsconductedby,oronbehalf

of,policydepartments,institutionssubsidisedbytheministry,andmultilateralinstitutionsbydrawingupgenerallyapplicablequalityrequirementsandguidelinesforconductingevaluations,advisingonthedesignandimplementationofindividualevaluations,andconductingsystematicassessmentsoftheseevaluationsorpeerreviewsofotherevaluationservices;

4)tostrengthencapacitytoconductevaluationsinpartnercountriesbysupportingeducationandtrainingandparticipatinginjointevaluations;

5)topromotethedistributionandcommunicationofevaluationresults.

Re1:IOB’smaintaskistoconducthigh-qualityevaluationswithintheframeworkoftheMinistryofForeignAffairs’evaluationprogramme.TheseevaluationscanrelatetoallareasofforeignpolicythatfallundertheHGIS.Inthiscontext,evaluationcomprisescriticalandthoroughinvestigationofapolicyorprogrammeandexplainingtheresults.IOBevaluationsdonotstandinisolationbutareinprinciplepartofthepolicycycle,inwhichplanning,monitoringandevaluationareseenasanintegratedwhole.

Page 14: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

12

3 Basic principles of IOB’s evaluation policy

Evaluationisthereforeanessentialcomponentofthe‘knowledgefunction’withintheministry.Knowledgeontheimplementationandimpactofpolicysupportstwoconcretefunctions:

• itprovidesthebasisforaccountingtoparliamentandsocietyatlargeforpolicyandexpenditure(accountabilityfunction);

• itgeneratesinformationforthepurposesofimprovingpolicyformulationandimplementation(learningfunction).

Re2:AsstatedintheRPE2006systematicevaluationmustbeafixedcomponentofeverypolicyprocess.Eachpolicyareamustbeperiodicallyevaluatedatthelevelofoperationalobjectives.TheRPE2006thusformsthebasisofevaluationprogramming.Theprogrammingisalsodeterminedbyspecificrequestsfromparliament,ministersandstatesecretaries,andbudgetholders.WithintheMinistryofForeignAffairs,IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperationandadvisesonprogramminginotherareasofforeignpolicy.

Re3:IOBisnottheonlyunitattheMinistryofForeignAffairsconcernedwithevaluation.Policyadvisorsalsohavetodealwithevaluations,asthecontactforevaluationprogrammingwithintheirpolicydepartment,asa‘client’commissioninganevaluation,sometimesas(joint)implementer,orasauseroftheresults.Inadditiontoconductingevaluations,IOBthereforehasanadvisoryfunction.ThisentailsformulatingqualitycriteriaandguidelinesforimplementingevaluationsandadvisingontheTermsofReferenceforevaluationsthatIOBdoesnotconductitself.Duringtheannualsettlementoftheprogramme,agreementsaremaderegardingtheextenttowhichIOBwillplayanadvisoryroleinevaluationsconductedbyoronbehalfofministrydepartments.IOBalsoadvisesonthedesignoftheministry’sevaluationfunction,theevaluationsystemandinstruments,andcoordinationwithplanningandmonitoring.InadditionIOBperformsqualitytestsonprogrammeevaluationsconductedbycofinancednon-governmentalorganisations(MFS),withtheaimofstrengtheningtheirevaluationcapacity.

ForthemultilateralchannelIOBisinvolvedininternationalreviewsoftheevaluationservicesofrelevantinstitutions.

Re4:Strengtheningevaluationcapacityinpartnercountriesisanimportantthemeindevelop-mentcooperation.Independentevaluationcapacityinpartnercountriesisindispensableiftheaimofestablishingasystembasedonmutualaccountabilityistobeachievedinreality.IOBseesitasitstasktocontributetothisbysupportinginternationalinitiativesaimedatstrengtheningevaluationcapacity.Inaddition,wherepossibleandsafeguardingitsowncriteriaforindependenceandquality,IOBworkstogetherwithresearchersfrompartnercountriesinconductingevaluations.

Page 15: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

13

Evaluation policy and guidelines for evaluations

Ad5:Evaluationsgenerateknowledgeonthebasisofexperiencefromthepastregardingwhatworksandwhatdoesnot,andespeciallywhy.IOBconsidersitimportantthattheresultsofevaluationsaredistributedaswidelyaspossible.Toachievethis,itdiscussesthefindingsinstaffmeetings,withministersandstatesecretaries,andwithothernationalandinternationalpartiesinvolvedintheevaluations.Inaddition,wherepossibleandwhereitdoesnotconflictwithitsownresponsibilities,IOBwilloffersupportinpreparingpolicyresponsesanddescriptionsofevaluationfindingsintheMinistryofForeignAffairs’AnnualReports.

Page 16: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

4Definitions and criteria for evaluations

Page 17: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

15

Evaluation policy and guidelines for evaluations

4.1 What is evaluation?

Thereisawidevarietyofdefinitionsoftheconceptofevaluation.Inthe1980sand1990s,theOECD/DACworkedonasharedframeworkofconceptsforevaluationpracticesinthefieldofdevelopmentcooperation.TheresultingdefinitionofevaluationservesasabasisforIOB’swork.

‘Anevaluationisanassessment,assystematicandobjectiveaspossible,ofanongoingorcompletedproject,programmeorpolicy,itsdesign,implementationandresults.Theaimistodeterminetherelevanceandfulfilmentofobjectives,developmentalefficiency,effectiveness,impactandsustainability.Anevaluationshouldprovideinformationthatiscredibleanduseful,enablingtheincorporationoflessonslearnedintothedecision-makingprocessofbothrecipientsanddonors.’3

TheOECD/DACdefinitionreflectsanunderstandingofevaluationthatisbroadlysupportedintheinternationalcommunityofevaluators.Thedefinitionisintendedforevaluationsinthefieldofdevelopmentcooperation.SinceIOB’sworkextendsfurtherthandevelopmentcooperation,conceptslike‘developmental’and‘recipientsanddonors’willnotapplytoallIOBevaluations.

Evaluationisdifferenttoreviewsandmonitoring.TheOECD/DACGlossarydescribesareviewasanassessmentoftheperformanceofanintervention,periodicallyoronanadhocbasis.ForIOBthemaindifferencebetweenareviewandanevaluationisthatareviewisconsideredamanagementinstrumentforoperationalmonitoringoftheimplementationofanactivity,andthereforefocusesonoperationalaspectsandrecordingresults,whileevaluationisamorein-depthassessmentoftheaims,implementationandresultsofanintervention,withthequestionofattributionasitsmainfocus.

Monitoringisaninstrumentofinternalmanagement.Itaims,onthebasisofafixedsetofindicators,toprovidearegularinsightintotheprogressmadeinimplementingactivities.Monitoringallowsabudgetholderortheimplementerofaninterventiontostayuptodateoninputssuppliedandactivitiesperformed(outputs).Monitoringshowswhetherintendedactivitieshaveactuallybeencarriedout.Good,systematicmonitoringisofgreatimportanceforevaluations,asitprovidesasignificantpartofthedataonwhichevaluationisbased.

3 DAC principles for Aid Evaluation, OECD/DAC, 1991 (the definition was also included in the DAC Glossary

of Key Terms in Evaluation and Results-based Management in 2002).

Page 18: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

16

4 Definitions and criteria for evaluations

4.2 Types of evaluation

TheOrderonPeriodicEvaluationandPolicyInformation2006(RPE2006)prescribesasystemofevaluationbasedontheso-calledoperationalobjectives,OD.Thesystementailsperiodicalevaluationoftheseoperationalobjectives.

TheRPE2006distinguishesbetweenpolicyreviewsandex-postimpactstudies.Bothconceptsdeviatefromtheevaluationterminologyusedinternationally,butaretheequivalentofthemorecommonterms‘policyevaluation’and‘impactevaluation’.Thestartingpointisthatpolicyreviews,whicharelinkedtotheoperationalobjectives,areasfaraspossiblefedbyimpactstudies.Inprincipletheyresemblesynthesisstudies.IOBgivesprioritytothesetypesofevaluation,where–especiallyinthefieldofdevelopmentcooperation–policyreviewsareprecededbyaseriesofex-postimpactstudies(impactevaluations).

OnthebasisofthesystemprescribedintheRPE2006,IOBimpactevaluationswillgenerallyfocusoninterventions(rangingfromprogrammestovariousformsofbudgetsupport).Policyevaluations(orreviews)addressthemesorsectors.Inaddition,integratedforeignpolicycanalsogiveriseto‘decompartmentalised’evaluationsofcountryorregionalpolicies.

Besidesthese–usuallyextensive–evaluations,IOBcanalsoconductshortinternalevaluationsthatarelimitedinnatureandnecessarytomonitortheprogressofprogrammes.

PartoftheevaluationsinIOB’sprogrammingfocusessolelyonDutchprogrammesandactivities.ItislogicalfortheseevaluationstobeconductedbyIOB.However,itisbecomingincreasinglycommontoworktogetherwithinternationalpartnersinthefieldofdevelopmentcooperation.Projectsbyindividualdonorsaregraduallybeingreplacedbyjointfinancingofgovernmentpolicyinpartnercountries.Itisthenalsologicalthatdonorsgivegreaterpreferencetothejointprogrammingandimplementationofevaluations.Incaseswheredonorsworktogether,forexampleinprovidingjointsupportforsectorpolicyorotherfar-reachingformsofjointfinancing(suchasbasket-funding),IOBevaluationswillinprinciplebeconductedincooperationwithotherdonors,aslongasIOB’scriteriaforindependenceandqualityaremet.Thenatureofthecooperationcanvaryfromactualparticipationbyseveraldonorsinanevaluation,agreeingonadivisionoftaskswithinasharedTermsofReference,toagreementonthedivisionoflabourbetweendifferentthemes,sectorsorcountries.

Page 19: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

17

Evaluation policy and guidelines for evaluations

4.3 Evaluation criteria

Evaluationcallsforatailor-madeapproach.Thatdoesnotmean,however,ignoringqualitycriteriaorabandoningconsistencyinapproach.Inallevaluations,theevaluationcriteriamustbeappliedinanunambiguousand,aboveall,transparentway.Here,inparticular,standardisationisparamounttoenableevaluationsandtheirresultstobecompared.TheevaluationcriteriaappliedbyIOBarelargelyidenticaltothosedrawnupbytheOECD/DAC.TheOECD/DACdefinitionofevaluationspecifiesfivecriteriaforevaluationofdevelopmentinterventions:efficiency,effectiveness,impact,relevanceandsustainability.Theevaluationcriteriarefertodifferentresultlevelsfoundedonthelogicalframeworkorresultchainsbasedonit.Together,thesecriteriaprovideaframeworkwithinwhichresearcherscanpresenttheirfindingsinanorderedandcoherentway.Theextenttowhichequalattentioncanorwillbepaidtoallcriteriawillvaryperevaluation.Furthermore,additionalcriteriacanbeapplied:evaluationof‘decompartmentalised’policy,forexample,mayrequirethatIOBdevotesexplicitattentiontopolicyconsistency.

4.3.1 Efficiency Efficiencymeasureshoweconomicallyresources(inputs)andthewaytheyareappliedareconvertedtodirectresults.Theconceptthereforeillustratestherelationshipbetweeninputandoutputandreferstoalevelofresultsthattheimplementercancheck.

Inevaluatingefficiency,boththequantityandqualityofinputsandoutputsareassessed.Themosteconomicalorcost-effectiveinputisnotalwaysthemostappropriateandtherecanbesignificanttradeoffsbetweenthequantityandqualityofoutputs.Assessingefficiencyalsocallsforcomparisonwithalternativeapproaches,withwhichthesameoutputscanbeachieved.Thisisbynomeansalwayseasybecausethereareoftennopreviouslyformulatedbenchmarksandtheymighthavetobedevelopedonthebasisofpracticalexperience.Thatismorelikelyinthecaseof,forexample,interventionsinvolvingconstructionactivitiesthanofthoseaimingtobringaboutsocialorinstitutionalchange.

Thefollowingelementsshouldbetakenintoaccountwhenevaluatingefficiency:

• Haveappropriateinputsbeendeployedatthelowestpossiblecost?• Haveactivitiesbeenconductedinanuncomplicatedmanner?• Havedecisionsbeenmadeattherightlevelandhasbureaucracybeenavoided

asfaraspossible?• Haveoverheadcostsbeenkepttoaminimum?• Hasduplicationbeenavoided?• Haveconflictsduringimplementationbeenpreventedorsolved?• Haveoutputsbeenachievedwithintheplannedperiodandbudget?

Page 20: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

18

4 Definitions and criteria for evaluations

4.3.2 Effectiveness Effectivenessrelatestotheextenttowhichthedirectresultsofinterventions(output)contributetothesustainableachievementofpolicyobjectives(outcome).Policyobjectivesaretheobjectivesthatinterventionsareintendedtoachieveandshould,inprinciple,beformulatedintheprojectdocument.Insomecasespolicyobjectivescanbeidenticaltotheoperationalobjectivesspecifiedinthebudgetorspecificcomponentsofthem.

Aninterventionisconsideredeffectiveifitsoutputshavemadeademonstrablecontributiontoachievementoftheintervention’sintendedobjectives.Efficiencyandeffectivenessrefertotwosuccessivelevelsintheresultschain.Apositiveassessmentatoneleveldoesnotautomaticallyimplyapositiveassessmentattheother.Aninterventioncanbeefficientwithouthavinganeffect,andaneffectiveinterventioncanbeinefficientlyimplemented.

Theeffectsofaninterventionarebeyondthedirectcontroloftheimplementer.Interventionsgenerateoutputswhichareassumedtogeneratethedesiredeffects.Evaluatingeffectivenessaimstoestablishthiscausality.Thecoreproblemisthatitwouldbeidealtoknowwhatwouldhavehappenediftheinterventionhadnotoccurred(thecounterfactual).Onlythencanobservedchangesbeattributedtotheinterventionandisitpossibletospeakoftheneteffects.Inotherwords,thereisaproblemofattribution.Aslongasotherfactorshaveavisibleimpactontheobservedchanges,theproblemisrelativelysimpletosolve.Itismoredifficultwheninfluentialfactorsarenotimmediatelyvisible(the‘unobservables’).Researcherscannotknowwhattheydonotknow.Evaluationstrytofindasolutiontothisproblem(see6.6).

Inpractice–especiallyinthecaseoflargeandcomplexinterventions,butalsowithinterventionsthataimtobringabouteffectsthataredifficultorimpossibletoquantify–evaluationsoftenmakeuseofplausibilityargumentation.Itisthenimportantthattheargumentistransparentandcredible.

Inessence,threestepscanbedistinguishedintheevaluationofeffectiveness:

• Themeasurementofchangesintheeffectvariablesincomparisontothesituationatthestart(baseline);

• Attributionoftheobservedchangestotheintervention;• Assessmentofchangesobservedandattributedtotheinterventionintermsofthe

objectives.

Page 21: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

19

Evaluation policy and guidelines for evaluations

4.3.3 Impact Thebroadconsensusinthecommunityofevaluatorsabouttheconceptsofefficiencyandeffectivenessislessevidentwhenitcomestoimpactandrelevance.

TheOECD/DACdefinesimpactasallsignificanteffectsproducedbyadevelopmentintervention,directlyorindirectly,intendedorunintended,ontheultimatestakeholdersandthirdparties.Whileeffectivenessrefersonlytotheintendedeffectsofanactivity,inthisterminologyimpactreferstoallpossibleeconomic,social,political,technicalandecologicaleffectsatlocal,regionalornationallevelthathaveadirectorindirecteffectonthetargetgrouporotherparties.

Inthestrictercontextofthelogicalframework,theconceptofimpactusuallyreferstothehighestlevelofresults.Thequestionisthentowhatextent,withtheaidofoutputs,theeffectsachieved(outcomes)havecontributedtobroader,morefar-reachingobjectives.Thesecanbeoperationalobjectivesorobjectivesatahigherlevel,suchasmoregeneralpovertyreduction.

Intheinternationalliteratureonimpactevaluations,theconceptofimpactisusedfortheneteffectsattributedtotheintervention,i.e.atthelevelofeffectiveness.

Oneinterpretationofimpactisnotbydefinitionbetterthananother.Itisimportantthatconceptsaremadeexplicitinevaluationreports.Intheinterestofmutualcomparisonitisdesirableinex-postimpactstudiestousetheinterpretationofimpactevaluationsmostcommonlyappliedintheinternationalliterature.InthisrespectIOBfollowstheinstructionsintheRPE2006.

4.3.4 Relevance Theconceptofrelevanceisalsosubjecttodifferentinterpretations.TheOECD/DACdefinesrelevanceastheextenttowhichtheobjectivesofadevelopmentinterventionareconsistentwithbeneficiaries’requirements,countryneeds,globalprioritiesandpartners’anddonors’policies.Thatmakesrelevanceacontainerconcept,withinwhichnochoicesaremade.Thisdefinitionisunsatisfactoryforseveralreasons.First,theassessmentofwhetheraninterventionisalogicalconsequenceoforiscompatiblewithpolicyisinessenceanexantequestionthatshouldbepartoftheapprovalprocedure.Second,donorpolicy,thepartner’spolicyandtheneedsofthetargetgroupdonotnecessarilyallperfectlycoincide.Thatwouldimplythataninterventioncanberelevantandirrelevantatthesametime.Third,policyprioritiescanchangeovertime,suchthataspecificinterventionisrelevantatonemomentandnotthenext,orviceversa.Lastly,thisdefinitionimpliesthattherelevanceofthepolicyitselfisnotupfordiscussion.Inotherwords,itexcludesproblemswithpolicyitself.

Inthelogicalframework,relevancereferstotheextenttowhichtheeffectsofinterventionsmakeasustainablecontributiontoachievingtheultimateobjective(theimpact).Aninterventionhasbeenvaluable,orrelevant,totheextentthatithas

Page 22: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

20

4 Definitions and criteria for evaluations

generatedeffectsthatbringachievementoftheultimatedevelopmentobjectivecloser.IOBwillapplytheconceptofrelevanceinitsevaluationsinthisrespect.

4.3.5 Sustainability Sinceanactivitycanhardlybeconsideredeffectiveiftheeffectithasachievedisnotlasting,sustainabilityisinfactanaspectofeffectiveness.4Atthesametimetheconceptofsustainabilitycomprisesavarietyofdimensionsthatarerelevantforassessinganintervention.Inthedonorcommunitythesustainabilityofinterventionsislinkedtosevenfactors,whichcanbeusedtoanalyseinterventions.Whichfactorsorcombinationsoffactorsareusefulforassessingsustainabilitywilldependonthenatureoftheintervention.

• Theinvolvementoftherecipientisusuallyrecognisedasafactorthataffectsthesuccessofanintervention.Involvementisexpressedthroughagreementonobjectivesandtheprovisionofhumanandmaterialresources;

• Whendeployingtechnologyitisimportanttotakeaccountofthefinancialandinstitutionalcapacitiesoftherecipientandthedegreetowhichtheconditionsformaintenanceandrenewalareguaranteed;

• Interventionscanhaveaneffectontheenvironmentintheshortorlongterm.Whileshort-termeffectsareusuallyenvisagedandarepreventedormitigated,itisespeciallyimportanttoincorporatelong-termenvironmentaleffectsintheassessment,astheycanneutralisepositivechangesorhaveotherharmfulconsequences;

• Thedegreetowhichchangesaresustainabledependsverystronglyonsocialandculturalfactors,whichareoftennotdirectlyvisible.Studyingperceptionscanprovidebetterinsightintothisdimensionofsustainability;

• Institutionalandcapacitydevelopmentplayanimportantroleindeterminingsustainability.Whenassessinginterventionstheattentiondevotedtotheseaspectswillhelpdeterminetheextenttowhichsustainabilityisachieved;

• Financialandeconomicsustainabilityreferstothedegreetowhichmeasureshavebeentakentoguaranteethatactivitiescancontinueandcompletedworkscanbemaintainedinthefuture;

• Interventionsalwaystakeplaceinawidernatural,political,economicandsocio-culturalenvironment,whichcannotbeinfluencedinanywayatallbutwhichitselfhasastronginfluenceoneffectivenessandsustainability.Changesinpoliticalstability,economiccrises,ornaturaldisasterscanhaveadecisiveeffectonthesustainabilityofinterventions.

4 In the context of humanitarian aid this is, of course, different. It usually entails short-term,

punctual interventions, the effects of which it is hoped will only be necessary for a brief period.

Page 23: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

21

Evaluation policy and guidelines for evaluations

4.3.6 Policy consistency Thisconceptreferstotheabsenceofinherentcontradictionsbetweendifferentpolicyareas.ConsistencyisparticularlyimportantforIOBintheevaluationofdecompartmentalisedpolicy.Developmentobjectivescanconflictwithinternationaltradeinterestsorforeignpolicyconsiderations.Thatiswhyitisimportant,particularlyincaseswhereobjectiveshavetobeachievedinthecontextofdecompartmentalisedpolicy,todeterminetheextenttowhichinterventionsperformedfromdifferentperspectivesarecoherentwithanddonotobstructeachother.Theaimistohelpachieveintendedobjectiveswithouttheeffectsofoneinterventionneutralisingthoseofanother.

Page 24: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

5Guiding principles and ethical standards

Page 25: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

23

Evaluation policy and guidelines for evaluations

IOB’sguidingprinciplesreflectthecorevaluesonwhichitsevaluationsarefounded.Theyareinterrelatedandmutuallyreinforcingand,assuch,theymustbeseenasprovidinganoverallframework,withwhichallaspectsofIOB’sevaluationworkmustbeconsistent.Theprinciplesdeterminewhatcanbeexpectedfromaninspectorandotherresearchersinvolvedinanevaluation,fromevaluationprocessesandproducts,andfromthemethodology.TheyareconsistentwiththeOECD/DAC’sPrinciplesforEvaluationofDevelopmentAssistance(1991)andEvaluationStandards(draft2009).

AllIOBevaluationsmustbeconsistentwiththefollowingguidingprinciples:

Independence Thejudgementsoftheresearchermustnotbeinfluencedbyexternalpressureoraconflictofinterests.Organisationsorinstitutionsthatarecommissionedtoperformanevaluationandmembersofevaluationteamsmaynothavebeenpreviouslyinvolvedintheinterventionsthataretobeevaluatedorinformulationofthepolicyonwhichtheyarebased.

Objectivity Theevaluationmustbebasedonverifiablefacts.Theresearchermustmakeeveryefforttoensurethatthedataonwhichtheevaluationisbaseddoesnotcontainanyinaccura-cies.Thepresentationoffactsshouldbeclearlyandrecognisablydistinguishedfromjudgments.

Transparency Theevaluationmustbeconductedinawaythatcanbefollowedclearlybystakeholdersandthirdparties.Thequestionstobeaddressed,thedatabase,theresearchmethods,findingsandconclusionsmustbepresentedinthereportinaclearandaccessibleway.

Validity TheevaluationmustmeasurewhattheTermsofReferencespecifiesasrequiringmeasurement,andmustdothisinawaythatthereadercanunderstand.

Reliability Theresultsoftheevaluationshouldbestable,i.e.thatarepeatoftheevaluationwillproducethesameresultsandconclusions.

PartnershipInthecaseofevaluationsinthefieldofdevelopmentcooperation,wherepossibleandinsofarasitdoesnotconflictwithotherprinciples,partnersfromtherecipientcountrieswillbeinvolvedintheimplementationofevaluations.Everyeffortwillbemadetocooperatewithindependentresearchersfromthepartnercountries.

Page 26: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

24

5 Guiding principles and ethical standards

Feasibility Evaluationsareonlyconductedifthedatabaseisaccessibleandtheassociatedmethodologyisavailable.

Correctness Theevaluationmustnotcauseharmtoindividuals.

Cost effectiveness Theevaluationmustbeconductedatthelowestpossibleprice,takingaccountoftheotherguidingprinciples.

Usability Theevaluationmustresultinconclusionsandpointsforattentionthatarevaluablefortheuseroftheevaluationbysuggestingactionsthatliewithintheuser’scapacity.

Evaluationisaformofresearchthatentailsajudgment.Itdiffersfrompureacademicresearchinthatitalwaysresultsinanopinionontheresultsofconcretehumanactions.Researchersthereforehaveagreatresponsibilitytowardsstakeholdersinvolvedintheevaluation.ThisappliesequallytoIOBinspectorsandtoexternalresearcherscontractedbythem.Researcherscanthereforebeexpectedtoadheretointernationallyacceptedethicalstandardsforevaluations.

ThefollowingethicalstandardsapplytoIOB’swork5:

Respect and understanding for other cultures Manyevaluationsareconductedinadifferentculturalenvironmenttothatoftheresearchers.Theycanthereforebeexpectedtocomplywithlocalculturalcodesduringinterviewsandotherformsofdatacollection.

Universal values Someculturalcustomsandpracticesmaybeincompatiblewithfundamentalrightsandfreedoms.Althoughresearchersshouldinprincipleshowrespectforothercultures,theymustnotignoretheeffectofcertainculturalvaluesongenderrelationsorminoritiesandotherspecificgroups.InsuchcasestheyshouldactinaccordancewiththeUniversalDeclarationofHumanRightsandtheassociatedinternationalagreements.

5 The standards are largely based on the evaluation policies of the Danish International

Development Agency (DANIDA) and the Canadian International Development Agency (CIDA).

Page 27: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

25

Evaluation policy and guidelines for evaluations

Gender Evaluationsshouldbecarriedouttakingaccountofthedifferentrolesperformedbymenandwomenandtheaimofachievinggenderequality.Inpolicyareaswheregenderdifferencesareadimensionoreffectofthepolicypursued,theyshouldbetakenintoaccountinthedesignandimplementationoftheevaluation.

Anonymity and confidentiality Researchersareexpectedtorespectrequestsfrominformerstotreattheinformationtheyprovidewithconfidentialityandtotakestepstoensurethatconfidentialinforma-tioncannotbetracedbacktothesource.

Responsibility Inmanycountriesacriticalstanceregardingthegovernmentcanhavesevereconse-quencesforresearchersfromthecountryitself.Inevaluationsinvolvinglocalresear-chersitisimportantthatsuchrisksberecognisedintimeandstepstakentoavoidthem,forexample,bymakingclearagreementsaboutwhoisresponsibleforthecontentofthereport.

Omissions Researchersmayobtaininformationthatisnotdirectlyimportantinansweringthequestionsaddressedintheevaluation,butisneverthelessrelevantinthelightoftheaccountabilityandlearningfunctionsoftheevaluation.Insuchcases,theclient(inthelastinstance,theIOBDirector)willbeconsultedtodeterminewhetherthisinformationshallbeincludedinthereportandinwhatform.

Assessment of individuals Evaluationsarenotassessmentsofindividuals.Yetevaluationreportscanincludefindingsonleadershipandthemanagementorotherqualitiesofimplementersthatcomeclosetopersonalassessments.Researchersshouldthenmakeeveryefforttopresentthefactsofeasilytraceablecasesinanabstractform.Ifindoubt,theyshouldconsultwiththeIOBDirector.

Evidence of malpractice Evaluationsarenotinvestigationsintolegitimacy.Neverthelessitispossiblethatmalpracticescometolightduringevaluations.InthatcasetheresearchershouldsubmitthecasetotheIOBDirector.Ifthereissufficientreasontodoso,theAuditDepartment(ACD)willbeinformed.

Page 28: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

6The evaluation process

Page 29: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

27

Evaluation policy and guidelines for evaluations

6.1 Programming

Thequalityofevaluationbenefitsfromacarefulandtransparentdecision-makingprocess.Thisprocessstartswiththeprogrammingofevaluations.IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperation.ThatmeansthattheIOBDirectorapprovestheprogrammingafterithasbeendiscussedwiththeFinancialandEconomicAffairsDepartment(FEZ)andtheotherpolicydepartmentsinvolved.Withrespecttotheotherareasofforeignpolicy,IOBadvisesontheprogramming,whichisapprovedinthemeetingoftheSecretary-GeneralandtheDirectors-General.IOBappliesthebasicprincipleslaiddownintheRPE2006forsystematisationoftheprogramming.Theevaluationprogrammingdistinguishesbetweenthreetypesofevaluation:policyreviews(policyevaluations),ex-postimpactstudies(impactevaluations)andotherevaluations.Policyreviewsandex-postimpactstudiesfallundertheRPE2006.TheRPEprescribesthateachpolicyareamustbeperiodicallysubjectedtoapolicyrevieworpolicyevaluationatthelevelofoperationalobjectives.Apolicyreviewshouldmakeasmuchuseaspossibleofex-postimpactstudies.PolicyreviewsandimpactstudieslargelydeterminetheIOB’sresearchagenda.

InadditiontotheRPE2006therecanbeotherreasonsforincludingevaluationsintheprogramming.

First,therearepoliticalconsiderations.Atcertaintimespolicyissuescanbethesubjectofpoliticalorsocialdebate.Itisthenimportanttofeedthedebatewithinsightsobtainedthroughthoroughandindependentevaluation.

Policyrelevancecanalsoplayarole.Thisisdeterminedbytheprioritygiventoanissueandthemomentinwhichtheevaluationiscarriedout.Ifitisclearwhenresultsarerequiredfortheformulationofnewprogrammesorpoliciesandanevaluationcanbeconductedwithinthatperiodwithoutlossofquality,IOBensuresthattheevaluationisprogrammedandimplementedintime,irrespectiveofthescaleorcomplexityofthesubject.

Inthefieldofdevelopmentcooperationthereisinternationalpressuretoimprovecoordinationbetweentheevaluationprogrammingofindividualdonorsandtoconductjointevaluationsasfaraspossible.Jointevaluationsincludethosecarriedouttogetherwithevaluationservicesintherecipientcountries,multidonorevaluations,andvariouscombinationsofthetwo.Workingtogetherwithevaluationservicesinrecipientcountrieshelpsstrengthentheirevaluationcapacityandimprovemutualaccountability.ThechoiceofwhichjointevaluationsIOBparticipatesinwilldependoninitiativesdevelopedduringinternationalconsultations,opportunitiestoenterintostrategicallianceswithlike-mindedevaluationservices,theextenttowhichIOB’scriteriaforindependenceandqualityareguaranteed,andwhethertheycanbefittedintoIOB’sexistingprogramming.

Page 30: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

28

6 The evaluation process

IOBalsoconsidersitimportanttoallowspaceinitsprogrammingforadhocrequests.ThesemayberequestsfromwithintheministryforIOBtoconductshortevaluationsofalimitednaturetodeterminetheprogressofprogrammesortosupportthedecision-makingprocess.Inaddition,ministers,statesecretariesorparliamentcanrequestIOBtoconductspecificevaluations.IOBmakeseveryefforttocomplywiththeserequests.

Theevaluationprogramminginthefieldofdevelopmentcooperationrelatesnotonlytobilateralcooperationbutalsotothenon-governmentalandmultilateralchannels.Inthecontextofthecofinancingsystem(MFS)IOBconductssystematicassessmentsofprogrammeevaluationscarriedoutundertheresponsibilityoftheco-financednon-governmentalorganisationsthemselves.Inaddition,IOBselectsanumberofthemesorcountriestoconductitsownevaluations.Todate,IOBhasbeensomewhatreservedinprogrammingevaluationsofmultilateralcooperation.Togetherwithotherdonors,IOBhasencouragedmultilateralinstitutionstoimprovethequalityoftheirownevaluationsthroughpeerreviews.Inthefuture,wherepossible,IOBwillparticipateinjointdonorinitiativesaimedatamoresystematicevaluationofmultilateraldevelopmentcooperation.

ThedecompartmentalisationofforeignpolicyandthesubsequentbroadeningofIOB’smandatemeanthatIOBevaluatesnotonlyinterventionsinthefieldofdevelopmentcooperation,butalsoactivitiesinotherareasofforeignpolicy.Itisnotunusualforotherforeignpolicyinstrumentstobedeployedtoaddressaspecifictheme,alongsidedevelopmentcooperation.Thiscanbegoodreasontoconducta‘decompartmentalised’evaluation.Inmanycases,onlytheMinistryofForeignAffairsisinvolvedinsuchanevaluation,butnotalways.Therearealsoissuesofforeignpolicyinwhichotherministriesplayaprominentrole.ForIOBthatisreasontoexplorethepossibilitiesforconductingevaluationsjointlywithotherministries.Insuchcases,andifIOB’sconditionsregardingindependenceandqualityaremet,theseevaluationswillbeincludedintheprogramming.

Page 31: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

29

Evaluation policy and guidelines for evaluations

6.2 Preliminary study

Onceasubjectforevaluationhasbeenincludedintheprogrammingapreliminarystudycanbeinitiated.Especiallyinthisinitialphase,anumberofimportantdecisionshavetobemadethatwillhavefar-reachingsignificanceforthefurtherprogressoftheevaluation.Theinspectorwillhavetobecomefamiliarwiththepolicyareatobeevaluatedandtheavailablesourcesofinformation,andformsomeideaofthequestionstobeaddressedintheevaluationandthemostadequateapproach.Withlarge,complexevaluationstheinspectorcandrawupastart-upreport.Thereportdescribesthereasonfortheevaluationandtheobjective.Itcanalsogiveabriefdescriptionofthepolicyareatobeevaluatedandformulatecorequestionsforthepreliminarystudy.Itcansuggestwhichtypeofevaluationwouldbemostappropriateandhowitmightbeorganised(theexpectedduration,whetherfieldmissionsareconsiderednecessary,andwhenandhowexternalconsultantsmightbeneeded).Lastly,thestart-upreportcanofferaninsightintotheexpectedproductsoftheevaluationandthebudget.

Start-upreportsarediscussedwiththeIOBmanagementandtheinternalpeerreviewersdesignatedfortheevaluation(see7.1).Afterdecisionshavebeenmadeontheabovementionedmainpoints,thepreliminarystudywillmoveontodraftingdefinitiveTermsofReferencefortheactualevaluation.

Thepreliminarystudycomprisesanumberofactivities:

Reconstruction of policy theory and intervention logic Theextenttowhichasubjectcanbeevaluatedisimprovedbyclearlyformulatedpolicyprinciplesandprogrammeobjectives,adescriptionoftheactivitiesundertaken,andalogicalrelationshipbetweenpolicy,objectivesandactivities.Practiceshowsthatthesecriteriaarebynomeansalwaysmet.Generallyspeaking,thisappliestobothdevelopmentcooperationandotherareasofforeignpolicy.Especiallyatthelevelofoperationalobjectives,policyisoftenbroad-rangingandnotveryspecificallyformulated,aimsandeffectsarefrequentlyunclearandthelogicofinterventionsisnotsufficientlyexplicit.Insuchcases,onthebasisofthepreliminarystudy,inspectorsthemselveswilldevelopapolicytheoryandareconstructedinterventionlogicbasedonit.Theywillthenpresentthetheoryandthelogictothepolicymakertodeterminewhethertheyareinlinewithwhatthelatterhadinmind.

Defining the evaluation objectPolicythemesandsectorsareoftenbroadlyformulatedandcoverawidevarietyofactivities.Thisappliesinparticularattheleveloftheoperationalobjectivesspecifiedintheministry‘sbudgetdocument.Thepreliminarystudyshouldthereforeestablishthelimitsoftheevaluationobject–alsointhecaseofwide-rangingevaluations–suchthatithasaclearfocusandthusremainsfeasible.Thisoffersabetterguaranteethatthefocusoftheevaluationremainsclearandthatonlythosequestionsareformulatedthat

Page 32: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

30

6 The evaluation process

canreasonablybeexpectedtobegivenavalidanswer.Establishingthelimitsofanevaluationrelatestothetimedimension,thegeographicaldimensionandthenatureoftheactivities.Itisbasedonconversationswithrelevantinformants(stakeholders,externalexperts)andonliteratureandfilestudies.

Identifying result indicators Animportantcomponentofreconstructingpolicytheoryisformulatingaresultschain,distinguishingvariouslevelsofresults.Measurableresultindicatorsareidentifiedforeachoftheresultlevels,toenableperformancetobeassessedatthatlevel.

Identifying data sources Identifyingthescaleandqualityofexistingsourcesofinformation(files,literature,informants,statisticaldatabases)andtheopportunitiesforgatheringsupplementaryinformationdeterminetheextenttowhichthesubjectcanbeevaluatedandhowtheevaluationquestionsareformulated.

Formulating the research questions TheresearchquestionsincludedintheTermsofReferencearedeterminedbytheultimateaimoftheevaluation.Theyrelatetotheself-evidentneedforinformationonresultsachievedatthelevelofoutputsandeffects.Thisinformationisofcoursevaluableforboththeaccountabilityandlearningfunctionofanevaluation.Inaddition,supplementaryquestionsrelatingtoaspecificlearningneedcanbeformulated.

Determining the method of evaluation Thechoiceofresearchmethodsdependsonalargenumberoffactors.Itwillbeinfluencedbythenatureofthesubject,thetimeandbudgetavailablefortheevaluation,datathatisalreadyavailableandneedstobecollected,andthenatureoftheresearchquestions.

6.3 Terms of Reference

TheTermsofReferenceestablishtheconditionsunderwhichtheevaluationwillbeconductedandformtheresultofthepreliminarystudy.TheToRmustcontainatleastthefollowingcomponents:

• thereasonandjustificationfortheevaluation:ashortexplanationoftherelevanceoftheproposedevaluation;

• the(external)aimoftheevaluation;• thecorequestions:formulationofthecentralresearchquestionandthesecondary

questionsderivedfromit;• thedemarcationandscopeoftheevaluation:explanationofhowthefieldof

Page 33: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

31

Evaluation policy and guidelines for evaluations

researchwillbedemarcatedintermsoftheme,geographicallocationandtime;• approachandchoiceofmethodology,includingfocusonsubsidiarystudies:

descriptionofthedesignoftheevaluation,theindicators,informationsourcesandresearchtechniques;

• representativeness:explanationofhowandtheextenttowhichrepresentativenessplaysaroleintheevaluation;

• organisation:explanationoftheorganisationoftheevaluation;deploymentofexperts,organisationofworkshops,missions,etc.;

• products:indicationoftheexpectedsubsidiaryproducts,includingsubsidiarystudiesoradatabase.Alsoanindicationofwhetherthereportwillbepublishedorwhethertheevaluationisinternal;

• timepath:thetimepathfortheevaluation;• budget:aspecifiedoutlineofthecosts.

IftheTermsofReferenceareforapolicyreviewthequestionsprescribedbytheRPE2006mustalsobetakenintoaccount.Policyreviewsmustaddressthefollowingquestions:

• Whatproblemgaverisetothepolicy?Doestheproblemstillexist?Whatisthecauseoftheproblem?

• Whydoesthegovernmentconsidertacklingtheproblemasitsresponsibility?Whyistheresponsibilityatcentralgovernmentlevel(andnotatlocalorEUlevel)?Whatformdoesthatresponsibilitytakeandwhy?

• Whatobjectivehasthegovernmentformulatedtosolvetheproblem?• Whatinstrumentsarebeingdeployed?Howcoherentarethey?Isthereoverlap?

Whatisknownabouttheimplementationofthepolicyandtheefficiencyofoperationalmanagement?

• Whatistheeffectoftheinstrumentsontheformulatedobjectives(solutiontotheproblem)?Doinstrumentsinotherpolicyareasalsohavesignificanteffectontheformulatedobjectives?Whatarethemainpositiveandnegativeside-effects?

• Howarethebudgetsdetermined?Onwhatbasis?

ThedraftTermsofReferencearenotonlyputontheagendaoftheIOBdepartmentalmeetingbutshouldalsobediscussedinadvancewithdirectstakeholdersandexternalexperts(preferablyintheformofareferencegroup,see7.2),andwiththeinternalpeerreviewers.Thedegreetowhichandthewayinwhichcommentsaredealtwithmustbecleartoallparties.TheTermsofReferenceforanevaluationareapprovedbytheIOBDirector;ifthesubjectoftheevaluationliesoutsidedevelopmentcooperation,theToRwillbeapprovedbytheDirector-Generalconcerned.

TheTermsofReferenceformthebasisonwhichabudgetwillbereservedfortheevaluation.Italsoprovidesaframeworkforcontractingexternalresearchers.ByapprovingtheTermsofReference,theIOBDirectoralsoapprovesthebudgetincludedintheToR.

Page 34: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

32

6 The evaluation process

6.4 Deploying researchers

TheIOBinspectorbearsfullresponsibilityforsettingup,conductingandcompletingtheevaluation.Inmostcases,theinspectorwilldeployresearcherstoconducttheevaluation.Themomentatwhichtheywillbedeployedwillvary,aswilltheextenttowhichtheyoperateautonomously.

Researcherscanberesearchassistantsorconsultantsfromresearchinstitutions,universitiesorconsultantagencies.

Researchassistantsareusuallyrecentgraduateswhosupporttheinspectorinconductingtheevaluation.Theactivitiestheyperformvaryfromfilestudies,assistancewithfieldmissionsorinterviewinginformantstodraftingpartsofthefinalreport.Thewayinwhichtheyaredeployedwilldependontheirexperienceandcanchangeovertime.Researchassistantsarefullmembersoftheevaluationteam.

WiththemajorityofevaluationsIOBcontractsexternalconsultants.6Twocriteriaaredecisive:theexpertiseofexternalexpertsmustbecomplementarytothatoftheinspector,andtheymusthaveampleresearchexperience,preferablyinthecontextofevaluations.Variousmodalitiesarepossiblefordeployingexperiencedexperts.Oneormoreexpertscanbedeployedasadirectcounterpartfortheinspectorandadivisionoftasksagreedbetweenthem.Anotheralternativeisforwholeorpartoftheevaluationtobecontractedouttoaresearcheroraresearchbureau.Thewayinwhichexternalexpertiseisdeployeddependsonthenatureoftheservicesrequiredandassociatedcosts.Inthecaseofjointevaluationswithotherdonors,thewholeevaluationisgenerallycontractedout.

6.5 Cooperation with researchers in partner countries

InconductingevaluationsIOBaims,wherepossible,toworktogetherwithresearchersfrompartnercountries.Thathelpstobuildindependentresearchcapacityinthesecountries.Furthermore,researchersfrompartnercountriescanbeinvaluableastheyareoftenwellinformedonlocaldevelopmentsandrelations.However,notallevaluationsareofcoursesuitedtosuchcooperation.Thepossibilityofmakinguseofresearchersfrompartnercountriesdependsinthefirstplaceonthesubjectoftheevaluation.Somesubjectsdonotwarrantsuchachoice,forexample,becausetheevaluationwilladdressonlyDutchactivitiesorinterventionsofaglobalnature.Wherethedeploymentofexpertsisalogicaloption(forexample,evaluationofsectorpolicyinaspecificcountry)IOBactivelyexploresthepossibilities.

6 Contracting out and entering into contractual relationships are determined by the procedures laid

down in the Operational Procedures Manual (HBBZ). They will not be considered further in this document.

Page 35: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

33

Evaluation policy and guidelines for evaluations

6.6 Methodology

Evaluationisaformofresearchthatentailsajudgment.Thecustomarymethodologyislargelyderivedfromthesocialandeconomicsciences.TheTermsofReferencegiveageneraloutlineofthemethodologytobeappliedintheevaluation.Whentheevaluationistobecontractedouttoexternalresearchers,theyareusuallyaskedtoelaboratethemethodologytheywilluseinaninceptionreport.

InaccordancewiththeOECD/DACEvaluationQualityStandards,IOBrequiresthatasoundmethodologyisappliedandthatitisaccountedforinthefinalreport.Themethodologicalchoicesmadeinanevaluationaredeterminedbytheextenttowhichtheyleadtothecollectionofreliabledataandprovideabasisforreachingvalidandreliablejudgments.Themethodsandtechniquesusedcanvaryperevaluation.

Thevalidityofanevaluationisdeterminedbytheextenttowhichthedesignoftheevaluation,themethodofdatacollectionandtheanalysisofthedataleadtoreplicableanswerstothequestionsposedintheTermsofReference.Validityrelatesto:

• thestatementoftheproblem(theclaritywithwhichtheproblemisstated,theformulationoftheresearchquestionsandthedefinitionoftheevaluationcriteria);

• theevaluationobject(thedefinition,operationandlimitationoftheevaluationobjectandsituatingitinthepolicyandinstitutionalcontext);

• thepolicytheory(thepresentationorreconstructionoftheinterventionlogicandresultlevels,andtheoperationalisationofresultmeasurementonthebasisofindicators);

• theanalysis(datasources,datacollectionandtheprocessingandargumentationofconclusionswithfindings).

Thereliabilityofanevaluationisdeterminedbytheextenttowhichtheresultsarestable,i.e.thatrepeatingtheevaluationwillproducethesameresults.Reliabilityisrelatedto:

• theresearchmethods(specificationofandaccountingfortheresearchmethodsused,verificationofdataandtriangulation);

• thescope(representativenessofthesampleandselectionofcasestudiesandspecificationofthelimitsoftheevaluation);

• independence(ofthesourcematerialandoftheresearchersinrelationtostakeholders);

• theprogressoftheevaluationandqualitycontrol(reportingontheprogressoftheevaluationandorganisedqualitycontrolthroughinternalandexternalmonitoring).

ToguaranteethevalidityandreliabilityofevaluationsasmuchaspossibleIOBchoosesacombinationofresearchmethods(‘mixedmethods’).Thiscombinationrelatestothedesignoftheevaluationandtoqualitativeandquantitativemethodsandtechniquesofdatacollectionandanalysis.

Page 36: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

34

6 The evaluation process

The design of the evaluationThemajorityofIOBevaluationsaredesignedaccordingtotheguidelinesfor‘theory-based’evaluation.Thatislogicalbecausetheaimistoassesstheefficiencyandeffectivenessofpolicyimplementation.Thatrequirestakingpolicytheoryasastartingpoint.Themodeloftheory-basedevaluationisverysuitedtopolicyreviews.Theory-basedevaluationsfocusonadetaileddescriptionofwhathashappenedinthepastandwhy,andanalysingthedemonstrablecausesofthesedevelopments.Todothatinaresponsiblemanner,itisimportantinthefirstinstancetoreconstructthepolicytheory.Theevaluationthenassessesthevalidityofthepolicytheory.Themosttriedandtestedmethodofidentifyingaimsandinterventionstrategiesisthroughthelogicalframeworkortheoftenmoredetailedresultchain.Thismodeloffersaframeworkforsystematicdatacollection,identifyingindicatorsforsystematicassessment,anddeterminingtherelationshipsbetweenthedifferentresultlevels.

Analternativeforthisevaluationdesignistotakechangesinrealityasastartingpointratherthanthepolicytheory.Supportersof‘realistevaluation’advocatethiskindofapproach,whichfirstdevotesattentiontothecontextinwhichchangesoccurandthemechanismsinvolved.Onlyinthesecondinstancedoesitattempttoattributechangestospecificactivities.

Inrecentyearstherehasbeenincreasingemphasisinternationallyontheneedtoprovidestrongerevidenceoftheeffectivenessofgovernmentpolicy.Commonevaluationpractice,whichgenerallyprovidesplausibleexplanationsoftherelationshipbetweeninterventionsandtheireffects,isnolongerconsideredadequate.Thereisaneedtoidentifytheneteffectsofinterventions,ifpossibleinquantifiableterms.Thatmakesitnecessarytoconductimpactevaluations.ThedesireforharderimpactstudiesisalsoexpressedintheRPE2006,whichexplicitlydistinguishesex-postimpactstudiesasaseparatecategoryinadditiontopolicyreviews.InitsevaluationprogrammingIOBthereforeincludestheimplementationofaseriesofimpactevaluationsinappropriatepolicysectorsandthemes.Inpractice,however,theoptionsarenotunlimited.‘Randomisedcontroltrials’(RCTs),inwhicha‘treatmentgroup’iscomparedwitha‘controlgroup’andwhichineffectreproducealaboratorysituation,arenotapplicableinthegreatmajorityofcases.ThemostcommonalternativeforRCTsisaquasi-experimentalresearchdesign.AlthoughthiscloselyresemblesafullyexperimentalRCT,itrecognisesthattheresearcherdoesnothavefullcontroloverthecompositionofthetreatmentgroupandthecontextinwhichthemeasurementsaremade.Theselimitationsareaccountedforintheevaluationreport.

ForthesakeofcompletenessitshouldbemadeclearthatIOB’simpactevaluationsarealsodesignedonthebasisofatheoryofhowtheinterventionstobeevaluatedwork.Themostimportantdifferenceisthatthistheoryisnotexclusivelybasedonareconstructionofpolicy,butislargelyfuelledbyexistingscientificinsights.

Page 37: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

35

Evaluation policy and guidelines for evaluations

Qualitative and quantitative methods of data collection Variousmethodshavebeendevelopedtomeasuretheeffectivenessofinterventions.Generallyspeakingadistinctioncanbemadebetweenqualitativeandquantitativeresearchmethods.Qualitativeresearchmethodsfocusmoreonprocessesandemphasisethebehaviourofactors.Theyareflexible,oftenbecausetheyhaveaneclecticapproach,andcanplacefindingsintheirculturalandpoliticalcontext.Aqualitativeapproachcangiveaplausibleexplanationofwhyacertaininterventionhasorhasnothadaneffectinagivencontext.Qualitativemethodsplayanimportantroleindetermininginternalvalidity.Itishowevernotalwayspossibletogeneralisethefindingsofqualitativeresearch.Quantitativeresearch,ontheotherhand,explicitlyaimstomakegeneralstatements(externalvalidity).Statisticaltechniquesmakeitpossibletoisolatetheeffectsofaninterventioninacomplexenvironment.Thismethodis,however,verydata-sensitive.Itrequiresalargequantityofreliableinformationtoallowtheformationofsamplepopulations.

Quantitativeandqualitativetechniquesarenotinterchangeable,butcomplementary.Quantitativetechniquescanprovideastatisticalbasisforamorequalitativeapproach,whileaqualitativeanalysecanexplainrelationshipsestablishedbyquantitativeresearch.Withoutatheoreticalframeworkandqualitativeresearchintothemechanismsbehindassumedrelationships,statisticalanalysishasnosubstance.InitsevaluationsIOBappliesbothquantitativeandqualitativetechniques.

Methods of data collection Thevalidityandreliabilityofanevaluationstandsorfallswiththereliabilityofthedatacollection.Wherethedataiscollecteddependsstronglyonthesubjectoftheevaluation.Whendevelopmentinterventionsarebeingevaluatedsomeofthedatawillusuallybecollectedinthepartnercountriesconcerned.Iftheyareavailable,existingdatabasescanbeused.

Themostcommonformsofdatacollectionare:

• File and literature studiesToobtainaclearpictureofthefieldofstudyitisnecessarytobecomefamiliarwiththecompleterangeofwrittensourcesavailable.Thisincludesinformationrelatingtointerventions,morepolicy-relateddocumentationandacademicliterature.Itrequiresworkingthroughthearchivesingreatdetail,preferablyusingasystematiclistofpointsforattention.Oneimportantpointforattentioninafilestudyistoassessthereliabilityoftheinformation.

• Interviews with key informants Generallyadistinctionismadebetweenstructured,semi-structuredandnon-

structuredinterviewmethods.Whichmethodisapplieddependsontheroleoftheinterviewintheresearchandthenatureoftheinformationthatisbeingsought.Especiallyintheinitialorientationphaseinterviewswillgenerallybeopentoallowasmanydifferentopinionsandviewpointstobecollectedfromtheinterviewees.

Page 38: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

36

6 The evaluation process

• Group interviews Variouskindsofgroupinterviewsareusedinevaluations.Oneofthemost

commonformsisthefocusgroupdiscussion,inwhichtheinteractionbetweentheintervieweescanalsobeavaluablesourceofinformation.

• Direct measurement Alotofquantifiableinformationcanbeacquiredbysimplerecordingand

counting.Especiallyatthelevelofoutputs,thisisthemostlogicalapproach.Directmeasurementisparticularlyusefulifdataisavailablefromabaselinestudy.

• Direct observation Observationmethodscanbelimitedtothephysicaloutputsofanactivity,butcan

alsorelatetocertainprocesses.Observationtechniques(withorwithoutthedeploymentofaudiovisualaids)havealwaysplayedanimportantroleinfieldresearch.Itshouldbeaddedthattheycanbeverytime-intensive.Evaluationsgenerallymakeuseofaquick(moreorlessstructured)tourofphysicaloutputs,a‘transect’,and/orattendingoneormoremeetingsofthetargetgroupsconcerned.

• Surveys Surveysaimtocollectalargequantityofstandardiseddatafromalargepopulation.

Suchdataisverysuitableforstatisticalanalysis.Surveysmakeuseofpreviouslydrawnupquestionnaireswhicharepreferablytestedfirstandwhichcanbeinwrittenorverbalform.Theyareessentialindeterminingthestartingsituation(baseline)andareindispensableinimpactevaluations.Becausesurveyscanbetime-consumingandcostly,impactevaluationsoftenusetheresultsofpreviousquestionnairesormodulesaddedtopendingsurveys,sothattheycanservemultiplepurposes.

• Case studies Casestudiesarein-depthstudiesofasmallnumberofspecificinterventions.They

offerthepossibilityofconsideringtheperceptionsofstakeholdersorreconstructinghistories.Casestudiesareavaluablesupplementtobroadersurveys,becausetheyprovideanopportunitytoexamineandexplainrelationshipsthatbecomeapparentfromstatisticalanalysisingreaterdetail.Ifanevaluationisbasedexclusivelyonaseriesofcasestudies,itisessentialtodeterminetowhatextenttheselectedcasesarerepresentative.

Page 39: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

37

Evaluation policy and guidelines for evaluations

6.7 Reporting

ThefinalreportofanevaluationisapprovedbytheIOBDirector,aftershe/heissatisfiedthatthecommentsofthereferencegroup,internalpeerreviewersandtheIOBDepartmentalMeetinghavebeenincorporatedadequately.IOBthensendsthereporttotheministerorministersconcernedtobeforwardedtoparliament.Onrequest,IOBcanalsoprovidetheministerwithadditionalinformationonthefindingsoftheevaluation.Ifanevaluationisaddressingapoliticallyverysensitiveissue,IOBwillinformtheministerconcernedingoodtimeontheprogressoftheevaluationand,asfaraspossible,theexpectedoutcomes.

Inprinciple,allIOBevaluationreportsareaccessibletothepublic.Thisrulecanonlybedeviatedfrominexceptionalcases.Anevaluationmight,forexample,beveryconfidentialinnatureormayonlyserveaninternallearningfunction.Thedecisionnottopubliciseanevaluationreportmustalwaysbetakenbeforetheevaluationstarts.

Thereportshouldtakeaccountofthefactthatthefindingsoftheevaluationaretobepresentedtodifferentstakeholderswithvariousinterests.ThebasicprincipleisthatIOBreportsaredesignedtoaccounttobothparliamentandsociety,andtofulfilalearningfunction.Thatsetshighstandardsforthereporting.Parliamentbenefitsfromashortandconcisepresentationofthemainfindingsandconclusions.Accountingfortheresultsandthedeploymentofthepublicfundsrequiredtoachievethemhaspriority.Policydepartmentswillbeinterestedintheextenttowhichthereportprovidesinformationthatoffersopportunitiestoadjustpolicyformulationorimplementation.Inallcasesitisimportanttoaimforatransparentpresentationoffindings,analysisandassessment.

IOBevaluationsidentifyanumberofburningissuesthat,inthelightoftheevaluationfindings,requireattention.IOBthereforeavoidsmakingconcreterecommendations,puttingitselfinthepositionofthepolicymakerandtherebyofabutcherinspectingthequalityofhisownmeat.Furthermore,IOBworksaccordingtotheOECD/DACdefinitionofevaluation,whichrefersonlyto‘lessonslearnt’,andtheguidelinesforpolicyreviewintheRPE2006.

IntheinterestoftransparencytheTermsofReferenceshouldalwaysbeattachedtotheevaluationreportasanannex.Theresearchquestionsshouldbesystematicallyaddressedinthereport,alsoifsomeofthemcannotbeanswered.Ifnecessary,additionalinformationshouldbeprovidedintheannexesontheresearchmethodsemployedandanyproblemsthatariseduringimplementationoftheevaluation.

IftherearefundamentalandsubstantiateddifferencesofopinionbetweenIOBandthestakeholdersregardingthemainconclusionsofthereport,theseshouldbeexplicitlymentionedinthereport.ThedecisiontothiswillbetakenbytheIOBDirector.

Page 40: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

38

6 The evaluation process

Manyevaluationreportsareasynthesisofsubsidiarystudies.Especiallywherethesesubsidiarystudiescanalsobereadindependentlyitisimportanttomakethemaccessibleforpolicydepartmentsorotherstakeholders.Theyoftencontaindetailedinformationthatisnotincludedinthemainreport.TheevaluationreportsarepublishedbothinprintformandontheMinistryofForeignAffairswebsite.Subsidiarystudiesarealwaysavailableonthewebsiteandareinsomecasesalsopublishedinprintform.Inthecaseofveryextensiveevaluations,intheinterestsofaccessibility,IOBproducesasummarisedversionofthereport.BoththesummaryandthefulltextaresubmittedforinclusioninOECD/DACnetworkforevaluationservices’evaluationinventory.Completedimpactevaluationsarealsoenteredinthedatabasesetupby3IEandNONIEforevaluationsofthiskind.

Page 41: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

39

Evaluation policy and guidelines for evaluations

Page 42: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

7Quality control

Page 43: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

41

Evaluation policy and guidelines for evaluations

TheIOBinspectorisresponsibleforconductingindependentevaluations.Thequalityoftheworkcarriedoutis,however,aconcernforIOBasawhole.TheIOBmanagementhasitsownresponsibilityinthiscontext,asitapprovestheevaluationreports.Toprepareforthisdecision-makingprocess,IOBhasaqualitycontrolsystemcomprisingfiveinstruments:internalpeerreviewers,referencegroups,thedepartmentalmeeting,theadvisorypanelandtheself-evaluation.

7.1 Internal peer review

Qualitycontrolbenefitsfromintensivecooperationandasystematicintercollegialmonitoring.IOBthereforehasasystemthatensuresthatinspectorscanrelyonsubstantivesupportfromcolleagueswhenconductingevaluations.Start-upreports,TermsofReference,subsidiarystudiesandthedraftfinalreportarediscussedwithtwointernalpeerreviewers,fellowinspectors,beforebeingsubmittedtothedepartmentalmeeting.Theinspectorchooses,inconsultationwiththeIOBDirector,whichcolleaguesaretoactasinternalpeerreviewers.TheDirectorappointsinternalpeerreviewersperevaluation.Theyareselectedandappointedonthebasisofthefollowingcriteria:

• coachingoftheinspector;• internalreadershipasalearningfunctionforrecentlyappointedinspectors;• evaluationexperience;• affinitywiththeevaluationsubjectandexpertiseinthethemeconcerned;• avoidingconflictsofinterest;• anequaldistributionoftheworkloadwithinIOB.

Notallcriteriaareequallyimportantandaspecificcriterioncanbedecisive.

Thetaskoftheinternalpeerrevieweristogiveadvicetotheinspectors.Thisadviceisnotbindingbutinspectorsdohavetoindicatewhattheydowithit.LastingandfundamentaldifferencesofopinionmustbesolvedinconsultationwiththeIOBDirector.

Inspectorsmustincludethecommentsandadviceofinternalpeerreviewersandashortreportoftheirdiscussionswiththereadersinthefileontheevaluation.

Theinternalpeerreviewersarenotpartofthereferencegroupfortheevaluation.

AstheIOBDirectorisresponsibleforapprovingevaluationproductshe/shecannotbeaninternalpeerreviewer.

ThedecisionwhethertomakeuseofinternalpeerreviewintheeventofjointevaluationsbetweenIOBandtheevaluationservicesofotherdonorsislefttotheIOBinspectorresponsiblefortheevaluation,inconsultationwiththeIOBDirector.

Page 44: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

42

7 Quality control

7.2 Reference groups

AsecondformofqualitycontroloverIOBevaluationsisreferencegroups.Thesegroups,whicharecreatedforeachevaluation,arepartlyexternal,consistingofrepresentativesofthepolicydepartmentsinvolvedintheevaluation,organisationsand/orlocalbodies,andexternalexperts.AreferencegroupischairedbytheIOBmanagementandmeetsatkeymomentsduringtheprocess(e.g.todiscussthedraftToR,partialreportsandthedraftfinalreport).Theinspectorisresponsibleforreportingonreferencegroupdiscussionsand/orprovidingsummariesofwrittencommentsbyreferencegroupmembers.

Forevaluationsthatentailintensivecooperationwithresearchersinaspecificpartnercountry,asoundingboardgroupissetupinthecountryconcerned.Theformthegrouptakeswillvaryfromcasetocase.

ForjointevaluationswithotherdonorsaDutchreferencegroupisusuallynotsetup.Thereisajointsteeringgroupcomprisingrepresentativesoftheparticipatingdonors.Inthecaseoflarge-scale,complexevaluationsasmallermanagementteamissetupfromwithinthesteeringgrouptoruntheevaluationonadailybasis.Thesteeringgroupisthenprimarilyconcernedwithqualitycontrol.Inaddition,withtheseevaluations,theresearchbureauconductingtheevaluationisexpectedtoarrangeitsownqualitycontrol.Thisisoneoftheaspectstakenintoaccountwhenassessingtenders.

7.3 Departmental meeting

TheIOBdepartmentalmeetingplaysapartinqualitycontrolattwomomentsduringtheimplementationofanevaluation:whendiscussingtheTermsofReference(orsometimestheinceptionreport)andduringdiscussionofthedraftfinalreport.TheToRarealwaysdiscussedatafulldepartmentalmeeting.BeforethismeetingitmustbecleartotheinspectorwhichcolleagueshavesubmittedcommentsontheTermsofReferenceandthesubstanceofthecomments,ingeneralterms,sothathe/shecanprepareforthediscussion.Draftfinalreportsarethendiscussedonceagaininaseparatefinalmeetingbetweentheinspectorandtheinternalpeerreviewers,chairedbytheIOBDirector,beforetheDirectorapprovesthereport.Thisfinalmeeting,whichisopentoallIOBstaffmembers,hastwopurposes:itofferstheopportunityforafinalassessmentofthequalityoftheproduct,andhelpscolleaguestolearnfromeachothers’experiences.ItalsoensuresthatthefinalproductofanindividualinspectorbecomesaproductofIOBasawhole.

Page 45: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

43

Evaluation policy and guidelines for evaluations

7.4 Advisory panel

ToincreasetheusabilityanduseofevaluationsinthefieldofdevelopmentcooperationtheMinisterforDevelopmentCooperationmadeapledgetoparliamenttoestablishanAdvisoryPanelofprominentexternalexperts.Thepanel,withitssecretariatatIOB,willplayaroleinqualitycontrolbymonitoringtheusabilityofevaluationsandofferingadviceonrequestoronitsowninitiative.Itwillbecomeoperationalinthecourseof2009.

7.5 Self-evaluation

Everyevaluationconcludeswithacriticalreviewofhowitproceeded,withtheaimofimprovingtheapproachtofutureevaluations.Theseself-evaluationsarewrittenbytheinspector,sometimestogetherwitharesearchassistantandtheinternalpeerreviewers.TheTermsofReferencefortheevaluationdeterminetheformatoftheself-evaluation.BysystematicallygoingthroughthevariouscomponentsoftheToR,theself-evaluationprovidesaclearinsightintothesuccessfulandlesssuccessfulelementsoftheevaluation.Itshouldinanycaseaddress:(1)theextenttowhichtheresearchassignmentcouldbeimplemented,(2)anydifferencesbetweentheplannedtimepathandtheactualdurationoftheevaluation,and(3)differencesbetweenestimatedandactualcosts.Self-evaluationsarediscussedinternallyandperiodicallyatthedepartmentalmeeting.

Page 46: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

8Communication andfeedback

Page 47: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

45

Evaluation policy and guidelines for evaluations

Communicationandfeedbackrelatetothefeedbackofevaluationresults,withaviewtothetransparencyoftheevaluationprocess,theaccountabilityfunctionandtheintegrationoflessonslearnedinfuturepolicyformulationandimplementation.Feedbackisnotlimitedtothephasefollowingcompletionoftheevaluation.Consultationwithandfeedbacktostakeholderstakeplacethroughoutthewholeperiodofimplementation:duringthepreliminarystudy,inrelationtotheTermsofReference,andduringdiscussionofsubsidiarystudiesandthedraftfinalreport.Wherenecessary,consultationandfeedbackisalsoprovidedtothepartnercountries.Whilefeedbackduringtheevaluationisimportantatimplementationlevel,aftertheevaluationhasbeencompleteditismoreimportantatpoliticalandmoregenerallevel.

Theextenttowhichcompletedevaluationsfulfilthelearningfunctiondependsonanumberoffactors.Inthefirstplace,itisamatteroftheusabilityoftheevaluationitself.Thisisdeterminedby:

• theclaritywithwhichtheexternalaim(i.e.outsidetheevaluationitself ),for whichtheevaluationresultscanbeused,isspecified;• theclarityandcompletenesswithwhichtheessenceoftheevaluationandespecially

itsmainfindingsarepresentedintheevaluationreportandthesummary;• thecompletenesswithwhichallresearchquestionsareansweredbythe

conclusions;• theextenttowhichtheresultsoftheevaluationsuggestactionsthatliewithin thecapacityoftherecipientofthereport.

Inthesecondplacelearningfromevaluationsdependsontheextenttowhichtherecipientsandusersarereceptivetothelessonsdrawnfromthestudies.Thisisrelatedtotheinstitutionalcontextandtheorganisationalculture.IOBhasnoinfluenceonthisanditisnotIOB’sresponsibility.Itisthedirectresponsibilityofthemanagementoftheorganisation(theministryorotherorganisationsthatreceiveIOBreports).

IOBdoeshoweverhavearesponsibilitytocommunicateonevaluationreports.Ascommunicationwithpolicydepartmentsiscrucialtoincreasethelearningeffect,itiscrucialthatevaluationsarediscussedintherelevantstaffmeetings.Withregardtoexternalcommunication,IOBdrawsupacommunicationplanforeachevaluation,inconsultationwiththeministry’sInformationandCommunicationDepartment.IOBincreasestheaccessibilityofevaluationresultsandensurestheirwiderdistributionbypreparingmediapresentations,givinglecturesandinterviews,writingarticles,andorganisingconferencesandseminars.

IOBevaluationsarealsoaninstrumentforrenderingaccountabilitytoparliament.WithinthreemonthsofthereportbeingapprovedbytheIOBDirector,theministersendsittoparliament,usuallytogetherwithapolicyreaction.Thetextofthereaction,whichisusuallydraftedbythepolicydepartmentconcerned,containsindicationson

Page 48: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

46

8 Communication and feedback

howtheresultsoftheevaluationaretobeacteduponinthelightoffuturepolicy.Inthisway,accountingtoparliamentalsocontributestothelearningprocess.

Inthecaseofevaluationsinthefieldofdevelopmentcooperationthereportsarealsopublishedinthepartnercountries,offeredtothegovernmentandmadewidelyknownthroughtheorganisationofmeetings.WithaviewtomutualaccountabilityIOBinvestssystematicallyinsuchfeedbacktothepartnercountries.

Page 49: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

47

Evaluation policy and guidelines for evaluations

297 2004 Over solidariteit en professionalisering

Evaluatie van Gemeentelijke Internationale Samenwerking (1997-2001).

isbn 90-5328-341-2297 2004 On solidarity and

professionalisation Evaluation of Municipal International Co-operation (1997-2001). isbn 90-5328-342-0 298 2004 Onderzoek naar de kwaliteit van in

2002 afgeronde decentrale evaluaties

Eindrapport. isbn 90-5328-344-7299 2005 Een uitgebreid Europabeleid Evaluatie van het Nederlands beleid

inzake de toetreding van Midden-Europese landen tot de Europese Unie 1997-2003

isbn 90-5328-347-1300 2005 Aid for Trade? An Evaluation of Trade-Related

Technical Assistance isbn 90-5328-349-8301 2006 Van Projecthulp naar Sectorsteun Evaluatie van de sectorale

benadering 1998-2005 isbn 90-5328-351-x301 2006 From Project Aid towards Sector

Support An evaluation of the sector-wide

approach in Dutch bilateral aid 1998–2005.

isbn 90-5146-000-7302 2006 Evaluatie van het Nederlandse

mensenrechtenbeleid in de externe betrekkingen

isbn 90-5328-352-8

303 2006 Dutch Humanitarian Assistance An Evaluation isbn 90-5328-352-8304 2007 Evaluatie van de vernieuwing van

het Nederlandse onderzoeksbeleid 1992-2005

isbn 978-90-5328-353-0304 2007 Evaluation of the Netherlands'

Research Policy 1992-2005 (Summary)

isbn 978-90-5328-353-0305 2007 Impact Evaluation: Water Supply

and Sanitation Programmes Shinyanga Region, Tanzania 1990-2006

isbn 978-90-5328-354-7306 2007 Chatting and Playing Chess with

Policymakers Influencing policy via the Dutch

Co-Financing Programme isbn 978-90-5328-355-4307 2008 Beleidsdoorlichting seksuele en

reproductieve gezondheid en rechten en hiv/aids 2004-2006

isbn 978-90-5328-358-5 308 2008 Het Nederlandse Afrikabeleid

1998-2006 Evaluatie van de bilaterale

samenwerking isbn 978-90-5328-359-2308 2008 Het Nederlandse Afrikabeleid

1998-2006 Evaluatie van de bilaterale

samenwerking (Samenvatting) isbn 978-90-5328-359-2308 2008 The Netherlands’ Africa Policy

1998-2006 Evaluation of the bilateral cooperation (Summary) ISBN 978-90-5328-359-2

Evaluation studies published by the Policy and Operations Evaluation Department (IOB) 2004-2009 Evaluation studies published before 2004 can be found on the IOB website at www.minbuza.nl/iob

Page 50: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

48

Evaluation studies published by the Policy and Operations Evaluation Department (iob) 2003-2008

309 2008 Het Vakbondsmede-financieringsprogramma

Een evaluatie van steun gericht op versterking van vakbonden en vakbonds- en arbeidsrechten

isbn 978-90-5328-357-8309 2008 The Netherlands Trade Union

Co-Financing Programme An evaluation of support for trade

unions and trade union and labour rights (Summary)

isbn 978-90-5328-357-8309 2008 El Programa de Cofinanciamiento

Sindical Una evaluación del apoyo orientado

al fortalecimiento de sindicatos y de derechos sindicales y laborales (Resumen)

isbn 978-90-5328-357-8310 2008 Clean and sustainable? An evaluation of the contribution of

the Clean Development Mechanism to sustainable development in host countries

isbn 978-90-5328-356-1311 2008 Impact Evaluation: Primary

Education in Uganda isbn 978-90-5328-361-5312 2008 Impact Evaluation: Primary

Education in Zambia isbn 978-90-5328-360-8313 2008 Xplore-programma isbn 978-90-5328-362-2314 2008 Primus inter pares Een evaluatie van het Nederlands

EU-voorzitterschap 2004 isbn 978-90-5328-364-6315 2008 Impact Evaluation: Support to Rural

Water Supply and Sanitation in Dhamar and Hodeidah Governora-tes, Republic of Yemen

isbn 978-90-5328-364-6316 2008 Be our guests Beleidsdoorlichting Nederland als

gastland van internationale organisaties en Samenvatting

isbn 978-90-5328-370-7316 2008 Be our guests Policy review on hosting international

organisations in the Netherlands and Summary

isbn 978-90-5328-371-4

316 2008 Be our guests Examen de la politique meneé par les

Pays-Bas en tant qu'État hôte des organisations internationales (Sommaire)

isbn 978-90-5328-372-1317 2008 Sectorsteun in milieu en water isbn 978-90-5328-369-1318 2008 Samenwerking met Clingendael Evaluatie van de subsidieovereen-

komst tussen de ministeries van Buitenlandse Zaken en Defensie en

Instituut Clingendael isbn 978-90-5328-365-7319 2008 Meer dan een dak Evaluatie van het Nederlands beleid

voor stedelijke armoedebestrijding isbn 978-90-5328-365-3320 2008 Het tropisch regenwoud in het

OS-beleid 1999-2005 isbn 978-90-5328-374-8321 2009 Maatgesneden Monotoring 'Het

verhaal achter de cijfers' Beperkte beleidsdoorlichting

Medefinancieringsstelsel 2007-2010 isbn 978-90-5328-374-5322 2009 Draagvlakonderzoek Evalueerbaarheid en resultaten isbn 978-90-5328-375-2323 2009 Preparing the ground for a safer

world Evaluation of the Dutch efforts to control landmines and explosive remnants of war 1996-2006

isbn 978-90-5328-377-6

Page 51: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

Evaluation policy and guidelines for evaluations

49

Page 52: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

50

Page 53: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

51

Evaluatiebeleid en richtlijnen voor evaluaties

Page 54: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

52 Publishedby:

MinistryofForeignAffairsPOBox20061|2500EBTheHague|TheNetherlandswww.minbuza.nl

Layout:Eindeloos,TheHaguePrinter:OBT,TheHague

©MinistryofForeignAffairs|October2009|Ordercode:BZDR6662/E

Page 55: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

ratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe

nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa

aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum

houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMia

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono

dIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MpalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisVelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgH

openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauW

restBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/K

etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntweenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKa

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBa

asChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKiazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSan

alle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

furt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaMan

ekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono

adHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhou

openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusW

BernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWtislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIst

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthe

iroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeR

okioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroenouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

aDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

enenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaP

obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo

elsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiaortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonodIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTjnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavanna

aoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisV

elgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWrestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStad

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwe

enosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsP

eheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBaasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKi

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSanalle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

alaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

ashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitK

obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCa

HanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideo

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono

JubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHel

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonW

tislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheiroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe

nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa

aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum

houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo

Page 56: Evaluation policy and guidelines for evaluations · 5 Evaluation policy and guidelines for evaluations This document describes the evaluation policy and guidelines for evaluations

IOB

Evaluation policy and guidelines for evaluations

Published by:

Ministry of Foreign Affairs PO Box 20061 | 2500 EB The Hague | The Netherlandswww.minbuza.nl© Ministry of Foreign Affairs| October 2009 | Order code: BZDR6662/E

olicy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department

This document describes the evaluation policy and guidelines for evaluations of the Policy and Operations Evaluation Department (IOB) of the Dutch Ministry of Foreign Affairs. In publishing these guidelines, IOB wishes to provide transparency to third parties: colleagues at the Ministry of Foreign Affairs, the Dutch parliament, organisations and institutions affected by IOB evaluations, users of the evaluation results, and interested citizens. IOB will periodically review its policy and the guidelines and, if necessary, revise them.