WBI Evaluation Briefs I REPORTING ON CLIENT AND STAFF LEARNING AND CAPACITY DEVELOPMENT PROGRAMS - rn rn

Nidhi Kttattri, Acting Manager Institute Evaluation Group March 2005

Trust Fund Learning and Accreditation Program: Short-Term Impact

The Challenge A training program designed to impart the appropriate

Trust Funds in the World Bank are financial arrangements whereby knowledge and skills to Bmk staiT would result io the proper and bilateral and other external donors provide grants in support of the eficient management and use of trust funds.

Bmk's poverq reduction mission. These grmts are used by Limiting TF managemen1 to accredited staff would provide an borrowers for project preparation, techcal assistace, advisory incenljve for participlion in training. services, debt relief, post-co&d transition, and the cofinancing of . ~~w~~ should ~~k ms use irust fund5, bur should invcshcnt projects. The Bank's trust fund (TF) podolio ha$ dso be open to other Bank s& who help implmmt trust fund$, expanded over the years; contributions received from donors in IT04 totaled $4.9 billion, which was a 10 percent increase over those received in Fi03.I Bank staff play an importani role in mboaging the appropriate use of trust funds. In WOI, a comprehensive Trust Fund Process Review identified the need to enhance Bank sM howledge of the management and applications of trust funds and of related World Bank policies and procedures.

World Bank Intervention The Trust Fwd Learning and Accreditation Program (W) was developed in response to the findings of the Trusc Fund Process Review. TLAP comprises an e-laming curriculum that indud6 17 1e;rrning modules, a d-mwsment test, and a 48-item on-line accreditation exam based on the learning modules. The progt-dm was piioted ia October 2002 and rolled out Bank-wide in December 2002 for alI staff involved in the management and execution of trust funds. A formative evaluation of the pdot recommended improvements that were implemented prior to h e Bank-wide ro l lo~ t .~ Thc Bank's TF management required that beginning in FY04 only accredited Task Team Leaders ('MIS) would be allowed to manage and execute trusl funds. By July I, 2003, 1868 Bank staff had been accredited.

Underlying Assumptions TLAP was designed and implemented based on the following assumptions:

Noncomplidnce with World Bank policies and pmedures reIafed to TF management occurred because staff lacked adequate howledge or understandmg of these policia and procedures.

Box t : What Respondem Said abwt the Trust Fund baming and Acereditation Program

Benefits FmmlTls: "It MS useful in rmhrng our knowkdge. Most ofus were fwni/ior wirh m e TFs,PHRD, CTF; etc. The program mode clearer the row of optbns ovorbble onrn..." ~progrmmos~usw~~~ond~tonmi~.Theopenbook ~ ~ h w i t h ~ a o b i b t y t o s t u d y w f i e n ) r m f i k m e ~ ~ e # e n t T h e S e y - a s w s s m e m ~ & l t o ~ r ~ p m g r e s o f s t r r d y o n d w e r e g o o d fhqlwmon for the re01 &st"

Fmm TFCslTFMs: "It come m tk rrght trme.AWng was gdng on at the some time. b mmed onowmwith~.TheygoetoWthot~isnottheBonkbudget"

'The filled o void with resped to mfbnnofbn on p m d r e s , poho~~, and o c c o u m o b r ~ ft helped tremendously and brought o w o ~ of f iduwry responsibilibk.. .ln terms of i m m we hove seer^

o big change.. ." 'Very VwyfuI to those who ore just sorting. They h e dewloped o sense Ofocwuotobi~."

Recommendations FmmlTls: 'The amdtmhn exom should Be foarsed on ?he stoge of leornins the m n c e ~ l framework for the Bonk's use of Trust F u d ond which oust funds ore used for what yps of ossisrance.The TF Una shou(d then ~ r e o s e r o n d p h e , ~ n g f b r o U T F ~ s & f w mke minmg on the p-ng ofTFs hrn start to $nlsh with ckar gd&m fbr e w y step ofthe dwld not be guesswork fir TTLr thot are new to the Bonkl bureouctuq''

% need o pod monual m refer w.There shou(d be more @rt put into o referem dorurnent"

1 . Thse figures reflect conkributions received on a cash basis for all IIWL runds excepl Heavily !n&led Pmr Counvies (HIIT). h e Global Environmnk Facility (GEF), and the Global Fund to Fight AAID Tuberculosis, and Malaria (GFATM). for which contributions are recorded on an accrual basis {World Bank Annuul Repon 2W).

2. Khallri, N.. %. Shi, M. Palrnisano. and M. Fein. "liusr Fund Lc<irning o d Accwditutron Pwgrttrn PiIot Fomwfive Evuluutiun." WBI Evaluation Studiw. No. EG03-065, D e c e h r 2002.

40578P

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

ed

W O R L D B A N K I N S T I T U T E

Figurn I. Task T m Leader and Trust fund CoodiniitorManager Ratings of W ' s Usefulness in Building Skilb and Improving Compliance

I Knowledge and Skills IbrTF Manqaememt

l T L Rating: ~ o w ~ ~ h m s t h e ~ ~ t n ~ % o p l n g ~ o w i undemmdhg ofTF managemmtl

I NOT a t a11 userul ~~" T F W M Ratings P l e a ~ ~ Indlcate whether h p m of Tb,. has

b e e n b & e r o r ~ a d n m a ~ a g a ~ d t o knowledge & skllls in TF rnmgmmt

I Compliance with TF Policies and Pmcedums

- - - - * --- -

"Much worse" "Much betEer" TowhatexrenrhastheP helped the ilk i m p m wrlcmwk&e ndlm ma%'-- fnmagmmt

~ the n u l a u s polldar and procad- manage?

"Nor at all"

T F m M -gs: Plem indicate wherher the pet%nnance dl-b., has k e n ' betwr or m e since a year ago for ~0mplk tn~~ with @Ides and procedures "Hi bettw"

To what extent has the helped the TTLs improve

.+t&Pcir" . .

including Ts;sk Team k k n t s dTE+rm&ors (no).

Evaluation Methods and Instruments The learntng Hoard requested that the World Eank Institute Wuation Group (EG) madud an duahlotl of llAF ia R04 to idatiEj the short-term impact of the program.' This stndy f o c d on the extent to which the program has supported the Bank's fid- responsibilrty &ugh:

Providing infocmadon tbat is usefuI for developing Tns' a d other staff's ~dmtmding of TP lmagement and maltion. providing hfomatiofoa that helps TTh and other st& comply with the polides and procedures of the tfilst fan& they mamge. Impmvlng ITh' and other sta£f's spec& ImowIedge and d d h in managing aust funds*

~~ TIZ qaestionnaire data from n repem* w e of acc& TTh and a sample of accredited Ti& (b) d- ad&&red T'FC questionnaire data horn

Vice Presldmdd Unit CVPU) TFC and a ~ e c t d s a m p k o f T m t P u n d ~ 1 5 ~ {rn) of progmmatic tmt funds to obtain hdepmdent wesamts on the effects of the program on TTiq (c) w m and focus groups with ms, TFCs, and ?IPMs; (d) hkrvim with W k staff Imowldg&k about SAP rn fund da@ including the TF mounting st& in C h q aad (e) revim of reports issued by the Trust Pund Qualib and C o ~ c e {TQC) unit. Data were collected from February to hy, 2004.

Participants and Response Rates The popnlation for this evaEuation were 4 W TTIs wtro were W - h d d t e d a of Jme 3,2003, a d who were $ade GF o r h i g h e c . ~ o f t h e l & a m & d ~ staE,asti.atl&edsmple d704lThwere chosen for the ITL questimmh. Responses were rewived from 385, for a

response rate of 54 percent. In addition, & lTAs dso responded to he TTL q u e s t i o m v t h g 50 percent of the d t e d m,

TheTFC questioanaIrewas dmhisteted tb TFCs d the 14 WUs md TPMs or TP mdinabrs of 24 progammatic trust hds, The combined TEVDM response rate ms 54 permpercent. The data from TKs, TPMs, and other st& working on TF- retated issues were collected @ally to obtain a more obj- asemat d the influence of TLAPfn mhmldpand perfommce and to minimim pdble & w i n self-reported daEa.

Evaluation Results 1. Most TIZs hund the program useful.

Seventy-one pmt of the lYL ~ r a t e d T L A P a s W I F f n developing heir understanding of trust fund mmgment T i t p e n t of rapondents the as wefal in prmiding them with he howkdge and skih for

W O R L D B A N K I N S T I T U T E

nlanaghg their current trust funds. Wmty percent indicatd that the p r o g * r m w a s & i n h ~ ~ e m t o complywith the polides and procedures ofthetmstfunds t b q r w e . 'The program was deml to Tms' managanent of their current Tk. On average, I T h found thai 59 percent of the p r o p bad such relmce. W l e v e l ~ d e d o n ~ m o f udnm. Hmvmq such d i f h m in ~ u s e 6 a l n e s s w e r e n o t ~ c r m t when mbolhg for a few p m d charadeastta, jncludhg l d o n (h- or c o w *I, gab a n d t h e n u m b t l r o f ~ ~

2. TPCs a d TFMs indimed that, fmm IT03 to lT04, T l b bad improved their howledge and hikiIls as well as their comphce with TF policies and procedures. These improvements were attributed In great part to TLAP (see Figure 1). The mean rating among TPCs and TPAls far the improvement of Tl l howledge md skills was 5.21 on a sale horn 1 "much worse" to 7 "much her ." The average rating for the program's help with this improvement was 5.50, on a s a l e from I "not at all" to 7 "a lot." The TFWFMs avemge rating r e p i n g change in 'ITL compliance was 5.10, and tbe rating for tbe program's help with improving compliance was 5.20.

3. O v d , respondents Indicated that TLAP was M v e . The mean rating--averaged across 18 arm of trust fund management-was 4.65 on a scaIe of 1 "not at a11 effective" to 7 "dremely effe~tive."~ Sixty percent rated the program as effective ( f i g s of 5, 6, and 71, and thirty-four p m n t rated it as being extremely e£fective (mtings of 6 or 7) (see Egure 2)

4. W n p the p p m ' s ~ ~ I n ~ ~ ~ o f t n M t frmd mmgment varied WWP;

The two areas rated as most effective were 'The dilkence betweedl went work and Bank work" (average lTL rating 5,22mdwersl$eTWRM rating 505) and "the types d acthities that can be funded under mst funds" (average TIZ rating 5.18 and amrage TwrFM d n g 5.30).

Tbe two areas rated lowest in effectlyenas were "How to use SAP for monitoring trust fund financial activity" (werage ITL rating 3.64 and average TWRM rating 3.70) and 'The critical informdon ad integration points that must be created in SBP for trust fond mqemenc ' (average TI'L rating 3.81 and avenge IFWFM rating 3-89].' f i e s e ~ ~ h a v e b e e n l o w a s TW was not dsigned to teach the

Figure 2. TTL and TFCrrfM Wngs of P q p m E f k d k m e s s in 18-

Diffmnces bemen mdphnt work and ,,wok I

T - of that can be funded under c d t a n t TF

The a l l d e uses of TFs I 5.1 I Whether aTF should be Lnk-execured or

ne iprem-md

When and how to h aTF

M&ods of set- c o n m b m for amWa to be financed thmugh TFs r

Procedures and requrrnmems for 4.t acr[vadng aTF

M& d sel consdthg Arms Tor ~ ~ t ~ m c d t h ~ ~ h 1-

Arrsngemena that need to be in place for TF dkbuments m

Requiments for d i s h i n g an InmaingBhefbrrlT 1-

How to align a cmsukntb TOR with I@ requlremems and Bank @Ides

Requimentr for dewdqirrg an lnlttad ~or tcep t~o tebra? 1-

Formal documemtIon r e q u l m t s kr managing TFs I

Wan and how ro amandTF zpenwmts m The Bank's TF disclosure policy

Cridcal Idormadon and Inoegraclon polm thatinu6tbernSAPforTFma~gwnent

How to use SAP lor mdtorlngTF c- l Tn.Ratlngs dhrandaI mivlty TFCrrFM Ratings

6. RmBpwsdentsl wid alw K, "mt appppH& (MIA)" If they a M y had auffidcnt ~ l e ~ l l u in the arta prior to pmgram paaidpadon w "not perdmnr" i f fhe area was not related ta their woct

7. Fw *Hw 10 tw SAP Fortwnlt&ng m W flnancial acrivity: the tstlngs were unlforrnly below the midpoint of tht &. The SAP-KW areas m y have Wvod low mdngs due la he wording d h m y questim. The TFCdlTkh sad ITh may have imarpmed them items ta refer to fhe acquidtim of skills in vaiag SAP, which the pgmm was tm &signad m d.

W O R L D B A N K I N S T t T U T E

skUrsneededforustngWbutinsteadtohdp~understand the SAP monltorlng functlon in TP management, The rehive eEfedimess mkqg of the 18 areas were sindm across VPUs. Elowever, the ratings differmi, with T k in MlddIe Bast rand North Africa Region (MNA) providing mnsistentIy lower ratings of ~ ~ e s s than llh from Afrim Regiorl (AFR), South Asia won (SAR) , and the Networks on 14 m.8 TITS I d in Md p d higher mhg on the d p s s . h c e , and dechm of the program than those b a d in Washtngon D.C? These higher rating rmly reflect a lack of ready references on TF management for c o m t y M sM. lTAs provided si@mtIy higher ratings for p m p m usefulnm and &&mm compared to lTb altho@ the ~ O f T r A ~ w a s ~ t o t h a t o f m .

5 . ~ ~ ~ m ~ ~ t 0 m o a p ~ i ~ t k ~ a f p ~ o g ~ s m ~ e s s . h f a d , t h e t o p a a d b o # o m threeratedareasofdkhm~ are thesameforh~rwpondent pups (saeEigm2). fiesehdhghdicate that overallindependent m e n b Q£ TlAP &mess provided by TFCs/rPMs converge with those gfven by the program e t s t h w d w , thereby mw c o ~ c e in the evlthtatm d t s , '

6. Focm p u p support tbe qumtitstive smey ~ . M c i p a n t s a g m d h t t h e p ~ h a d b m u s e f u t f m enErancing or their knowledge of trust fund management, but they a h noted that m e parts of the progrrrm seemed unreaIistIc and that some policies and Informahton sourm needed ~ c a a t t o n (see Box 1).

ImpIications and Conclusions Oven& the &ding hdim that ?ZAP has been successful in Moping a general ~ d m h n d h p md howledge of trust Enads for a huge and diverse W-wide ITL population. In ~ regard, using an an-he h m h g approach d ready reference materials was particcularfv w d d RecommendAions that mqed from this duation include the followfns

T h e r a u l t s f m T T L d ~ m m q s m g g a t t M t h e r e may be a need to focus on SAP-dated areas, i~ they my create a b o t h & in trast funds management. Efforts should be made (a) to assesswhether TKhowIedgedsktIIsh~SAPare wentiid for meeting the Bank's fihdary responsihilif and, if so, (b) to provide the appropriate training,

Refemce materials on TP -ent should be d y adable to those who have comp1eted the program, inclaw country-based staff. Such materials would fadlitate the updating of needed T1Z knowledge and skills as policies and p h C h . 6 n g e .

A more modular approach to TF-related learoIng should be developed, with the program smhg as aa orlentatio~ cam and other specific follow-up modules being offered to address issues that may be of particular rdemce to different types of trust funds.

Acknowledgement The evaluation summrhd in this brief was conduded with the h c i a l support of the h m q Board of the World hk