A

Twaweza East Africa

Annual Report 2014

Contents

Contents....................................................................................................................................2

Introduction...............................................................................................................................3

1. Programs................................................................................................................................5

1.1 Programs Uganda.................................................................................................................5

1.2 Programs Tanzania...............................................................................................................7

1.3 Programs Kenya..................................................................................................................10

1.4 Experimental Interventions (KiuFunza)..............................................................................14

1.5 Sauti za Wananchi..............................................................................................................18

1.6 Uwezo.................................................................................................................................23

1.7 Strategic Engagement.........................................................................................................31

2. Learning, Monitoring and Evaluation...................................................................................35

3. Communications...................................................................................................................41

4. Operations............................................................................................................................46

5. Governance and Management.............................................................................................49

3

Dear friends,

What a year it has been! 2014 was the sixth year of our initiative, and the end of our first strategic

period. We have spent the year consolidating our activities, refining our partnerships, reflecting on the

very essence of what we are trying to achieve, learning some deep lessons about what has been working

and where we have been failing, and adapting to the changing context and new realities. We also

received a largely positive evaluation of our Tanzania program, spanning the first strategic period; the

exercise was spearheaded by one of our largest supporters, the Swedish International Development

Cooperation Agency (SIDA).

By the end of the year, we had completed an intense process of realignment and transition in the

following three dimensions. First we articulated a clearer Theory of Change to guide our new strategy

for 2015 to 2018. Second, we completed the merger between Uwezo and Twaweza into a single

organization and incorporated Twaweza East Africa, an independent company limited by guarantee.

Finally, we organized our management style and structure, and after a global search, found a new chief

executive to institutionalize Twaweza beyond its founder, Rakesh Rajani and lead the organization in the

new strategic period.

It was also a year of tremendous achievements. As our reflection and realignment was taking place, we

maintained the intensity and momentum of our activities. Even more significant, the impact of our

ideas, activities and lessons began to show in a number of very concrete ways. Several of these deserve

special mention.

First, it is a rare social experiment to test a government policy the results in a full adoption of its

processes and results within two short years of its inception. This is exactly what we achieved with our

KiuFunza initiative, Africa’s largest randomized evaluation in education. We proved that capitation

grants could be successfully disbursed directly to schools. The government of Tanzania announced on

three separate occasions, that it intends to start transferring capitation grant funds directly to all

schools.

Second, our Sauti za Wananchi platform, Africa’s first nationally representative mobile phone survey,

was used by Tanzania’s Ministry of Education and Vocational Training to poll parents about school

subject combinations. This official recognition of our methodological rigour, transparency and

independence was reinforced by the huge media coverage of our political poll in October. We co-

authored a manual on mobile phone surveys with the World Bank that will be published in mid-2015.

Third, Uwezo, our large, citizen-led learning assessment continued to be sought after as a credible input

into national education debates in Kenya and Uganda. A major achievement was our invitation by

Uganda’s Ministry of Education, Science, Technology and Sports (MESTS) to be a key panellist in the

2014 Education and Sports Sector Review (ESSR). That was a clear demonstration of government

recognizing Uwezo as a key actor in improving and sustaining learning achievements in literacy and

numeracy. Our Uwezo team in Kenya helped finalize the National Education Support Programme (NESP)

that will lead the sector until 2018.

Fourth, at the global level, we were instrumental in developing a strong “Response Policy” for the Open

Government Partnership (OGP) to protect against intrusions on civic space. We also drafted a set of

4

principles for the data revolution that were included in the report of the UN Secretary General’s Expert

Advisory Group on the Data Revolution.

Fifth, we sharpened our monitoring, evaluation and learning activities. Monitoring was implemented

around every major activity in Uganda and for the majority in Tanzania. Twelve new Monitoring Briefs

were published on our website and they have proved useful in communicating internally and with

partners on how Twaweza gathers information to learn and improve. In addition, at least three external

evaluations of our activities provided important findings that influenced the strategy for 2015-2018.

I am confident that this Annual Report will give you a good picture of our year in 2014. Since this the last

Annual Report in our first strategic period, we begin with a reflection of the main lessons learned over

the last five years. Then, each unit provides a narrative description of what was done, followed by a

deep dive into one particularly interesting story, and a summary of what was achieved against the

annual plan for 2014. Dashboards by unit are provided at the end of the report. Greater details are

available in matrix form upon request.

***

A key lesson from our work in 2013 was that we needed to move from being neutral brokers to

becoming involved actors who are “generating evidence, ideas and stories, and engaging with specific

agenda in mind.” We made this move in 2014. And everything that we accomplished is a result of the

energy, creativity, care and passion that the entire Twaweza team brought to the work every day.

Ultimately, this is a testament to the inspiring leadership of Twaweza’s founding head, Rakesh Rajani. He

enthused everyone with the sense of wonder, excitement and thirst for justice that is an integral part of

the trailblazing Twaweza adventure.

As I write this, it has been six weeks since I formally started at Twaweza. For me, the team’s sense of

mission, the spirit of thoughtful exploration and the commitment to doing excellent work, are already

palpably powerful. I feel privileged to be part of the Twaweza family, and to lead the team into an

exciting future.

Aidan Eyakuze, Executive Director

5

Reflections from the first strategic period (2009-2014)

In 2015, as we begin our second strategic period (2015-2018), we find ourselves with many lessons from the first five years – based both on our successes and our failures – and so with renewed and more focused energy to make a real difference in the lives of East African citizens in the coming five years. Below are some of the highlights of our learning journey from the first phase.

Learning from what we got right Citizen Monitoring: Uwezo Since 2010 we have engaged tens of thousands of volunteers selected from local communities to test hundreds of thousands of children in their homes across Kenya, Tanzania and Uganda. District-level partners help coordinate this effort and engage in local communication activities and debates. Uwezo has become a well-known brand for independent learning assessment in East Africa. In the course of the 5 years, evaluations suggest that Uwezo has changed the discourse on primary education, from attention to the relatively successful provision of education inputs (enrolment, buildings, materials, etc.) to the general failure in achieving learning outcomes. In all three countries there is now a general sense of urgency to ensure that children are not only in class but that they also learn. We are promoting the idea that action taken to improve the education sector should be measured on its success by improved learning outcomes. Citizen Monitoring: Sauti za Wananchi Sauti za Wananchi (Voices of Citizens), in mainland Tanzania, is Africa’s first nationally representative mobile phone survey, with a sample of 2,000 respondents. It combines statistical rigor with the flexibility and frequency of a quick poll. The platform lets Tanzanians talk back to their government on issues that concern them. Since 2013, Sauti za Wananchi has produced 17 briefs, covering a variety of topics. The briefs are publicly launched and receive wide media coverage, often sparking debates that resonate long after the briefs were first published. Like Uwezo, Sauti za Wananchi is well respected in Tanzania because of its rigor, which contributes to a key Twaweza value: using facts and evidence to guide policy and implementation. The Sauti za Wananchi concept is already being adopted by others. In 2015, Twaweza will, with the World Bank, co-publish a handbook on high quality mobile surveys. Engagement with the media Twaweza set out to invest in existing networks that are central to our Theory of Change (TOC) – information flow, citizen agency and accountable public service delivery – and that already successfully reach citizens directly. Of the networks that we had in mind from the start, media proved to be the most diverse, imaginative and effective. We have worked with a large variety of different initiatives, from six large media houses in multi-year contracts, to small, creative and targeted popular programs, to radio news agencies serving local radio stations, satirical shows, investigative journalism and citizen journalists. Many partnerships, because of their unconventional nature, gave us a fair share of issues to think through and deal with, but the sum total has been a formidable reach with generally high quality. Connecting local with global The government, and especially local governments, have a persistent culture of secrecy and keeping information confidential, which makes it almost impossible for citizens to engage with the leaders who serve them. In 2011 Twaweza was at the heart of co-founding the Open Government Partnership (OGP).

6

This unique global partnership has designed and adopted a set of concepts and guidelines, including drafting of specific country plans and independent review mechanisms. Many critics predicted the OGP would lose momentum and fizzle out, like many global initiatives. But OGP continues to draw new member states, up to 65 in total at the beginning of 2015. For two years Rakesh Rajani, then Head of Twaweza, co-chaired the OGP as the Civil Society chair, and helped to shape and provide direction to the Global initiative. Twaweza continues to be member of the national Steering Committee in Tanzania. Concrete results in Tanzania are the tabling of the Access to Information Bill in parliament in early 2015, the ambitious second national OGP Action Plan and the gradual shift of government towards more openness. Taking learning seriously From the beginning, learning has been at the core of what we do. Despite that appreciation, it took us a few years to establish a system and posture of learning that works for us. It has evolved into a framework in which monitoring, learning and evaluation come together and mix with internal staff development and engagement with local and international experts. To mention a few: we use different ways to test communication products before setting them loose; an elaborate monitoring framework guides our measuring of reach, quality and effects of our partnerships; we hold many and regular learning sessions, guest lectures and discussions (formal and informal), organize the occasional hands-on training; hold regular reading clubs in our library; we recruit colleagues who are curious by nature and eager to learn; we do research; we hold learning conferences and retreats to share findings with our regional and international partners; we contribute to international conferences and publications.

Learning from what we got wrong Straight to scale, no iteration In our enthusiasm and rush to make change happen, we have often jumped on opportunities and brought them to scale without testing and adjusting. At times this has led to failure, (which we have described and presented at various moments) or, more often, did not bring the maximum results. Though we do intend to reach scale with everything we take up, over the 5 years we have learnt to slow down, start at modest scale and take time to iterate: think hard, engage the intended audience or users, try, test, adapt, test something different learn, and test again, and so on. Though we are not 100% there yet, we have become much better at it which pays off in effect and cost. No differentiation of citizens We assumed that by partnering with networks that already directly engage with millions of citizens, the mere scale of our effort will foster an ecosystem of information and ideas that will eventually reach the people who can drive change. There would be “outliers” who use the information to be the prime-movers for change in communities all over the region, and the general population will be informed and speak out, debate in public and take action, and on the way build a bouquet of options and ideas to improve their lives. It was a heroic assumption. The scale, coordination and investment it would require to create such an orchestrated ecosystem is simply unattainable. We did achieved scale, possibly even an unprecedented scale in Tanzania, but it proved impossible to get all the coordinates right to achieve a viral effect. We have learnt that a smarter, almost surgical approach to select strategic segments of communities and carefully determine how they could drive change and how we best reach them has a much higher chance of success. This does not mean that reaching millions is not important; it certainly is

7

– as even quiet endorsement of the masses enables sustainable change by the avant-garde elements to take root. We assumed that information alone would work In the first years we designed different creative methods and products to reach people, comparative information and stories of change which we assumed would help people imagine the world could be different and would help them figure out how to solve some of their problems. We knew there are constraints that hold people back, such as deep inequities, fear of reprisal and lack of resources, but we thought people would get organized and work together to overcome those. We found that this was not the case, at least not as a general rule. And though we are convinced that information is absolutely crucial in moving towards a more prosperous and democratic society, it generally takes more than that: people must have the ability to act (whether this is knowledge, or skills, or self-efficacy), they must be motivated to act (by believing their action has a chance of bringing change, for instance, or by not fearing risk of reprisal), and there must be opportunities for people to act (whether through a responsive and interested media, or through an institutional feedback mechanism, etc.). Brokering at policy level as a ‘by the way’ After decades of policy dialogues, advocacy, annual budget meetings, poverty reduction strategies, multi annual integrated programs, life for the ordinary person in an average village has changed but very little. Too little. As Twaweza we firmly turned away from the “policy circus” that had lost its traction. We felt that we all had spent too much time repeating discussions and making the same resolutions again and again that in the end only seemed to help feed ourselves, and not the majority of the population. Instead, we focused all our effort on reaching citizens directly. Therefore, it did came as a surprise that some of our notable successes were achieved precisely in the policy environment and less where we had expected it: change driven by citizens. We acknowledge that by ignoring the policy realm we missed potential opportunities to become more strategic and to strike a balance between citizen and policy engagement in a synergetic way. Crucially, we still maintain that policy debate is insufficient unless it results in actual implementation, and it is this cross-section of the realm of governments and citizens that we are most interested in.

We have achieved much in the first strategic period, and we have learned a lot. We have also discovered that driving our efforts to scale and achieving major impact will take time; that context matters, that an inertia born of cynicism runs deep and that achieving lasting positive change is a complex task in which authorities play a central role. Building on these lessons, and many others, we are well equipped and ready for the new phase.

8

2014 Annual Report

Programs Tanzania: Narrative & Story The process of scaling down the number and breadth of partnerships began in 2013 and was actualized in 2014. The focus for the year was on successful partnerships with a strong focus on learning. During the year we closed our two major media framework agreements with Mlimani Media and Sahara Media Group. The former, due to struggles with reporting, and the latter due to discussions not taking off as planned. Most importantly, however, the Programs Unit worked with LME and Communications to evaluate the quality of content produced within these partnerships. The findings for Mlimani Media were largely negative; in many cases Mlimani performed worse than the control group on some key indicators of media quality such as number and transparency of sources. For Sahara Media the results are more positive although there is still room for improvement. Sahara Media Group performed the strongest on including ordinary citizens’ voices in their programming and in putting figures into context. Minibuzz continued to be a strong partner for Twaweza and monitoring data showed that 26% of Tanzanians have watched the show, 85% of these get useful information from the show, and 63% discussing topics on the show with others. In the second half of 2014 we adopted a more content-driven partnership approach; Minibuzz produced and broadcast 25 shows on Twaweza topics of interest. This meant that, for example, all Sauti za Wananchi topics were covered on the show. Monitoring of this new style of partnership was largely confined to internal quality reviews of shows with Twaweza topics. The team found that, although the program is innovative and engaging, there is room for improvement in how topics are moderated, gender balance in the discussions and the presentation of data and statistics. Minibuzz also attempted to implement feedback loops by running mini focus group discussions on the bus but these proved to be problematic in urban areas, leaving passengers confused and discontent with answering the types of questions being asked. In the absence of polling mechanisms similar to those in Uganda, the team will have to work hard to try to get some immediate and quick feedback from viewers of the show. We continued to work with Femina but again adopted a new approach. Twaweza was the main collaborator on their annual issue on citizen engagement, contributing to discussions about the theme of the magazine, reviewing all articles and crafting suggested action points. We also reviewed the citizen agency elements of other issues throughout the year. Given the selective distribution of Femina, it does not lend itself to assessing reach through nationally representative surveys. However, Femina themselves maintain regular contact with their readers through digital and face-to-face communication and receive feedback which feeds into their content development process. Part of the agreement also included the mapping of Fema Clubs, essentially groups that are formed around the magazine. The results for this have just been submitted but will provide insight into whether these are potential spaces to mobilise collective action. We also closed off a partnership with Gaba Africa to produce and distribute 250,000 copies of each issue of Kingo magazine for a total of 1.5 million copies over the course of the partnership. Twaweza provides the theme for the topic and key facts and messages around the issue. The team at Gaba Africa then incorporate this information into stories and comics in their unique satirical style. More details can be found in the story below.

9

We moved forward with a partnership that was many years in the making. One of Twaweza’s five networks or channels for information was religious networks and organisations. But we have had difficulties getting traction in this space. A breakthrough in 2013 meant that a contract was signed with the Christian Social Services Commission, a natural partner for Twaweza as they are the coordinating body for service delivery for the majority of Christian networks. Even more excitingly, the partnership offered the opportunity to understand more about factors influencing collective action as we were supposed to distribute materials to existing bible prayer groups. The hypothesis was that the Christian value orientation and existing cohesion of these groups would make them fertile ground for collective action on social issues. However it took some times to really join hands and understand each other’s ways of working such that the materials were only ready for pre-testing at the end of 2014. Moving forward we will continue to work on this partnership and see if our assumptions bare out. Twaweza also supported Well Told Story (WTS) to expand into Tanzania and the star of their Shujaaz FM character, DJ Boyie, is well on his way to becoming an East African star. In 2015 we will work with WTS on embedding a governance storyline into the comic in Tanzania and understanding its effectiveness through a randomized control trial and qualitative research in partnership with MIT and Coca Cola (who can offer a controlled distribution opportunity). Finally we worked with the Legal and Human Rights Centre (LHRC) to produce and distribute three booklets about the constitution to tie in with the ongoing review process. To our knowledge these are some of the only public-facing and accessible sets of materials about the constitution and its relationship to people’s everyday lives. A total of 750,000 copies of three different booklets were distributed across the country through LHRC’s network of paralegals and human rights monitors. We followed up on distribution and quality with this network and found that 85% of respondents had received at least one of the three booklets and the majority of them found the information in the booklets useful. Overall implementation was lower than anticipated, in part because of the reflective nature of the year and the slow but steady scaling down of partnerships and staff. At the same time, we are proud to have genuinely embedded monitoring into every activity so that we have some indications of the utility and value of the partnership agreements. Moving forward, partnerships as a standalone piece of the organisation have been eliminated, instead we will try to work more practically with different partners, particularly in media, to communicate around very specific issues to contribute to Twaweza’s new success statements.

Reaching a national captive audience Two major challenges to the transmission of information designed to prompt behaviour and norm shifts are the poor reading culture in Tanzania and the difficulties of adequately disseminating printed matter in a cost effective way. Even the most popular tabloid newspapers in the country have a circulation of around 75,000 maximum. For Twaweza this makes our work altogether more difficult. As much as video material can be transformative, the advantages of print material (permanence, the ability to convey more specific information, the ability to share) make it a critical pillar of any work Twaweza does. In 2013, we partnered with Gaba Africa, producers of Kingo magazine. Kingo blends satirical humour and relies on

10

imagery as much as written text. Gaba Africa also approached us with a unique distribution proposition to install magazine racks in upcountry buses. Over the course of the partnership, Twaweza input into six different issues of the magazine with 250,000 copies each time for a total of 1.5 million magazines. Gaba Africa also distributed over a million copies of Twaweza’s own printed cartoon booklets alongside the magazine to leverage the cost of the magazine racks and partnerships with the bus companies for maximum utility. Gaba Africa had agreements with a number of bus companies that meant the magazine was on 70 buses a day, leaving the central bus station with extra copies for the route back. The topics covered included health, water, the constitution and education. As a nice aside related to this, the issue of the magazine about the constitutional review process featured the head of the Constitutional Review Commission, Judge Joseph Warioba, depicting him as being pulled between different political camps but staying true to trying to reflect citizens’ views. Members of staff of a hotel where Judge Warioba was staying placed a copy of the magazine in his hotel room, almost just as a joke, yet he was so impressed that he actually called Gaba Africa to congratulate them, and left a signed copy of the magazine for them! We also conducted two monitoring exercises to understand more about perceived quality and effects of the magazine. Focus group discussions were held with young people in Dar es Salaam and Arusha which threw up rich insights about how people interacted with and understood the magazine. And we also had a questionnaire for bus passengers which was less in depth but found that the majority would take the magazine home and share it, and found it informative and useful. We did find that the two challenges of a low reading culture and distribution issues were addressed. On top of that, the overall response to the magazine was extremely positive; participants valued the magazine’s style and creativity and overridingly took away a message of personal responsibility in making change happen. At the same time, all our learning so far has shown that these type of generalised sentiments of citizen agency do not yield attitudinal or behavioural shifts. So we find ourselves with a popular and meaningful product but one that is unlikely to feature heavily moving forward given our new strategic orientation.

11

Programs Kenya: Narrative & Story In Kenya we continued to work with a small group of strong partners; we briefly describe the core activities below. Tazama is a TV program focused on young people, combining a number of different segments including town hall debates, small news style features on citizens making a difference and a soap box section. Tazama came up with a number of creative segments, very much in line with Twaweza's emphasis on rural coverage, diversity, citizen driven stories of change and investigative stories. Tazama’s popularity did not go unnoticed. The Ford Foundation contacted Tazama and is funding its entire next season, which we take as a success and a valuable contribution to the media landscape. For Twaweza the show will continue to hold opportunities, particularly if we wish to suggest topics for investigation. Although there were delays in implementation, the show quickly picked up. Monitoring data show that 11% of Kenyans have ever watched Tazama which is significant considering it is a new show. The main reasons that people watch the show are to get information on current topics and to hear from Kenyan citizens. Our work with Well Told Story and their stable of media products, including the popular Shujaaz FM comic, continued to grow from strength to strength. In particular, WTS have incorporated an ambitious learning agenda into their plans, in part motivated to do so by Twaweza and the space our funding has given them. They are interested in positive deviance, have developed their own methodology (Ground-Truthing) for audience driven content generation, carried out standard research on reach and uptake of their products and have engaged MIT to help them assess effects. They also, with Twaweza’s support, finalized their three year strategy with a well-articulated theory of change in order to attract core, instead of comic to comic, funding. Thus far they have managed to attract some of these funds but not enough to cover all their costs. And finally they continued with their core business of producing the comic and found themselves the recipients of a second Emmy award. Buni Media have continued to produce and broadcast the XYZ Show in 2014 as planned. It remains a high quality show which is funny, critical, and attracts a large viewership, even in rural areas. Monitoring data revealed that 24% of Kenyans have watched the XYZ Show. Among viewers, half watch the show for its comical nature, but half watch to see their leaders exposed, and to learn about current issues. Seven out of ten viewers appreciate the jokes about leaders and coverage of under-addressed topics. With all its success and uniqueness, the XYZ Show has always been a partner that was slightly on the margin of Twaweza's strategy. Now that we are starting our new strategic period, with more focused and targeted media work, the XYZ Show no longer qualifies as a core partner. The partnership with Royal Media Services (RMS) has been both frustrating and successful. The contract with RMS was one of Twaweza’s very first media framework agreements and came to an end in 2014 after four years of implementation. Overall, as with the other media framework partners, RMS provided us with excellent reach and distribution, including large numbers of PSAs at very good value. The Stories of Success carried by the 14 local radio stations proved popular and RMS is continuing them without our support, also emphasizing the integration of citizens’ voices in various different programs, including the daily news. We intend to engage RMS specifically for Uwezo in 2015, mostly banking on channelling district level results through the local radio stations, and media attention during the Uwezo launches. HiviSasa, a startup local news agency, was our only new partner in Kenya in 2014. This innovative platform provides daily local news, online, through citizen journalism. Our support has enabled HiviSasa

12

to expand to more districts, experiment with Uwezo volunteers as citizen reporters and publish Swahili news. See the example for more details. Having limited staff capacity in Kenya has meant that we needed to keep our portfolio limited yet we feel that each of these partners has delivered exciting content and reached millions of citizens. Although very few of them will have a prominent role in the new Twaweza strategy, we are proud of what they have accomplished so far in ensuring dynamism and energy in the Kenyan media landscape.

It’s like this right now Printed news is popular in Kenya but mostly unavailable and poorly distributed, not locally relevant, and often biased. Although there are plenty of local radio stations, they are often under-resourced and cannot easily collect all local news and events. At the same time, the development of solid local news faces many obstacles. Traditional news media like the printed press require heavy capital investment, skills, and well-developed distribution channels. Media ownership is highly centralized which tends to discourage expansion and specialization. Consequently local newspapers in particular are scarce in East Africa (see also the newspaper map). HiviSasa is a unique new social business start-up that aims to address this niche. Despite our decision to focus on proven partnerships, we are always willing to support unique initiatives that have great potential. It is a Kenyan daily that provides county-level news, usable on any internet-enabled phone (available for as low as USD 40). It satisfies the need for hyper-local content in a cost-effective manner by leveraging the energy and passion of ordinary citizens to write news from their community for a fee. News stories are generated by ‘citizen reporters’: anyone wanting to write news stories can register in one minute and start submitting news. All that is required are 200 words and a photo. Once published, the citizen journalist is paid KES 100 (USD 1.15) per article via mobile money and starts building up a citizen journalist profile. Senior editors select and edit incoming news stories on the back end of the site, and provide feedback to authors. They verify key facts through local contacts, and ensure that the content is acceptable for publication. Anyone with a data-enabled mobile phone can access the stories and locally-relevant information for free (e.g. job opportunities, information on service delivery, market information, political events, or local government issues). HiviSasa also enters into partnerships with radio stations in counties where they are active, to provide a news agency type function. We supported them to continue their operations but also to expand their activities to seven new counties, to offer opportunities for submission of Swahili articles, solicit contributions from Uwezo volunteers and promote submission of education related stories through an added incentive payment. Even though the news site currently produces more than 50 news articles per day, viewership is growing slowly. For HiviSasa to be profitmaking it will require thousands more daily page views. Monitoring by Twaweza is currently in process, using Google Analytics and by tracing the work of Uwezo volunteers. Early findings indicate that citizen journalists are submitting stories in Swahili since the option was introduced; in general pick up of stories is lower than anticipated, and the incentive payment and callout to Uwezo volunteers caused an increase of 129% in education related submissions. Moving forward, HiviSasa would be largely relevant to Uwezo and will allow us to publicize county-level

findings to local citizens and authorities, however we will need to ensure that the content developed is

read by a larger audience.

13

Programs Uganda: Narrative & Story Partnerships in Uganda continued to be media-based. In 2014 a stronger collaboration between Programs, LME, Uwezo and Communications in Uganda resulted in a number of synergies between activities and messages, and a strong monitoring portfolio for all of these. In many ways this serves as a preview of the new Twaweza strategy which envisions this type of collaboration between units and a less silo-style approach to work as the norm. In total we managed nine partnerships and reached millions of Ugandans; managing to build a portfolio of creative media partnerships that bring something new to the media landscape and appear to resonate well with target audiences, particularly young people. We adjusted the media framework agreement with Vision Group. Historically, there have been reporting challenges with these large media contracts throughout implementation across all three countries. Early in the year, as part of our work with Buzz Events, we changed course in our agreement with Vision Group. Vision Group supported Twaweza to run a campaign around socially-conscious and/or active musicians. With the wide media coverage and success of the campaign, and Vision Group’s successful submission of a timely report, we sensed there was an opportunity to work with media outlets in a different way. An experimental intervention was designed; Vision Group were offered a general contract for a series of outputs and a bonus incentive offer. The purpose of the contract was the commitment to increase parents’ awareness of the capitation grant, and Vision Group proposed the right mix of programs and sponsorships within the budget available and were then offered a bonus incentive if they could increase parents’ knowledge. Initial findings show that knowledge of the grant moved from 9% to 38%. Uganda Radio Network (URN) continued to perform solidly and steadily, providing local news to radio stations around the country. The contract with them and model of work has not really changed so much over the past three years, however we took the opportunity of the end of our first strategic period to evaluate the effects of URN on their partner radio stations. Findings were extremely positive. Over the course of the partnership, URN produced more stories (4,744 in 2010 to 7,277 in 2013), partnered with more radio stations (43 to 65), and redesigned their website resulting in 273,321 downloads in 2013 as compared to 86,758 in 2010. News editors also reported that they found URN to be a trusted source of news. Minibuzz continued to go from strength to strength. Again we trialed a more explicitly instrumental or content driven partnership which resulted in the production and airing of an average of one show a week on education issues. In total, 26% of young Ugandans have ever watched Minibuzz and 32% of them said they watched it to get information on current topics. We continued to work with artists through Buzz Events and, over the course of the year, supported artists to record eight music videos for songs about citizen agency. Buzz Events also organized a special event to celebrate musicians who composed songs about social issues. Subsequently a category of award for these type of songs was introduced into the hugely popular Buzz Teenie Awards. This area of work dovetailed with the Communications Unit interventions with artists. We also partnered with popular musician Bobi Wine to infuse his reality TV show with messages about parental involvement in education and the Capitation Grant. A total of 26 episodes of the show were

14

produced under this agreement and while the monitoring data are still being processed, initial indications are positive in terms of reach and shifting attitudes. Coverage is high as with many partners in Uganda; 28% of young Ugandans have watched the Bobi Wine show. Lastly, we tried two entirely new partnerships to bring something new to the landscape. SoloFx worked with two popular comedy groups to help them inject social issues into their comedy routines. Their weekly performances were broadcast on TV for a total of 52 skits over the course of the agreement. Monitoring largely focused on an expert view of the quality of these skits rather than coverage or effects; the baseline survey in Uganda revealed that 75% of young people watch comedy shows at least once a month. Though views on whether comedy is just entertainment or can bring up social topics were varied, 77% of respondents agreed that shows can present topics in ways that normally would not get discussed in public. The partnership with ReefKnot aimed to bring the satirical success of Buni Media’s XYZ Show (also supported by Twaweza) to Uganda. The political environment in Uganda, however, is such that the project was delayed by four months while the producers sought feedback from a range of different government personnel to ensure that they were comfortable with this type of satire and depiction of leaders. Production and broadcast only commenced in December 2014.

Modeling parenting Uwezo results reveal that children are not learning and parents’ roles in their children’s education are fairly limited. At the same time, evidence from around the world shows that engaged parents can boost children’s learning levels. In Uganda, many parents do not have a clear idea of what increasing their role in education really looks like. Together with the Communication for Development Foundation Uganda (CDFU), we designed a radio drama that modeled parental involvement in children’s learning. CDFU, creators of the radio drama ‘Rockpoint 256’, ventured into new territory for this partnership. Rockpoint is best known for health behavior change communication, and we used 2014 to see if they would be as effective in education. The drama depicts characters that struggle with a particular behaviour and move from a point of pre-contemplation to knowing and then finally taking action. In this case, characters in the drama demonstrated what it means to be involved as a parent, to help a child learn and to improve their performance. The characters attend school meetings, visit schools to discuss learning with their teachers and are supportive of their children’s efforts. Fifty episodes were produced and translated into three local languages. The episodes were broadcast on 22 radio stations reaching over 7.6 million listeners. Monitoring data show that 36% of young Ugandans have listened to the show and 34% of those watch it because the characters provide examples of how to deal with issues; with CDFU’s own qualitative evaluation finding a similar positive impact with parents reporting to have changed their attitudes towards involvement in children’s learning. A case study from this evaluation shows how a mother moved from using physical discipline to punish her child for poor performance to a more sympathetic approach, and approach the mother reports has improved her daughter’s performance in school. Given the success of the Rockpoint approach in health communication and our success in transferring the model to another sector, Rockpoint will be a key partner for Twaweza in our new strategic period.

15

Reflections on the Annual Plan: Programs Tanzania, Kenya and Uganda

In the 2014 Annual Plan we articulated: How did we do in 2014? Major Aims Did we keep with the major aims…? 1) Maintaining the two track approach in Uganda and

Kenya focusing on Media partners only. Scale down to 15 – 18 core partners in three countries

2) Put emphasis on learning and experimenting

3) Continue work with popular media and creative arts in Uganda

1) We maintained focus and retained the most successful partners in the media networks in Uganda and Kenya. In Tanzania we concentrated on a smaller portfolio of successful partners. For 2015 we continue formal partnership with in total 17 partners in the three countries.

2) All partnerships in the three countries had a clear learning plan. With some partners we tested a campaign style approach, through baseline and follow up measures. A number of partners (e.g., URN, Buzz, Stream Ideas and TracFM) were evaluated on achieving stated effects, and lessons were documented. With all the partnerships internal feedback loops were promoted, to learn from target audiences, most of which were adopted by partners. We have invited WTS and Compass to our own learning agenda around Positive deviance.

3) We continued our work with creative arts in Uganda. In 2014

two new partnerships in this line were Solofx (Comedy) and Reefknot (TV puppetry, modeled on Kenya’s XYZ concept). We maintained and strengthened our work with Buzz events and increased our work with musicians, supporting music and videos with key messages and brokering their appearances on TV and radio to talk about education.

What has changed since previous plan? How did we implement the changes…? 1) Step up the intensity of our engagement with fewer

partners and play a stronger advisory and brokering role so we can get more out of our partnerships.

2) Introducing a much more systematic monitoring

system right from the start of each partnership with quicker feedback loops to and by partners

3) Planned to continue with the integration of Twaweza

1) The level of involvement in partners’ work was intensified at the beginning of each partnership. This was done to ensure that partners understand Twaweza’s intentions and goals. We ensure focus on key messages, suggest topics and communication outcomes, review scripts and story boards. This level of involvement continued regularly with various partners during the year. Most partners welcome the clarity and are receptive to advice and engagement.

2) Most partners received monitoring feedback by Twaweza (see LME unit). Introducing feedback mechanisms and loops to partners took off in 2014. All new partnerships brokered had an element of how to get feedback on the implementation, through a variety of mechanisms, such as social media, toll free hotlines, SMS platforms, group discussions, registration of participants for follow up, opinion polls using telephones, polls on inter-city buses, and more.

3) This worked well in Uganda and Tanzania, but less well in

16

partnerships with Uwezo, building on and learning from the progress made in Uganda in 2013, capitalizing on a more harmonized and concerted effort of pulling the work of the units together.

Kenya due to absence of a PO Programs in Kenya. With all partnerships where the platforms allowed, UWEZO messages were integrated in the communication. Minibuzz, Solofx, Buzz Events, Rock Point 256, URN, Vision Group, Femina, SMG and Stream Ideas all used UWEZO information in their respective content creation processes. The messages to be used were provided by UWEZO and in some cases orientation face to face meetings with partners were held with the UWEZO team.

Risks and Risks Management Did we face the risks and how did we address them…? 1) Uganda: Potential risks include clamp down on

messages that call for and appear to incite change, as the regime in Kampala is very cautious and unaccommodating of any calls for change.

2) Kenya: Miscellaneous Amendment Bill adopted, limiting funding for NGO’s

1) The risk was faced early in 2014 when the PO was arrested at a Twaweza-supported concert because the posters and messages bore the word “change.” The situation was managed diplomatically and we took extra precautions for the remaining concerts. For Reefknot (political satire using the President’s caricature) we did all the due diligence with the broadcasting regulators and sought partnership with a government leaning media house for broadcast.

2) The Bill has not been adopted yet and is still under discussion.

Learning and Experimenting Aims How did we learn and experiment…? 1) In 2014, we will motivate our partners to learn and

experiment. Our task is then to work with each partner to introduce ways that will increase the number and frequency of feedback loops, aiming at improving quality and effect of their work.

1) This was largely done, though learning and experimenting can always improve. With a number of partnerships we experimented at small or larger scale (for instance VG and URN providing bonus on achieved outcome) and/or we introduced strong feedback loops using SMS platform, opinion polls, social media, group discussion, surveys, and follow up phone calls.

17

Experimental Interventions (Kiufunza, “a thirst to learn”): Narrative & Story 2014 was a relatively smooth year for KiuFunza, following a successful first year in 2013 when all the tools, processes, and personnel were new. There was a balance of the familiar, for example district teams used instructions and scripts which they already understood, with some improvements based on learning from the first year of implementation. All student level test data was captured in order to make the endline process more secure, and to enable the management team to countercheck all student level passes and conduct the teacher level aggregation centrally. Given the specialization required and the administrative burden of processing teacher payments, a decision was taken to subcontract the payment process in 2014 rather than managing this process ourselves. Still, as anticipated and promised, the KiuFunza team delivered two timely disbursements of the capitation grant to 156 schools in June and in August, providing a welcome contrast to their experience of delayed and incomplete government disbursements. A continuing challenge is the apparent discrepancy between intervention test data and survey research test data. These two tests do not take place at the same moment (September to November for the survey, and November for intervention) and they are done in slightly different ways by different teams. The good news is that we have not found, with the exception of one school-test team combination in 2013, any evidence of test gaming; the team has developed a dedicated test integrity document that describes the thorough data checks undertaken to prevent gaming. However what we do see is that intervention test pass rates are slightly higher across the board. This measurement issue is being checked in detail before we can release the preliminary impact estimates for 2014. Aside from running the largest randomized evaluation in education in Africa smoothly, the KiuFunza team scored another big success in terms of policy engagement. Ultimately, across the year, three separate announcements were made by different government officials that the Government intended to start transferring capitation grant funds directly to schools. Outreach and engagement were built into KiuFunza from the outset. Twaweza sought full collaboration and buy-in from different government departments on the design and launch of the intervention, and we continued this pattern in 2014. The KiuFunza principle investigators visited Dar es Salaam to present the preliminary results of the study and discuss the use of randomized evaluations in policy to a group of stakeholders at the Government’s Commission for Science and Technology (COSTECH). Towards the end of the year, we also organized a “field” visit at Mbweni Primary School during which teachers talked about their experiences of both intervention arms, and participants saw the testing process firsthand. The impressive list of attendees included representatives from the Ministry of Education and Vocational Training (MoEVT) and the Prime Minister’s Office – Regional Administration and Local Government (PMO-RALG), the President’s Delivery Bureau (PDB), donors and civil society as well as a number of members of Parliament (MPs). It was at this event that the Deputy Minister for Regional Administration and Local Government reiterated their commitment to transfer capitation grants directly to schools.

Making policy change Next to teachers’ salaries, capitation grants take up the largest share of the Government of Tanzania’s education budget, and are meant to be one of the major avenues to support learning in the thousands of primary schools across the country. A problem that has haunted implementation from the start is that real per student receipts at the school level have been declining and overall have been very low

18

compare the officially stated level of TZS 10,000 per student. Many schools report not having received any capitation grant funds at all. One of the school treatment arms of the KiuFunza randomized evaluation is the direct transfer of the capitation grant to schools. This part of the experiment has several distinguishing features: First, it pays the money directly into school accounts instead of district level authorities, with all payments communicated to the responsible Ministries (Education and Local Government) and receipt confirmation at the school level. This intervention provides a clear example to policy makers of the practical do-ability of this approach. A second element is that the capitation grant of TZS 10,000 per child is paid in full and on time, as per policy, and based on enrolment numbers from the relevant ministry. A challenge with any intervention of this kind is convincing education authorities to pay attention to the experiment and adopt elements of it as policy. The good news here is that the capitation grant is very easy to implement and is already part of the current education system. On the other hand, most rigorous education studies point out that increasing school inputs as a stand-alone intervention will not increase learning outcomes - our main outcome of interest. Preliminary results show that the capitation grant alone does not improve learning outcomes; in the combination arm, however, there is a positive impact on student skills. At the end of this second year of the intervention the Government of Tanzania announced that KiuFunza-type direct capitation grant payments will be adopted as policy going forward. This was announced on several occasions, including one of the policy engagement events organized in December 2014. Although these announcements have not yet translated into practice, we are pleased that the Government of Tanzania has taken note of the intervention in this way. This type of policy influence was part of the overall goal of KiuFunza; as this type of influence is really quite rare, KiuFunza as a component and Twaweza as a whole have a concrete policy impact to be very proud of. A challenge is, of course, to keep the pressure on and make sure the policy statements are actually implemented. Moving forward we will explore how a similar policy influence can be established for teacher incentives and accountability; this is likely a much harder sell, but the conversation has started.

19

Reflections on Annual Plan: Experimental Interventions

In the 2014 Annual Plan we articulated: How did we do in 2014? Major Aims Did we keep with the major aims…? 1) The big goal for 2014 is to consolidate KiuFunza,

keep it running successfully while improving where necessary. Some challenges still need to be resolved, e.g. teacher payments in Jan 2014.

2) A second goal is to learn from year 1: take time to analyze the data, start writing up quantitative results and discuss them. Also to support active policy engagement led by the strategic engagement unit. If KF is to succeed at a macro-level, the groundwork needs to be done in 2014.

3) A third goal is to incorporate a qualitative,

ethnographic component into KF which will use much less structured types of interviews to collect information.

4) A fourth goal is to reflect, review other relevant experiences and consider new experimental arms in KiuFunza, outside or inside the current experimental sample.

1) We managed keep close to our plan, see the budget performance and the timelines followed.

2) We managed to write up the results of year 1 and shared them publicly. However, the intensity of the implementation work means we did less analysis and writing than planned. The public engagement started late and was under-resourced but led to an unexpected PPE success: the GOT announced in September 2014 that following the KF example Capitation Grants will be sent directly to schools; the commitment was reiterated during a KF public event in December. (However, at print time, this commitment is yet to be implemented).

3) This was realized in close collaboration with the LME unit. The qualitative study yielded important in-depth information, useful in understanding the experience of KF and in communicating the study results.

4) We have managed to develop a new COD “gains”

intervention arm. This provides an exciting start to the second phase of KiuFunza starting in 2015.

What has changed since previous plan? How did we implement the changes…? 1) Very little change in the design: constancy and

uniformity of implementation is in the DNA of a randomized evaluation. Having invented many wheels in year 1 we have, compared to the start of 2013, a better idea of what activities will be done and how much they cost.

1) No changes were planned.

Risks and Risks Management Did we face the risks and how did we address them…?

20

1) A potential risk issue is at District implementation level, where we face varying partner implementation capacity. To mitigate the KiuFunza team will review all partnerships and restructure as necessary.

1) The KiuFunza team reviewed 2013 performance of all teams, particularly the implementation of the endline testing. One of the team was replaced because of malpractice. For the new team as well as for other teams viewed as weaker, support during training and implementation was provided by members of the intervention management team.

Learning and Experimenting Aims How did we learn and experiment…? 1) KiuFunza as a whole is about learning and

experimentation. However, Twaweza has a lot to learn about evidence based “policy engagement” generally and how to pitch the KiuFunza results specifically. The future challenge, if the learning results are positive and “large,” will be to convince the education sector and stakeholders (Ministry, Districts, MPs, Union, and Donors) to implement the incentive system.

1) Twaweza as an organization did not manage to prioritize the public engagement around KiuFunza. However, a lot of effort went into the new strategy and organogram, in which PPE figures prominently. Also, a new senior staff member has been recruited to take charge of the KiuFunza public engagement with the education sector in 2015.

21

Sauti za Wananchi: Narrative & Story As Africa’s first nationally representative mobile phone survey, Sauti za Wananchi fills an important gap in data about people’s views and realities, and allows for frequent collection of both. The survey was launched formally in 2013, following a baseline survey in 2012. If 2013 was the year in which we fine-tuned the machinery for this high frequency data collection effort, 2014 was the year in which we finessed the communications around our findings. Led by the Communications Unit, we established a fixed monthly launch date for the year. All the same, because events naturally provide opportunities for quick feedback and adjustments, small tweaks were made to the timing and format of these events over the course of the year, to capitalize on media coverage and relevance to what was happening at the time. In total we held 11 launch events and generated over 300 pieces of media coverage; while attendance varied depending on the topic and even mundane logistical details such as start time. Overall we had between 25 and 300 participants at any one launch event, with the most popular events being those with more political themes. A particular highlight – and challenge – was the production and launch of the political poll brief in November 2014, a brief containing data from three Sauti za Wananchi surveys on the political affiliation and preferences of Tanzanians. The findings were rich and varied, and there was a wealth of information that one could glean from the data, yet the media coverage focused on candidate names and ratings. These data and the subsequent debate, critique, and commendation reminded us about a number of key aspects of the survey. First, that there is a scarcity value on this type of poll data which makes the findings big news. Second, the media cannot tell a nuanced story with this type of data in particular, but also in general. We noted that the focus tends to be on one big headline item, often at the expense of the full story. With the huge volume of comment pieces and reactions, and in some cases forceful attacks on Twaweza that resulted, the need for impeccable analysis, transparency in methodology, and swift response to accusations of bias or data manipulation were all brought home. Another significant, if quieter, success was a direct request from the Permanent Secretary in the Ministry of Education and Vocational Training to make use of the Sauti za Wananchi infrastructure. As a result we ran a call round on parental views about school subject combinations and career perspectives with the Ministry. This is one of the great ambitions for Sauti za Wananchi: to provide a quick snapshot of reality that can feed into and inform policy decisions, evaluate the success of interventions from citizens’ perspectives and shed light on relatively unknown problems. Engaging key policy actors to use the infrastructure is part of the Sauti za Wananchi outreach strategy, but success often comes through opportunistic use of the Twaweza network of contacts and through seizing opportunities as they arise. Aside from the public side of Sauti za Wananchi work, the background machinery was kept running and was refined. We managed to come very close to the ambitious quantitative target for 2014 in terms of household calls: we implemented 15 standard household call rounds, as well as a call round related to the revisit. This resulted in budget performance of slightly over 100%. We did less well in terms of the number of facility and citizen monitor calls. The team is also preparing a “how to” handbook on mobile phone panel surveys with colleagues at the World Bank, and participated in two writing workshops in 2014. A draft handbook was finalized in December 2014, to be jointly published in 2015. The idea is that the lessons learned from various mobile

22

phone panel survey experiences will be shared in an accessible manner, and it will also serve as an internal standards document for Twaweza staff.

The Sauti Effect One of the main problems in Tanzania – as in many developing countries – is the lack of reliable, up-to-date data about citizens’ lives. How long do people take to collect water? What are their experiences with their local, government health clinic? Answering these questions often relies on irregularly collected, highly expensive surveys conducted by government or international organizations, or is done by issuing sweeping generalizations. Sauti za Wananchi addresses this gap by providing policymakers with rapid, reliable data every month. But how reliable are the data? While Sauti za Wananchi models itself on the rigor of global-standard surveys such as the World Bank’s longitudinal surveys, or the National Bureau of Statistics’ National Panel Survey, there is still the question of whether rigour can be maintained when one switches from field-based to phone-based surveying. There is little existing evidence on the rigor of mobile phone-based surveys, and thus there is little knowledge of whether these surveys are more or less reliable. This was therefore an important “meta” issue for Sauti za Wananchi to address: how could it collect data on its own reliability? In order to address this question, we designed a quasi-experimental study which would piggyback on the pre-existing respondent revisit plan. Originally, a field revisit had been budgeted to check up on the Sauti za Wananchi respondents, provide information, and – if needed – refresh any household data. The Sauti za Wananchi team decided to use this unique opportunity of engaging physically with respondents to design a quasi-experimental study. The study would answer two questions: First, is mobile phone data as reliable as data collected in the field? This would address the intrinsic rigor of a Sauti za Wananchi-style survey. Second, has being part of the Sauti za Wananchi sample for two years influenced respondents in any way? This addresses the potential loss of rigour in these types of surveys, in case samples are progressively influenced – and thus biased – over the course of a multi-year panel. The revisit-cum-meta-analysis-study took place between November 2014 and February 2015. All 2,400 originally selected Sauti za Wananchi respondents were targeted for an interview, and this included respondents who had dropped out of the survey, as well as reserve respondents that had yet to be asked to join the survey. In order to test the first hypothesis, the sample of still-active Sauti za Wananchi respondents were called on the phone and interviewed in the field with a select number of identical questions. The balance here was between having enough questions to capture any differences between face-to-face and phone responses, and not so many questions so as to trigger an “annoyance bias”. In order to test the second hypothesis, enumeration areas between 5km and 10km away from Sauti za Wananchi enumeration areas were randomly selected using GIS software. In these enumeration areas, households were randomly selected according to the same process used in the original selection of respondents. These households were then given the same questionnaire as the Sauti za Wananchi respondents. In this way, their responses could be compared to the Sauti za Wananchi respondents’ answers. If Sauti za Wananchi has an effect - for example, by periodically reminding respondents about

23

government accountability issues – we would expect the Sauti za Wananchi respondents to be more active, engaged citizens than a random sample of the overall population. The data will provide this answer. Data from the revisit/meta-analysis are currently being processed, and the Sauti za Wananchi team will analyze them to detect any statistically significant effects of either mobile phone surveying or Sauti za Wananchi as an intervention. In the meantime, the process of the revisit/meta-analysis has provided important monitoring and management information. For example, Sauti za Wananchi (and Twaweza) is still erroneously associated with the Freemason movement; a belief that leads some respondents to refuse to participate. In learning about this, the Sauti za Wananchi team has begun devising strategies to clarify its origins and increase community sensitization for the upcoming second Tanzania baseline, and first Kenya baseline. Furthermore, a considerable number of drop-out respondents (around 100) have chosen to re-join the sample. Once analysis of the data is complete, it will provide more important lessons for the future of Sauti za Wananchi. If we find a strong effect on respondents’ perceptions of accountability, government service delivery, and so forth, that will be an argument in favor of refreshing the sample more frequently. If we find that mobile phone surveying is not as reliable as field-based surveying, we will have to re-envision the Sauti za Wananchi process. Overall, this is an important monitoring and evaluation activity both for internal Twaweza purposes, as well as for the general literature on data collection.

24

Reflections on the Annual Plan: Sauti za Wananchi

In the 2014 Annual Plan we articulated: How did we do in 2014?

Major Aims Did we keep with the major aims…?

1) Sauti za Wananchi (SzW) to collect data via call

centre mobile phone interviews from 2000

Households across Tanzania, 100 Heads of Health

Facilities, 100 Water Group Leaders and 120 Head

Teachers of Primary Schools and 200 citizen

monitors. There is also a significant increase in call

rounds from previous year.

2) The data will be analysed, policy briefs written and

launched monthly in a public event

1) We achieved nearly fully the ambitious quantitative

target in terms of the household calls. We did less

well in terms of the number of facility and citizen

monitor calls.

2) We produced significant amount of content and we

met our quantitative goal of doing 11 public launch

events. We managed to stir debate, particularly

around the constitution drafts and the election poll

results, as witnessed by wide and prolonged media

coverage of the data.

What has changed since previous plan? How did we implement the changes…?

1) The call frequency increased to 18 household call

rounds.

2) A monthly external brief launch event was created to

increase visibility.

3) A field revisit was scheduled, to provide feedback

and to test the representativeness of the sample

after two years.

4) We started documenting the project in a handbook

on mobile phone panel surveys.

1) We managed to implement 15 standard household

call rounds, as well as a call round related to the

revisit.

2) The monthly launch event series was started

successfully. Attendance varied over rounds but

overall it was high with considerable media coverage;

the coverage particularly around political topics was

significant.

3) Field revisit was finalized early 2015. The data will

provide a unique insight into the dynamics and

feasibility of running a high frequency mobile phone

panel survey for a long period.

4) A draft handbook was finalized later 2014, to be

jointly published in 2015 with World Bank.

Risks and Risks Management Did we face the risks and how did we address them…?

1) A perennial risk for panel surveys is attrition, which

may lead to low response rates, low statistical power

and/or biased estimates.

2) A second risk is political. When messages are

1) SzW has a number of features built in to address the

attrition risk ex-ante, such as the within group

linkages, group leader, reserve households. Ex-post

we correct for possible bias using response weights

that are based on observable characteristics of those

who respond and those who don’t.

2) The best strategy is to first be sure that the data

25

perceived as being against certain interests, a

pushback against the findings and/or methodology

can be expected.

quality and analysis are flawless; then let the data

speak and respond point by point to criticism. This

approach worked well in 2014, particularly around

the political poll. There was pushback from different

sides of the political spectrum, which in fact

underlined our independence.

Learning and Experimenting Aims How did we learn and experiment…?

1) Sample revisit to keep in touch with respondents and

investigate whether sample is providing answers

comparable to a “fresh” sample.

2) The SzW team to help write and publish a practical

handbook on mobile phone panel surveys.

1) Data are just in at the time of writing. We will use this

to test whether asking questions over the phone

leads to answers that are different from face-to-face

interviews; and, similarly, whether respondents that

have been interviewed over two years respond

differently from a “fresh” sample.

2) Lessons learned from various mobile phone surveys

are shared in a “how to” handbook on mobile phone

panel surveys in Africa. Much of the empirical

material and case study comes from Sauti za

Wananchi.

26

Uwezo: Narrative & Three Stories Uwezo adopted the organizational mantra of learning and experimenting, and rolled out a number of trials and tests across the assessment and communications process. In order to facilitate this the assessment was scaled back in Tanzania and Uganda, covering 50 and 28 districts respectively. In Kenya the assessment was undertaken at full scale in 158 districts. Part of the logic of this strategy was also to understand whether collecting this more limited dataset would still allow for the same policy-level resonance. In Uganda this was the first year in which partners were recruited directly by Uwezo. Previously partner recruitment and liaison was the responsibility of the Uganda NGO Forum, which had originally been the host of Uwezo in Uganda. The new direct approach led to the recruitment of a large number of new partners to ensure compliance with organizational standards, policies and procedures. Uwezo Uganda also trialed a new initiative: the recruitment, from the same partner organizations, of specific communications personnel to share findings at the local level. In other countries, district coordinators from the partner organizations are expected to carry out assessment and communications activities. Uwezo Kenya, with more established and long-term partners, also supported some of them to trial their own new interventions in promoting learning outcomes. These activities will be completed in the first half of 2015. We also undertook critical steps in leveraging the Uwezo infrastructure and process: in Uganda we cemented the local language assessment and began defining protocols for the same. We also started the process of defining test protocols for higher-level tests, and, perhaps most excitingly, we piloted using the assessment process for gathering critical health data. In Kenya the tools and processes were developed in partnership with the Ministry of Health and then replicated for Tanzania and Uganda. These data will allow us to link health and learning outcomes but also to expand the scope of the assessment into other sectors. The central pillars of Uwezo are the people; it is on the quality of personnel that the cascade model and therefore Uwezo itself rises or falls. In recognition of this, we developed a cadre of East African trainers who participated in trainings in all three countries and increased harmonization between them; we tested different lengths and “class sizes” for trainings; and we introduced the role of village coordinator to support household listing, assessment logistics and to backstop volunteers during the assessment. In general, we placed greater emphasis on training because this has such a significant impact on the quality of the data and final products. Part of this included breaking down the training manual to match the various phases of training. Instant feedback continued to be a critical part of engaging tested households, alongside the communication material distributed at the point of assessment. However we have learned through our monitoring that quality assurance of this process is problematic and the feedback varies hugely between volunteers. Some critical thinking in Tanzania about how to transmit this feedback more consistently did not come to fruition; the basic issue is that the data are reliable when aggregated up, and we do not wish to place too much emphasis on providing individual assessment scores more permanently and prominently because of the possibility of errors. Moving forward this is still an issue that needs to be resolved.

27

Aside from instant feedback, Uwezo has also always relied heavily on launching the assessment findings as a sort of kick start to the annual program of communications. Unfortunately in 2014, we did not manage to launch the 2013 reports. The pressures of achieving everything we had set out to do, got the better of us; in the midst of all the changes we were going through and things we were contributing to in 2014, while still delivering the assessment which is still huge despite the scaling back of the number of districts. It is still inexcusable, and we intend to rectify this by holding two sets of launches in 2015 – for the 2013 and 2014 data – no small feat, but it shall be done. In both Kenya and Uganda, we continued to use radio to spread awareness and generate debate around findings. In Kenya a new longer-term engagement with two radio stations resulted in 48 talk shows, 18 on-ground forums and 18 news items. Again in Kenya, 15 journalists also participated in the assessment process, further driving the Uwezo agenda. In all three countries, Uwezo country coordinators and the regional manager participated in tens of interviews across the year, particularly around critical education moments, ensuring that we contributing to the thinking on education in the three countries and even globally. Similarly, Uwezo participated in a host of education related events in each country. From national and global education week in Tanzania to local education days and barazas in Uganda to critical policy meetings in Kenya, Uwezo is viewed as a reference and essential participant in national education events. We also continued to maintain strong links with teacher bodies in each country, attending national conferences and cementing the positive working relationships established. Similarly in Kenya and Tanzania we worked with national coalitions to develop joint messaging and shared agendas. In Uganda we have not yet made inroads with the national education coalition and intend to rectify this in 2015. Uwezo continued to make its mark on the global stage as well: participating in post-2015 discussions to emphasise learning outcomes in any new development goals; cementing our relationship with the global family of citizen-led assessments by hosting the annual meeting in Kampala; and supporting other countries in Africa who wish to undertake citizen-led assessments of learning outcomes. In particular we lent significant support to Africa’s largest country as they geared up to undertake their pilot and first assessment round. Monitoring has been built into the assessment process from the outset. Given the reliance on citizens to conduct the assessment, it is crucial that we can make clear statements about how we guarantee the quality of the data collected. There were three particular highlights for 2014. First, we delimited monitoring to be the sole core function of regional coordinators who are slightly outside the core assessment process and are therefore more likely to be impartial. However the quality of their work varied and in many cases, regional coordinators may not have the right skills and experience to perform this function well. Following the 2013 success in Uganda, the process re-check was conducted using the Lot Quality Assurance Sampling (LQAS) method in all three countries, which provides insight into the implementation of the assessment as well as parents’ recall of communications material and instant feedback. Finally we received reports from the Results for Development (R4D) evaluation, and the evaluation of the Australian Council for Educational Research (ACER) which will provide valuable lessons and insights for moving forward.

28

Uwezo Uganda: Ideas for reaching people in 80 districts, with a team of two One of the major challenges for Uwezo is communicating our findings meaningfully to authorities and citizens across the country, particularly at sub-national level. Beyond using mass media (radio in particular), which we know to exert limited influence in norm and attitude shifts, how can we achieve the type of scale in communications which we do in the assessment, in a cost effective manner? We have always been compelled by the idea of leveraging the assessment infrastructure to also perform communications functions. However, different types of expertise are required as well as entirely new types of contracts with the district partners. Since we have often struggled in getting the best out of our contracts with partners even for the assessment, we were loath to significantly change these. In addition, Uwezo Kenya has been testing ways in which to support their existing assessment partners to undertake communications work. So in Uganda, we decided to try something different. We developed contracts with communications staff in the same partner organisations in four districts to trial the idea of having district communications contact persons to carry out this type of work. The four partners we worked with in 2014 attended and contributed to debates on learning at barazas, school meetings, radio talk shows, planning and budgeting meetings and church services among others. From the monthly activity journals for a period of four months they attended 94 events between them. To complement this work, and to trial something new in the space of engaging with authorities and trying to stimulate responsiveness, and public agency, we also worked with Local Council III Chairpersons (LC3s) in eight districts. LC3s are elected officials and so are incentivised to serve the populations, and are part of the establishment and thus bring some level of decision-making power. They essentially inhabit the liminal space occupied by many local government officials whereby they wish to be seen to be on the side of the people, making demands for their constituency but also, in many ways, represent the authority that they might advocate to on behalf of local populations. The LC3s used their existing work and agendas - that require them to address community gatherings on a regular basis - as a platform to transmit messages on learning outcomes. They reached parents at the community level at various events including village meetings, talk shows on regional/community radios, marriage ceremonies, burials, leadership meetings and church services. When they speak, they are likely to be a trusted source, not necessarily in terms of their overall credibility, which may vary between individuals depending on their track record, but because of their office. During their work with us over five months, LC3s shared our findings at a total of 193 events with an average of 200 people at each event. The most exciting finding has been that a number of LC3s in particular have continued to promote the Uwezo message even after the expiry of our support to and contracts with them. This is exactly the type of organic growth that Uwezo has always wished to spur, and we were delighted that a number of the LC3s continued to engage through meetings and promoting Uwezo testing. This was established through follow up phone calls after the project. For DCCPs we did not find evidence of the same continuation but this may simply be because their work does not naturally place them in positions where they can transmit this message as much as LC3s’ work does. Unfortunately beyond ‘bean counting’ events attended and people reached, despite intensive briefings on monitoring, the activity journals recorded by both groups did not really delve into the meat of their

29

work for Uwezo. We wanted them to capture more: a clearer sense of what these engagements meant to them and audience reactions. Moving forward, particularly as we scale these activities up, we will need to develop better monitoring tools and tactics. Most importantly we should not rely on the partners alone to provide this type of ‘thick’ information without fully focused training. Our work with outcome mapping may provide some support in this regard, however we are confident that the Uwezo message reached, and we would guess ‘touched’, more people much more directly than ever before.

Uwezo Kenya: A question of heart or mind? After five years of implementing the Uwezo assessment consistently and making enough noise with our communication of these findings, we are confident that we can assert a policy effect at the national level in all three countries. This is particularly evident in Kenya where the Uwezo team is constantly called upon to input into critical policy discussions. Yet we were much less confident about our sub-national reach and effectiveness. It is all very well for centralised policy actors to have heard and absorbed our call for education to focus on learning outcomes, but what about those whose work is directly connected to implementing national policies and rolling them out locally? We have always felt that we have been lacking in this regard. Given that Uwezo collects data that are representative to district level, we feel there is a big gap in our communications. So in 2014, on top of our usual repertoire of local radio programs and a high-profile launch of a district ranking poster attended by over 500 policy actors, we tried something new. Our district partners identified key local policy actors who were then invited to participate directly in the assessment. We secured participation from a number of directors of quality assurance, and even a deputy county governor. For a few hours, they joined our teams of volunteers and were able to see first-hand the test process and the difficulties children have in mastery of the basic skills we test for. The reactions were overwhelmingly positive. Participants moved from distrust and scepticism to acceptance and concern, wondering what could be done to address the learning crisis in Kenya. Of course, words are relatively cheap and it remains to be seen whether any further actions from these participants ensue, but we can honestly say that we have never had these types of reactions to the actual reports of our findings or any of the other communications materials that we have produced. Internally, debate has raged on, because those who directly participated in the assessment tended to be the more junior officials, thus there was some sense that this exercise had failed to attract the right level of district policy actor. If the participants do not hold real decision-making power, there was the sense that it was not worth bringing them to the assessment. At the same time, it is clear that directly experiencing the assessment process really helped to build our credibility among this group, and ensured that, at the very least, they took our findings seriously and understood the rigour of the process better. Having this type of inside voice singing from our hymn sheet, even from a more junior position in the hierarchy, is still an important inroad for us. Seeing our guests react to the assessment experience, hearing them tell their stories and describe their reactions, we believe we have created a group of officials who are more than ready to hear our message and take it on-board in their work. Ultimately, we believe that what we need is a visceral reaction from local policy actors, reaching out to them in such a way so as to move them from paying lip service to learning outcomes towards a sense of

30

a crisis in learning and a strong feeling of personal responsibility. Surely this is best achieved by the individual experience of seeing a child being unable to read basic English or Kiswahili, unable to add numbers. This, surely, is a question of the heart.

Uwezo Tanzania: Armed for learning Although we feel that Uwezo has achieved many of its policy prioritisation goals at the national level, this has tended to occur only in directly relevant ministries. One of the challenges that anyone working in public service delivery can face is that the debate and conversation remain housed within the sector both within and outside government. We cannot achieve real gains in learning outcomes if citizens and authorities alike both view this as a challenge for the Ministry of Education alone. Ultimately for true national success, we need to see the state of learning discussed as part of a generalised, not specialised, national conversation. This dialogue needs to take place in the street and the corridors of power of all government institutions and bodies. So we were delighted in 2014 to be invited to present to a group of security personnel from across the region. Members of the Armed Forces from Kenya, Nigeria, Zambia, Uganda, Malawi, Burundi, Rwanda, and Tanzania invited Uwezo to facilitate a training session, and give a presentation, on education and learning outcomes. We were also able to make a strong link between education and national security, which resonated with the audience. It was uplifting to see the response, with many participants immediately seeing the challenge posed to the Armed Forces and national security by the learning crises in our countries. It truly was unusual to see the army’s top brass flipping through the reports we distributed at the event, reading voraciously through the material. Following the presentation, Uwezo was invited to hold a series of seminars at the National Defence College to spread our message to more army personnel. Internally, we see this as the next major national policy challenge: to bring on board strategic and powerful government officials from different sectors and spaces, in the battle to improve learning outcomes.

31

Reflections on Annual Plan: Uwezo

In the 2014 Annual Plan we articulated: How did we do in 2014?

Major Aims Did we keep with the major aims…?

1) Implement the annual learning assessment:

collect robust data on children’s competency

levels, analyse and publish data.

2) Achieve greater public understanding and debate

about learning, focusing on parents and teachers.

3) In concert with others, ensure that policies,

programs and practices at both national and local

levels better reflect a focus on learning.

4) Achieve rigorous monitoring of Uwezo work and

effect, learn and use lessons to inform planning

and execution.

5) Greater coordination of Uwezo tools and

processes across the 3 countries.

6) Document and utilize Uwezo lessons to influence

thinking at national, regional and global levels.

1) Uwezo assessment was implemented in all three countries

following organizational policies and regulations. A total of

234 districts across the 3 countries were assessed. Analysis

of 2013 data was completed, publication slated for 2015.

2) The year was light on public communication, and fewer

communication materials were distributed at the

household level than previously.

3) Interventions targeting policy engagement were carried out

in the 3 countries, e.g., working closely with the National

Teachers Unions and presenting Uwezo at national

teachers’ conferences; formal presentations to Ministry of

Education officials and at events such as the annual

Education and Sports Sector Review (UG), the Big Results

Now conference (TZ) and at the Education for All 2013/14

Global Monitoring Report launch (all three countries).

4) Monitoring of Uwezo was implemented in all three

countries in the pre-, during and post-assessment. Process

recheck was conducted using the LQAS method. Data were

analysed to inform future implementation.

5) The use of Uwezo EA trainers, joint development of

manuals, and a combination of tools to monitor standards

of tests were among core initiatives.

6) Uwezo’s evidence and lessons were shared widely,

including presentations at the 20th Nigerian Economic

Summit; Comparative International Education Society (CIES)

conference in Canada; UNESCO Institute of Statistics

conference on oral reading assessment, Canada; US-Africa

leaders’ summit in USA; the Africa Network Campaign on

Education For All meeting in Ethiopia; the World Innovation

Summit for Education in Qatar.

What has changed since previous plan? How did we implement the changes…?

1) Creation of space to implement tests and studies

aimed at strengthening Uwezo.

2) An extra layer of support for conducting the

annual learning assessment.

1) Scale was reduced in Tanzania and Uganda, assessing 50 and 28

districts respectively, while maintained at full scale in Kenya.

2) Village Coordinators were introduced in all three countries,

three in each district, to strengthen the assessment cycle.

32

3) Moving Uwezo beyond assessment of basic

literacy and numeracy.

4) Consulting more on Uwezo standards, especially

on training, tests and methodology ahead of the

new strategy.

3) In collaboration with ASER, we plan to assess higher grades

starting in 2015 piloting in each country. Uwezo+ was piloted in

Kenya, with the assessment of nutrition status of children; some

survey items were also adapted by Tanzania and Uganda.

4) Sampling and methodology experts from the respective national

bureaus of statistics were brought together to revise the

methodology standards.

Risks and Risks Management Did we face the risks and how did we address them…?

1) Drastic changes in the plan could affect

implementation, 2014 being a year of writing new

strategy.

2) Reduction of scale in Uganda and Tanzania may

mean weakening the infrastructure, while during

the year of learning, Kenya may be overwhelmed.

3) Uwezo communications may be affected by

bandwidth, as we engage in much

experimentation.

1) The pace of implementation of was slower than previous years.

A number of activities kicked off late in the year and will be

finalized early 2015.

2) Kenya completed all assessment activities, and over 90% of

retirements within the year. Uganda had chance to newly

establish its infrastructure, independent from UNNGOF. In

Tanzania, many partners remained engaged through the

KiuFUnza initiative, continuing collaboration.

3) Many activities within this cluster could not be completed,

though reasonable ground was covered. Priority was given to

policy-level communications. Connection to the media networks

also remained strong in all three countries.

Learning and Experimenting Aims How did we learn and experiment…?

1) The main work of the unit this year is to enable

Twaweza to learn as much as it can from its own

experiments, from external evaluations, as well as

from other available international evidence.

1) We tried new approach to handling the Uwezo contracts with

district partners. We learnt that conducting due diligence at

onset, holding meetings with heads of organizations, clarity of

expectations, utilizing an accounting manual and clear forms,

and monitoring for accountability are key success components.

2) We experimented with different models of training volunteers

(e.g., more volunteers trained for shorter time, vs. fewer trained

for longer, etc.). We conclude that the 2 days of volunteer

training are adequate, combined with strictness and discipline

in time management. Improving support from Village

Coordinators during the training may be more strategic than

lengthening the duration of training.

3) We tried longer contracts for radio communication in Kenya;

such contracts are better in building relationships with media.

4) We engaged the District Communications Contact Persons in

Uganda, as well as the engagement of LC 3 chairpersons. This

initiative has shown promise in strengthening feedback and

stirring debate at the local level.

33

Strategic Engagement: Narrative & Story Twaweza’s Head, Rakesh Rajani, continued to play a highly influential role on the global stage to shape the policy environment around open government. As civil society co-chair of the Open Government Partnership (OGP) until late in the year, he was instrumental in developing a robust policy for the OGP on protecting against intrusions on civic space. A “Response Policy” with this goal was formally adopted by the OGP in September. Rakesh was also appointed to the UN Secretary General’s Expert Advisory Group on the Data Revolution. Through this role, the Strategic Engagement unit was able to influence the group’s report – and thus the UN’s future work in this area – to include strong focusses on national and sub-national data needs, transparency and open data, and on data users (in addition to generators). A set of principles for the data revolution, drafted initially by Twaweza, was included in the group’s final report. At the national level within Tanzania, the Government was supported to develop a strong second action plan for the OGP, informed by analysis of the (lack of) progress made under the first such plan. The plan includes bold commitments on the freedom of information law, open budgets and open data, all closely in line with Twaweza’s recommendations. Twaweza established a practice of regularly carrying out small analyses of interesting data across a wide range of policy topics. Highlights included a video on wealth and inequality in East Africa, using data from Credit Suisse and Forbes, analyses of political polling data from Afrobarometer and Sauti za Wananchi surveys, and analysis of data on the quality of health and education services collected by the World Bank. One such analysis, of the implications of a leaked contract for the extraction of offshore natural gas, provoked considerable debate, initially around the terms of the contract and later on contract transparency. This second aspect was taken up by the Public Accounts Committee (PAC), which called on officials from the state-owned Tanzania Petroleum Development Corporation (TPDC) to provide parliament with copies of the contracts. TPDC initially refused to do so, but later changed their mind and have promised to present all contracts to Parliament over the coming months. Less progress than expected was made on developing web-based tools for showcasing data on education, water and public opinion and for promoting public debate. The delays were largely the result of bandwidth challenges, given the packed schedule Twaweza had in 2014; difficulties in finding capable partners also contributed to these delays, though a solid foundation has been made for this work in 2015 and beyond. The need to balance political expediency with frank and honest criticisms continues to be a tight-rope. On the OGP, Twaweza largely played an inside game and delivered considerable policy commitments in this way. However, even in this case, some stringent criticisms were provided, including at high profile public fora, to maintain the pressure for strong commitments and for delivery. In the case of the Sauti za Wananchi political poll, which attracted a very high degree of public and political interest, and some fierce criticism, Twaweza was able to defend itself on the basis of a strong pre-existing reputation for independence, rigour and credibility of data gathering and analysis.

34

Gas, contracts, and transparency Expectations for the impact of major new offshore natural gas discoveries in Tanzanian waters are high, both in government and in wider society. The quantity of gas found is such that it has the potential to transform the Tanzanian economy, though many questions remain. One key question is whether the government has negotiated a fair deal for sharing natural gas revenues with the extraction companies? Without access to the contracts – the Production Sharing Agreements (PSAs), which have always been secret – it is impossible to know, and Tanzanian citizens are simply asked to take on trust that the contracts represent a fair deal. This has really pushed the drive for commitments to openness and transparency in Tanzania, without which secrecy around contracts (and many other important details) will continue. In mid-2014 one such contract, between the TPDC and Statoil, was posted online by an unknown leaker. This attracted some attention from extractive sector experts, but no one was analysing the contract’s terms and implications. Engaging with various analysts and politicians active in the sector, Twaweza’s Strategic Engagement unit carried out a rough-and-ready analysis of the contract, which was published on the Mtega blog (run by Ben Taylor) and the African Arguments blog. This analysis attracted considerable media and political attention in Tanzania, led by opposition politician Zitto Kabwe and generating numerous front-page newspaper articles. Having brought the leaked Statoil contract to wider public attention, focus was then shifted onto the broader principle of contract transparency. A further seven blogposts on the topic were published on the Mtega blog. Working with leaked information comes with various risks. In this case, because a Twaweza consultant was prominently involved in simply analysing the leaked PSA, he was later labeled by the media as “the leaker”. This was easily and quickly resolved by reaching out to the editor of the newspaper, which then published a correction. Zitto Kabwe, in his role as chair of Parliament’s PAC, drew on Twaweza’s analysis to call on TPDC, the Ministry of Energy and Minerals, and the gas companies to make their PSAs public. At one point, two senior officials of TPDC were briefly arrested for failing to comply with a demand from PAC to provide the PSAs. Finally TPDC agreed to make all 27 gas PSAs available initially to PAC, and then to the wider public. This process has started, with the first group of contracts provided to PAC. This was purely opportunistic work that was carried out in an unplanned, exploratory manner. Nevertheless, it capitalised very effectively on a window of opportunity – the leak of an interesting document – to raise an important issue of open government, and to deliver meaningful change. The informal partnership with a range of sector analysts and politicians provided a very effective mechanism for bringing about change. In particular, Twaweza was instrumental in bringing the issue to wider attention through conducting much of the analysis, and the different players in government brought the political influence to bring the matter to the fore.

35

Reflections on Annual Plan: Strategic Engagement

In the 2014 Annual Plan we articulated: How did we do in 2014?

Major Aims Did we keep with the major aims…?

1) Open Government: To identify and act on key opportunities to advance Twaweza aims on transparency and open government through policy action at national and global levels.

2) Evidence-based policy: To inform policy actors and help them make better use of data and evidence, including citizen voices.

3) Learning Outcomes: To inform policy actors with evidence based thinking on how to make education reforms more effective and provide better value for money.

1) In addition to extensive work on the OGP (including supporting development of a second OGP plan in Tanzania, and, at global level, developing the OGP Response Policy), the unit played an influential role on the UN Secretary General’s Expert Advisory Group on the Data Revolution.

2) The unit conducted analyses of data across a wide range of policy themes, such as inequality in East Africa and natural gas contract terms, and supported the Sauti za Wananchi surveys with analysis and communications.

3) The unit supported the communications work of the

Uwezo initiative, including representing the initiative and presenting its findings to high level policy actors both within East Africa and beyond, including DFID social sector advisors, the ODI and the World Bank.

What has changed since previous plan? How did we implement the changes…?

1) Twaweza sees a greater need to explicitly engage with policy actors at both national and global levels, and to better coordinate ‘bottom up’ public engagement and ‘top down’ policy engagement.

2) Managers will continue to play an

increasing role so as to have greater capacity in policy engagement in the organization beyond the Head.

3) A long term consultant and an additional staff person will be recruited to provide adequate staffing to the unit.

1) Twaweza built and maintained relationships with policy actors including politicians from across the political spectrum and leading thinkers and practitioners in the development and aid community. We brought evidence from ground level to national and global debates – e.g. through Sauti and Uwezo – and informed ground level work with cutting edge thinking and evidence from elsewhere in the world.

2) Twaweza’s new strategy clearly emphasizes the greater role of managers in policy engagement.

3) A long term consultant was available throughout the

year, though no additional staff member was recruited. As a result, a considerable burden was borne by the Head.

Risks and Risks Management Did we face the risks and how did we address them…?

1) A risk is that Twaweza could be tempted to water down its public engagement work or analytical critique in fear of jeopardizing relations with government actors nationally or globally. While

1) There is always a balance to be struck between protecting good working relationships with key policy makers and speaking out critically and publicly where necessary. To some degree, this requires sensitive political antennae and considered decisions on

36

maintaining a civil and reasonable tone, we will need to continue to define our signature as based on evidence and not afraid to say things straight so as to maintain our credibility.

different approaches to address given problems. On the OGP, for example, Twaweza has played the inside game, and has earned notable policy commitments as a result. However, the best protection against this risk is credibility among both policy actors and the wider public – to produce data and analysis that is trusted. On several occasions, most particularly the Sauti political brief, Twaweza’s strong reputation as a credible and independent source was extremely valuable in protecting the organization against politically-motivated criticisms.

Learning and Experimenting Aims How did we learn and experiment…?

1) We will play close attention to the value of coordinating public and policy engagement better, and studying what forms of engagement are more effective. However, there will be no formal experiments.

1) Twaweza has gained valuable experience of combining both an inside (providing policy advice) and outside (mobilizing public agency) game concurrently, including re-recognising the importance of formal, invited spaces for policy influence either as a complement to public action or where there are opportunities to move policy forward in advance of public opinion.

37

Learning, Monitoring and Evaluation: Narrative & Story In 2014, our year of thinking, questioning, reviewing and re-strategizing, the LME unit had a significant

role in the development and articulation of the new Twaweza strategy and the logic which underpins

this new exciting organizational period. The unit spearheaded the adoption of the problem-based

approach to situation analysis, as well as the use of reverse logic in planning; both of these approaches

are described in our new 2015-18 Strategy. That said, the unit also rolled up its sleeves practically, and

carried out a number of significant monitoring and evaluation activities, many of which took place in

Uganda. In fact, the Uganda learning, monitoring and evaluation portfolio really took off in 2014.

In Tanzania, we assessed the distribution network used for communication materials about the new

constitution (in partnership with the Legal and Human Rights Center). We also assessed the distribution

of Open Government booklets to schools, and coverage of the Hot Seat satirical television fillers

produced by Vuvuzela. The LME unit also gathered feedback and sought views on the perceived

quality from the target audience of Kingo magazine. A comprehensive quality of content analysis

exercise was conducted on Mlimani and Sahara Media, two large media houses supported by Twaweza

for several years. This exercise was conducted jointly with Tanzania Media Fund, who participated in a

lot of the thinking on how this exercise was going to be shaped and carried out.

In Kenya, the main partners we worked with as articulated by the Programs unit meant that we assessed

coverage, and reasons for watching, of the political satire the XYZ Show and the Tazama investigative

journalism show.

Uganda is where we had the most traction and material to work with, and we started by measuring the

distribution of Twaweza materials through the innovative video hall partnership, the recall and feedback

related to a series of Ni Sisi concerts targeting young people, and coverage of capitation grant-related

issues by the media house Vision Group.

We also looked at coverage of a number of flagship communication partnerships (through a

nationally representative survey of young people), including RockPoint and MiniBuzz. Our

partnerships with the Bobi Wine show, the Uganda Radio Network, TracFM, and an experimental

approach of working with selected comedy groups all needed evaluating, which resulted in some

interesting findings which are shared in our monitoring briefs.

Finally, also in Uganda, a baseline survey focused on young people; in addition to measuring the

coverage and perceived quality of a number of communication partnerships, the survey also captured

indicators related to self-efficacy of young people and their community engagement – the intermediate

outcomes that the communication initiatives were designed to promote. We plan to repeat selected

measures in 2015.

Many of the above resulted in Monitoring Briefs, of which there are now 12 posted on the Twaweza

website. These are short, engaging communication pieces, describing the initiative assessed, methods,

and the main results in simple, accessible language. They have been useful internally to debate results

38

and engage colleagues from different units; they have also proven to be useful in communicating with

partners, donors, and other interested parties on how Twaweza gathers information to learn and to

improve our practice.

In addition to making more active use of our monitoring information for learning, the internal learning

agenda was generally implemented well in 2014. The target was nearly met in Tanzania for the variety

of sessions held (skills labs, learning sessions, etc.), great improvements in frequency and engagement

by staff were seen in Uganda; the Kenya pace of the learning agenda remains slow, perhaps due to the

absence of an LME staff person there. One big change in 2014 was that we skipped our much-treasured

immersion: in the year of re-strategizing, experimenting, etc. we simply did not have the bandwidth to

pull off the immersion exercise in a meaningful, thoughtful way. We look forward to immersing

ourselves again in 2015.

In external evaluation, one of our long-standing teams – the consortium of MIT, Princeton and UCLA

professors (known as “LPT”) wrapped up its work in mid-2014. The team had produced numerous

important outputs, such as published papers, qualitative research reports, blogs and presentations; the

core of these, focused on lessons from Uwezo-related research, we are currently compiling into an

overview research brief. We have also continued engagement with MIT and Princeton in 2014, through

exploratory qualitative research focused on community and parental engagement with schools, and the

perception and engagement of young people with local and national government. We hope these will

lay the groundwork for further relevant research, directly related to our implementation, in 2015 and

beyond. Also in evaluation, the Amsterdam Institute of International Development wrapped up its multi-

year “sikiliza” qualitative component; preliminary analysis using this data suggest that there is indeed

support for the Twaweza hypothesis that an increase in information is linked to greater citizen agitation,

which is linked to action and problem-solving in public service provision. To complete the overall

evaluation, we will be fielding the quantitative follow-up survey in early 2015, and are looking forward

to the exciting analytical combination of the qualitative panel with the quantitative pre- and post-

surveys.

Joining the dots in Uganda Uganda is a young country: the median age is 15.5 years, and well over two-thirds of the population is

less than 30 years old. Twaweza’s communication portfolio in Uganda has tended to focus on this

population group, primarily encouraging young people to engage in their communities and with relevant

authorities to bring about positive social change. Yet until this year we didn’t have a very clear picture

on whether young people were effectively reached through our core initiatives, nor did we completely

understand some of the important drivers of young people’s participation in community and socio-

political spheres. That being the case, in 2014 we designed a monitoring and evaluation strategy

alongside the implementation plans, to both understand our audience better, and to get a measure of

our own effectiveness.

We started by conducting a baseline survey of young people across Uganda; making use of an

established survey mechanism (the Omnibus), we achieved a nationally representative sample at a

fraction of the usual cost for such surveys.

39

The baseline survey showed us that our flagship communication partnerships (such as the RockPoint

radio drama, and MiniBuzz televised debate show) are listened to and watched widely; that our hunch

to work with popular media figures, musicians and comedy groups, to get across messages on

participation and civic engagement resonates well with young people. For instance, 93% of young

Ugandans say that if their favorite artist is an important role model for them, and that if the artist

promoted a specific action, they would be more likely to follow that action themselves (e.g. if the artist

promoted it, 91% say they would join a group in their community, and 75% say they would vote in the

next election). This survey was also the first instance that we know of which measured internal self-

efficacy and political self-efficacy among Ugandan young people.

Moreover, we were interested in active civic participation: we found out that almost 1 out of 3 young

Ugandans (31%) say they have personally taken an action to address a core issue in their community

(among top issues reported were lack of infrastructure, poverty, and environmental degradation).

Among respondents who report not having taken action, nearly half (49%) say they would consider

taking action but only one out of three (34%) say they know, or could figure out, how to go about doing

so. Women are half as likely as men to take action, and older respondents are somewhat more likely

than younger respondents to act. Then, we added self-efficacy into the mix, and it turns out that young

Ugandans with the maximum reported self-efficacy are seven times more likely to take action than

those with the minimum self-efficacy.

And when asked who in their community they would turn to in order to solve a problem, 49% chose the

local government.

Through the year, the Uganda team implemented a variety of communication activities targeting young people and promoting community and civic engagement, building on the findings on the survey. The LME team worked closely with the implementing units and developed measurement approaches to provide meaningful insight. In doing so, we found that the series of concerts with top Ugandan artists with messages encouraging community engagement resulted in high recall of the core messages. An assessment of our partnership with a popular singer and TV star to promote education-related messages suggested that young parents who watched the show gained knowledge on school and education-related issues, and appreciated the artist as a role model for young people’s involvement in community issues. We also evaluated our long-standing partnerships with the Uganda Radio Network (URN) as well as with TracFM – given that radio remains, by and large, the most common source of news and information for all Ugandans. Both evaluations suggested positive contributions of the initiatives: URN in bringing high-quality, trustworthy news to the various corners of Uganda’s fragmented media landscape, and TracFM in bringing citizen’s voices and opinions to the forefront of radio debates.

40

Reflections on Annual Plan: Learning, Monitoring, and Evaluation

In the 2014 Annual Plan we articulated: How did we do in 2014? Major Aims Did we keep with the major aims…? Learning 1) Strengthen our own internal learning architecture by

engaging with available evidence and applying lessons to our practice.

Learning 1) We used both our own and external evidence

rigorously in designing our new strategy, and selecting the set of problems (and successes) on which the strategy focuses.

Monitoring 1) Integrate monitoring into various units of Twaweza and

strengthen this aspect among our partners (including feedback loops from end-user audiences).

2) Engaging creatively with experimenting and testing new ideas, together with implementation approaches.

3) Monitoring public discourse in the media on topics of interest to Twaweza.

Monitoring 1) Monitoring plans were developed around activities in

all implementing units; some have also become keen to engage pro-actively with monitoring and feedback from audiences (e.g., Kingo, Minibuzz). In Uganda, monitoring was implemented around every major activity; about three-quarters of implemented activities were monitored in Tanzania, and a few (Uwezo, and two media partners) in Kenya.

2) Experimentation was the desire for 2014, although only a fraction of the planned / desired experiments took place. LME unit actively participated and supported the testing of these ideas; most were implemented around the Uwezo assessment, and with communications / program partners.

3) In Tanzania, we kept a comprehensive log of

Twaweza in the media and have analysed the quantity and quality of our influence in the media space. Moreover, jointly with TMF, we conducted a thorough content analysis of the media products Twaweza has supported through Framework Agreements with Sahara and Mlimani media houses. In Kenya and Uganda, media monitoring was conducted on as-needed basis, mostly related to Uwezo public events.

Evaluation 1) Facilitate ongoing external evaluations and to make the best

use of results as they become available.

Evaluation 1) Results from external evaluation collaborations were

key in informing internal strategy discussions, though some not yet made public (e.g., Mukuru study), and some not yet finalized (e.g., AIID study). Preliminary AIID analysis suggests support for the core hypothesis linking an increase in information, citizen “agitation” and subsequently action to solve an identified problem. On the other hand, the Mukuru study in Kenya demonstrated that even very popular and powerful communication has only a limited effect in the short-term.

41

2) To identify key research questions relevant to the new

strategic period (including identifying possible external partners)

2) Evaluation strategy for the new strategic period

developed at a slower pace than planned. However, encouraging discussions have been taking place with potential evaluation partners.

What has changed since previous plan? How did we implement the changes…? 1) A distinct monitoring plan articulated for all implementing

units of Twaweza, with the purpose of improving the frequency and quality of meaningful monitoring.

2) Greater focus on experimenting while implementing.

1) The plans were a great management tool, allowing the LME unit to hold frequent discussions with units on progress and changes to implementation, and to tailor the execution of the plan accordingly.

2) LME supported experimental implementation in the Uwezo unit, and the Communications unit.

Risks and Risks Management Did we face risks and how did we address them…? 1) Potential risks include expectation for partners to implement

feedback or monitoring mechanisms and will depend on their interest and capacity.

2) Capacity and human resources need to be increased within

the LME unit, to enable us to carry out this ambitious agenda.

3) We run the risk of ending the year without the full evaluation

findings. We will need to be closely involved with the external evaluation teams to make use of available findings to guide us when designing the new strategy throughout 2014.

1) Some partners were keen to engage and take on the feedback exercises (e.g., Minibuzz, Kingo), others less so. We encouraged and promoted those that were keen, while ensuring we get the essential measurements from all partners.

2) An LME assistant program officer was engaged in TZ. KE office continues to be without an LME staff member. The new organogram includes 2 new mid-level (and regional) positions in the LME unit, acknowledging that the unit is under-powered.

3) All three original external evaluations (LPT,

Georgetown, and AIID) have provided important and evaluation findings in time for strategic thinking, although not all have been finalized.

Learning and Experimenting Aims How did we learn and experiment…? 1) The main work of the unit this year is to enable Twaweza to

learn as much as it can from its own experiments, from external evaluations, as well as from other available international evidence.

1) Available evidence from our internal monitoring and short-term evaluations, from external long-term evaluation, and internal smaller experiments was used extensively in setting the problems & successes for the new strategic period.

2) The LME unit in collaboration with Uwazi designed and oversaw the qualitative study of KiuFunza, yielding significant and useful results, deepening our understanding of the intervention.

3) We also continued to engage MIT (Professor Tsai) and

Princeton (Professor of Lieberman) in explorative studies to lay the groundwork for possible future research / evaluations on young people’s engagement with government, and on parental and community support of basic education.

42

Communications: Narrative & Story The Communications unit extended and better articulated its support function to the rest of the organization in 2014. The support element is reflected in the large number of no budget items in the 2014 plan, and yet the unit plays a role in amplifying the work of every programmatic unit within the organization. The Communications unit also spent time learning about the effectiveness of different approaches to engagement and different pieces of content – allowing for a more targeted approach to how we communicate. For the 11 Sauti za Wananchi launches, we created and distributed policy briefs and press releases (each in two languages) for a total of 55,000 publications of briefs; reviewed seven issues of partner magazines; worked with partners to produce four dialogues broadcast on 14 local radio stations, 13 episodes of a TV series, 11 concerts, three radio and TV public service announcements and four different posters; edited and designed 12 monitoring briefs; produced one documentary and three shorter films / clips; and input into a number of different Uwezo products. Indeed, 2014 was a very busy year for us. In Uganda the work consisted of four specific calls to action over the course of the year, reflected in a range of interventions. These included the Ni Sisi concert tour, consisting of five concerts, 245 radio ads, 112 TV ads, and other publicity activities; production and extensive distribution of public service announcements by artists repeating the calls to action; a TV show featuring different artists in schools; and six concerts with Gravity Omutujju – a Ugandan ‘Luga flow’ artist. Preliminary monitoring data indicate positive findings, but the design of implementation meant that comprehensive monitoring of effects was not possible. Again in Uganda, which has the most developed network of video halls / informal cinemas, content was inserted into breaks in DVDs of popular films and series and made available to video libraries or distributors. In leveraging existing infrastructure, the idea was to use these distributors, generally concentrated in the urban centres of districts, to reach videos halls or informal cinemas which are more dispersed. We wanted to see if we could tap into the film supply chain and achieve significant reach for fairly low cost. Monitoring of this activity found poor results, with difficulty in reaching a sufficient sample of video halls and only half of those being able to recall seeing Twaweza content. Subsequently a new approach was tried through a different partner, and though independent verification is not yet complete, self-reporting includes precise data like photographs and GPS coordinates and shows positive results in terms of reach. An entirely new and fairly substantial series of work around communicating the monitoring, evaluation and learning aspects of Twaweza was initiated. The monitoring briefs series on different areas of work was instituted with a defined visual identity. An LME study on stock-outs in Tanzania was launched and disseminated to key stakeholders in the sector. The Acting Director of the Medical Stores Department, the body responsible for medicine supply and distribution, was a speaker at the event. And finally we supported the LME Unit with organizing a program of events for external visitors to share insights and knowledge with civil society in Tanzania. Moving forward, greater specialization and role specificity has been built into the new Twaweza structure and strategy. We also know that we need to continue to focus on monitoring to ensure that we can really tell the full story of our interventions. One piece of feedback we hear quite often is that we fail to pull our work together so that the lessons are easy to decipher; unearthing the story is a bit of

43

a treasure hunt – you know there is something spectacular to be found, but the finding of it remains a challenge.

Revolution songs We know that expectations of government among citizens, particularly young people, are low. Governance norms involve unaccountable relationships and an extractive approach to ruling. We also know that these types of norms are extremely hard to shift given their basis in years of experiences interacting with the state and because they are influenced by so many different factors that are both knowledge driven, emotional and based on people’s sense of internal and external efficacy. At the same time, we continue to be intrigued by the influence that musicians exert on their fans and believe that this could be an exciting route to behavior change and reframing norms, particularly when it comes to apparently intractable problems. Across the year in Uganda, we embarked on a series of activities with musicians, building on our learning and constantly tweaking ideas to ensure we were cost effective and met our objectives. The core idea was to shift young people’s norms and expectations around their own responsibility in making change happen. The main challenges to implementation were really in coordination. Delay in one activity throws off the entire package and means that you cannot have the same certainty about effectiveness. The monitoring of each activity was rich and informative but thus far does not allow us to tell a concrete and coherent story of overall effects. However we did find that:

Ni Sisi concerts: 93% of the concert goers said that they learned something from attending the

concerts and 87% recalled messages about contributing to positive change in society;

Airing PSAs: we independently verified that they were aired through Ipsos;

TV show: focus groups were run during school visits but data are not yet in.

Most importantly, a large representative survey of young Ugandans found that:

93% of young Ugandans say that their favorite artist is an important role model;

A large proportion of respondents (89%) say they like their favorite artist because of the issues that

s/he brings up

An equally large proportion (91%) say that if their favorite artist encouraged citizens to organize to

take care of problems in their communities, they would be more likely to join a group themselves;

87% say that their favorite artist is person like them;

Three-quarters of respondents (75%) say that if their favorite artist encouraged citizens to vote in

the next election, they would be more likely to do so themselves.

These findings bode well for the potential impact of our work with artists. Moving forward we will need to think more intelligently and holistically about monitoring, particularly around effects, of this type of work. This will require additional human resources, to ensure that these complex multi-faceted campaigns can be rolled out in a coordinated and effective way.

44

Reflections on Annual Plan: Communications / Public Engagement

In the 2014 Annual Plan we articulated: How did we do in 2014?

Major Aims Did we keep with the major aims…?

Creative and action-orientated content

1) Ensure all Twaweza and partner content is creative, imaginative and effective.

2) Learning about what constitutes effective content for social and behaviour change.

Creative and action-orientated content

1) The Communications Unit continues to function as the custodian of Twaweza and Uwezo content, undertaking in-depth editing and review of all Twaweza outputs. Notable achievements include: created and distributed 11 policy briefs (each in 2 languages) and press releases, for a total of 55,000 publications; reviewed 7 issues of partner magazines; worked with partners to produce 4 dialogues broadcast on 14 local radio stations, 13 episodes of a TV series, 11 concerts, three radio and TV public service announcements; produced one documentary and three shorter films / clips.

2) 2014 saw deeper engagement with a smaller group of content partners, editorial discussions, content review and input of Twaweza data and research were among the roles played by the Unit.

3) Established contract to pre-test all mass communication

content.

Engaging relevant audiences for content

1) Supporting public and policy engagement such that Twaweza products reach relevant audiences.

2) Learning about what constitutes effective engagement.

Engaging relevant audiences for content 1) Developed hard and soft copy dissemination systems.

2) Regular events held to release Twaweza research, foster discussion and share expertise, 19 events held in the year.

3) Commenced trial to distribute information to bible

discussion groups. 4) New partnership with Jamii Forum, Tanzania’s largest social

media channel. 5) New series of online monitoring briefs to share LME

monitoring data with interested parties. 6) Organised Twaweza’s largest strategic engagement event

to mark departure of Founder and arrival of new leadership.

Experimenting to learn

1) Trying new things, small scale, learning about what content is effective in what circumstances. Targeting young people.

Experimenting to learn 1) Small-scale trials proved too labour intensive so a series of

larger activities were completed, such as trialling two different models of holding concerts with key messages,

45

keeping value for money and maximum exposure in mind.

2) Using basic infographics to present Uwezo and Sauti za Wananchi data.

3) Improved quantity and quality of interventions but still challenges in implementation and coordination result in delays.

What has changed since previous plan? How did we implement the changes…?

1) Removal of mass communication work ‘Ni Sisi’ except in Uganda with artists.

2) Clearer breakdown of support function to other units.

1) We produced a lower volume of mass content to focus on the year of learning. Exception was Uganda, initiatives targeting young people.

2) Supported all units as required, including advance planning with units.

Risks and Risks Management Did we face the risks and how did we address them…?

1) Human resource challenges of recruitment and absence of dedicated staff in Kenya and Uganda.

2) Make use of existing capacity from other units and emphasize the support function of the Unit.

1) Worked extensively with Programs Officer in Uganda to implement a full program of work. A more modest scope of support lent to Uwezo.

2) In Tanzania, ongoing attempts to recruit an additional staff member, four sets of interviews.

3) Constant re-orientating and re-focusing when it seemed

activities were too much of a challenge to implement.

Learning and Experimenting Aims How did we learn and experiment…?

1) Learning about mass media and behaviour change through stories of change.

2) Testing effective ways to engage young people.

3) Extending the reach of video beyond TV.

1) Radio stories of change completed at year end and production of TV/radio series ongoing. Monitoring will be conducted subsequently.

2) Youth engagement was trialled in Uganda and focused on work with artists. We now have a substantial body of work and monitoring of different models of and outputs around artist engagement, such as engaging artists to promote calls to action, and understanding more about musicians as persuaders.

3) Production of Twaweza’s first feature film with a Ugandan celebrity.

4) Implementation delays meant communication efforts were not as holistic and multi-channel as planned.

5) Tried to reach informal cinemas / video halls through new

network of video distributors in Uganda with limited success. Subsequently worked with a new distributor which distribute directly to video halls every two weeks. Preliminary results look promising.

46

Operations: Narrative & Story The Operations unit continued its support function to the rest of Twaweza, in particular supporting smooth, efficient and cost-effective procurement of goods and services for implementing program units. We were integral to identifying an exciting supplier for KiuFunza payments, providing backend logistical support to the Uwezo assessment and systematizing regular/recurring services required by the organization, such as printing. In addition the Operations unit has a specific role in recruiting and retaining staff and managing smooth and efficient administrative systems. Human Resources The 2014 staff establishment required a total of 59 employees across East Africa. In January 2014, five of these positions were vacant. Given historical difficulties in attracting and retaining high quality staff, we employed a range of recruitment strategies including head hunting, advertising and offering bonuses to Twaweza staff for candidates they recommended who were retained. Nonetheless only two of the vacancies were filled in this way. Towards the second part of the year recruitment was suspended because contracts could only be offered until the end of the year since the new strategy would drive a new organizational structure with different positions. The Management Team of Twaweza was restructured to tie in with the new strategy, through the support and expertise of an external consultant. We created a new tier of Directors, who then built their teams, to a final staff establishment of 72 positions across the three countries. Offer letters were issued to staff for these positions towards the end of 2014, after consultation and discussion. Whilst the focus was on Twaweza’s needs first, it was also important to acknowledge and incorporate staff inputs; sharing information at different key moments ensured that the team was kept abreast of the many changes. In the end, the organization did not retain four members of staff and a further four departed of their own volition. Offices and Procurement In Nairobi, Twaweza relocated its offices in July 2014. All IT systems and office setup were completed on time so that operations could continue during the transition. In Uganda, a theft late in the year resulted in the loss of assets worth USD 16,386. However, Twaweza’s rigorous systems and policies came into their own during the incident and the insurance payment covered USD 11,698 while the security company will cover 50% of the remaining loss. During this difficult time, the team really came together to work through the loss of computers and equipment with minimal interruption to activities. To avoid such incidents in the future, CCTV and alarm systems have since been installed. Following a review of financial and administrative processes around the Uwezo assessment by students from the University of Michigan’s Ross School of Business, a number recommendations were made, and implemented, related to how we could strengthen the processes related to the assessment and the incredibly varied work related to managing and implementing Uwezo. Office assets were thoroughly checked and reconciled with financial records. A total of 70 assets that were found to be broken beyond repair were disposed of as per policy. The list of assets was exported to the new asset management software, SAMANAGE, allowing for better management and tracking of assets. For Twaweza, 2014 was set to be a transformational year. A new strategy, a new leader, new ways of working, new learning and new structures to name just a few of the transitions.

47

Reflections on Annual Plan: Operations

In the 2014 Annual Plan we articulated: How did we do in 2014? Major Aims Did we keep with the major aims? Human Resources:

1) Have the right people in place particular in some key units eg. LME.

Policies and systems

1) Continue to ensure that systems and processes are robust and of high standards, and well documented.

2) Maintain value for money and accountability in the implementation of activities

Human Resources: recruitment strategies included: 1) Engaging various recruitment firms.

2) Twaweza employees where a motivational bonus was introduced if one manages to communicate good candidate and is considered for the job.

3) Liaising with partners and continue advertising on the website.

4) One position for LME unit was filled, but still crucial positions could not be filled. Since Twaweza was in the preparation of the new strategy more search was put on hold until the new organization’s structure is established.

Policies and systems

1) Twaweza already had effective systems and processes in place. To ensure that everybody is aware and follow them, Ops supported by Head’s office organized training on procurement, contracts and payment workflows. Four Michigan students were also invited to study specifically the way various workflow work and advice the organization. Twaweza received recommendations and started implementing some of the advised e.g. when a request is not as per standard it should be communicated immediately to the previous level and it should not be returned to the initial stage unnecessary in order to avoid delays. All workflows were to be incorporated into salesforce system by January 2015.

2) Twaweza continued to ensure that whatever is purchased is

according to set and approved requirements. This was made possible by involving key stakeholders with different expertise in any activity. Maintained high level of transparency, competition in all procurement and recruitment processes to ensure that the organization is informed of different prices and quality of required goods and services available in the market in order to make informed decisions.

What has changed since previous plan How did we implement the changes…?

1) To engage a consultant to carry out a job evaluation of all positions as per staff establishment.

1) Twaweza strategy was ending December 2014 therefore, a new strategy was already developed by end of quarter three. From the new strategy, it was clear that the organization structure was going to change to accommodate the new strategy. This meant changes in job description which is among key document required during job evaluation exercise. In this case, it was decided to postpone the exercise until 2015, in order to evaluation new jobs to ensure that the exercise is meaningful.

48

Risks and Risks management Did we face the risks and how did we address them…?

1) Job evaluation exercise is potentially sensitive.

1) The activity was postponed.

Learning and Experimenting Aims How did we learn and experiment…? 1) Engage potential candidates even

when it was clear that they did not fully meet job specifications but give them an opportunity to work with Twaweza with the aim of mentoring them so that they can gradually learn, adapt and meet the set job specifications

1) About two positions were filled under this arrangement. Mentoring was done, and it proved to work at a rate of 75%. The remaining 25% means that candidates required more time to catch-up than Twaweza had expected. Initially, Twaweza had thought that the mentoring and hence one meeting set specifications could take upto six months, but it appears that this could take up-to 2 years which means additional costs to the organization as it may be forced to consider consultants to support or/and requirements of more support and guidance from unit head than anticipated.

2) However, this experiment could not be practiced fully because as

new strategy was being developed, recruitment was suspended until we were clear of the new organizational structure for 2015.

49

Governance and Management: Narrative & Story A global search, through a premier recruitment firm, for the best candidate to lead the organization into a new strategic period and institutionalize Twaweza beyond our founder, was successful. By December 2014 we had identified a new Executive Director and begun a series of induction meetings with the management team and exiting Head. The search method also inspired us to think about our own recruitment strategies, and how these could be sharpened to identify the best candidates out there. We also needed to ensure that we created and reserved space for deep thinking and homework in articulating the new strategy. This process was begun in 2013 and led us to demarcate 2014 as the year of experimenting and learning. The aims of this year were twofold: to ask hard questions about current practice to ensure that we really were implementing in the most effective and efficient manner. Even those things that we had been doing for a number of years needed review and reflection. Secondly, we wanted to test and trial new ideas in small ways so as to feed into the new strategy. Although some of these plans did not materialize, we were true to the spirit of reflection, adaptation and learning. We were conscious of the need to reconsider our management style and structure to avoid over-centralisation and to ensure we had enough variety of positions to attract senior candidates to implement ambitious and thoughtful new areas of work. We recruited a specialist consultant who devised a new structure, more in line with the new strategy and with an emphasis on devolving powers from the Head. After feedback this was finalized and adopted. Another major achievement was our independence. Since inception Twaweza has been housed by Hivos, which provided the incubation space for our early development. As we matured as an organization and became more ready to stand on our own, the plan was always for the organization to be registered independently in all three countries. At the time of writing, this has been successfully achieved with Twaweza fully registered in Tanzania and with certificates of compliance to operate in Kenya and Uganda. Each of these on its own is a mammoth task, but in addition to implementing the 2014 Annual Plan, drafting the strategy for 2015-2018, and ensuring that we remain agile enough to address any exciting opportunities, it was intense, exhilarating, exhausting, and (surprisingly or unsurprisingly) achieved.

50

Reflections on Annual Plan: Governance and Management

In the 2014 Annual Plan we articulated: How did we do in 2014? Major Aims Did we keep with the major aims…? 1) Second and third phases of SalesForce rollout,

including contracts, procurement and payments workflows. In the second phase internal request systems for administrative aspects will be set up, to manage operations across the three countries. Use of SalesForce as the main internal system for communication, collaboration, and workflows.

2) Twaweza’s internal processes will also become more transparent and free for the public to scrutinize. We are looking to fully implement IATI and IRIS standards.

1) SalesForce continues to be the central communication platform. Active use of SalesForce continues to increase among staff across the three countries, with most staff logging in every day to keep abreast of major announcements.

2) A South African firm was selected from our procurement

processes, and Q3 begun by customize these workflows. They were completed with slight delay, and procurement and payments are still in development.

3) Promotion of Twaweza values and developing of a

new Theory of Change. 3) Twaweza published data on budgets and expenditures in

line with the International Aid Transparency Initiative (IATI) standards. The data is available through the IATI registry. This is a first step in a longer process, which will see Twaweza data further disaggregated, taking full advantage of the potential of the IATI standard, and fully meet best practice in terms of the data being published. Nevertheless, it is a significant step towards making Twaweza one of the most transparent organizations working in international development.

4) 2014 was devoted to learning and developing a new strategy. This included a number of internal sessions, review of analytical and planning models and available evidence, and joint thinking with sharp minds from around the region and the world. A new strategy was developed, which includes a revised Theory of Change, new values and new organizational setup.

What has changed since previous plan? How did we implement the changes…? 1) The new Twaweza 2015-2018 strategy will be

discussed, drafted, and finalized. 2) Availability of as many internal documents online,

such that we are accountable to the citizens of our countries and to our donors, will be key to achieving this

1) The year started with an engaging and inspiring strategy session with key thinkers who helped us to challenge and then better articulate some of our thinking. Much of the year was devoted to drafting, writing and vetting of the new strategy.

2) Through developing a GlassPockets-inspired framework, we were able to gauge ourselves against these high global standards on transparency and accountability. We were able to see that in terms of compliance we actually performed very well in ensuring that as many of our internal documents and processes were available online to ensure openness and accountability.

51

Risks and Risks Management Did we face the risks and how did we address them…? 1) Implementation risks when staff do not follow

workflows. 2) A three-day training on greater delegation was held in

Dar es Salaam in January 2014 for all staff. The intention was to ensure that all staff were aware and had time to practice their role in the workflows, allowing them to take greater ownership of processes and tasks. Similar sessions were led by the Head in Uganda and Kenya to ensure the entire team was trained. These same workflows, of contracts, payments, and procurement were then customized for Twaweza’s workflow management platform, SalesForce.

Learning and Experimenting Aims How did we learn and experiment…? 1) Increased delegation of authority and responsibility

to perform day-to-day tasks expected of all unit managers.

2) Additionally, using reverse logic in our day-to-day

activities, starting with the goals we aim to achieve, and working backwards to identify our starting points.

1) Greater delegation of authority meant the Head could free up time for strategic engagement that would be of benefit to Twaweza prior to his departure. Together with the management team, who exercised greater responsibility, it allowed for this to be a time dedicated to shaping the future of the organization.

2) The reverse logic framework was used as a planning

methodology for our November Annual Planning Retreat and was met by the team with great appreciation. Though the task was a mammoth one, staff were able to comprehend why this way of planning makes sense, and the 2015 Annual Plan was developed using this thinking.