abbreviations - home page - uk · web viewevaluation of the post programme in uganda report...

Evaluation of the POST programme in Uganda, July 2015.

Evaluation of the POST Programme in Uganda

Report prepared by Dr Chandrika Nath, Deputy Director, Parliamentary Office of Science and Technology (POST) and Dr Kerry Holden, external consultant, July 2015

1 |


TABLE OF CONTENTS

ABBREVIATIONS.............................................................................................................................4

ACKNOWLEDGEMENTS...................................................................................................................4

EXECUTIVE SUMMARY....................................................................................................................5BACKGROUND........................................................................................................................................5AIMS AND OBJECTIVES OF POST’S PROGRAMME IN UGANDA............................................................................5THE NATURE OF THE PROBLEM..................................................................................................................5FINAL EVALUATION OF THE PROGRAMME....................................................................................................6LESSONS FOR THE FUTURE......................................................................................................................10FUTURE DEVELOPMENTS........................................................................................................................11

1. INTRODUCTION.....................................................................................................................131.1 WHY SCIENCE AND TECHNOLOGY MATTER TO PARLIAMENTS..............................................................131.2 THE POST AFRICA PROGRAMME ..................................................................................................141.3 CONSULTATION PROCESS FOR PROGRAMME DESIGN.........................................................................141.4 RATIONALE FOR WORKING IN UGANDA..........................................................................................151.5 THE FINAL EVALUATION AND THIS REPORT.....................................................................................16

2. THE POST PROGRAMME IN UGANDA....................................................................................182.1 AIMS AND OBJECTIVES...............................................................................................................182.2 SUMMARY OF ACTIVITIES TAKING PLACE UNDER THE PROGRAMME......................................................18

3. FINDINGS: IMPACT ON PARLIAMENTARY STAFF....................................................................283.1 THE NATURE OF THE PROBLEM....................................................................................................283.2 HOW HAVE THINGS IMPROVED ?..................................................................................................303.3 CHALLENGES/AREAS FOR IMPROVEMENT.......................................................................................34

4. FINDINGS: IMPACT ON MPS..................................................................................................364.1 THE NATURE OF THE PROBLEM....................................................................................................364.2 HOW HAVE THINGS IMPROVED....................................................................................................374.3 CHALLENGES/AREAS FOR IMPROVEMENT.......................................................................................41

5. FINDINGS: BENEFITS TO POST...............................................................................................445.1 THE NATURE OF THE PROBLEM – THE SITUATION IN 2009................................................................445.2 HOW HAVE THINGS IMPROVED?...................................................................................................445.3 CHALLENGES/AREAS FOR IMPROVEMENT.......................................................................................45

6. FINDINGS: WIDER BENEFITS TO DONORS..............................................................................466.1 THE NATURE OF THE PROBLEM....................................................................................................466.2 HOW HAVE THINGS IMPROVED?...................................................................................................466.3 CHALLENGES............................................................................................................................47

7. LESSONS FOR THE FUTURE....................................................................................................497.1 BUILDING INSTITUTIONAL CAPACITY..............................................................................................497.2 ACTIVELY ENGAGING SENIOR STAFF...............................................................................................497.3 REDUCING LONG-TERM RELIANCE ON INTERNATIONAL EXPERTISE........................................................507.4 EXPERTISE IN CAPACITY BUILDING AND PARLIAMENTARY PROCESSES....................................................507.5 FOCUSING ON A SINGLE COUNTRY................................................................................................517.6 THE RESTRICTIONS OF FOCUSING ACTIVITIES ON PARLIAMENTS...........................................................527.7 PREPARING FOR ELECTIONS.........................................................................................................537.8 WORKING WITH PARLIAMENTARY STAFF........................................................................................537.9 ACKNOWLEDGING THE RISKS OF DEVOLVING MANAGEMENT..............................................................537.10 REDUCING CONFUSION THROUGH LACK OF COORDINATION...............................................................54

2 |


8. FUTURE DEVELOPMENTS......................................................................................................568.1 RECOMMENDATIONS FOR FURTHER ACTION IN UGANDA...................................................................56

9. REFERENCES..........................................................................................................................59

APPENDIX A: RESEARCH DESIGN...................................................................................................60METHODOLOGY....................................................................................................................................60METHODS...........................................................................................................................................60RESEARCH ETHICS.................................................................................................................................62

APPENDIX B: GENERIC INTERVIEW SCRIPT....................................................................................63GENERIC INTERVIEW SCRIPT....................................................................................................................63

APPENDIX C: PEER REVIEW OF PARLIAMENTARY BRIEFINGS.........................................................64SUMMARY OF FINDINGS.........................................................................................................................64METHODOLOGY....................................................................................................................................65QUANTITATIVE FINDINGS........................................................................................................................68QUALITATIVE FINDINGS..........................................................................................................................69LIMITATIONS OF THE ANALYSIS................................................................................................................71

3 |


ABBREVIATIONS

ASADI African Science Academy Development InitiativeATPS African Technology Policy Studies NetworkBTVET Business, Technical, Vocational Education and TrainingDFID Department for International DevelopmentINASP International Network for the Availability of Scientific PublicationsNCHE National Council on Higher EducationNGO Non-Governmental OrganisationOECD Organisation for Economic Cooperation and DevelopmentP of U Parliament of UgandaPOST Parliamentary Office of Science and TechnologyRS Royal SocietyS&T Science and TechnologySTI Science, Technology and InnovationUNAS Ugandan National Academy of SciencesUNCST Ugandan National Council on Science and TechnologyUNESCO United Nations Educational, Scientific and Cultural Organisation

The following abbreviations are used to classify interviewees: Ug CS Ugandan Civil SocietyUg G Ugandan GovernmentUg MP Ugandan Member of Parliament or former Member of ParliamentUg P Ugandan Parliamentary staffUg Sc Ugandan Scientific CommunityUK NGO UK based Non-Governmental OrganisationUK P UK parliamentary staffUK MP UK Member of the House of CommonsUK Sc UK scientific community

ACKNOWLEDGEMENTS

We would like to thank the 41 interviewees based in the UK and Uganda who gave their time to participate in the evaluation. Special thanks are due to Harriet Nabunnya for organising and conducting interviews in Uganda. We would also like to thank interviewees and other stakeholders who took part in a data analysis workshop in late March 2013, and all those who reviewed policy briefings that had been written by staff from the Parliament of Uganda.

4 |


EXECUTIVE SUMMARY

BACKGROUNDParliaments play a crucial role in scrutinising the work of government and debating and passing laws. They also act as an important forum for public debate. Members of parliament (MPs) are increasingly required to handle issues of a highly technical nature and to keep track of modern developments in science and technology (S&T) to fulfil this role.

MPs rely on parliamentary staff to provide impartial, policy-relevant information. In many cases this information is of a highly technical nature and needs to be provided by specialist staff with scientific backgrounds. In the UK, the Parliamentary Office of Science and Technology (POST) has provided objective advice and support to UK parliamentarians for 20 years.

African MPs have far fewer specialist staff to support them than their counterparts in the developed world. However, S&T is moving rapidly up the agenda in African countries, where it is seen as crucial to socio-economic development. MPs have to tackle issues ranging from climate change to biotechnology, but they lack the skills, knowledge and resources needed to make informed decisions.

AIMS AND OBJECTIVES OF POST’S PROGRAMME IN UGANDAIn 2007 POST received a grant of £240,000 from the Gatsby Foundation to initiate the POST Africa Programme. The programme focused on the Parliament of Uganda, and its objectives were set after consultation with potential beneficiaries, UK parliamentary colleagues and non –governmental organisations (NGOs). The aims of the programme were to help the Ugandan parliament find ways of improving its scrutiny of S&T issues and to build up a body of knowledge which would guide other work in this area.

To achieve these aims, a number of different outcomes were identified, each focusing on a specific target group:

Strengthening the ability of Ugandan parliamentary staff to support MPs on issues related to S&T by: developing their skills in identifying and using S&T resources; building their confidence in handling S&T issues; and helping develop their networks.

Strengthening the ability of Ugandan MPs to scrutinise issues relating to S&T by: building up their interest, their take-up and use of resources, and their networks; and building the profile of the Standing Committee on Science and Technology.

Building POST’s understanding of S&T issues in the developing world and its network of contacts in the developing world, learning about working methods applicable in the UK context.

Greater awareness amongst partner organisations of the importance of building S&T capacity in parliaments, and greater understanding of how to interact with parliaments.

The programme ran from September 2007 until September 2012. It was managed by POST, working in close collaboration with the International Network for the Availability of Scientific Publications, a UK based NGO.1 Much of the day to day running of activities was devolved to the Ugandan National Academy of Sciences (UNAS), particularly in the latter part of the programme.

THE NATURE OF THE PROBLEM To understand the needs of the Ugandan parliament in greater depth, POST conducted detailed research in years 1 and 2 of the programme. This consisted of:

1 www.inasp.info

5 |

http://www.inasp.info/


Interviews and focus groups with MPs, staff and scientists. An expert review of internal policy briefings on S&T produced by Ugandan parliamentary

staff. An analysis of a selection of Ugandan parliamentary debates on S&T. A review of the work of the Standing Committee on Science and Technology of the

Parliament of Uganda (referred to hereafter as the S&T Committee).

This research showed that the Parliament of Uganda was failing to scrutinise government effectively in areas related to S&T and that it was not using research evidence effectively in its scrutiny of government. Ugandan MPs voiced enthusiasm for S&T, but this very positive attitude did not always translate into action. Debates on S&T were poorly attended and contained many factual inaccuracies. Internal briefings on S&T produced by parliamentary staff made poor use of evidence and contained scientific inaccuracies. Some key underlying factors were that:

MPs and staff lacked confidence in handling S&T, and MPs did not see S&T as relevant to daily life.

The S&T Committee, established in 2002, had poor attendance rates and a low profile within and outside parliament.

Staff lacked the skills and resources needed to provide MPs with objective, well referenced briefings on S&T. In particular they lacked skills in identifying reliable evidence on the internet.

Both MPs and staff had limited networks with the external scientific community, particularly the scientific community in Uganda.

The results of this research were made publicly available in the form of a joint report (referred to hereafter as the ‘baseline study’) by POST, the Parliament of Uganda and UNAS.2 The direction of the POST programme was continuously revised, based on findings from this research and from ongoing evaluation of activities.

FINAL EVALUATION OF THE PROGRAMMEThe POST Africa programme ran from 2007 to 2012. From November 2012 to May 2013, the programme was evaluated to assess its impact and influence. This findings set out in this report are based mainly on interviews conducted with stakeholders in the UK and Uganda, once the programme had drawn to a close. These interviews were supplemented by desktop research to assess how the quality of policy briefings and the work of the S&T Committee have evolved since the start of the programme. In addition, a focus group was held with UK-based donor agencies in May 2013. For more information about research design, please see Appendix A.

The evaluation findings below are structured around the four main programme objectives, as set out above.

BENEFITS TO UGANDAN PARLIAMENTARY STAFF

The data indicate that the ability of Ugandan parliamentary staff to support MPs on issues related to S&T has improved over the course of the POST programme:

There has been an improvement in the quality of internal briefing papers on S&T produced by staff for MPs. Experts rated samples of internal briefings from before the start of the programme and towards the end of it. Ratings indicate that subject coverage, scientific

2 Use of Scientific and Technological Evidence within the Parliament of Uganda, a joint report published by the Parliamentary Office of Science and Technology, the Parliament of Uganda and the Ugandan National Academy of Science, October 2011.

6 |


accuracy, use of evidence, objectivity and clarity have all improved. However, the experts’ comments indicate that there is still room for improvement, particularly in referencing information, and in using the best available evidence. Also, the briefings contain policy recommendations; the experts questioned where these originate and whether it was appropriate for parliamentary staff to make policy recommendations to MPs (see Appendix C of main report)

The number of staff with specialist knowledge of S&T has increased. At the start of the programme there was only one parliamentary researcher allocated to working on S&T issues. There is now a dedicated section for S&T within Department of Research Services. It is currently staffed by five researchers, but the aim is to employ up to six researchers with specialist knowledge in areas of S&T identified as important to national development. However, in the interviews, a lack of resources for training both existing staff and new recruits was highlighted as an issue.

Staff are better able to access information on S&T. in the interviews, staff said that they had access to an increased range of resources (such as databases and journals) as a result of linkages made with other donors through POST. Managers report that staff are better at finding information on the internet. These observations correlate with the observed improvement in use of evidence in the internal policy briefings. However, staff say that accessing locally relevant content is still an issue.

Staff have a more positive attitude to S&T. Interviews show that levels of interest in S&T have increased. One clerk said, “Now, anything to do with science and technology, I am very passionate about it … All my colleagues want to be a clerk for science and technology.” Staff are less afraid of handling S&T: one parliamentary researcher told us that “science was always thought of as something very technical and complicated ... but [POST’s] training was able to show that it can be … communicated easily and simply … in simpler terms for people, for all people to understand.”

Staff have better networks. Staff who were directly involved in POST’s activities (such as MP-scientist pairing) say they have better networks with the outside world. According to one parliamentary researcher, “Just the fact that I networked with people through the pairing scheme means it’s much easier for me to contact these people, because sometimes without creating a rapport with people it’s quite difficult to get information.”

The main benefits of the POST programme appear to be focused on those individuals who participated directly in POST’s activities, rather than the institution as a whole.

BENEFITS TO UGANDAN MPS

The S&T Committee is more active than before the programme started, and it has a higher profile. In addition, comments made by interviewees indicate that parliament as a whole is more open to discussing S&T and that levels of interest in S&T have increased.

The S&T Committee is more active and has a higher profile. In the eight parliament (2006 to 2011) the S&T committee had a low profile and was not very active. For example, it did not scrutinise any bills and only produced one report. In the ninth parliament (2011 to present) it has been much more active. For example, it has played a leading role in driving forwards the National Biotechnology and Biosafety Bill, first presented to parliament in November 2014. In the second session alone (July 2012 to May 2013) three of its reports were debated by parliament. There is a marked improvement in structure and clarity of these reports in comparison with the single report available from the eighth Parliament. However, the underlying analysis behind recommendations made in the reports is not always made clear.

Parliament as a whole is more open to S&T. Interviewees from government, parliament and the scientific community in Uganda say that parliament is now more open to considering

7 |


S&T issues. According to one government official, “Engagements like ones with POST helped them appreciate that science is important and that there is need for something to be done at a policy level to address these.”

MPs have better access to resources on S&T. MPs say that they are more aware of what their researchers can offer and that they are making better use of their research department. According to one MP, “One thing that it [POST programme] encouraged me to do was to seek knowledge from our research department, especially people handling science.” One member of staff told us that “They [POST] brought us closer to our clients, the members of parliament.”

MPs have better networks with external scientists. MPs directly involved in the MP–scientist pairing scheme say the scheme has brought them closer to the scientific community in Uganda. Some of the MPs who took part in the scheme in 2008 are still in touch with the scientists they were paired with.

It is difficult to attribute any changes seen in the Ugandan parliament to a single cause, as there are many donor initiatives running in parallel. Moreover, at national level, S&T has continued to rise up the agenda, and this is also likely to have influenced attitudes towards S&T. According to one parliamentary researcher:

“Cause and effect might not be pinned down but [they are] broadly positive – [there are] lots of initiatives, not just POST. For me I think the most important part of it was the gaining of skills and knowledge, which we benefitted from activities which were arranged by POST. That was very, very important. I may not separate what I got from another institution and what I got from POST – the cause and effect may not be known but we did benefit.”

WHAT BENEFITS HAS THE PROGRAMME HAD FOR POST AND THE UK PARLIAMENT?POST’s core remit is to provide scientific advice to UK parliamentarians. Although the main focus of POST’s programme was to improve scrutiny of S&T within the Parliament of Uganda, POST and the UK parliament have benefited in a number of ways:

The programme has furthered understanding of best practice in the UK parliament. By exploring how to have an impact in the Parliament of Uganda, POST has learned more about how to maximise its impact in the UK. Lessons learned from carrying out the baseline study in Uganda helped inform POST’s own evaluation exercises in 2009 and 2012. One MP said that the baseline study contains many lessons for the UK Parliament: “In many ways, from looking at the report and talking to the staff at POST, some of the most pertinent recommendations are actually also quite applicable to the UK.” (UK MP1)

UK parliamentary staff are better able to work with developing country colleagues. As a result of the POST programme in Uganda, parliamentary staff (in POST as well as in other departments) have interacted more closely with their counterparts in Uganda. This has helped UK staff appreciate the constraints that colleagues in developing countries face and has given them better skills in communicating with overseas colleagues. The UK parliament hosts many incoming delegations from these countries. As a result of the POST programme, POST plays an active role in working with these and ensuring that the role of S&T, and of research evidence more generally, is highlighted during their visits.

The programme has extended POST’s network of contacts in the developing world. POST has worked with a wide range of NGOs, think-tanks, research organisations and policymaking institutions as part of its programme in Uganda. This has put POST in a better position to provide accurate and unbiased coverage of issues in developing country. For example, with the help of contacts made as part of the POST Africa programme, POST is designing a horizon scanning exercise which will help the office identify development issues relating to S&T which are relevant to the UK Parliament.

8 |


The programme has boosted POST’s reputation in the UK and overseas. POST’s work in Africa has been praised on the floor of the House of Commons. Andrew Miller, former MP for Ellesmere Port and Neston (1992-2015), cited the programme in a debate about the future of POST in November 2012: “This House has influence well beyond the shores of the United Kingdom, and when I was in Uganda with the select committee I was delighted to meet a fellow who had been on one of the POST fellowships through that scheme.”3

THE PROGRAMME’S IMPACT ON DONORS

Before the POST programme started, very little parliamentary capacity building work focused specifically on S&T or looked at how research evidence was used in a parliamentary context. Interviewees say that the POST programme has had a positive impact in this area in several ways:

POST has helped other partners appreciate the subtleties of working in parliaments. Many organisations are working to improve the use of evidence in policymaking. POST has helped such organisations appreciate the role parliaments play in the policymaking process. According to one donor, “Everyone talks about parliament being there to scrutinise government but in actually recognising the subtleties of what that means in practice, I think that’s been very, very helpful interacting with POST.” (NGO 09)

POST’s research in Uganda has influenced other partners’ work. A number of interviewees have pointed to the baseline study as the first detailed, well defined analysis of how evidence is used in a parliamentary context. According to one donor, “That was a very influential report, and that report keeps coming up in conversations with people from other sectors. I think it has had an influence there.” (UK NGO 07)

Partners have built upon POST’s initial work. The POST programme generated a high degree of interest: during the course of the programme, additional funds were brought in that amounted to over 40% of the original grant, not including in-kind support.

Since the POST programme started, interest in promoting evidence-informed policymaking in the developing world has increased markedly. In December 2012, the Department for International Development (DFID) launched a call for proposals to build the capacity of decision-makers in low and middle income countries to use research evidence and rigorous data to inform their decision-making, with total funding of £13 million. One donor who was interviewed said the POST programme had played a role in influencing DFID:

“I think it was really important to start that ball rolling … Basically it was only because of that that the whole INASP programme started, and that’s now, I think, doing some really great work. And I think you could make the argument that it was because of that INASP work, or that that played a relatively big role in influencing DFID now to launch a new call for work to do specifically that.”

However, the interviewee did not think the POST programme had a significant impact in Uganda:

“I’m not sure that the POST Africa project did an amazing job of actually dealing with capacity gaps. I wouldn’t say it did. I think it identified a problem, highlighted it, and then other people have gone on to deal with that better.”

Below we explore some of the factors which could have increased the impact of the programme and discuss lessons learned for the future.

3 http://www.publications.parliament.uk/pa/cm201213/cmhansrd/cm121108/debtext/121108-0002.htm The Member is referring to a meeting he had with a Ugandan parliamentary researcher who had spent three months at POST on a commonwealth professional fellowship.

9 |

http://www.publications.parliament.uk/pa/cm201213/cmhansrd/cm121108/debtext/121108-0002.htm


LESSONS FOR THE FUTURE Interviewees were clear about the benefits of the programme to individuals – for example to Ugandan parliamentary staff and MPs who were directly involved. However, they were less clear about the wider benefits to colleagues and the extent to which the programme had built long-term capacity in the Ugandan parliament. Questions were also raised about what kind of organisations should undertake this kind of work in future and what it should involve.

TO BUILD INSTITUTIONAL CAPACITY, INTERNAL PROCESSES FOR SHARING BEST PRACTICE NEED TO BE IN PLACE

A number of factors would need to be addressed within the target institution(s) to ensure that the benefits of any future activities extended beyond the individuals involved:

More effective mechanisms for sharing learning: Ugandan staff who have been directly involved in POST’s training activities say they did not always have the opportunity to feed back what they had learned to their colleagues after the training was completed. In future, mechanisms for sharing learning with other colleagues could be formally integrated into the design of training activities. However, staff need support from their own organisations to ensure that they have the time and the opportunity to do this.

More effective managerial structures: According to one donor, “Individuals have benefitted very much from the programme, but there’s a sense that you take individual capacity and feed it into the organisation. I think that’s where things were broken a bit – unstable managerial structures.” A lack of guidance from managers prevented some training activities taking off. In some cases weak management and high workloads presented a challenge. Future activities could try to engage senior staff more actively, or even offer training opportunities to senior staff who are in a position to initiate changes within the organisation.

Better internal infrastructure and processes: The Parliament of Uganda still lacks the infrastructure to facilitate effective information sharing – for example there is no centralised system for the electronic storage of documents although an electronic repository is now under development.4

FUTURE PROGRAMMES SHOULD BRING TOGETHER EXPERTS IN BOTH CAPACITY BUILDING AND IN PARLIAMENTARY PROCESSES.

The interviews have highlighted some of the challenges faced when a capacity building programme is driven forwards by a parliamentary body such as POST, for whom capacity building is not the core remit. A lack of dedicated resources and lack of capacity building experience were two key issues raised. According to one donor, “There was a real understanding of … the aim of what they were trying to get to, but there wasn’t necessarily such a good understanding of how one could get there .” (NGO 07) Nevertheless, interviewees say that the parliamentary expertise of the POST team provided valuable insights when combined with external expertise in capacity building.

FOCUSING ON A SINGLE COUNTRY ALLOWS MULTIPLE APPROACHES TO BE TRIALLED IN A SINGLE CONTEXT

Focusing a programme on a single country is an unusual approach in capacity building in this area and is seen as being more risky than working in several countries. In this case, however, the focus on a single country has allowed greater continuity between activities and created more opportunities for follow up. It also provided the opportunity to test out multiple approaches in a single context.

FOCUSING CAPACITY BUILDING ACTIVITIES ON PARLIAMENTS HELPS TO BUILD AWARENESS OF THE IMPORTANCE OF SCRUTINY PROCESSES

The exclusive focus of the programme on parliaments was controversial. Some donors argued that parliaments in African countries are often weak institutions with little power. Therefore, even if they

4 Personal communication with Director of Research Services at Parliament of Uganda, June 2015

10 |


have the skills and resources to tackle S&T issues effectively, this will have little impact over the policymaking process as a whole, unless the scrutiny function of a parliament is recognised and supported.

“Even if they were handling science and evidence really well, it wouldn’t necessarily mean much, because in a lot of countries they don’t really have any power, so they’re not fulfilling this kind of scrutiny function that maybe you might hope that they’re doing.”

The fact that parliaments are weak institutions is not seen as a reason to avoid working with them. Indeed, many capacity building activities focus on parliaments for this very reason. However, this comment draws attention to the importance of not setting over ambitious targets such as ‘achieving more effective scrutiny’ which may depend on factors outside the programme’s control.

DISRUPTION CAUSED BY ELECTIONS NEEDS TO BE ANTICIPATED AND PLANNED FOR

Parliamentary elections took place in Uganda in February 2011. In the lead-up to these elections, as well as in their aftermath, many parliamentary activities were suspended, so it was very difficult to interact with MPs or to implement programme activities. Future programmes should take account of election dates, and activities should be timed such that they do not require interaction with MPs when they are likely to be campaigning.

WORKING WITH PARLIAMENTARY STAFF AVOIDS LOSS OF CONTINUITY AT ELECTIONS

There was a high turnover of MPs: many of the MPs who worked closely with POST at the start of the programme lost their seats. According to one scientist, “Unfortunately every term we have a new lot of MPs come in and the other ones have gone. They will go with that knowledge and we have to start again.”

However, because the programme focused on staff as well as MPs, loss of continuity was minimised. It should also be noted that working with MPs is not a wasted investment even if they lose their seats, as former MPs can go on to influence policy in other ways. One former MP told us:

“I ended up actually joining the National Biotechnology and Biosafety committee ... So we are now the ones championing the bill. Now I am the other side…”

FUTURE DEVELOPMENTSPOST does not currently have funding to support further work in Uganda. However it continues to provide informal support and participate in activities on an ad-hoc basis. Moving beyond Uganda, POST is currently investigating further opportunities to work with policymaking institutions in low and middle income countries and to share the lessons learned from its Ugandan programme. For example POST is currently collaborating with African Institute for Development Policy (expand) as part of the DFID funded SECURE health consortium which seeks to strengthen the capacity of policymakers to utilize research evidence in health policymaking in Kenya and Malawi.

11 |


1. INTRODUCTION

1.1 WHY SCIENCE AND TECHNOLOGY MATTER TO PARLIAMENTSThe role of a parliament is to hold the government to account by scrutinising its activities and its expenditure. Parliaments also act as an important forum for public debate and passing laws. Across the world, they are therefore a crucial component of effective governance.

Members of Parliament (MPs) are increasingly required to handle issues of a highly technical nature and to keep track of modern developments in science and technology (S&T). Across the world, S&T is increasingly seen as being crucial to socio-economic development. An understanding of S&T underpins all policy areas, from climate change adaptation to public health.

Sometimes politicians have to make decisions on whether and how to regulate an emerging technology, for example biotechnology. On other occasions decisions may relate to how to handle the impacts of an established technology such as nuclear energy. Even when parliament deals with policy areas which are not explicitly linked to science and technology, it needs to ensure that its scrutiny is robust and based on an informed assessment of the available evidence, including research evidence.

Parliaments in most developed nations provide extensive support to their MPs to help them fulfil their roles. In particular, they rely on parliamentary staff to provide impartial, policy-relevant information. In many cases this information is of a highly technical nature and needs to be provided by specialist staff with scientific backgrounds.

In the UK, this role is fulfilled by the Parliamentary Office of Science and Technology (POST), which is an internal office of the UK’s Houses of Parliament, as well as by staff working within the parliamentary libraries and committees. POST has been providing advice and support to UK parliamentarians for 20 years. Its team of specialist advisers provide oral briefings to members on a wide range of issues. It also provides support to parliamentary committees and acts as an intermediary linking parliament and the scientific community.

“Politicians arrive in politics because of their prejudices, in a positive way, and their ideologies, and yet we learn from Archimedes that when you have an idea about something the important thing is to test it. So a hypothesis needs to be tested and basic scientific, evidence checked [to develop an] approach to policymaking that actually works rather than ideologies that just pretend to work.” (UK MP)5

“POST’s value lies in the concise and accessible nature of its POSTnotes. The impartial and lucid analysis is an invaluable resource in preparing briefings and speeches.” (UK MP’s researcher)

In general, African MPs have far fewer specialist staff to support them than their counterparts in the developed world. They may rely on parliamentary clerks, librarians and researchers, most of whom have no specialist background in S&T. Briefing parliamentarians on S&T is usually only one part of their job. Moreover, there is very wide variation in staffing levels in legislatures across the world. For example, the Parliament of Uganda has 34 research staff serving a parliament of 385 MPs and officials. Some African parliaments have only one or two librarians and no researchers at all. In comparison, in the United States, the Congressional Research Service has around 700 staff to serve 435 representatives.5 POST (2010) POST Evaluation: Summary of Findings 2009 (MP 11).

12 |


1.2 RATIONALE FOR POST INVOLVEMENT IN CAPACITY BUILDING Over the past decade POST has seen increasing levels of interest in its work from African parliamentarians and their staff, both in terms of interest expressed by visiting delegations from African parliaments and approaches over email. Many African MPs have asked POST whether it could help them advise their parliaments on how to handle scientific issues, or even whether POST could provide direct advice to their parliaments.

The majority of initiatives focused on S&T in Africa have focused on building its science base. Those that have focused on governance have concentrated largely on the executive, rather than on building parliamentary capacity to scrutinise the government.

One exception is a series of meetings run by UNESCO called the Regional InterParliamentary Forums on Science and Technology, which started in 2003. The focus of the forums was to achieve “better governance of science, technology and innovation policies based on building dialogue between parliamentarians, scientists, science journalists and civil society”. 6

Attending these forums gave POST staff the chance to have informal discussions with donors, parliamentarians and scientists and revealed a high demand for capacity building activities in S&T directed specifically at parliaments. While the UNESCO project was very high level, POST staff felt that the office’s strengths lay in working more at grassroots level and sharing knowledge and experiences with counterparts in African parliaments. POST’s core funding comes from the UK parliament and so could not be used for capacity building activities. However, driven by the high levels of external interest, POST investigated options for funding its own capacity building work in Africa, and in 2007 it was offered a grant of £240,000 from the Gatsby Foundation for this purpose.

From POST’s perspective, it was felt that this area of work would give the team valuable insight into development issues, which would be beneficial in terms of its work for the UK parliament. Over the years, POST’s Africa programme has enhanced the development aspects of its own work in the UK. For example, it has enabled POST to expand its portfolio of briefings on development issues and to provide more support to parliamentary committees working on development issues.

1.3 CONSULTATION PROCESS FOR PROGRAMME DESIGNExtensive consultation took place with UK and international stakeholders before the programme started. After obtaining formal approval for the programme within the UK parliament, POST also sought input on what the key needs were in this area and what it was best placed to accomplish without duplicating the efforts of others.

Within the UK parliament, POST sought the approval of its board before the programme started, and it kept them up to date with the programme throughout.7 POST also consulted with senior officials and parliamentarians from across both Houses of Parliament and with groups such as the All Party Africa Group, the Commonwealth Parliamentary Association, and the Overseas Offices of both Houses of Parliament.

6 For further details on the launch of the Pan-African Parliamentary Forum see http://www.unesco.org/new/en/natural-sciences/science-technology/single-view-sc-policy/news/launch_of_pan_african_parliamentary_science_forum/. Another exception is NEPAD’s Consolidated Plan of Action (2006). See a press report published by SciDev: http://www.scidev.net/en/news/panafrican-parliament-scheduled-to-talk-science.html.7 All POST’s work is governed by a parliamentary board. This board is made up of members of both houses, senior officials, and scientists, and it makes the final decisions on all POST’s activities.

13 |


POST also sought advice from donors, non-governmental organisations (NGOs) and academic and scientific bodies such as DFID, the Royal Society, the Overseas Development Institute and SciDev.net. To understand the range of challenges faced by African parliaments, POST met and sought advice from visiting parliamentary representatives from a number of countries including Kenya, Tanzania, Swaziland, Malawi, Ghana, South Africa, Botswana and Uganda. POST also met with the representatives from the Network of African Science Academies. In 2007 POST helped organise a conference in Nairobi on science in parliaments, run by the Kenyan National Academy of Sciences. This conference was attended by representatives from a number of countries including Kenya, Uganda, Ghana and Tanzania. It gave POST an opportunity to make contacts and to assess the feasibility of initiating activities in a range of countries.

1.4 RATIONALE FOR WORKING IN UGANDADue to the limited resources available, both POST and the Gatsby Foundation were keen that the programme should focus on one country rather than spread resources across a number of parliaments. The choice of country was guided by the following key questions:

To what extent was S&T recognised as a national priority? To what extent was S&T recognised as a priority for Parliament? What level of support (for example infrastructure or staff time) was available from the

recipient parliament and the indigenous scientific community? What level of interest and engagement was there?

Was POST in a position to provide the type of support which the parliament needed in the area of S&T (based on the initial assessment)?

On this basis, POST selected the Parliament of Uganda to be the focus of the programme. There were a number of indications that S&T was a priority at national level.8 Recent years have seen an increase in spending on research and development in Uganda and, moreover, the Millennium Science Initiative ran from 2006/07 to 2010/11 and was aimed at strengthening science, technology and innovation (STI) in Uganda.9

The Parliament of Uganda had taken a number of steps towards strengthening its handling of S&T issues. For example, in 2002 it set up the Standing Committee on Science and Technology (hereafter referred to as the S&T Committee), composed of 20 members. It also had a team of researchers within the Department of Library and Research, whose role it was to provide information to MPs on a range of topics. At the time, only one researcher was assigned the task of handling issues relating to S&T, but this number has since increased to five.

Setting up capacity building programmes with a parliament is very challenging, because the many protocols in place mean that it can be difficult to identify the right people to approach within the organisation. The Parliament of Uganda has a Parliamentary Donor Coordination Office, which was initially very helpful in this respect. It helped us identify contacts and ensure that the right protocols were followed. In addition, the Parliament of Uganda has consulted and set out its own strategic priorities in its Parliamentary Strategic Investment and Development Plan which can help prospective donors gain an understanding of what kinds of support the parliament most needs (although this sets out needs in very broad terms and does not specifically mention science and technology). Another factor influencing POST’s decision was the enthusiasm and engagement of MPs and staff within the Ugandan parliament as well as the scientific community, through the Ugandan 8 See UNCST (2011) Status Report 2009/2010, UNCST (2009) National Science, Technology and Innovation Policy. 9 Ecuru J. et al (2008) Research in Uganda: Status and Implications for Public Policy, pp. 16–17, World Bank (2011) Science, Technology and Innovation in Uganda: Recommendations for Policy and Action.

14 |


National Academy of Sciences (UNAS). While enthusiasm is invaluable, this evaluation shows that it does not necessarily guarantee that the beneficiaries have the resources and commitment to see the programme through. This issue is discussed further in Chapter 7.

1.5 THE FINAL EVALUATION AND THIS REPORTThe POST Africa programme ran from 2007 to 2012 and involved a series of training workshops (in collaboration with other NGOs and civil society organisations), as well as fellowship, mentoring and networking activities. These were mainly focused on Uganda, but some staff and MPs from other parliaments were involved in specific activities. The programme also included research activities to understand the challenges the Parliament of Uganda faced when handling issues related to S&T. It focused on parliamentary staff as well as MPs and was led by POST, working in close collaboration with INASP and UNAS.

From November 2012 to May 2013, the programme was evaluated to assess its impact and influence. This report sets out the evaluation findings. It is intended that the report will add to the stock of knowledge on capacity building and will be useful information for NGOs, donor agencies and civil society organisations working on similar programmes. It reflects on the challenges and rewards of planning and implementing capacity building programmes within different political and cultural contexts, and considers how expectations are managed, how capacity building is monitored and accounted for, and how improvement is assessed.

In keeping with the aim of the programme to strengthen capacity in-country, this report assesses the extent to which the POST programme became embedded within the political culture of the Ugandan parliament. The methodology is drawn from real-time and retrospective evaluation data. This refers to data collected during the lifetime of the programme and after it had finished respectively. The main dataset used to inform the report is based on qualitative methodology: in-depth, semi-structured interviews conducted with a wide range of professionals involved in, or with knowledge of, the POST programme. The interviews were conducted retrospectively and complement the real-time evaluative data collected at the beginning and mid-way through the programme. For more information about research design and methodology, see Appendix A.

The report is structured as follows:

Chapter 2 introduces the programme’s activities. The findings of the evaluation are structured around the four programme objectives and are

presented in Chapters 3 to 6. Chapter 7 discusses lessons that can be learned for future work. Chapter 8 sets out future developments and includes recommendations for further action in

Uganda.

15 |


2. THE POST PROGRAMME IN UGANDA

2.1 AIMS AND OBJECTIVESThe programme aims and objectives were set after extensive consultation with potential beneficiaries, UK parliamentary colleagues and NGOs. They were clarified in a ‘logic model’ drawn up in consultation with an evaluation expert in 2009.

The intended impacts were very high level. First, the programme set out to achieve better use of evidence by the Parliament of Uganda in the scrutiny of government. Second, the programme set out to build up a body of knowledge of how to help developing countries and agencies use S&T in parliament.

There is no single way of achieving these ambitious objectives. A number of different outcomes, each focusing on a different target group, needed to be met in order to achieve these impacts:

Strengthening the ability of Ugandan parliamentary staff to support MPs on issues related to S&T by: developing their skills in identifying and using S&T resources; building their confidence in handling S&T issues; and helping develop their networks.

Strengthening the ability of Ugandan MPs to scrutinise issues relating to S&T by: building up their interest, their take-up and use of resources, and their networks; and building the profile of the Standing Committee on Science and Technology.

Building POST’s understanding of S&T issues in the developing world and its network of contacts in the developing world, learning about working methods applicable in the UK context.

Greater awareness amongst partner organisations of the importance of building S&T capacity in parliaments, and greater understanding of how to interact with parliaments.

As the programme evolved, it became clear that the issues faced in handling S&T issues were common to many subject areas. Rather than simply facing a capacity gap in handling S&T, much of the problem lay with parliament’s ability to access and use research evidence more generally in its scrutiny of government. This is discussed in subsequent chapters.

A number of tangible outputs were linked to these outcomes in order to develop a framework against which progress could be measured:

Increased availability of internal and external resources on S&T.10 An improvement in the quality of material on S&T produced by parliament. Increased participation in activities relating to S&T. Leverage (as an indication of support from other donors).

2.2 SUMMARY OF ACTIVITIES TAKING PLACE UNDER THE PROGRAMME The initial direction of the programme was set after brainstorming meetings in 2007 and 2008 with Parliamentary Donor Coordination Office, Ugandan MPs, parliamentary staff and representatives from UNAS. Broadly, the programme has focused on two main areas:

Training parliamentary staff to build their skills in providing reliable briefings (focusing on S&T) to their MPs.

10 Note that as well as increased access to these resources, we use ‘availability’ in a broader sense, i.e. users should also have a greater ability to use these resources in the scrutiny of government. ’Resources’ does not just mean access to literature but implies increased access to expertise, through enhanced networks, etc.

16 |


Building MPs’ interest in S&T and strengthening their links with Ugandan scientists, with a view to raising the profile of S&T (and the S&T Committee) across parliament.

The decision to focus the programme on staff as well as MPs was a crucial one: the turnover of MPs in Ugandan elections is very high. Focusing on staff as well as MPs is one way of ensuring that capacity is built in a sustainable way.

Ugandan colleagues had a wide range of suggestions for possible areas of focus, but many of these reflected their perceived needs, rather than actual needs. For example, there were requests for POST to fund journal subscriptions or to pay for MPs to attend international conferences and go on study visits. However, such ‘one-off’ activities are not considered to be a sustainable way of building capacity. It was necessary to reject some suggestions – such as funding conference attendance – at an early stage.

To understand what the ‘needs’ of the parliament really were, POST conducted detailed research in years 1 and 2 of the programme. The resulting report has been made publicly available. It is referred to hereafter as the ‘baseline study’ and its findings are discussed in more detail in Box 1. The direction of the programme was continuously revised based on findings from the baseline and from ongoing evaluation of activities.

Table 1 summarises activities conducted in each year of the programme. POST aimed for the programme to be driven from within Uganda as far as possible. However, it was still essential for staff in the UK to set up and oversee activities. In the first two years of the programme UK staff helped direct the programme and run individual activities. After that, a parliamentary liaison officer was taken on by UNAS to shift the day to day running of operations to Uganda.

ADDITIONAL FUNDS AND IN-KIND SUPPORT

POST encountered a great deal of interest in this area from other organisations, and was able to tap into this interest to support its own activities as illustrated in the table. The programme also triggered a number of spin-off activities. In total, the programme attracted an additional 40% of funding. This meant that the Gatsby grant of £240,000 lasted longer than originally anticipated (three years) and the programme was stretched out over four and a half years to allow for some continuity following the 2011 election.

In terms of funds, a number of organisations have contributed to the programme’s activities, either by sponsoring activities or co-funding activities. For example:

POST obtained fellowships for four parliamentary staff from Uganda to spend three months in the UK parliament, based at POST. Funded by the Commonwealth Scholarship Commission, this provided an opportunity for intense one-to-one training, which was highly valued by the individual staff and their managers.

Many organisations, including the Royal Society and INASP co-funded activities such as MP pairing and staff training.

The Ugandan parliament sponsored a study visit for seven science and technology committee members in late 2011. Some activities were also supported by grants from the Wellcome Trust.

UNAS obtained matched funding from the African Science Academies Development Initiative for a number of the programme’s activities.11

11 African Science Academy Development Initiative.

17 |


Our estimate of the total value of this external support is around £100,000, which is over 40% of the original grant.

A few organisations generously provided in-kind support. For example, the programme was based at the UK Parliament and therefore used POST’s office space, IT facilities, etc. INASP and the Parliamentary Science and Technology Information Foundation12 assisted with management of funds. Various collaborators offered their services (for example consultancy or briefing reviews) free of charge or at much reduced rates.

12 The Parliamentary Science and Technology Information Foundation is a charitable trust that liaises with the Parliament and Science Committee providing long-term support for scientific and technological issues in parliament.

18 |


TABLE 1Date Activity Main beneficiaries Organisers Funding source Additional notesSeptember 2007 Conference in Nairobi on

science in parliaments –POST organised a workshop on methods of delivering science advice

Attendees: East African scientists, African policymakers, UK parliamentarians and staff

KNAS (with input from POST and Royal Society)

Kenyan government, KNAS, Royal Society, Gatsby13 (sponsored Ugandan participation)

Networking and initial scoping/market research for POST programme

August/September 2008

MP/scientist pairing 2008 in P of U14

Ugandan MPs and scientists (also parliamentary staff, journalists)

POST, Royal Society, UNAS

Gatsby, RS, UNAS (via ASADI) 5 MP-scientist pairs

September 2008 Science communication workshop for parliamentary staff in Kampala

African parliamentary staff from 13 countries (also the organisers benefited from working in a new area)

POST, SciDev.net,P of U, ATPS

Gatsby, SciDev.net, ATPS

Ongoing in 2008 Spin-off activities i.e. other donors providing funds to support similar work. Examples include:

Various: Ugandan MPs funded to attend

various conferences P of U staff funded to visit

Nigeria by INASP (peer to peer exchange)

October 2009 Recruitment of programme officer to manage day to day programme activities

POST, INASP Gatsby, INASP INASP agreed to co-fund a programme officer based at POST, working 50% on POST work and 50% on INASP

January to March 2009 Commonwealth professional fellowship at POST

Ugandan staff, POST team POST Commonwealth Scholarship Commission

This fellowship supplemented training provided in Uganda

Ongoing in 2009 Baseline study – policy briefing review

Scientists and policymakers from East Africa and UK, staff from P of U, POST team

POST, UNAS Gatsby

Ongoing in 2009 Baseline study – analysis of interviews and survey results

MPs, staff from P of U POST, UNAS Gatsby

Ongoing in 2009 Recruitment of UNAS parliamentary liaison officer to run activities with UNAS

UNAS staff, POST team POST, UNAS Gatsby

August 2009 Information literacy training in Nairobi

Parliamentary staff from several African countries

POST, INASP Gatsby, INASP

November 2009 Information literacy training in P of U

P of U staff and some external observers

P of U Gatsby, INASP

November 2009 Seminar on capacity UK parliamentary staff, MPs, POST, INASP Gatsby, INASP Start sharing learning from programme with other

13 i.e. via the POST programme. 14 Parliament of Uganda.

19 |


building in UK parliament

external donors and research organisations involved in capacity building

donors and stakeholders in UK

Ongoing in 2009 Spin-off activities (only highlights given)

INASP funding up of capacity building activities in many other African parliaments

POST team briefing Lord Sainsbury

Programme manager invited to speak at National Science week in Uganda

Review of Overseas Development Institute paper on science and legislatures

January to March 2010 Commonwealth fellowship 2010

Ugandan staff POST Commonwealth Scholarship Commission

Supplement training provided in-country

June 2010 Discovery of oil in Uganda: What are the implications to the environment?

70 people attended including MPs from relevant committees, parliamentary staff, government agencies, academia, civil society organisations and the private sector

POST/UNAS ASADI

Gatsby/UNAS ASADI

June 2010 Pine tree planting seminar

MPs, parliamentary staff, school and college students and key stakeholders in natural resources including representatives from the forestry industry, the National Forestry Authority and the Faculty of Forestry and Nature Conservation, Makerere University

POST/UNAS ASADI

Gatsby/UNAS ASADI

20 |


Throughout 2010 Spin-off activities (only highlights given)

POST staff involved in design of ‘internet detective’ science on the internet tutorial funded by INASP

PM supervised research study on Nigerian legislature, funded by INASP

PM guest edit of INASP newsletter on science in parliaments

Ongoing advice to other donors

Spread methods piloted in Uganda to other countries

July 2011 Summarising skills training in P of U

Ugandan parliamentary staff and observers

POST, INASP, UNAS

Gatsby, INASP

2010/2011 MP/scientist pairing for 17 pairs in P of U

Ugandan MPs, staff, scientists, Royal Society, Gatsby, UNAS (ASADI)

Dec 2010 to January 2011

Interim evaluation survey in P of U

12 staff from library and research department, P of U

POST, UNAS Gatsby

January to March 2011 Commonwealth fellowship

Ugandan staff

Throughout 2011 meetings between staff and scientists in in P of U

Ugandan staff UNAS Gatsby Build on links made during pairing week

May 2011 Training in health policy brief writing in Nairobi

Parliamentary staff, civil servants, researchers

INASP, WHO Gatsby (P of U participation only), INASP, WHO

October 2011 Publication and dissemination of baseline study

Donors, researchers, beneficiaries POST Gatsby, INASP (contributed to the editing)

25 November 2011 Climate change networking event in P of U

Attendees: MPs, scientists, civil servants and staff, NGOs

UNAS Gatsby

December 2011 Show me the Evidence seminar in UK parliament

Attendees: Ugandan MPs from S&T committee, UK donors, parliamentary staff and MPs, civil servants

POST Gatsby

Ongoing in 2011 Spin-off activities (only highlights given)

Ugandan parliament sponsors visit to UK for members of S&T Committee

INASP programme of capacity building with parliaments continues to expand

POST P of U

January to February 2012

International conference on evidence informed policy making15 in Nigeria

Policymakers, researchers and donors from 16 countries across Africa, Asia, South America, Europe

POST, INASP, NACETEM16

Gatsby, INASP, NACETEM, Wellcome Trust

Spreading lessons learned from POST programme (largely baseline study) to other countries beyond Uganda

15 Conference report available on http://nacetem.org/events/report_on_international_conference.html.16 Nigerian governmental science organisation http://www.nacetem.org/.

21 |

http://www.nacetem.org/

http://nacetem.org/events/report_on_international_conference.html


Ongoing in 2012 Dissemination of learning (highlights only)

Dr Nath gave briefings on baseline and programme learning to: Sir David King and team DFID funded PISCES bioenergy

consortium in Tanzania World Water Forum

(Marseilles) African Union/UNESCO

meeting on African parliamentarians and STI in Cairo (TBC)

Overseas delegations visiting UK including (Chinese, US, Ghana (JW), Mozambique, Rwanda (SA), Uganda principal clerk

Various conference organisers Spreading lessons learned from POST programme (largely baseline study) to other countries beyond Uganda and other sectors

Ongoing in 2012 Spin-off activities P of U staff invited to attend at World Water Forum

Post staff mentoring INASP research project in Uganda

2013 commonwealth professional fellowship secured for P of U

World water forum, INASP

September 2012 Information literacy training in P of U

P of U staff POST Gatsby Advice received from INASP

July 2009 to July 2012 Day to day networking in P of U by UNAS Parliamentary Liaison Officer

P of U staff, UNAS staff, Ugandan MPs

Gatsby

Throughout 2011/2012 Management of remote mentoring scheme in P of

P of U staff POST, INASP Gatsby

22 |


BOX 1. Baseline Study in Parliament of Uganda

A programme of research initiated was in 2008 to establish how effectively the Ugandan parliament handled scientific and technological evidence during the seventh and eighth Parliaments (2001 to 2006 and 2006 to 2011). Conducted by a team of staff from UNAS, POST and the Parliament of Uganda, this research included the following types of ‘data’:

An expert analysis of 17 S&T-related policy briefings produced by Ugandan parliamentary researchers. A survey of 121 Ugandan MPs from across parliament in the first quarter of 2009. Semi-structured interviews with seven Ugandan MPs and nine Ugandan staff. A focus group with the Ugandan S&T Committee. Desktop research: an analysis of selected Ugandan parliamentary debates on various topical STI issues.

The team found that Ugandan MPs voiced enthusiasm for S&T but that this very positive attitude did not always translate into action. Two-thirds of surveyed MPs said that there was a need for more parliamentary debates on science issues and that debates should be more evidence-based. S&T did not get much attention on the floor of the house, and debates were poorly attended. This was attributed to many diverse factors, ranging from the fact that most MPs did not see S&T as relevant to daily life through to poor internal communications meaning that they did not receive enough warning of what was going to be debated on a given day.

A key issue was MPs’ ability to obtain reliable information on S&T in time for debates and their ability to distinguish it from unreliable information. The team found many instances of incorrect scientific information being cited during debates. Both MPs and staff lacked links with the science community in Uganda. The quality of internal briefings produced by parliamentary staff for MPs varied significantly. A number of common areas for development for parliamentary staff were identified, including a need for greater objectivity, better exploitation of available sources of information (such as online resources) and improved understanding of the basic scientific principles underlying issues being debated. Staff training activities have focused on these areas.

The S&T Committee had a low profile both within parliament and outside it. MPs on the committee complained of low attendance at committee meetings and of the committee being bypassed when it came to scrutinising legislation with an S&T component.

The baseline research was done specifically to help steer the POST programme, but the results have also been useful to other donors. The study’s findings were published in full in October 2011 in POST’s report on Use of Scientific and Technological Evidence in the Parliament of Uganda17 and presented at POST’s ‘show me the evidence’ seminar in December 2011,18 which looked at the role of evidence in the work of both legislatures and governments in the developing and developed world.

There is little detailed research into how evidence is handled by policymakers in developing countries, particularly in a parliamentary context. Many decisions on where to focus capacity building activities in this area are based on incomplete information. For this reason, POST, along with INASP and the Nigerian governmental organisation NACETEM, ran a workshop on conducting research into evidence based policy making, in Nigeria from 27 to 29 February 2012.19 The aim was to share best practice on how to conduct this type of research and to stimulate research in this field.

17 http://www.parliament.uk/documents/post/Baseline%20Report%20131011.pdf 18 http://www.parliament.uk/mps-lords-and-offices/offices/bicameral/post/events/past-events/show-me-the-evidence-81211/ 19 See http://www.oxfamblogs.org/fp2p/ for a commentary on the Nigerian conference

23 |

http://www.oxfamblogs.org/fp2p/

http://www.parliament.uk/mps-lords-and-offices/offices/bicameral/post/events/past-events/show-me-the-evidence-81211/

http://www.parliament.uk/mps-lords-and-offices/offices/bicameral/post/events/past-events/show-me-the-evidence-81211/

http://www.parliament.uk/documents/post/Baseline%20Report%20131011.pdf


BOX 2. Overview of Uganda’s research base

STRATEGIC MILESTONES IN THE DEVELOPMENT OF UGANDA’S RESEARCH BASE

In 1990, the Ugandan National Council for Science and Technology (UNCST) was established by an Act of Parliament. UNCST was set up to advise government on policy and regulation in areas related to science and technology. UNCST regularly produces status updates20 on the state of the Ugandan science base, which feed into international benchmarks issued by the World Bank, World Economic Forum and UNESCO.21 In maintaining strategic oversight over the Ugandan science base, the UNCST commissions regular reports on politically salient issues. However, it was only with the publication of the 2009 National Science, Technology and Innovation Policy that the UNCST was integrated into national development policymaking. The STI policy identified a coordinated and enabling policy environment as central to the strategic development of a Uganda knowledge economy.

In 2000, The Ugandan National Academy of Sciences (UNAS) was launched as an honorific organisation that brings together scientific expertise across different specialities. UNAS is part of the umbrella organisation Inter-Academies Programme Global Networks of Science Academies and Network of Africa Academies.

In 2000 it was announced that Uganda would receive a $30 million grant as part of the Millennium Science Initiative. The objective of this project was for Ugandan universities and research institutes to produce more and better qualified science and engineering graduates, and higher quality and more relevant research, and for firms to utilize these outputs to improve productivity for the sake of enhancing S&T-led growth. The Ugandan government took the decision not to renew the loan, and the project came to an end in June 2013. Uganda also participates in the United Nations Millennium Task Force on Science, Technology and Innovation. The task force was established to encourage developing nations to incorporate science and technology into national growth strategies. Low to middle income counties are advised to harness science and technology to meet the millennium goals of eradicating poverty and raising living standards. The strategic overview of Uganda’s scientific development is presented in UNCST 2009 policy document.

OVERVIEW OF DEVELOPMENTS IN UGANDA’S RESEARCH BASE

The most recent UNCST status report22 highlighted that the total science budget for 2009/2010 was Ushs 479 billion (£120 million). This has increased from Ushs 366.6 billion (£93 Million) in 2008/2009, an increase of 30.7%. While this increase shows a consistent upward trend in expenditure, it is not as steep as the 40.5% increase between 2007/08 and 2008/09. Over 60% of expenditure was spent on scientific and technological services; 12.2% was spent on scientific and technical education and training (a decrease from the 22.6% spent in 2002/03); 25.9% was spent on research and development (an increase from 17% spent in 2002/03).

In terms of human resource, in 2009/10 the number of people working in STI rose to 4,002, an increase of 35%. This comprised 1,703 researchers, 1,194 technicians and 1,105 support workers. Of this number, 524 researchers and 713 technicians and support staff were working full time. The figure of 4,002 represents approximately 0.03% of the working population, estimated at mid-2009 to be 11.5 million.23 This is way below international averages of between 5% and 18%.

20 UNCST (2011) Status Report 2009/2010.21 UNCST (2009) National Science, Technology and Innovation Policy.

World Bank (2011) Science, Technology and Innovation in Uganda: Recommendations for Policy and Action.22 UNCST (2011) Status Report 2009/2010.23 UBS (2010) National Household Survey 2009/2010. 82% of the working population are based in rural areas.

24 |


Uganda recognises the importance of nurturing its human resources through higher education, yet its economy depends on a steady supply of vocational and technical skills rather than the academic training. The BTVET (Business, Technical, Vocational Education and Training) is the national qualification awarded through community polytechnic colleges. While enrolment on BTVET courses continues to rise, the failure rate is as high as 72%. This is attributed to a lack of adequate resources such as library books and teaching facilities. Amongst those who do qualify, results are often poor. These inadequacies in the BTVET result in a skills deficit, especially in areas such as agricultural science, where vocational skills are as important to developing the research base as scientific training.

The expansion of human capital is further hindered by the low take-up of postgraduate training in science, technology, engineering and mathematics subjects. The percentage of PhD students as a proportion of all graduates is 0.14% and the percentage of masters students is 7.6%.

In 2009, Uganda awarded 35 doctoral degrees, with agricultural sciences attracting the highest number. Uganda has 29 degree awarding institutions, but only five offer doctoral training. In a recent survey of 450 Ugandan doctorate holders, just over half had received their degree from a Ugandan university.24

In addition to its universities, Uganda has 20 active research and development initiatives and five private laboratories.25 Research institutes and centres are mostly funded by external organisations, such the Wellcome Trust and Medical Research Council. In its study of STI in Uganda, the World Bank (2011) reported significant gaps, including: a lack of incubator sites and science parks; a lack of availability of information and insufficient information and communication technology; and a lack of expertise on knowledge transfer. While a basic infrastructure was in place to promote research and development, the World Bank recommended strategic investment in developing ICT networks, access to scientific know-how and expertise in targeted areas.

24 UNCST (2012a) The Careers and Productivity of Doctoral Holders (CDH) Survey.25 World Bank (2011) Science, Technology and Innovation in Uganda: Recommendations for Policy and Action.

25 |


3. FINDINGS: IMPACT ON PARLIAMENTARY STAFF

3.1 THE NATURE OF THE PROBLEM The baseline study highlighted a number of issues which affected the ability of staff to support members on S&T related issues.

AVAILABILITY OF INFORMATION

Staff interviewed as part of the baseline study highlighted the need for more up-to-date information on science and technology in the library, both in hard copy and online.

“There is need for an up-to-date library service with recent information on science and technology. The library lacks up-to-date reports, brochures and newsletters from research institutions, NGOs, CSO, and embassies.” (Clerk, baseline study)

They also cited difficulties in accessing locally relevant content:

“The problems I have had in the past while trying to access and use information is that a lot of science information on Uganda technology is not fully documented, and proxy indicators which happen elsewhere are used.” (Librarian, baseline study)

One of the librarians interviewed for the baseline study was not sure how to do an effective search for information on S&T, and he spent a lot of time on searches. He said he also faced challenges accessing information through online sources, because many sources were not free and the internet connection was not fast enough.

Availability of information also depends on staff skills in identifying reliable resources – particularly those which are freely available on the internet. This in turn depends on what training is available to the staff and what networks they can draw upon for assistance. These issues are discussed below.

SCIENTIFIC AND INFORMATION LITERACY

The 2009 review of internal policy briefings on S&T (see Appendix C) showed that the quality of reports produced by staff for MPs varied widely. Many briefings contained factual errors and scientific inaccuracies. While some briefings were reasonably well referenced, others were highly subjective and did not make use of the wealth of publicly available sources online. Staff had poor skills in identifying reliable information on S&T (and on research more generally). Reviewers highlighted the poor writing style and incoherent structure of many briefings. One reviewer commented that:

“This brief seems to be based on a single report produced by an organic lobby group that has not been peer reviewed. The International Assessment of Agricultural Science and Technology for Development (IAASTD) report and/or peer reviewed papers would have provided a more reliable source of information” (Reviewer, 2009 policy briefing review)

These findings are reiterated by NGOs who have built on the work POST started in Uganda:

“In a lot of the training that I’ve done with staff from parliaments ... you could give them a piece of paper, an A4 piece of paper with some writing about some topic and they would find it extremely difficult to summarise and say what is the meaning of this brief, what’s actually being

26 |


said here. So people are … very good at reading and learning things, but actually then being able to extract and summarise information seems to be really be lacking.” (UK NGO 7)

NGOs also highlight problems with understanding nuance and uncertainty in scientific research:

“People expect scientists to go away somewhere and come back with facts, indisputable facts. And there’s a worry that actually there’s a lack of understanding of the scientific process … science is valued when it supports someone’s position, but even then … no one accepts ambiguity within policymaking circles. There’s a tension that people expect scientists to produce evidence and I’d argue that scientists and researchers more broadly produce data. Someone decides if it’s evidence because evidence is only evidence in a particular context.” (UK NGO 09)

Based on these findings, POST (in close collaboration with INASP) focused on delivering training to parliamentary staff, mainly in Uganda, between 2008 and 2012. As well as single workshops, POST offered number of two- to three-month fellowships to staff from Uganda, and piloted a remote mentoring scheme. The training activities focused on staff rather than MPs, since MPs rarely have time to conduct their own detailed research, and they rely on staff to do it for them. They covered areas such as how to find information on the internet, how to summarise information, and how to communicate information to policymakers. The training also aimed to build awareness of the nature of the scientific process.

NETWORKS

At the start of the programme, staff had limited external networks, particularly with Uganda’s scientific community. This affected their access to information on S&T, which in turn affected the quality of policy briefings. The MP-scientist pairing scheme was designed to build links between parliament and Uganda’s scientific community. In the second round, in 2010, a particular effort was made to involve staff as well as MPs. Other schemes, such as the remote mentoring scheme for staff, were also aimed at building up links between staff and external experts.

RESOURCES AND PROCESSES

A common problem raised by staff interviewed for the baseline study was the lack of time available to produce briefings.

“The biggest challenge we have is time constraints. We have given guidelines to MPs asking them to submit their requests at least two weeks in advance (before the day they need the report), but many of them fail to do this. And then if the speaker comes and says, ‘I want this tomorrow,’ … you have to put everything aside.” (Library and research interviewee 08, baseline study)

The lack of a peer review process for reviewing the quality of internal briefings was highlighted as a reason for the low quality of some briefings. However, the quality of the briefings was attributed in part to lack of time:

“Most reports are done in a hurry and, as such, there are some mistakes.” (Library research interviewee 08, baseline study)

MPs and staff alike complained that there were too few specialist staff available to tackle S&T issues:

“A research officer sits on the Science and Technology Committee and takes care of research needs … Unfortunately, this researcher is shared with the Committee on Agriculture, Animal

27 |


Industry and Fisheries. You can see how this resource is divided among different roles.” (Clerk01, baseline study)

“The research department needs to be given muscle and training so that they can be able to handle the demand for science and technology information. Otherwise, it will be like a blind man leading a blind man.” (MP on S&T Committee, baseline study focus group)

The lack of a centralised repository for storing briefings meant that sharing information internally was a challenge, and this led to some duplication of effort.

3.2 HOW HAVE THINGS IMPROVED?

AVAILABILITY OF INFORMATION

In the end-of-programme interviews, staff point to an improvement in their ability to access to information on S&T online and in accessing research information more generally.

According to one librarian, access to research information has improved as a result of linkages with other donors, made through the POST programme:

“Well it has improved a lot, because there’s some other program that … we got in contact with as a result of the POST program – the INASP PERii, which came because there are some people who are working in POST and also working in INASP. So that’s how we happened to get access to databases and journals, which we now subscribe to, and it has improved a lot, the access is very good.” (Ug P 8)

One MP comments about the marked improvement of the library, although as he points out, this cannot be attributed to a single organisation:

“We see a great improvement in the library in terms of organisation, in terms of keeping of records … In terms of also their ability to quickly pick out a document from the pile, I think it’s improved tremendously. Whether this is a result of the activities of POST alone or helped by other program, is something that the library department alone can tell us, but I think I have seen a great improvement in what is going on in the library.” (Ug MP1)

However, access to locally relevant content is still a concern.

“There’s a lot of information there on journals and about what people have done [elsewhere], but what has been done in Uganda you will not find there ... because that’s what MPs need to know, you know: something local which they can relate to … there’s no aggregation of those publications. There are many organisations actually publishing those things but there is nowhere they aggregate it, where you can find them in one place like you can find journals of other countries.” (Ug P8)

Lack of internet access was not raised as a key problem.

“We are fully subscribed, not fully but largely, to online resources and we have full time access to the internet.” (Ug P 11)

Access to hard copies of periodicals and books was not raised as a key issue in the final interviews. Interviewees clearly appreciated that more up-to-date information could generally be obtained online.

28 |


“At least I can say that [online resources] are available unless there is a network interruption. We also rely on the library but sometimes the book may not be up to date, so you are compelled to rely on online resources.” (Ug P 6)


A comparison of internal policy briefings on S&T produced before the start of, and towards the end of the programme (see Appendix B for more details) indicates an improvement in subject coverage, accuracy, evidence base, objectivity and clarity.

External reviewers rated the briefings according to a number of criteria established by POST. Average ratings increased across all criteria.

Comprehensiveness increased from 2.50/5 in 2009 to 3.25/5 in 2013. There were major omissions in ten out of fifteen briefings reviewed in 2009 but in 2013 only two of the fifteen briefings had major omissions. However, many briefings were still judged to lack the detail required for a reader to make an informed judgement about the issue in question.

Accuracy increased from 2.55/5 in 2009 to 3.43/5 in 2015. In 2009, seven briefings out of the fifteen contained major factual inaccuracies and/or incorrect science. Three had minor inaccuracies. In the post-2010 review, only one briefing had major factual errors and two had minor inaccuracies. Thus, for the majority of the post-2010 briefings, reviewers had no problems with the factual accuracy of what was presented. However, in many cases a lack of detail, a failure to reference claims, and a failure to present data clearly still had a negative impact on accuracy.

Use of evidence increased from 1.92/5 in 2009 to 2.81/5 in 2013. In 2009, ten of fifteen briefings were criticised for making poor use of the available evidence and for missing out references to key sources. In 2013, four out of fifteen briefings were criticised for making poor or limited use of the available evidence, for missing out key sources or for using references of dubious quality.

Objectivity increased from 2.83/5 in 2009 to 3.02/5 in 2013. In 2009, two briefings were considered to be very biased or speculative. Only one was judged to be ‘reasonably objective’. In the post-2010 review, two briefings were considered to be biased, but six of them were considered to be reasonably objective.

Clarity increased from 2.74/5 in 2009 to 3.52/5 in 2013. In 2009, three of the briefings were assessed as being quite easy to read and well structured. The others received a mixture of criticisms, for example: confusing use of statistics and graphs, use of jargon, poor language, lengthy paragraphs, and the excessive use of copying and pasting of text. In the 2013 review, ten briefings were considered to be well written, of which six were also assessed as being well structured.

In spite of this progress, there is still considerable room for improvement. Particular issues for the future include:

Attribution: Five out of fifteen briefings reviewed in 2013 were criticised for not attributing claims, facts and figures made in the briefings. This meant that reviewers did not know where the claims originated and were not able to judge their validity.

Making recommendations: Eight out of the fifteen briefings reviewed in 2013 were criticised for making unsupported recommendations. Many reviewers have highlighted the fact that the briefings need to be much clearer about how any recommendations have been arrived at. A few reviewers have pointed out that other neutral parliamentary bodies (such as POST) do not consider making recommendations to be within their remit.

29 |


It is acknowledged that the numbers of briefings reviewed was small (15 in both 2009 and 2013). However, when the results are considered alongside the results of the interim review conducted in January 2010 and comments made in the end-of-programme interviews, they support a conclusion that there has been an improvement in both scientific and information literacy. However, the benefits of the programme appear to be focused more on the staff who were directly involved in POST’s activities, rather than the institution as a whole (see Chapter 3).

In the interim review, staff were very positive about the impact of the training. Overall they felt that POST had strengthened their department’s:

Understanding of science and technology. Access to information on S&T. Ability to advise MPs on science and technology. Prioritisation of evidence -based policy.

They also thought it had some effect on the quality of briefings and the strength of quality control of briefings.

Both MPs and senior managers in the end-of-programme interviews said they thought that staff skills had improved as a result of the POST programme:

“Their [the staff’s] performance has, I think, improved because now as supervisors we see that they are able to find and reference useful information from the internet. So generally speaking our capacity to find useful information on the internet has improved.” (Ug P 12)

“Well, I saw the training programme going so well. I interacted with members of staff who did go for this training programme, some people in the library, and others who went to London for an attachment to POST and when they returned they came back more productive ... So for me I think that was well organised, for science background and training and I think that was well done.” (Ug MP1)

The end-of-programme interviews indicate an understanding of what ‘research evidence’ is and why it is important.

“Research evidence? Well I understand it to be that … truth that has been arrived at after research and which can be proved and … if you did the same research again, you would reach … the same result, and if anybody else did the same research, they would still meet the same result.” (Ug P 8)

Staff say that they are more confident at handling S&T issues and that they are treated as more knowledgeable.

“People are treating me like I am more knowledgeable ... but I find it means that some things will not happen because I’m not there. ‘Oh, XXX is not around, you go, you come back later.’” (Ug P 8)

“I feel I’ve gained more knowledge in areas that I previously wasn’t confident about. Also looking at how the POSTnotes are written and how we produce our work, I think I can make improvements on how I write my reports back home, and also I hope to share this information with my colleagues and see how to improve things.” (Ug P 6)

30 |


NETWORKS Some interviewees said that as a result of the POST programme, they were better linked with the outside world and knew who to go to if specific issues came up. However, such comments were only made by those who were directly involved with the programme activities, particularly the MP-scientist pairing scheme.

“Just the fact that I networked with people through the pairing scheme, it’s much easier for me to contact these people because sometimes without creating a rapport with people it’s quite difficult to get information but now it is very easy for me to contact these people.” (Ug PS6 (2010 MP-scientist pairing scheme))

One librarian mentioned that links with other librarians had improved as a result of the POST programme. (Note that it is not always clear which training activities are being referred to in the interviews. In this instance the training in question is likely to be one organised by INASP.)

“And we had a training in … with the National Council for Science and Technology, and we organised that, so from that we have established that network of librarians. So now I can always know where to get what and to call on librarians when I need something, and they can always call on parliament.” (Ug P8)

The mentoring scheme also helped build links:

“Then through this mentoring programme, my mentors knew some of the staff in Makerere University who have knowledge on solid waste management. So somehow I got in touch with them, so it’s much easier for me to get in touch with these people. Even still, through this same mentoring programme I met many people in the Kampala Capital City Authority.” (Ug P 6)

Information sharing between parliament and government departments and agencies can still be problematic, according to interviewees on both sides.

“You see, they have a research desk but it is not competent in all fields of science. While there are a number of competent agencies within government that can provide the relevant policy support for parliament, there is very little work done by collaboration.” (Ug G1)

“In my case, I sometimes go to the National Environment Authority and the library. The other option is of course the ministries, but getting information from the minstries is never an easy thing.” (Ug P 6)

RESOURCES

A notable change has been in the recruitment of more staff with a scientific background in the research department.26 When the programme first started, there was only one member of staff handling scientific issues. Now there is a dedicated section on S&T.

“The most important and key achievement of POST for us is the creation of science and technology section in the research department.” (Ug P3)

This dedicated section is currently staff by 5 researchers, but the aim is to employ up to six researchers with specialist knowledge in areas of S&T identified as important to national development: agriculture, energy, engineering, infrastructure and ICT. This rapid expansion reflects the national trend, but also the breadth of issues related to science and technology, which span across a broad range of policy areas. Many interviewees pointed out that issues relating to S&T

26 At the start of the programme the library and research sections were together in a single department. However, following restructuring, they are now two separate departments.

31 |


intersected with the work of several committees. This is a marked change from the start of the programme, when science and technology was perceived as a difficult subject which had little bearing on other areas of work.

However, many of the problems identified in the original study remain. For example, the rapid turnover of work in the research department still impacts on the quality of briefings. There is still no peer review process.

“If an MP brings an assignment and says he wants it that day, then I will have to produce it within that timeframe … Most of the time the quality of the work I can produce and the length of the papers more or less depends on how much time I am given to produce the work.” (Ug P 6)

“What is lacking now is the peer reviewing of their work [in the Research Department] … so it becomes difficult to know whether the reports have done an improvement or whether the reports are still at the same level. They need to create a mechanism of peer reviewing of the work they do. You know the problem within parliaments is that they always say ‘I want a report’ to present the next day.” (Ug CS2)

There are still issues with information sharing across parliament, as highlighted in the baseline study. In particular, there has been limited progress towards establishing infrastructure to facilitate better information sharing. The research department still lacks a centralised place to store policy briefings electronically.

3.3 CHALLENGES/AREAS FOR IMPROVEMENT

HAS CAPACITY BECOME EMBEDDED?Ensuring that the benefits of capacity building are not just confined to individuals has been a challenge – success has been mixed. (However, as a participant in the end of programme workshop pointed out, improvements in institutional capacity are harder to measure - interviewees generally feel more confident making statements about their own strengths and weaknesses than their organisation as a whole). Several interviewees felt that the improvements in staff performance were mainly confined to those staff who were directly involved in the training. According to one member of staff who was involved in the mentoring programme:

“It has only been the three of us [who] actually gained, so my thinking is, in terms of improving of how we write our papers and things like that, if more staff were in the mentoring programme I think on an individual basis the staff would gain more.” (Ug PS6)

One donor argues that the small number of individuals involved in the programme means that the chances of achieving a systemic change are minimal:

For the individual staff members, the sort of parliamentary libraries and things, great. It’s an opportunity to train, it’s an opportunity for network building etc. For the MPs the default answer is going to be yes. Are you interested in science? Default yes. Would you like to see science strengthened? Of course, yes … there is clearly a demand … and there’s clearly some positive impact on beneficiaries. But it’s a kind of a systemic change. And that’s really hard to catalyse from a relatively small number of individuals.” (UK NGO 8)

32 |


Poor mechanisms for sharing information and learning were also a factor. In the majority of cases, staff who had taken part in training were not required by their managers to feed back to the wider team (except in the form of formal reports, which had little impact).27 While all the activities were endorsed by management, active engagement from senior staff would have been necessary to ensure that wider benefits from training and were better integrated into the programme.

The impact of weak management structures was seen in the case of the remote mentoring scheme, in which a member of staff was linked up with an external expert, in order to producing a policy briefing under the expert’s guidance. The first time the scheme was piloted, it failed to take off, because the staff member did not engage with the mentor. The manager responsible did little to help get the project back on track.

“I know that individuals have benefitted very much from the programme, but there’s a sense that you take individual capacity and feed it into the organisation. I think that’s where things were broken a bit: unstable managerial structures.” (UK NGO 9)

In principle, decisions on who should participate in activities are best made by managers, who should understand the competencies of their staff and the needs of the organisation. This tends to be the protocol when working with parliaments. However, due to the issues of weak management already highlighted, the wrong staff were sometimes put forward for activities, thus reducing their impact.

One senior staff member says that due to lack of resources, the department itself only offers limited training opportunities and is not able to tailor these to new staff (for example through induction programmes). Thus there is still a heavy reliance on donors for training activities, and there is little evidence that in-house provision of training has improved.

CAUSE AND EFFECT

Overall, interviewees report many improvements – for example more research staff allocated to scientific issues, better organisation in the library, and a more active S&T Committee. While these changes cannot be directly attributed to POST, the momentum and enthusiasm generated by the POST programme appear to have had a positive impact on parliament. They have also raised the profile of S&T and of research within parliament more generally.

Chapter 7 explores further what can be learned from the POST programme to help build long-term capacity.

27 Informal communications from staff.

33 |


4. FINDINGS: IMPACT ON MPS

4.1 THE NATURE OF THE PROBLEM In spite of MPs expressing considerable enthusiasm for S&T, it had a low profile at the start of the programme. S&T issues did not receive much attention on the floor of the house, and those debates that took place were poorly attended. The key issues identified in the baseline study were as follows:

Availability of information: MPs identified the need for more up-to-date information on S&T from the parliamentary library and research department, although they praised the approachability of the staff. Turnaround times for research requests were also raised as an issue, with MPs expecting quality work to be delivered on faster timescales than was feasible. According to one MP, “Our library, I could say, is still lacking with science and technology materials, but they offer you [good] service when you approach them.”(MP04, baseline study)

Scientific and information literacy: analysis of selected debates on S&T highlighted that many MPs were lacking in a basic understanding of scientific concepts. The baseline study also identified a lack of critical skills needed to identify, interpret and communicate information on S&T.

Networks: MPs on the S&T Committee highlighted a lack of expertise amongst members as one reason why the committee lacked authority when dealing with STI issues. MPs said that parliament’s links with indigenous scientific organisations needed to be strengthened.

Resources: MPs interviewed for the baseline study highlighted a lack of parliamentary staff who could handle S&T issues: “The Research Services are ok, except that the staff are overwhelmed by the work they have to accomplish.” (MP03, baseline study)

Parliament’s openness/receptiveness to handling science: a number of interviewees we spoke to for the baseline study argued that parliament took longer to pass bills requiring a greater understanding of S&T. Seven out of a total of 23 bills still pending at the end of the eight parliament (2006-2011) had an S&T focus.28

Impact of the S&T Committee: o Low profile: MPs interviewed for the baseline study thought that the committee had

a low profile – attendance at committee meetings was very low. It was often overlooked in favour of higher profile committees when it came to taking the lead on scrutinising bills. According to one committee member, “People have too many commitments, and as a result very few people attend science and technology meetings, which appear as non-priorities.” (Focus group participant, baseline study, 2009)

o Limited performance against mandate: part of the committee’s remit is to review the scientific and technological content of all bills laid before parliament, as well as to initiate new bills. In the eighth parliament the committee did not review or initiate any bills. Committee members felt that there were many bills that ought to have been referred to them but that the committee was often overlooked.

o Limited outputs: the committee only produced one report on its activities during the eighth parliament. A review of the report (conducted by the programme manager) concluded that the committee could have improved the way in which its messages were communicated by making the report more concise and giving it a clearer structure. Another key issue was that the basis for the recommendations made in

28 The Narcotics & Psychotropic Substances (Control) Bill 2007, The Plant Variety Protection Bill 2010, The Uganda National Meteorological Authority Bill 2010, The HIV/AIDS Prevention & Control Bill 2010, The Uganda Forestry Association Bill 2010, The Plant Protection and Health Bill 2010, The Anti-Counterfeiting Goods Bill 2011.

34 |


the report was not always clear. For example, the report contained a number of recommendations relating to the biotechnology sector, but did not explain clearly how it arrived at these.

The POST programme carried out a number of activities focused on MPs which were aimed at addressing the above issues. Of these, the MP-scientist pairing scheme was a method that had not yet been trialled in Africa (see Box 4). The aim of the scheme was to build a network incorporating MPs and the scientific community, whilst boosting the profile of the S&T Committee and improving overall levels of interest in S&T across parliament.

4.2 HOW HAVE THINGS IMPROVED The end-of-programme interviews indicate that MPs’ ability to tackle S&T issues has improved, although it is not clear how much of this improvement can be attributed to POST.

AVAILABILITY OF INFORMATION One MP points out that he makes more use of the parliamentary research department as a result of the POST programme. This is corroborated by a member of staff:

“One thing that it (the POST programme) encouraged me to do was to seek knowledge from our research department, especially people handling science. And so, you know, it encouraged me to use other people to search for information, people who have the expertise ... So the best thing really that it brought to me was to seek knowledge from those who maybe have it readily or those who have more time to seek that knowledge on my behalf.” (Ug MP1)

“On emerging issues like oil, where we didn’t have enough trained capacity as a research and library, they [POST, INASP] tried to fill in the gaps. They brought us closer to our clients, the members of parliament.” (Ug P11)

However, the same MP feels that access to online resources is still limited. One of the librarians points out that this is partly because of difficulties in getting MPs to attend training:

“Personally I’ve been conducting research but, you know, when I look at the websites given to parliament, the ones that we can access without having to pay directly, it’s quite limiting. I also think that our researchers here need more training.” (Ug MP1)

“It [the use] is not as good as I would like it to be. Yes, but that’s because many … most of the people did not get the training in the first place. They find the whole thing very complicated because not everybody got their training, so they must come to the library and then we must take them through if they come. And that is easier for staff, but for members it is almost impossible to get them to train.” (Ug P8)


Assessment of whether there has been an overall improvement in scientific literacy amongst members is beyond the scope of this evaluation. The baseline study highlighted gaps in MPs’ scientific literacy evident in selected debates. However this did not allow scientific literacy levels to be quantified as results are highly dependent on the choice of debate and the profile of the MPs in attendance. Moreover, as one participant in the end of programme stakeholder workshop argued, debates are not necessarily a reliable way to measure literacy levels as they do not accurately reflect use or understanding of evidence.

Some of the comments in the end-of-programme interviews indicate that both MPs and staff have some understanding of what constitutes objective information. They also have some understanding

35 |


of the need to back up arguments with reliable evidence. However, analysis of policy briefings and committee reports does not always back this up (see 3 and 4)

“Our approach is to listen to both sides of the coin and then weigh whether there are merits in the arguments of those who are against... If they are not able to provide hard evidence against something, we put a big question mark, because it is also not wise to just have activism for the sake of being hard … [it must] be activism with facts. Science needs facts to be backed with empirical evidence before you adopt any move.” (Ug MP2 (S&T Committee member))

NETWORKS The POST programme was the first to pilot MP-scientist pairing in Africa (along with the Royal Society). Eleven of the interviewees had taken part in or been involved in the MP-scientist pairing scheme. The feedback from those who took part in the first round of the scheme, in 2008, was very positive. Five years later, they still remembered accurately what the scheme had involved, and they could point to benefits, such as advantages for their constituency, scientists being able to assist with legislation, and a general improvement in mutual understanding between MPs and scientists. Feedback was more mixed in 2010.

“I hope is hasn’t … you know, just been a one-off thing and gone, because for me … even now I still communicate with my scientist over different issues. But I don’t know whether the other MPs continued, or even those who are out of parliament continued, in communicating.” (Ug ex MP1 (2008 scheme))

As one donor points out, the impact of the scheme is heavily dependent on the choice of MP:

“I’m pretty sceptical about long-term outcomes and benefits from that. It has the potential to be awesome if you get the right MP and they’re hooked ... you can capitalise on a whole load of other change. But if you get the wrong MP and they’re just after a jolly, a per diem and you know some opportunity to match fund, then, and they lose their seat a year later, then you’ve achieved almost nothing.” (NGO 08)

Comments from UNAS indicate that from its side, links with parliament are perceived to be stronger as a result of the programme:

“UNAS has been one of the big partners fighting for the introduction of the biotechnology/biosafety bill which is now being discussed in parliament, and we are being part of that because we deal with the Chairman of the Science and Technology Committee and it has made our lives easier in pushing for that bill to be discussed in parliament.” (Ug Sc 2)

RESOURCES As highlighted in Chapter 3, the number of researchers handling S&T issues has increased since the start of the POST programme. In addition, MPs say they have observed improvements in the skills of existing staff.

“I saw the training programme going so well. I interacted with members of staff who did go for this training programme, some people in the library, and others who went to London for an attachment to POST, and when they returned they came back more productive.” (Ug MP1)

Box 4: MP-scientist pairing

36 |


The POST programme in Uganda demonstrated that the MP-scientist pairing can in principle work in Uganda, and that it helps build long lasting ties between MPs and scientists. However, the scheme needs close facilitation and should be timed to avoid the run-up to elections. Although POST tried to pass the management of the scheme to an in-country organisation (UNAS), this did not work well. Also, expectations need to be carefully managed so that participants (especially MPs’ constituents) are not misled to expect practical assistance.

“The scientist was able to come here, learn our procedure, see how we work, you know. Encounter our working environment. I was also able to go to the scientists’ workplace … study his environment, see the kind of challenges that he faces.” (Ug MP5)

“I went to Kibogo and had a meeting with the people of the constituency. They were handling a bill in parliament where they invited me to go. They planned for the bill. I actually submitted my interests so I actually reviewed the bill and made some proposals to the committee because I think at the time he was [involved in the Agriculture Committee].” (Ug SC3)

“The MP pairings scheme, I think what that’s had as an outcome has been to actually just raise the profile of science in parliament. And even just letting people in parliament know that there are scientists out there in Uganda, that UNAS exists, I think that’s an important outcome of that.” (UK NGO 7)

While in the UK the scheme is focused on encouraging dialogue between MPs and scientists, in Uganda it gave rise to expectations of practical benefits and thus led to disappointment. If attempted in future, the boundaries of the scheme should be clearly communicated to all involved, including constituents.

“Each time I visit this community they keep asking, ‘What has happened to out scientist, what has happened to our water project?’ And you know, they still continue to drink unsafe water, so whereas POST in this respect of MP-scientist paring schemes is spot on in terms of giving information, scientifically established and well researched information, where it requires it to be rolled out, you know, in practical terms, there needs to be a budget for this.” (UG MP1)

Many participants highlighted the problem of lack of continuity, with many of the MPs who participated in 2008 not returning to parliament. However, while this might be a loss to parliament, most of those MPs are still working in policy-related areas, so the benefits are not lost to the country as a whole.

One NGO representative thinks that the scheme runs the risk of scientists being given credibility, even when they are speaking about issues outside their field of expertise. This is a widely acknowledged problem with any involvement of ‘experts’ in policymaking and is not unique to MP-pairing. However, it could be more of an issue when conducted in an environment where there are few other links between parliament and the scientific community.

When the MP-scientist pairing scheme was run for a second time, in 2010, two things were different: first, POST and the Royal Society devolved the organisation of the scheme to UNAS; second, the scheme was organised just before a general election. Both these factors impacted on its success.

“The MP pairing scheme, I participated in it. It was a good idea, but it had a lot of challenges because parliamentarians … are a very busy lot. Even when they are given what they call a recess, like it is now, even now, during a recess, they are just simply such a busy lot. That [is why] the pairing did not work out very well in my opinion.” (Ug MP2)

However, there was positive feedback from the staff who were more closely involved in the scheme in 2010:

“For those who were active enough, I am very certain they got something out of it … I can say, to be honest, I think for most of the staff who have been involved it’s been a great experience.” (Ug PS 6)

37 |


PARLIAMENT’S OPENNESS/RECEPTIVITY TO HANDLING SCIENCE

In the end-of-programme interviews, a government official said he thought it was easier to get bills on S&T considered by parliament than in the past, and cited examples of bills which have been passed in recent years. He linked this to parliament’s increasing openness to S&T, which he attributed in part to the opportunities offered by POST’s programme.

“Then beyond that … parliamentary pairing scheme, we present yet another proposal, and this was not a simple proposal. This was to develop all-round national capabilities in science [for] national science and technology policy. Now this policy had been on the table for over 15 years because ...the issue didn’t seem to be of urgent nature to them at the time... and they had not got full appreciation of science and technology... Now some of these engagements, like ones with POST, had helped them appreciate that science is important and there is a need for something to be done at policy level to address these.” (Ug G1)

While these comments clearly indicate that the interviewee considers POST to have had a significant influence on parliament’s approach to S&T, it should be noted that there are many complex factors influencing the passage of a bill, and receptiveness to S&T is only one factor.

THE IMPACT OF THE STANDING COMMITTEE ON SCIENCE AND TECHNOLOGY

Overall, comments made in the end-of-programme interviews indicate that the committee is more active than in the previous parliament and has a higher profile inside and outside parliament. Many contributing factors are cited in the interviews. These focus mainly on the commitment of the committee chair and on increased interest in science amongst its membership. However, the influence of POST in the previous parliament is also cited as a factor.

More interest in the committee and better attendance at committee meetingsA number of end-of-programme interviewees say that attendance at committee meetings has improved and that members are more committed:

“It has not been that active until of late. Actually I think it is one of the most active committees to date. We register sometimes 100% attendance at the committee sitting … I think it is just the commitment of the kind of membership we now have – they are mainly scientists and have a lot of interests in the committee.” (Ug MP2)

“The members who came on the committee were very interested in that programme of MP and scientist pairing scheme. Actually, all the members on the committee are very interested in science and technology. They are vigilant, they love the committee and they wouldn’t even want to move to other committees because they are interested in what’s happening in the committee and what the committee handles.” (Ug P7)

One member thinks that the improvements are down to staff rather than MPs:

“Now, when it comes to conscription, then you find that the interest may not be there and therefore the contribution of a particular member of parliament is actually minimal. That is one failing … it’s a weakness that I noticed during my time as the chairperson of the committee and I think it’s continued to propagate itself within the current committee. Now the strengths about this is that, we now have, more than before, a fairly well trained and knowledgeable staff. The committee is no longer perceived as this foreign animal.” (Ug MP1)

A more active committee

38 |


The mandate of the committee has not changed since the start of the POST programme:

1) Review, discuss and make recommendations on the S&T content of all bills laid before parliament.2) Initiate bills on strategic issues of S&T for national development.3) Continuously monitor, assess and evaluate activities in public institutions and any other bodies engaged in S&T for national development.4) Examine and carry out research on the performance of the national S&T subsector.5) Examine, recommend and oversee adequate national budgetary provisions for the development of science and technology.

There have been three different committee clerks since the POST programme started. Some information on committee activities has been lost each time there has been a change of staff. Thus we were not able to obtain comprehensive information about everything the committee has achieved during the ninth parliament. However, the current clerk was able to give us some highlights. A summary of the committee’s activities in the second session (July 2012 to May 2013) is given in the table below.

Activity area Key Achievements Area of mandateReports tabled for debate, considered and adopted by parliament

African Space Research Programme (July 2012)

The status of science and technology in public universities (August 2012)

The status of science and technology in Uganda’s technical colleges (August 2012)

3, 4

Bills scrutinised by the committee

1st reading of National Biotechnology and Biosafety Bill, referred to S&T committee29 (February 2013)

Two agricultural bills scrutinised in conjunction with the Committee on Agriculture, Animal Industry and Fisheries

1, 2

Stakeholder consultations Over 20 different stakeholder groups from government, research institutes, farmers groups, NGOs forum, Doctors, public universities, plus a public hearing on the National Biotechnology and Biosafety Bill

1

Field visits for the bill Visits to 21 Ugandan research institutes, technical colleges, benchmarking tours to Kenya and South Africa.

1

Passing motions The committee passed a motion to create a ministry for science and technology (March 2013)

1

Future plans Initiation of private members bill on bioweapons is under consideration

Post-legislative scrutiny for National Biotechnology and Biosafety Bill (study visit to UK in planning)

2

New area

As well as scrutinising a high profile bill and taking part in extensive stakeholder consultations, the committee has also presented three reports to parliament, which the current clerk says generated debates on the floor of the house which were attended by members who are not part of the committee. In addition, a delegation of Members visited the UK in November 2013 to inform post-

29 Note that initially a Private Member’s Bill on Biotechnology and Biosafety was tabled by the Standing Committee on S&T. After a dialogue between the committee and the government, the government tabled a bill on the same subject on 5th February 2013, which was referred to the Standing Committee on S&T.

39 |


legislative scrutiny activities on the National Biotechnology and Biosafety Bill. POST coordinated this visit.

Increased profileMany interviewees have commented on the increased profile of the S&T Committee. There are a wide range of reasons for this, such as the increasing national profile of S&T and the commitment of individual members and staff.

“It may be more vibrant in the ninth parliament basing on the comments I hear from members. If a member who was an MP in the eight parliament asks whether the S&T is a new committee, yet it has been in existence for more than two parliaments, then that indicates that little was known about it.” (Ug P9)

One former MP draws attention to the fact that the committee took the lead on scrutinising the biotechnology bill:

“I think now even the fact that a bill can be given to them to discuss [is good] … because before the bills were probably given to the committee on agriculture … their profile has gone up, their reports are made to parliament, so at least they are put on the parliamentary agenda … I think the profile is really up because they are also budgeted for as a committee. They know activities have to be done and [they need to be] more recognised in parliament.” (Ug ex MP1)


CONTINUITY OF MEMBERSHIP AND ROTATION OF STAFF

After each parliament, the S&T Committee is reconstituted, so its membership changed from 2010 to 2011 as a result of the parliamentary elections. In addition, according to parliament’s rules of procedure, the membership of the committee will change after it has served a term of two-and-a-half years. Lack of continuity is therefore a key challenge.

“As members begin to appreciate the mandate, membership changes with the expiration of the term of a committee.” (Ug P9)

A further issue is that committee clerks are rotated every couple of years. The baseline report highlighted that this can lead to loss of long-term memory. There still appear to be some limitations in handover processes, as illustrated by the lack of comprehensive information on what the committee had achieved in the ninth Parliament. One development since the publication of the baseline report is that clerks are now required to provide a written handover on rotation.

TRANSPARENCY IN RECOMMENDATIONS/THE QUALITY OF SCRUTINY

A comprehensive analysis of the quality of scrutiny carried out by the committee in the ninth Parliament is beyond the scope of this evaluation. A study following the passage of the National Biotechnology and Biosafety Bill from its inception through to its implementation would provide valuable insight into the committee’s scrutiny processes. However, as the bill has not yet been passed, this cannot be undertaken yet.

Reviewing the three reports the committee tabled in the second session allows some initial observations to be made. Some positive developments can be seen. In particular, there is a marked improvement in structure and clarity of the reports in comparison with the single report available

40 |


from the eight parliament. However, the underlying analysis behind recommendations made in the reports is not always clear.

For example, the Committee’s report The Status of Science and Technology in Uganda’s Technical Colleges contains the recommendation that “In order to encourage science and innovation, Government and the private sector foundation should deliberately work with Colleges to establish incubation centres for research ideas”. While this may be valid recommendation to make, incubation centres are not mentioned anywhere else in the committee’s report. Thus, it is not clear from the report alone whether the recommendation is the opinion of the MPs themselves or the stakeholders they consulted. It is also not clear what analysis has been undertaken of the benefits of incubation centres over any alternative options before arriving at this recommendation.

These are just initial observations. A more detailed review of the committee’s reports, carried out by experts in committee scrutiny, would be a valuable future exercise.

41 |


5. FINDINGS: BENEFITS TO POST

5.1 THE NATURE OF THE PROBLEM – THE SITUATION IN 2009POST’s programme in Uganda was largely designed to meet the needs of beneficiaries. However, POST has benefited in a number of ways. Before the start of the POST Africa programme, links with developing countries were limited and ad hoc:

Limited network of contacts: POST had no formal links with parliamentary staff or MPs in developing countries. Contact was largely triggered by meetings at conferences or by requests for advice from POST received over email. POST addressed a wide range of development related issues in its briefing papers. However, much of the information for these briefings was sourced from experts in the UK.

Limited experience in evaluation and assessing impact: POST staff had limited experience of designing evaluation activities. POST had not undertaken systematic evaluation of its own impact.

Limited staff experience of capacity building: Staff had no experience of capacity building and few opportunities to interact with developing country colleagues.

5.2 HOW HAVE THINGS IMPROVED?POST’s core remit is to provide scientific advice to UK parliamentarians. Although the main focus of POST’s programme was to improve scrutiny of S&T within the Parliament of Uganda, POST and the UK parliament have benefited in a number of ways:

EXTENDING POST’S NETWORKS IN DEVELOPING COUNTRIES:POST has worked with a wide range of NGOs, think tanks, research organisations and policymaking institutions as part of its programme in Uganda. This has put POST in a better position to provide accurate and unbiased coverage of developing country issues. For example, as a result of contacts made as part of the POST programme in Uganda, POST is better able to identify development issues relating to S&T which are relevant to the UK parliament.

UNDERSTANDING OF BEST PRACTICE AND OF HOW TO EVALUATE IMPACT By exploring how to have an impact in the Parliament of Uganda, POST has learned more about how to maximise its impact in the UK. Lessons learned from carrying out the baseline study in Uganda helped inform POST’s own evaluation exercises in 2009 and 2012. One UK MP says that the baseline report contains many lessons for the UK parliament:

“In many ways, from looking at the report and talking to the staff at Post, some of the most pertinent recommendations are actually also quite applicable to the UK.” (UK MP1)

MORE EFFECTIVE RELATIONSHIPS WITH DEVELOPING COUNTRY COLLEAGUES As a result of the POST programme, parliamentary staff (in POST as well as in other departments) have interacted more closely with their counterparts in Uganda. This has helped UK staff appreciate the constraints that colleagues in developing countries face and has given them better skills in communicating with overseas colleagues. The UK parliament hosts many incoming delegations from these countries. As a result of the POST programme, POST plays an active role in working with these and ensuring that the role of S&T and of research evidence more generally, is highlighted during their visits.

“As a result of that programme, we’ve had POST involvement in a lot of bilateral programmes with other countries in the region as well. I mean Rwanda, we had some researchers come over from the parliament of Rwanda and POST were very helpful in talking about evidence briefing

42 |


and how they do that on scientific issues, because in the parliament of Rwanda they’re all generalists, so they don’t really have the luxury of having a speciality, so that’s always been quite an important thing to include in some of our programmes. And also we had the Chair of the Science and Technology Committee from South Africa, and he really enjoyed meeting POST and understanding about, you know, the sort of specific facility that we have, that we’re very lucky to have here, and that perhaps they’d like to establish in different parliaments eventually when the time comes.” (UK P2)

ENHANCED REPUTATION

POST’s work in Africa has been praised on the floor of the House of Commons. Andrew Miller, former MP for Ellesmere Port and Neston (1992-2015), cited the programme in a debate about the future of POST in November 2012:

“...this house has influence well beyond the shores of the United Kingdom, and when I was in Uganda with the select committee, I was delighted to meet a fellow who had been on one of the POST fellowships through that scheme.”30


LACK OF DEDICATED STAFFING

POST’s primary remit is to serve the UK parliament, so it was not possible to dedicate any POST staff to entirely working on the programme. The programme manager divided her time between work on the programme and work at POST. There were no staff working on the programme full time. One NGO representative comments that this might have given rise to confusion:

“Capacity-building work actually requires dedicated teams and resources and I think for a long time it was physically led by XXX, [who] has done a brilliant job but I’m sure it must have caused a lot of strain on the balance ...” (NGO 07)

THE SCALE OF A PROGRAMME COULD LIMIT ITS ABILITY TO BRING ABOUT SUSTAINABLE CHANGE

One donor questions whether a programme as small as the POST programme (in terms of resources and staffing) can realistically bring about sustainable change:

“[A fundamental question is] do you think it’s possible to do institutional capacity strengthening on a really small resource and person budget? Which I think is something that a lot of these much smaller organisations, much smaller organisations, like the programme that I used to run, really need to think about. Whether it is even feasible to create long-term meaningful change on whatever it is, half a person a week and a couple of hundred grand.” (NGO 08)

A participant in the end of programme stakeholder workshop commented on the limitations of the short timescales of the programme:

“to get to exploring what sort of interventions make sense can be quite difficult, especially when you’ve only got a short period of time, sometimes you can’t enter into a big process of analysis”

30 http://www.publications.parliament.uk/pa/cm201213/cmhansrd/cm121108/debtext/121108-0002.htm - the member is referring to a meeting he had with a Ugandan parliamentary researcher who had spent three months at POST on a commonwealth professional fellowship.

43 |

http://www.publications.parliament.uk/pa/cm201213/cmhansrd/cm121108/debtext/121108-0002.htm


6. FINDINGS: WIDER BENEFITS TO DONORS

6.1 THE NATURE OF THE PROBLEM Before the POST programme started, very little parliamentary capacity building work focused specifically on S&T or looked at how research evidence was used in a parliamentary context.

“I think there’s been loads of work that’s tried to do capacity building to push more and more research information out, but actually there’s very little that really looked at … ok, what about on the other side? What about the people who need to use that research, the policy makers? What’s going on there? What actually is the situation?” (NGO 09)

6.2 HOW HAVE THINGS IMPROVED?Interviewees say that the POST programme has had a positive impact in this area in several ways:

HELPING PARTNERS APPRECIATE THE SUBTLETIES OF WORKING IN PARLIAMENTS

Many organisations are working to improve the use of evidence in policymaking. POST has helped such organisations appreciate the role parliaments play in the policymaking process. According to one donor:

“Everyone talks about parliament being there to scrutinise government, but actually recognising the subtleties of what that means in practice, I think that’s been very, very helpful interacting with POST... You know researchers are going ‘Oh, no one reads your papers anymore,’ but do you know who your paper should be going to? Do you understand the role of the permanent secretary, or clerk, or you know, whatever, an understanding of how it should be and how it works in practice. POST has been very helpful in our conceptualisation of that and the importance of understanding that before trying to fix anything.” (NGO 09)

POST’S WORK IN UGANDA HAS INFLUENCED OTHER PARTNERS’ WORK

Although the POST programme was small scale in terms of resources and staffing, donors say that it has been valuable in generating debate about what methodologies and approaches work:

“In some sense, small programmes like POST and the methodologies they’ve used have made me think a huge amount about exactly what works in capacity building, some of these fundamental questions about whether there’s a programme size which is too small to be effective. So in that sense I don’t know whether they count as learnings, but they certainly count as I think moving the debate forward in the UK about what methodologies do and don’t work, how hard it is to do some of this sort of stuff, and some of the challenges about working in partnership.” (NGO 08)

A number of interviewees have pointed to the baseline study as the first detailed, well defined analysis of how evidence is used in a parliamentary context. The baseline study is flagged up by two donors as most influential aspect of the programme, because it furthered understanding of a problem which other donors were then able to tackle:

“I think that was certainly the first real study of policymaking environment that looked at the ability to handle evidence, and did it in a sort of objective clearly defined way. So I think that was a very influential report, and that report keeps coming up in conversations with people from other sectors, I think it has had an influence there.” (UK NGO 7)

44 |


“Without much to back this up, largely generalizable across a number of African sub-Saharan countries.” (NGO 08)

Its influence is also highlighted by a Ugandan government official, who says it drew attention across parliament, to the importance of handling science and technology effectively:

“Not only in the Science and Technology Committee but other committees, to aim to map out the structure and the pragmatic and procedural issues relating to the handling of science in parliament. It was very informative and educational to members of parliament, and also the committee and also members as individuals … what do they need to be able to effectively perform their function was, what POST wrote in their report and members of parliament were actually active participants in formulating those recommendations.” (Ug G1)

PARTNERS HAVE BUILT UPON POST’S INITIAL WORK

The POST programme generated a high degree of interest: during the course of the programme, additional funds were brought in that amounted to over 40% of the original grant, not including in-kind support.

“You can say that POST contributed to our success in raising about just under half a million pounds funding from DFID to extend the kinds of approach done in Uganda to other policymaking environments elsewhere in Africa, because we took the experience from the POST Africa project and thought, ‘Well, what can we apply elsewhere?’ So it was quite influential in that sense.” (UK NGO 2)

Since the POST programme started, interest in promoting evidence-informed policymaking in the has increased markedly. In December 2012, DFID launched a call for proposals to build the capacity of decision-makers in low and middle income countries to use research evidence and rigorous data to inform their decision-making, with a total funding of £13 million. One donor who was interviewed said that the POST programme had played a role in influencing DFID:

“I think it was really important to start that ball rolling. And I think as a result of that … basically it was only because of that that the whole INASP programme started, and that’s now I think doing some really great work. And I think you could make the argument that it was because of that INASP work, or that that played a relatively big role in influencing DFID now to launch a new call for work to do specifically that.” (NG0 09)

However, the interviewee did not think the POST programme had had a significant impact in Uganda:

“I’m not sure that the POST Africa project did an amazing job of actually dealing with capacity gaps. I wouldn’t say it did. I think it identified a problem, highlighted it, and then other people have gone on to deal with that better.” (NGO 09)

6.3 CHALLENGES DISSEMINATION OF LEARNING WAS LARGELY THROUGH DIRECT CONTACT

Most interviewees said they learned about the POST programme in Uganda through direct contact with individuals involved. Donors indicated that it would have been useful to have more regular feedback rather than relying on personal contact:

“What they’re doing with the parliament of Uganda of course is also very small, but it would be interesting to receive information on that on a more regular basis, without necessarily to touch base you know in a personal way.” (UK NGO 3)

45 |


46 |


7. LESSONS FOR THE FUTURE

In this chapter we explore some of the factors which could have increased the impact of the programme and discuss lessons learned for the future.

7.1 BUILDING INSTITUTIONAL CAPACITYTo build institutional capacity, internal processes for sharing best practice need to be in place. Interviewees were clear about the benefits of the POST programme to individuals – for example to Ugandan parliamentary staff and MPs who were directly involved. However, they were less clear about the wider benefits to colleagues and the extent to which the programme had built long-term capacity in the Ugandan parliament. Questions were also raised about what kind of organisations should undertake this kind of work in future and what the work should involve. A number of factors would need to be addressed within the target institution(s) to ensure that the benefits of any future activities extended beyond the individuals involved:

More effective mechanisms for sharing learning: Ugandan staff who have been directly involved in POST’s training activities say they did not always have the opportunity to feed back what they had learned to their colleagues after the training was completed. In future, mechanisms for sharing learning with other colleagues could be formally integrated into the design of training activities. However, staff need support from their own organisations to ensure that they have the time and the opportunity to do this.

More effective managerial structures: According to one donor, “Individuals have benefitted very much from the programme, but there’s a sense that you take individual capacity and feed it into the organisation. I think that’s where things were broken a bit: unstable managerial structures.” A lack of guidance from managers prevented some training activities from taking off. Future activities could try to engage senior staff more actively, or even offer training opportunities to senior staff, who are in a position to initiate changes within the organisation. However, in many cases weak management structures and high workloads present a challenge.

Better internal infrastructure and processes: The Parliament of Uganda still lacks the infrastructure to facilitate effective information sharing. For example, there is as yet no centralised system for the electronic storage of documents although there is one under development.4 This makes it difficult for colleagues to share best practice with each other.

7.2 ACTIVELY ENGAGING SENIOR STAFF Future programmes should explore ways of obtaining more active engagement from senior staff, who are in a position to bring about change. In Chapter 3 we discussed the lack of engagement from senior staff and how this limited the impact of training activities. For example, the staff who visited POST for two to three months came away with a wide range of valuable recommendations, which they presented to their departments. However, these staff had little success in bringing about sustained changes. This was in part due to the fact that those staff who undertook the fellowships were not in a senior position and therefore did not have the authority to drive such changes through. In the future, methods for more actively engaging senior staff or even providing training opportunities to senior staff could be considered.

Ugandan staff suggest integrating more detailed follow-up mechanisms into programme design. One staff member suggests more mentoring projects and opportunities to refresh knowledge.

47 |


“Refresher courses, because sometimes, when you leave stuff, somehow it goes away so if we have those refresher courses it would be good, and another thing is to have mentoring projects.” (Ug P6)

However, this depends on resources. In the case of Uganda, while extra resources have been allocated to allow more staff to be taken on, there is no evidence that in-house training resources have been scaled up to match this. At the same time, there is evidence that the high number of donors offering training opportunities creates confusion.

7.3 REDUCING LONG-TERM RELIANCE ON INTERNATIONAL EXPERTISEFuture programmes should identify ways of overcoming the perception that ‘outsiders are best’. Donors highlight the importance of working with in-country trainers, to reduce long-term reliance on international expertise.

“I think things need to be embedded within actual institutions, rather than being run as external workshops that people go off to, so things that are actually in-house, that’s important.” (UK NGO7)

“There are a couple of things we have decided to change: one is the reducing the single workshop model and there other is recognising that there are institutions with the remit to train and teach in country.” (UK NGO 9)

Some training workshops occurring under the auspices of the POST programme (for example the information literacy workshops organised by INASP in 2009) followed the ‘training of trainers’ model31 to ensure that the benefits of the training were not confined to individuals. The fact that there appears to have been an improvement in use of evidence across the research department could be linked to this (see Appendix C). However, as discussed previously, it is difficult to link any single activity to an observed change.

However, there is still a feeling amongst the beneficiaries, that training is more effective when delivered by an external trainer. There is a need to understand where the perception that ‘outsiders are best’ originates and to identify ways of overcoming it.

“I think it’s a natural tendency for human beings, when training is being delivered by someone you know, it’s like … but maybe the one by POST is delivered by someone different, who is… the people may take it differently from that one which is here, delivered by myself.” (Ug P 8)

7.4 EXPERTISE IN CAPACITY BUILDING AND PARLIAMENTARY PROCESSESFuture programmes should bring together experts in both capacity building and parliamentary processes. The end-of-programme interviews have highlighted some of the challenges faced when a capacity building programme is driven forwards by a parliamentary body such as POST, for whom capacity building is not the core remit. A lack of dedicated resources and lack of capacity building experience were two key issues raised. According to one donor: “There was a real understanding of … the aim of what they were trying to get to, but there wasn’t necessarily such a good understanding of how one could get there.” (NGO 07) Nevertheless, the parliamentary expertise of the POST team provided valuable insights when combined with external expertise in capacity building.

31 Reference about training of trainers

48 |


POST’s understanding of how parliaments work, and where research evidence fits into the parliamentary process, was seen by donors as being highly valuable. However, they also highlight POST’s lack of capacity building experience:

“I suppose the POST Africa programme, it has these really good points in that it’s run from parliament, so it’s people who really understand that. But what it doesn’t really have I think is expertise and knowledge in capacity building. I think that was probably was the negative side of it. NGO 07

The comments highlight the fact that to run a successful capacity building in a parliament, input should be sought from both those who understand how parliaments work, and from those have experience of capacity building. Being a practitioner may provide useful insights, but it does not necessarily give a person the skills to deliver effective training.

However, identifying and forging links with parliaments can involve a lot of bureaucracy and use up a lot of time. Being a parliamentary organisation, POST was able to forge links with other parliaments and thus create avenues for other organisations to work with parliaments.

Several interviewees pointed out the importance of taking cultural context into account – it should not be assumed that a model which works in one environment will work in another:

“These projects, you can write while sitting in an office and you take it to the place to be implemented but it cannot work, but it can work in another environment. What works in the UK parliament will not be exactly what works in the Ugandan parliament.” (Ug CS 2)

Consulting social scientists with country-specific expertise was raised during the May 2013 workshop as one way of building sensitivity to different cultural contexts.

7.5 FOCUSING ON A SINGLE COUNTRY Focusing on a single country allows multiple approaches to be trialled in a single context. This is an unusual approach in capacity building in this area and is seen as being more risky than working in several countries. In this case, however, the focus on a single country has allowed greater continuity between activities, and more opportunities for follow-up. It also provided the opportunity to test out multiple approaches in a single context:

“The long-term approach of working with the parliament, I think, was quite impressive, and I look forward to the overall report of the outcomes. I think they managed to trial a lot of really compelling approaches, the remote mentoring, the pairing scheme, the sort of workshop level things. And to do all that stuff within one context, I think that’s quite impressive. The tendency is to do one here, and go somewhere else and do it, so it actually allows you to look at multiple approaches; that’s been a great success.” (NGO 09)

It may be difficult to identify the sustained benefits arising from any single activity conducted as part of this programme. However, because POST focused on a single parliament over a long period of time it was able to build relationships and develop an in-depth understanding, not just of the organisation but of the strengths and weaknesses of different individuals. It was also able to ensure that each activity learned from those which had come earlier.

One donor argues that by working with a single parliament, POST maximised the chances of building long-term capacity.

49 |


“Intellectually, I’d say it is original. In practice it is, because most people recognise that it takes time, you need to have relationships, you know. Everyone says the words but very few people actually work that way. And the advantages to working that way are long-term capacity. But there are disadvantages, shocks when the relationships break down, and there are tensions, there’s a propensity to fail or there’s a higher chance of failure. But it’s a risk I think it’s worth taking. I think that POST was quite brave to take that risk and to work that way.” (UK NGO 9)

7.6 THE RESTRICTIONS OF FOCUSING ACTIVITIES ON PARLIAMENTS The exclusive focus of the programme on parliaments was controversial. Some donors argued that parliaments in African countries are often weak institutions with little power. Therefore, even if they have the skills and resources to tackle S&T issues effectively, this will have little impact over the policymaking process as a whole, unless the scrutiny function of a parliament is recognised and supported.

“Even if they were handling science and evidence really well, it wouldn’t necessarily mean much, because in a lot of countries they don’t really have any power, so they’re not fulfilling this kind of scrutiny function that maybe you might hope that they’re doing.” (UK NGO 5)

The fact that parliaments are weak institutions is not seen as a reason to avoid working with them. Indeed, many capacity building activities focus on parliaments for this very reason. One participant in the end of programme stakeholder workshop pointed out that working with the executive (i.e. government) might be more effective in terms of influencing specific policy outcomes, but it was important to work with parliaments to improve the quality of debate on an issue and engage civil society.

Participants in the workshop also highlighted the importance of not setting over-ambitious targets such as ‘achieving more effective scrutiny’, which may depend on factors outside the programme’s control.

“in every context, top level political decision making will seem arbitrary, but what you need to is to build up capacity to ensure issues are brought up for consideration, that’s all you can do”

One interviewee highlights that the Parliament of Uganda is a difficult environment in which to run a capacity building programme, due to the uncertainties in the political calendar and frequent recesses. Another issue is the high turnover of MPs at elections, as raised earlier in this report.

“However much you try to train a member of parliament, in the Ugandan circumstances, there are many challenges … The disruptions are many, even for those who want to do things. Yesterday, out of the blue, parliament went into recess.” (Ug CS2)

Another interviewee questions the value of working with parliaments to improve their handling of science, given their lack of impact in many African countries:

“Parliaments in many African countries … don’t really have much of a role anyway, so even if they were handling science and evidence really well, it wouldn’t necessarily mean much because in a lot of countries they don’t really have any power, so they’re not fulfilling this kind of scrutiny function that maybe you might hope that they’re doing.” (UK NGO 7)

50 |


There are a vast number of capacity building programmes aimed at parliaments, but these generally focus on strengthening governance processes rather than use of research evidence. This comment raises the question of how well established a parliament should be before it is worth concentrating efforts on enhancing its use of research evidence.

Another interviewee argues that working with parliaments alone is not enough – the problem is much more wide-reaching:

“It’s a systemic issue which requires strong institutions, research base, connection between the research base and the policymaking and policy scrutiny communities. It requires some funding for some of those intermediary type organisations. At a really basic level it requires a basic education system: you know you’re not going to be able to recruit good parliamentary clerks if your science teaching at primary school is awful. So it is one of those really challenging issues which requires … multiple different bits of the sort of innovation machinery before it actually becomes a reality.” (UK NGO 8)

7.7 PREPARING FOR ELECTIONSDisruption caused by elections needs to be anticipated and planned for. Parliamentary elections took place in Uganda in February 2011. In the lead-up to these elections as well as in their aftermath, many parliamentary activities were suspended so it was very difficult to interact with MPs or to implement programme’s activities. Future programmes should take account of election dates and activities should be timed such that they do not require interaction with MPs when they are likely to be campaigning.

7.8 WORKING WITH PARLIAMENTARY STAFF Working with parliamentary staff avoids loss of continuity at elections. There was a high turnover of MPs: many of the MPs who worked closely with POST at the start of the programme lost their seats. According to one scientist, “Unfortunately every term we have a new lot of MPs come in and the other ones have gone. They will go with that knowledge and we have to start again”.

However, because the programme focused on staff as well as MPs, loss of continuity was minimised. It should also be noted that working with MPs is not a wasted investment even if they lose their seats, as former MPs can go on to influence policy in other ways.

7.9 ACKNOWLEDGING THE RISKS OF DEVOLVING MANAGEMENT Devolving management to create sustainability brings risks as well as rewards. It is widely agreed amongst donors that devolving management of activities to in-country stakeholders is key to creating sustainability. With this in mind, from 2010 onwards management of the POST programme was largely devolved to the Ugandan National Academy of Sciences. A member of staff at UNAS was assigned the role of ‘Parliamentary Liaison Officer’ to manage POST’s activities on a day-to-day basis. This role was funded by POST. While this officer was in place, his role in linking parliament and the scientific community was valued by MPs and staff in the Parliament of Uganda.

“He was very resourceful and for the Chairperson, if we had a meeting and [the liaison officer] was not around he would say ‘Where is [liaison officer]? Did you tell him we have a meeting?’ I’ve told you, the Chairperson, he doesn’t have any science background so we always used to rely on [liaison officer] alone.” (Ug P7)

Comments from UNAS indicate that from its side, links with parliament are perceived to be stronger as a result of this role and UNAS’s involvement in the POST programme:

51 |


“UNAS has been one of the big partners fighting for the introduction of the biotechnology/biosafety bill which is now being discussed in parliament, and we are being part of that because we deal with the Chairman of the Science and Technology Committee. It has made our lives easier in pushing for that bill to be discussed in parliament.” (Ug Sc 2)

However, there have been a number of drawbacks to devolving activities in-country. For example, UNAS did not have the resources to maintain the role of parliamentary liaison officer once the POST programme had come to an end. Thus the long-term benefits of funding the role were limited.

Comments made about the organisation of activities indicate that in this case, devolving activities to an in-country organisation hampered the ability of the POST programme to meet its original objectives. A number of activities failed to take off once management was moved in-country. In the end-of-programme interviews it was highlighted that devolving activities in-country is heavily dependent on the capacity of the partner organisation as well as the scale of the programme.

“There’s a really fundamental question for smaller organisations like POST about to what extent is it possible to drive systemic change from a small budget and with partners like that.” (NGO 08)

7.10 REDUCING CONFUSION THROUGH LACK OF COORDINATION Lack of coordination amongst donors can confuse beneficiaries. A very large number of donors are working with the Parliament of Uganda, often without full knowledge of each other’s activities. Efforts to coordinate these activities have increased during the life span of the programme,32 but it is clear that the number of donors offering training can confuse the beneficiaries. In the final interviews, many interviewees were confused over exactly what training they had attended and who it was delivered by.

“I wasn’t trained by POST. Only one time I attended, what was it called, something breakfast, you know, at the time [POST] was still running.” (Ug P7)

POST was reported as being only one of many organisations running training programmes to strengthen capacity across parliament.

“I should say POST was not working in isolation … without trying to say that this came from that and that from this, in 2007, first of all our staff were trained in advanced internet searching. They were trained in information literacy so as to able to screen and authenticate valuable information from the junk and that was a very big step because internet has been developing bad and good. So in terms of training, I give credit to the institutions that enabled us to access that. Around the same there was this African Royal Society also based in the UK and came around the same time. They would research on areas of our interest and then send us reports.”

“The people you’re aiming to help may be working with six or seven other donors and they won’t see them as relevant to what you’re doing but they could be hugely relevant.” (UK P 3)

NGOs raise concerns that the saturation of training programmes creates fatigue:

32 The UK Commonwealth Parliamentary Association runs quarterly meetings where all Westminster based organisations involved in parliamentary capacity building activities come together to update each other on their work and identify overlap.

52 |


“There’s generally an initial enthusiasm, people kind of say yeah, yeah, yeah we want to be on this programme, but it doesn’t necessarily mean that there’s real energy to see it through.” (UK NGO 7)

One participant in the end of programme stakeholder workshop said that the focus of capacity building activities on training workshops often comes about because they are a good way of creating visibility and are relatively easy to deliver:

“It’s a lot easier to request and suggest a training workshop than to get into the nitty gritty of how to shift process and politics”

8. FUTURE DEVELOPMENTS

While POST does not currently have funding to support further work in Uganda, it will continue to provide informal support and participate in activities on an ad-hoc basis. For example, the Department of Research Services is developing a departmental report writing manual and research policy for the Parliament of Uganda and POST is liaising with them about helping review this manual. POST also helped co-ordinate a visit for the S&T Committee in November 2013. The purpose of the visit was for the committee to seek advice on the implementation of the National Biosafety and Biosecurity Bill (presented to parliament November 2014) from UK colleagues in parliament and government. Staff from the UK Parliament facilitated a training program on ‘Bill Analysis’ in April, 2014 for Ugandan parliamentary research staff. In addition, the team from the UK Parliament assisted in drafting a bill analysis manual which is now ready for printing.

Moving beyond Uganda, POST is currently investigating further opportunities to work with policymaking institutions in low and middle income countries and to share the lessons learned from its Ugandan programme. For example POST is currently collaborating with African Institute for Development Policy (AFIDEP) as part of the DFID funded SECURE health consortium which seeks to strengthen the capacity of policymakers to utilize research evidence in health policymaking in Kenya and Malawi.

Post will complement the in-country activities of the SECURE programme, by selecting four interns from the parliaments of Kenya and Malawi, to spend one month in post offices in Westminster during 2015/16, receiving one-to-one training on a range of key areas such as identifying, synthesising and appraising research evidence. It is envisaged that the internships will be the start of a longer term relationship which will enable POST to provide follow-up support to the selected interns when they return home.

8.1 RECOMMENDATIONS FOR FURTHER ACTION IN UGANDAThis evaluation report has helped identify a number of key actions which would help the Parliament of Uganda achieve the original objectives of this study, which could be taken forward either by parliament itself, or in collaboration with donors.

STRENGTHENING THE ABILITY OF UGANDAN PARLIAMENTARY STAFF TO SUPPORT MPS ON ISSUES RELATED TO SCIENCE AND TECHNOLOGY

Electronic repository: the establishment of an electronic repository to allow centralised storage of policy briefings produced by staff for MPs would help reduce duplication of effort and facilitate better sharing of information within parliament.

Peer review: mechanisms for peer reviewing policy briefings could be established. If there is not sufficient capacity to review each briefing, repeating the review process outlined in

53 |


Appendix C at regular intervals (for example once per parliament) in collaboration with an external organisation (to ensure independence) could be a valuable exercise.

Referencing and attribution: More effort could be made to improve referencing and attribution within the policy briefings themselves, so that it is clear what statements originate from the authors and what originate from third parties.

Policy recommendations: As an alternative to making policy recommendations, the briefings could simply set out a range of policy options with an analysis of their pros and cons. Where recommendations are made within the policy briefings, the underlying analysis behind them should be made clear, along with the reasons for advocating one specific policy option over other possible options.

Further training: POST has delivered a range of training workshops to Ugandan parliamentary staff, ultimately aimed at improving the quality of briefings they produce for parliamentarians. The programme has shown that such training is most effective when it addresses not only the ‘packaging’ of research information but also builds skills in identifying and synthesising research information. Thus, ongoing opportunities for continuing to provide training to new staff in areas such as information literacy, scientific methods and summarising skills could be investigated in collaboration with relevant donors. However, a more fundamental issue is the need to develop skills in critical thinking more broadly. This is beyond the scope of any individual capacity building programme.

Remote mentoring: POST trialled a scheme in Uganda whereby parliamentary researchers were linked up with international experts drawn from research and policy, who mentored them through the production of a policy briefing aimed at Ugandan MPs. This scheme was time intensive but costs were minimal. It could be repeated as a promising way of delivering training to staff, and helping them build links with the research community, without taking them away from the workplace.

Training follow up: Mechanisms for staff to feedback and follow up on training opportunities once they return to the workplace (for example after taking up overseas fellowships) could be integrated into the design of future training activities.

The recent restructuring of the parliamentary library and research department in the Parliament of Uganda may provide opportunities for many of the issues identified in this study to be addressed. For example, work is underway establish a centralised electronic repository for storing briefings and other information. Future developments in the area of S&T are likely to focus around the expansion of the science section within the newly established standalone research department. However finding the resources to training the new staff is a challenge.4

STRENGTHENING THE ABILITY OF UGANDAN MPS TO SCRUTINISE ISSUES RELATING TO SCIENCE AND TECHNOLOGY

Induction programmes for parliamentarians could be investigated. Incorporating training aimed at enhancing capacity to access and use (as well as communicate) research evidence, into the induction programmes of new parliamentarians could be valuable at the start of a parliament. Options for encouraging MPs to attend training in information literacy in particular could be investigated.

Committee mentoring: The S&T Committee could consider setting up an ongoing mentoring partnership with an S&T committee in another country, to share best practice and help reduce lack of continuity after elections and after the reconstitution of the committee. One of the issues that could be considered is whether reports produced by the S&T Committee in Uganda should include more detail on the underlying analysis behind any recommendations made, along with the reasons for advocating one specific policy option over other possible options. This was a preliminary conclusion of POST’s evaluation, but it needs to be backed up

54 |


by a detailed analysis of the S&T Committee’s work by an expert in committee scrutiny. This would be particularly useful once the National Biosafety and Biosecurity Bill has passed and has been implemented.

Linking parliament with researchers: If the MP-scientist pairing is continued as a means of maintaining links with the indigenous scientific community, it should be timed to avoid parliamentary elections, and expectations should be clarified with MPs constituents from the start. It should also involve MPs from other committees as well as the S&T Committee. There are a number of other ways to build long term relationships with the research community, including:

a. Fellowship schemes for researchers: for example POST operates a fellowship scheme in the UK Parliament, whereby young researchers spend three to six months based in the UK parliament, working with staff and parliamentarians, and delivering support in the form of written and oral briefings in a specialist area, often working for a specific parliamentary committee. As well as providing valuable training for young researchers, these schemes help parliament form long-term links with the research community.33

b. Fellowships for civil servants and parliamentary staff: for example the Cambridge Centre for Science and Policy runs a programme under which decision-makers (for example civil servants) are assisted over a period of two years in establishing a network of contacts in academia, in a policy area relevant to their responsibilities.34

33 See http://www.parliament.uk/mps-lords-and-offices/offices/bicameral/post/fellowships/ 34 See http://www.csap.cam.ac.uk/programmes/policy-fellowships/

55 |

http://www.csap.cam.ac.uk/programmes/policy-fellowships/

http://www.parliament.uk/mps-lords-and-offices/offices/bicameral/post/fellowships/


9. REFERENCES

Ecuru J., Nawegulo N., Lutalo R.B., Kasule D., Tujunirwe E., and Akampurira I. (2008) Research in Uganda: Status and Implications for Public Policy, Kampala, Uganda: UNCST

Ghaffar, D. A., et al. (2008) Changing Mindsets: Research Capacity Strengthening in Low and Middle Income Countries, Geneva: COHRED, Global Forum for Health Research and UNICEF, UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases.

Marjanovic, S. et al. (2012) Research capacity-building in Africa: networks, institutions and local ownership, Journal of International Development

NCHE (2010) The State of Higher Education and Training in Uganda: a Report on Higher Education Delivery and Institutions, Kampala, Uganda: National Council for Higher Education

NEPAD (2006), Africa’s Science and Technology: Consolidated Plan of Action, Pretoria, SA: New Partnership for Africa’s Development, Council of Science and Technology

Nabudere, D. W. (2008) Contextual Policy Framework for Developing a National System of Innovation in Uganda, Mbale, Uganda: Marcus Garvey Pan-Afrikan Institute

Parliament of Uganda (2006) Parliamentary Strategic Investment and Development Plan, Kampala, Uganda: Parliament of UgandaPOST (2011a) Use of Scientific and Technical Evidence within the Parliament of Uganda, London:

Parliamentary Office of Science and Technology and INASP (International Network for the Availability of Scientific Publications)

POST (2011b) POST Uganda: Staff Survey, London: Parliamentary Office of Science and TechnologyPOST (2010) POST Evaluation: Summary of Findings 2009, London: Parliamentary Office of Science

and TechnologyUBS (2010) Ugandan National Household Survey, Kampala, Uganda: Ugandan Bureau of StatisticsUNCST (2009) National Science, Technology and Innovation Policy, Kampala, Uganda: National

Council of Science and Technology, Ministry of Finance, Planning and Economic Development.

UNCST (2008) National Biotechnology and Biosafety Policy, Kampala, Uganda: National Council of Science and Technology, Ministry of Finance, Planning and Economic Development.

UNCST (2011) Status Report 2009/2010, Kampala, Uganda: National Council of Science and Technology, Ministry of Finance, Planning and Economic Development.

UNCST (2012a) The Careers and Productivity of Doctoral Holders (CDH) Survey, Kampala, Uganda: Ugandan National Council of Science and Technology

UNCST (2012b) The Quality of Science Education in Uganda, Kampala, Uganda: Ugandan National Council of Science and Technology

World Bank (2011) Science, Technology and Innovation in Uganda: Recommendations for Policy and Action, Washington D.C.: World Bank.

56 |


APPENDIX A: RESEARCH DESIGN

METHODOLOGYThe methodology builds on evaluative work undertaken by POST throughout the life of the programme. This includes data collected at two stages:

The baseline study conducted at the beginning of the programme to assess and understand the needs of the Ugandan parliament.35

The staff survey conducted midway through the programme and just prior to the 2011 general election that captured the impact and perspectives of the POST programme amongst 12 members of staff working in the Ugandan parliament.36

The methodology used to evaluate the POST programme is based on qualitative research conducted retrospectively once the programme had drawn to a close. This included semi-structured interviews, desktop research of policy documents and research reports, peer review of research reports, and a data analysis workshop held in the UK with interviewees and other professionals working in the sector. Quantitative secondary data was drawn from official reports and used to inform an understanding of recent developments in the Ugandan research base. A qualitative methodology allowed us to access the experiences and perspectives of interviewees in relation to the POST programme. We were also able to draw out some of the nuances inherent in evaluating capacity building programmes in other cultural and political contexts, which informed our understanding of the challenges of this type of work and signposted future considerations discussed in Chapter 8.

METHODS INTERVIEWING (FEBRUARY/MARCH 2013)

Semi-structured interviews were conducted with 41 individuals. In the UK, 15 interviews were conducted with individuals working in donor agencies and NGOs operating in similar areas of capacity building in science and technology as well as in the UK parliament. In Uganda, 26 interviews were conducted with MPs, clerks, researchers, librarians, members of civil society organisations, members of the scientific community, and members of intermediary, scholarly and policy offices. All interviews were semi-structured using a combination of open and closed questions. Interviews were recorded using a digital recorder. They were subsequently transcribed by one of three transcribers. The completed transcripts were uploaded to NVIVO (qualitative analysis software) for analysis by Dr Holden. Each transcript was coded using both emergent and applied codes. Emergent codes arise from the data and applied codes had already been devised from the outset of the evaluation to correspond with the baseline and initial skeleton report by capturing the availability of scientific information, levels of scientific literacy, and modes of communication and networks. Codes were sorted into themes that informed the discussion of findings set out in this report. Interviews in the UK were conducted by Dr Kerry Holden and in Uganda by Harriet Nabunnya. An interview script was drawn up for different interviewees depending on their location (UK or Uganda) and area of work.37 The interview script was designed to facilitate conversation and to allow the interviewee to respond to questions in their own words and to articulate extended answers. The average length of the interviews was one hour.

The interview opened by asking interviewees to give a brief biography that included describing their current role and career progression. They were then asked to describe the POST programme, its

35 POST (2011a) Use of Scientific and Technical Evidence within the Parliament of Uganda.36 POST (2011b) POST Uganda: Staff Survey.37 See Appendix B for a copy of the basic interview script.

57 |


activities and objectives. They were asked about the significance of research evidence and its use in parliamentary and governmental business. Finally, the interview script returned to the POST programme to ask more specific questions about its impact and influence.

Interviewee codes

The code will have a number behind it that identifies the speaker and creates consistency in the presentation of data.

KEY Description

Ug CS Ugandan Civil Society

Ug G Ugandan Government

Ug MP Ugandan MP or former MP

Ug ex-MP Former Ugandan MP

Ug NGO Ugandan based NGO

Ug P Ugandan Parliamentary staff

Ug Sc Ugandan Scientific Community

UK NGO UK based NGO

UK P UK Parliamentary staff

UK MP UK MP

UK Sc UK Scientific Community

DESKTOP RESEARCH

Peer review of Ugandan parliamentary briefingsPOST conducted review of internal briefings (also called ‘research reports’) produced by parliamentary researchers in Uganda. Further details are given in Appendix C.

DATA ANALYSIS WORKSHOP In early May 2013, a data analysis workshop was organised in order to review the data collected from interviewing. Extracts from 10 interviews with Ugandan and UK participants were chosen. The extracts were organised by themes that had emerged through initial data analysis. The workshop included interviewees as well as practitioners working in similar areas of international development and capacity building in science and technology. The workshop was recorded and used in the final evaluation report to comment on some of the obstacles and difficulties involved in undertaking and evaluating capacity building work. The workshop was also used to comment on the impact and influence of the POST programme on the development sector based in the UK.

SUPPLEMENTARY DATA

Baseline study (2008/2009)

58 |


The baseline study (see Box 1 in Chapter 2) was a series of interviews, focus groups and desktop research conducted in the first 12 months of the programme and provides a useful benchmark against which to measure progress.

Interim staff survey (January 2011)POST produced an interim evaluation of the programme in 2011. It was based a survey questionnaire with 12 members of staff. Seven of these worked in the parliamentary research section.

RESEARCH ETHICSEvery effort has been made to anonymise the data. The identifying details of interviewees have been replaced by a code (please see the above box). Identifying details in the data have been removed.

59 |


APPENDIX B: GENERIC INTERVIEW SCRIPT

GENERIC INTERVIEW SCRIPT This basic interview script was adapted to interviewees’ roles and their location.

1) Basic personal information (each interviewee was asked to give biographic information)2) Knowledge of the POST programme3) Perspectives on the use of information on science and technology in the Ugandan parliament.

a) What skills and knowledge do you consider important to the scrutiny of scientific and technological issues by parliament?

b) What resources are available to MPs and Parliamentary staff to support their work on scientific issues?

4) Involvement in the POST programme (if based in Uganda, interviewees were asked about each activity they participated in). a) What was your motivation for being involved with POST? b) How would you say you have benefitted from the events and activities of the POST

programme? 5) The impact and influence of POST (interviewees were asked to give their opinion on the impact

and influence of POST in the Ugandan Parliament)6) General perspectives on POST programme

a) What do you think are the most effective parts of the POST programme? And why?b) What do you think are the least effective parts of the POST programme? And why?c) Do you think it is important that POST has been involved with the Ugandan Parliament?

7) What do you consider the major challenges to improving the scrutiny of scientific and technological issues in parliaments?

8) How in your view should capacity building in this area in the Uganda parliament be taken forward?

9) What would you identify as areas for improvement? 10) Why might capacity building around science and technology be important to Uganda, in

particular?

60 |


APPENDIX C: PEER REVIEW OF PARLIAMENTARY BRIEFINGS

A key function of the parliamentary researchers in Uganda is to provide MPs with written briefings on a range of topical issues, to allow them to fulfil their role (for example to take part in debates, committee work, or even external conferences). This is broadly similar to the role of research services in parliaments across the world, including POST and the House of Commons library. At the start of the POST programme Ugandan MPs on the S&T Committee criticised the quality of the briefings on S&T produced by their parliamentary researchers. A key output of the POST programme was therefore to improve the quality of these briefings. Comments made in the end-of-programme interviews indicate a perception that the briefings have improved. To obtain a more objective view of whether there has been an improvement, POST conducted expert reviews of briefings produced both before the programme started and in its latter stages, and we compared the results. Below we summarise the results before discussing the methodology, findings and limitations of the analysis in more detail.

SUMMARY OF FINDINGS It is acknowledged that the number of briefings assessed is not enough to make the findings statistically significant. However, when considered alongside the comments made by both Ugandan MPs and staff in the end-of-programme interviews, the data support a conclusion that the quality of internal briefings on S&T has improved between the start and the end of the POST programme. Reviewers’ ratings have increased across all criteria and, in particular, there are fewer briefings containing major factual errors and omissions.

Comprehensiveness increased from 2.50/5 in 2009 to 3.25/5 in 2013. There were major omissions in ten briefings reviewed in 2009 but in 2013 only two briefings had major omissions. However, many briefings were still judged to lack the detail required for a reader to make an informed judgement about the issue in question.

Accuracy increased from 2.55/5 in 2009 to 3.43/5 in 2015. In 2009, seven briefings contained major factual inaccuracies and/or incorrect science and three had minor inaccuracies. In the post-2010 review, only one briefing had major factual errors and two had minor inaccuracies. Thus, for the majority of the post-2010 briefings, reviewers had no problems with the factual accuracy of what was presented. However, in many cases a lack of detail, a failure to reference claims, and a failure to present data clearly still had a negative impact on accuracy.

Use of evidence increased from 1.92/5 in 2009 to 2.81/5 in 2013. In 2009, then briefings were criticised for making poor use of the available evidence and for missing out references to key sources. In 2013, four briefings were criticised for making poor or limited use of the available evidence, for missing out key sources or for using references of dubious quality.

Objectivity increased from 2.83/5 in 2009 to 3.02/5 in 2013. In 2009, two briefings were considered to be very biased or speculative. Only one was judged to be ‘reasonably objective’. In the post-2010 review, two briefings were considered to be biased, but six of them were considered to be reasonably objective.

Clarity increased from 2.74/5 in 2009 to 3.52/5 in 2013. In 2009, three of the briefings were assessed as being quite easy to read and well structured. The others received a mixture of criticisms, for example: confusing use of statistics and graphs, use of jargon, poor language, lengthy paragraphs, and the excessive use of copying and pasting of text. In the 2013 review, ten briefings were considered to be well written, of which six were also assessed as being well structured.

However, there is still considerable room for improvement. Particular issues for the future include:

61 |


Attribution: five briefings reviewed in 2013 were criticised for not attributing claims, facts and figures made in the briefings. This meant that reviewers did not know where the claims originated and were not able to judge their validity.

Making recommendations: eight briefings reviewed in 2013 were criticised for making unsupported recommendations. Many reviewers have highlighted the fact that the briefings need to be much clearer about how any recommendations have been arrived at. A few reviewers have pointed out that other neutral parliamentary bodies (such as POST) do not consider making recommendations to be within their remit.

Given the wide range of factors affecting the quality of the briefings, the observed improvement could have a number of causes:

There has been an increase in the number of specialist staff (two additional staff with scientific backgrounds have been recruited between the two review periods).

Training provided to staff (for example by POST, another external body, or internally) may have had an impact.

Staff may be benefiting from increased access to external resources (as reported in the end-of-programme interviews).

It is not possible to identify one specific cause. It is likely that a combination of all these factors has led to the observed improvement. It is unlikely that the improvement is due to any improvement in quality control, or any increase in the time available to prepare the briefings: the end-of-programme interviews do not provide any indication that the review process for briefings has changed substantially between the two periods. Moreover, time pressure is still highlighted as an issue.

METHODOLOGY SELECTION OF BRIEFINGS

Fifteen internal briefings on scientific or technological issues were selected by POST from what we refer to as pre-2009 and post-2010 periods respectively. Note that these periods covered several years (Pre-2009 covers June 2003 to October 2008; Post-2010 covers Feb 2010 to November 2012) because of the need to obtain enough briefings for the analysis.

It is not possible to say what proportion of the total number of briefings on S&T produced during these periods the selection constitutes. For the pre-2009 period, the research division provided POST with a document that listed briefings written during that period. However, this document was not fully comprehensive, and a similar document was not available for the post-2010 period. For the post-2010 period, researchers were simply asked to share with POST all the briefings which they or their teams had produced in recent years (which were largely stored on individuals’ hard drives). The POST programme manager then selected briefings for the analysis.

The number of briefings selected from each period was limited to 15 due to the limited availability of reviewers and the need to select briefings which had sufficient S&T content to be relevant to POST’s programme. In the post-2010 period (which also covered an election during which no briefings were produced) only 23 briefings had an appropriate focus for this analysis; we only had resources to review 15 of these.

Below is a list of the titles and publication dates of briefings considered for the pre-2009 and post-2010 periods respectively. The pre-2009 review took place in 2009, while the post-2010 review took place in 2013 when the POST programme had come to an end.

Pre-2009 review

62 |


Date Topic Briefing numberJune 2003 The use of polythene and its

waste disposal in Uganda11

October 2005 Reduction in the levels of Lake Victoria

1

November 2005 DDT in public health: are there alternatives

10

September 2006 Gender and HIV/AIDS 2September 2006 What the government of Uganda

has done on reproductive health7

November 2006 Energy policy in Uganda 6November 2006 Impact of the current Electricity

Crisis8

November 2006 Fact sheet on the ban of the use of polythene bags

12

November 2007 Organic agriculture for better nutrition and wealth

9

July 2007 Climate change and global warming

14

July 2007 Fisheries development project evaluation

15

May 2008 The Ugandan Parliament and the fight against HIV/AIDS

3

September 2008 Diseases affecting women and girls (other than HIV/AIDS)

4

October 2008 Climate change, sustainable development models and renewable energies

5

October 2008 Cities, climate change and the role of the global parliamentarians forum

13

Post-2013 reviewDate (post 2010 review) Topic No.Feb 2010 The politics of breast cancer 12

August 2010 The role of legislators in expanding access to quality reproductive health care

7

October 2010 Food and nutritional security 4April 2011 Climate change in Uganda 3May 2011 Agricultural engineering 9June 2011 Genetically modified organisms -

impact regulation5

July 2011 Access to healthcare as a basic right

11

September 2011 Water production in Uganda 1April 2012 Progress parliament has made to

address the problem of polythene bags

15

April 2012 Promoting research and development in S&T

13

May 2012 Multilateral environmental agreements

8

July 2012 Agricultural financing and strategies for funding small scale farmers

2

July 2012 Crested cranes 10November 2012 Science, technology and

innovation6

November 2012 Impact of HIV/AIDs on women and girls

14

63 |


REVIEW CRITERIA Over the years, POST has devised a set of criteria to assess the quality of its own briefings. The Parliament of Uganda does not have its own written criteria, but staff of the Parliamentary Research Department say that the POST criteria are broadly applicable to their work. Experts were therefore asked to review the briefings both qualitatively and quantitatively on the basis of POST’s criteria. The questions posed to the reviewers are set out below.

1. CompletenessHow comprehensive is the briefing? For example:

Does it cover most issues that are relevant to the issue in question? Is there any key information missing? (If you’re not able to answer because the subject is outside your specialism, just make a note of this.)

Have they struck the right balance between providing technical background and providing policy context?

Please mark the briefing on a scale of 1 (not at all comprehensive) to 5 (very comprehensive) along with written comments.

2. Accuracy How technically accurate is the briefing? Are there any obvious technical inaccuracies ?

Please mark the briefing on a scale of 1 (very low technical accuracy) to 5 (very high technical accuracy) along with written comments.

3. Use of evidenceHow would you rate this briefing's use of evidence? For example, has the author made it clear where any key information comes from? Have they drawn on appropriate source material? Are there any key sources that have not been used that you could recommend to the author in future?

Please mark the briefing on a scale of 1 (very bad use of evidence) to 5 (very good use of evidence) along with written comments.

4. Objectivity Please rate this briefing for objectivity/impartiality. For example, is the language neutral? Does the briefing present both sides of the argument or is it one-sided? Does it make any recommendations? If so, is it clear who is making these recommendations and what their reasons are for doing so?

Please mark the briefing on a scale of 1 (very partial/biased) to 5 (very objective) along with written comments.

5. Clarity How easy was this briefing to read? For example was the language clear? Were any acronyms, units, and jargon, clearly explained? Were any figures, tables, etc., easy to understand?

How well structured was the briefing? Was the information set out in a logical way? Was it structured in a way that made it easy to read? Was it the right length? (Note: this last question doesn’t apply to conference papers, which are read out orally.) Please mark the briefing on a scale of 1 (very poorly structured) to 5 (very well structured) along with written comments.

64 |


Please mark the briefing on a scale of 1 (very difficult to understand) to 5 (very easy to understand) along with written comments.

SELECTION OF REVIEWERS

In both the pre-2009 and post-2010 reviews, the briefings were assessed by 15 experts, eight of whom were specifically chosen for their subject expertise and seven for their policy expertise (though all reviewers had a combination of both sorts of expertise). Although the original intention was to use the same reviewers for both periods, this was not possible because only seven reviewers from 2009 were still available in 2013. Substitute reviewers recruited in 2013 were chosen to have similar profiles where possible (for example people who had taken over the original reviewer’s role, or were working for the same organisation). However the post-2010 briefings covered some new subject areas that were not covered in the pre-2009 phase (for example breast cancer, and science, technology and innovation policy). Therefore, some new reviewers needed to be recruited. 38

It was not possible for every briefing to be read by every reviewer. The majority of the briefings were only reviewed by one person. However, in these cases the reviewer was assessed to have both policy and subject expertise. Reviewers were asked to review as many briefings as possible – in practice this ranged from one to four briefings.

QUANTITATIVE FINDINGS As set out above, reviewers were asked to assess the briefings both qualitatively and quantitatively, by providing comments and scoring each briefing on a scale from 1 to 5, according to the guidance sheet discussed on the previous page. All but two reviewers returned both qualitative and quantitative responses for every category.39

A comparison of average scores across all the briefings from the pre-2009 period and the post-2010 period respectively indicates an improvement in score. These figures on their own are not statistically significant because of the small number of briefings, but they are consistent with the comments made in the end-of-programme interviews as well as with the qualitative data (see qualitative findings, below).

38 One reviewer originally involved in the pre-2009 review was discounted for the purposes of the comparison. The reviewer was a former MP who was not available for the post-2010 review; an appropriate substitute could not be found. 39 Note that in some cases reviewers said they lacked very detailed knowledge of the subject area but they based their judgement on what they would ‘expect’ to see, based on their experience and on other similar literature they had read.

65 |


Comparison of average scores from the pre-2009 period and the post-2010 period

Completeness Accuracy Use of Evidence Objectivity Clarity0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

pre-2009post 2010

The graph below shows a comparison of scores which is based only on reviewers who were involved in both the pre-2009 and the post-2010 reviews. An improvement in average score across all categories can still be seen. Again, the figures are not statistically significant because of the small numbers of briefings involved, but they support a view that the observed improvement is unlikely to be simply due to the changes in reviewers across the two periods. 40 Comparison of average scores from the pre-2009 period and the post-2010 period using only those reviewers involved in both reviews

Completeness Accuracy Use of Evidence Objectivity Clarity0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

pre 2009post 2010

QUALITATIVE FINDINGSAn analysis of the qualitative review comments supports a conclusion that the quality of the post-2010 briefings is significantly higher than the pre-2009 briefings. However, the comments also indicate that there is still considerable room for improvement, particularly in areas such as use of

40 Note that where there was more than one reviewer for a given briefing, an average score was calculated for all reviewers of that briefing before calculating the total score across all the briefings.

66 |


evidence and objectivity. Below, we discuss the comments made under each category, and issues highlighted by reviewers in the post-2010 review, which could be the focus of future efforts. 41

COMPLETENESS In the pre-2009 review, major omissions were highlighted in ten briefings in terms of their coverage of subject area. In the post-2010 review, major omissions were highlighted in only two of the briefings. Nevertheless, for the majority of post-2010 briefings (ten out of fifteen) there was still a feeling that more in-depth coverage of the subject area was needed to allow the reader to make an evidence-informed judgement of the issue under discussion. For example reviewers in the post-2010 review said that:

Uganda-specific information would have been useful, or data comparing Uganda and other countries would have been useful (briefings 1, 4, 7, 11).

The views of a range of different stakeholders should have been represented (briefings 5, 15).

As well as listing possible policy options, it would have been useful to have more detail on the strengths and weaknesses of different policy options. (Briefing 2)

Too much ground was covered to go into anything in depth. (Briefing 3)

ACCURACY In the pre-2009 review, seven briefings contained major factual inaccuracies and/or incorrect science and three had minor inaccuracies. In the post-2010 review, only one briefing had major factual errors and two had minor inaccuracies. Thus for the majority of the post-2010 briefings, reviewers had no problems with factual accuracy of what was presented. However, in many cases a lack of detail, a failure to reference claims, and failure to present data clearly, still had a negative impact on accuracy:

Use of descriptive language: In two cases (11, 15) reviewers had issues with the accuracy of the language; phrases like “a good number of women” and “mortality rates continue to soar” could have been quantified to increase accuracy.

Lack of detail: In two cases (briefings 5,9) there was a lack of detail and the information provided was quite superficial, although reviewers had no problems with the accuracy of what was presented.

USE OF EVIDENCE In the pre-2009 review, ten briefings were criticised for making poor use of the available evidence and for missing out references to key sources. Another key criticism was that claims and statements made in the text were not always referenced, so it was not clear if they originated from the author or from another source. In the post-2010 review, four briefings were criticised for making poor or limited use of the available evidence, for missing out key sources or for using references of dubious quality. There were a number of other observations:

Climate change: The only paper available on climate change in the post-2010 review (Briefing 3) was judged to have made limited use of evidence, particularly from the International Panel on Climate Change. This was also a criticism made in the pre-2009 review.

Missing references: For three of the briefings (2, 8, 9), the evidence cited was considered acceptable in quality but was limited in scope – i.e. there was other relevant information

41 Some briefings were reviewed by more than one person. There were no cases where the different reviewers’ comments on the same briefing were inconsistent with each other, although reviewers emphasised different issues and some were more severe in their criticism than others.

67 |


available that was not used. One paper (1) did not list any references so it was not possible to judge its use of evidence.

Attribution: For five briefings (4, 5, 10, 12, 13) reviewers pointed out that although sources were listed it was not made clear where or how they had been used in the main text. There was not enough referencing to allow the reader to judge the reliability of information provided or the validity of the claims that were being made.

OBJECTIVITY In the pre-2009 review, two briefings were considered to be very biased or speculative. Only one was judged to be “reasonably objective”. The remainder received a mixture of criticisms – for example for using subjective language, for placing disproportionate emphasis on certain points, or for making recommendations without explaining the reasoning behind them. In the post-2010 review, two briefings were considered to be biased but six of them were considered to be “reasonably objective”. Other key comments included:

Unsupported claims or recommendations: in eight cases (briefings 1, 3, 6, 8, 10, 11, 13, 15) recommendations were made which were not backed up – reviewers were not clear how the recommendations were arrived at, or who was making them (i.e. whether the originated from the author or were taken from somewhere else and were merely being cited by the author). Two reviewers questioned whether it was the role of a parliamentary body to be making recommendations at all.

Emotive language: several briefings used non-neutral language (briefings 3, 5, and 15)

CLARITY In the pre-2009 review, three of the briefings were assessed as being quite easy to read and well structured. The others received a mixture of criticisms, for example for confusing use of statistics and graphs, use of jargon, poor language, lengthy paragraphs, and the excessive use of copying and pasting of text. In the post-2010 review, ten briefings were considered to be well written, of which six were also assessed as being well structured. Key comments included:

More tables and figures: In five cases the briefings would have benefited from more tables and figures. (Briefings 1,2,3,9,11)

Multiple authors: Two reviewers said their briefings were clearly written by multiple authors which made them harder to read. (Briefings 3 and 5)

Too long: Two briefings were assessed as being too long and detailed. (Briefings 3 and 11) Needing a summary: Five briefings were assessed as needing a summary. (Briefings 2, 10,

11, 12, 14)

LIMITATIONS OF THE ANALYSISAs already highlighted, there are a number of sources of potential inaccuracy in this analysis, which could be addressed if the review were to be repeated in future years:

The number of briefings assessed is small: this could be increased in future years by repeating the analysis on a larger scale and involving more reviewers.

Different reviewers were involved in the pre-2009 and post-2010 reviews. However, the results indicate that the observed improvement in briefing quality is unlikely to be a result of this.

There might be variations in the amount of time the authors spent in writing the briefings, which might account for some of the differences in quality. However, more detailed information would be needed from the authors to investigate this.

It is always possible that some bias has been introduced by inadvertently missing some relevant briefings out of the review process. However, this is unlikely: staff in the Ugandan parliament have shared all the briefings they are aware of. Moreover, POST selected

68 |


briefings for review solely on the basis of their relevance to S&T. No prior assessment of their quality was made before the selection.

69 |

abbreviations - home page - uk · web viewevaluation of the post programme in uganda report...

Documents