1

EVIDENCE BASE FOR COLLABORATING, LEARNING, AND ADAPTING Summary of the Literature Review, August 2016

SUBMISSION DATE: AUGUST 11, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Dexis Consulting Group.

Contract Number: AID-OAA-M-14-00015

i

EVIDENCE BASE FOR COLLABORATING, LEARNING, AND ADAPTING Summary of the Literature Review, August 2016

Submitted to:

USAID

Prepared by:

Dexis Consulting Group

DISCLAIMER:

The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

ii

TableofContentsACRONYMLIST............................................................................................................................................III

PURPOSEOFTHELITERATUREREVIEW........................................................................................................1

METHODOLOGY...........................................................................................................................................1

KEYFINDINGS...............................................................................................................................................1

WHATARETHEIMPLICATIONSOFLITERATUREREVIEWFINDINGSONUSAIDANDLEARN’SEFFORTSTOPROMOTECLA?............................................................................................................................................2

WHEREISTHEREEVIDENCETHATCOLLABORATING,LEARNING,AND/ORADAPTINGMAKEADIFFERENCE?................................................................................................................................................4

WHATARETHEKEYTHEMATICAREASEMERGINGFROMTHELITERATUREABOUTCOLLABORATING,LEARNING,ANDADAPTING?........................................................................................................................6

WHERE,WITHINTHECLAFRAMEWORK,ISTHERENOTMUCHEVIDENCE?..............................................11

WHATMETHODOLOGIESHAVEBEENUSEDTOSTUDYWHETHERCOLLABORATING,LEARNING,ANDADAPTINGMAKESADIFFERENCE?............................................................................................................11

WHEREAREPEOPLECALLINGFORMORERESEARCH?..............................................................................12

iii

ACRONYM LIST

CLA Collaborating, Learning, and Adapting CoP Community of Practice EB4CLA Evidence Base for CLA ICT Information and Communication Technology IP Implementing Partner IR Intermediate Result KM Knowledge Management M&E Monitoring and Evaluation NGOs Non-governmental Organizations PPL USAID’s Bureau of Policy, Planning, and Learning TOC Theories of Change USAID/PPL United States Agency for International Development Bureau of Policy, Planning, and

Learning

1

PURPOSE OF THE LITERATURE REVIEW

LEARN and USAID/PPL are managing an area of work known as Evidence Base for Collaborating, Learning, and Adapting (EB4CLA). The purpose of this work is to answer key learning questions:

● Does an intentional, systematic and resourced approach to collaborating, learning and adapting contribute to development outcomes?

● If so, how? And under what conditions? As we began this work, we identified the need to conduct a literature review looking at these questions to understand what is known, what remains unknown, and how others have tried to answer these questions to date. We were primarily interested in answering the question: has there been a comprehensive review of the evidence base on the effect or impact of collaborating, learning, and adapting (CLA) on development outcomes? In addition, we also considered:

● Is there evidence that collaborating, learning, and/or adapting impacts organizational effectiveness (perhaps prior to an effect or impact on development outcomes)?

● Does the literature identify any factors critical to CLA that are not currently included in the CLA framework (Version 6)?

● Who else is working on measuring the impact of CLA?

METHODOLOGY

We began the literature review by identifying and searching for keywords from the CLA framework. Recognizing that CLA is a construct used within USAID and among its stakeholders, the literature review here uses CLA to refer more broadly to the concepts of strategic collaboration, organizational learning, knowledge management, and adaptive management. The literature review draws primarily on academic articles and some grey literature. Given the relative novelty of measuring the effect, impact, and/or contribution of CLA, however, the team feels that a review of grey literature will be more critical in the future.

After identifying keywords, the research assistant responsible for the literature review looked primarily for summaries of existing literature and prioritized articles related to the field of international development. Based on preliminary keyword searches, the research assistant snowballed from some initial relevant articles, finding additional resources. Articles were organized according to the CLA framework in a spreadsheet, which included the link to the article, a summary of the article, and a summary of the methodology used by the researchers/authors.

This is a new and emerging field in many ways. As we were nearing the end of the time of the initial literature review period, we came across across several grey literature resources that were being published on an ongoing basis. We imagine the field will continue to grow as more researchers and practitioners become interested in the question of organizational learning and adaptive management in international development. Therefore, the literature review will be updated regularly by the LEARN team to identify if there are any additional resources available and maintain our technical evidence base. If you know of relevant research that should be incorporated into future updates of this literature review, please email us at info@usaidlearninglab.org with the subject line: Evidence Base for CLA.

KEY FINDINGS

Has there been a comprehensive review of the evidence base on the effect or impact of CLA on development outcomes? Is there evidence that collaborating, learning, and/or adapting impacts development outcomes or organizational effectiveness? Overall, we found that there is no

2

comprehensive review of the evidence base on the effect or impact of CLA on development outcomes. However, there are discrete pieces of evidence pointing to the importance of collaborating, learning, and/or adapting on both development outcomes and organizational effectiveness. These discrete pieces of evidence are typically in the form of case studies on development programs, though one recent empirical study stands out from the World Bank; it found a significant and positive correlation between intentional, high-quality M&E and development outcomes.

There are also some examples of a more systematic approach to organizational learning in the private sector (e.g. Southwest, Ford Lean Manufacturing, Motorola Sigma, etc.), and how these approaches have impacted the effectiveness of these organizations. Despite these cases, most of the the literature on CLA and its contributions towards organizational effectiveness and development outcomes remains predominantly theoretical or aspirational. Because of this, practitioners and researchers are calling for more comprehensive and credible studies on the effect and impact of CLA.

Difficulties in measurement are the main reason for the lack of comprehensive evidence about the impact of CLA on organizational effectiveness and development. The main measurement challenges include:

1. Finding a way to measure interventions−such as those that constitute CLA−that include relatively intangible aspects in a way that is meaningful and convincing; and

2. Making causal attributions between CLA and organizational effectiveness or achievement of development outcomes when a variety of other factors could be at play.

3. Because case studies are often the means by which CLA is studied within the international development context, it is difficult to aggregate across case studies to reach generalizable conclusions.

Does the literature identify any factors critical to CLA that are currently not included in the CLA framework? The literature predominantly reinforces the components and subcomponents found in the CLA framework. However, leadership is treated as an independent factor that significantly enables CLA in organizations. The current CLA framework treats leadership as a part of culture (insofar as leaders promote or inhibit organizational norms that may support or hinder CLA efforts), rather than a discrete condition. In addition, the current CLA framework does not explicitly place value on flatter organizations (which are believed to better support learning), though there is a focus on openness and relationship-building at all levels to support CLA.

Who else is working on measuring the impact of CLA? Several international development organizations and donors were found in the literature on CLA and development outcomes. These include: the Swedish International Development Agency (SIDA), the UK Department for International Development (DFID), the World Bank, Oxfam International, the Bill and Melinda Gates Foundation, the United Nations, the Overseas Development Institute, and the German Federal Enterprise for International Cooperation (GiZ). Specific sectors were also highlighted in the literature, including governance/public sector management, health management, and climate change.

WHAT ARE THE IMPLICATIONS OF LITERATURE REVIEW FINDINGS ON USAID AND LEARN’S EFFORTS TO PROMOTE CLA?

Based on the findings below, USAID/PPL and LEARN have identified the following key implications for how we promote greater CLA integration within USAID and among implementing partners:

Further invest in building the evidence base for CLA: Deepening the evidence base for CLA is an area that the literature identified for further research and should be a priority moving forward.

3

However, given the challenges in measuring the effect, impact, and/or contribution of CLA on development outcomes, we need to determine what constitutes credible evidence for our interventions and our target audiences.

Address/consider major institutional barriers to further integrating CLA: The literature highlights certain attributes of learning organizations, such as slack in resources (including time), risk-taking culture, and flat (rather than hierarchical organizational structures) that are at odds with USAID’s existing culture for a variety of reasons. How can these institutional barriers be addressed or, if not possible, be considered in planning? In addition, leadership and organizational culture are heavily emphasized in the literature. A clear strategy for addressing these aspects of the USAID system would be important.

Focus on learning among local partners and communities. Thus far, knowledge management and learning strategies in development have been based on private sector thinking that is organization-centric. Development, however, should focus on learning across all development partners and the field in general. In other words, “knowledge pooling” or knowledge sharing among development partners is encouraged.1 In addition, the literature speaks to significant power dynamics between Northern and Southern organizations when it comes to learning and determining whose learning matters. As a result, USAID’s CLA efforts should continue to encourage a move away from knowledge flowing only from North to South, and instead support USAID in working more closely with local partners and building local knowledge into programs and plans. This also means that jargon surrounding learning and knowledge management needs to be reduced to be accessible to those both within and outside USAID, including local partners.

Incentivize CLA among implementing partners: The literature highlights current donor practices, particularly those for monitoring and evaluation (M&E), that focus on accountability rather than learning. This often leads to identifying static results that are not easily adjusted during implementation. As a result, implementing partners are not properly incentivized to learn and adapt, for fear of losing future funding. For CLA to advance at the activity level under USAID funding, implementing partners will need appropriate incentives and encouragement from USAID counterparts.

Consider implications of differences in personality traits: Ultimately, it is individuals who take on the work of collaborating, learning, and adapting within organizations and across partner organizations. However, individual personality traits can affect who is more likely to take on these behaviors; these differences need to be considered when trying to promote behavior change. As with any change effort, generating trust and buy-in from stakeholders will be critical for CLA. USAID/PPL and LEARN can look to change management champions’ literature to account for these implications.

Tactical implications for CLA: Many of the findings below also require some tactical consideration on the part of USAID/PPL and LEARN. These include:

● Encouraging staff and partners to question theories of change throughout the Program Cycle, moving away from linear modes of thinking, and avoiding the use of static, unchangeable targets

● Making knowledge management more people-centric while also making better use of ICT for knowledge sharing internally and externally

● Creating opportunities for critical reflection throughout the Program Cycle ● Focusing on value-creating collaboration (both internally and externally) ● Encouraging learning networks and communities of practice (CoP)

1 Barnard, G., “Knowledge Sharing in Development Agencies: Knowledge fortress or Knowledge Pool?” Information Development 19 (2003) 280-288.

4

WHERE IS THERE EVIDENCE THAT COLLABORATING, LEARNING, AND/OR ADAPTING MAKE A DIFFERENCE?2

● Knowledge management/knowledge sharing/best practices:3 The literature discusses how organizations that are able to maintain and transfer knowledge from one unit to another are more productive and more likely to be successful. Knowledge management (KM) facilitates reflection and learning, and it is pivotal for making good decisions, and designing effective programs. The role of information and communication technology (ICT) is an area of interest in this field. However, the literature cautions against making KM only about storing information; instead KM should be people-centric.

● Reflective practice:4 The literature discusses the importance of reflecting often and changing course/adapting as needed. “Purposeful reflection on one’s accumulated experience leads to greater learning than the accumulation of additional experience.”5 Reflective practice requires development stakeholders to: reflect on development processes; challenge previous assumptions and instill dynamism in discourses; include multiple voices through a critical view of power relations; facilitate the creation and actualization of multiple approaches at the local level; create opportunities for these local imaginings to be synthesized at regional and global level, to enable a better understanding of global issues and advocate for the transformation of global regimes.6 The literature found that organizations and projects are much more likely to be successful if they adopt such practices and increase their agility. In addition, public reflection by individuals and government agencies is a useful strategy to enhance accountability and create a stronger onus for change. “Public reflection can bring the consequences of behavior to our attention in ways that might transform it.”7

● Internal collaboration and teamwork:8 Collaborative relationships among individuals and groups are important for innovation, knowledge production, and diffusion. Groups/teams

2 These takeaways synthesize lessons from numerous articles reviewed for the literature review. While footnotes identify the most pertinent articles that contributed to each takeaway, they are not the exhaustive list of articles found in the literature review. 3 Ramalingam, B., “Implementing Knowledge Strategies: Lessons from international development agencies,” Overseas Development Institute Working Paper 244 (2005); Cummings, J., Knowledge sharing: A review of the literature. OED Working Paper (2003) Washington, D.C., World Bank; Barnard, G., Knowledge Sharing in Development Agencies: Knowledge fortress or Knowledge Pool? Information Development (2003); King, K. and McGrath, S., 2003. Knowledge Sharing in Development Agencies: Lessons from Four Cases. OED Working Paper (2003) Washington, D.C., World Bank.

4 Hilden, S., Tikkamäki, K., “Reflective Practice as a Fuel for Organizational Learning,” Administrative Sciences 3.3 (2013) 76-95; Andrews, M., et. al. “Escaping Capability Traps through Problem-Driven Iterative Adaptation (PDIA)” CGDEV Working Paper 299, June 2012; Di Stefano, Giada, et al. “Learning by Thinking: Overcoming the Bias for Action through Reflection,” Harvard Business School NOM Unit Working Paper No. 14-093. March 29, 2015.

5 Di Stefano, Giada, et al. “Learning by Thinking: Overcoming the Bias for Action through Reflection.” Harvard Business School NOM Unit Working Paper No. 14-093. March 29, 2015. 6 Jakimov, T., “Answering the critics: the potential and limitations of the knowledge agenda as a practical response to post-development critiques,” Progress in Development Studies 8:4 (2008): 311-323.

7 Raelin, J. A., “Public Reflection as the Basis of Learning,” Management Learning 32 (1), (2001):11-30.

8 “Creating a Collaborative Organizational Culture” UNC White Paper (2014); Phelps, C., Heidl, R., and Wadhwa, A. “Knowledge, networks and knowledge networks: A Review and research agenda,” Journal of Management (2012); De Meuse, K. P., Tang, K. Y., and Dai, G. “Construct validation of Lominger T7 Model of team effectiveness” Minneapolis, MN: Lominger

5

develop “transactive (or shared) memory systems,” which enable better group goal performance.9

● Value of collaboration versus too much collaboration:10 ○ Companies with better collaborative management capabilities achieve superior financial

performance. Collaboration encourages innovation and boosts employees’ overall performance and loyalty. Effective teams are built on applying outstanding functional skills to address complex challenges or opportunities and leveraging strong, trusting relationships to deliver innovation and results. Components of successful teamwork include: external orientation, continuous learning, straight talk, team orientation.11

○ It is important to note that collaboration is not a panacea. Too much collaboration can lead to time wasting, high interaction costs that can slow decision-making, and have knock-on effects like poorer quality, interpersonal conflict, and loss of motivation. When collaboration is unstrategic, it can have negative consequences on staff engagement.

● External collaboration and specifically community-driven and locally led development:12 The literature emphasizes “thinking politically,” “politically smart” and “locally driven development.” Iterative, flexible, and politically informed programming should be pursued. Case studies of development initiatives showed iterative problem-solving, stepwise learning, brokering relationships, and discovering common interests were key to success, allowing actors to understand the complex development challenges they face, identify and negotiate ways forward, and find solutions that were both technically sound and politically feasible. Development, by and large thus far, has predominantly been led by Northern organizations and there is a top-down structure where the Northern organizations impose their

International: A Korn/Ferry Company (2009); Hackman, J. R. (2002). Leading teams: Setting the stage for great performances. Boston: Harvard Business School Press; Katzenbach, J. R., & Smith, D. K. (1993). The wisdom of teams: Creating the high-performance organization. Boston: Harvard Business School Press; Rubin, I. M., Plovnick, M. S., & Fry, R. E. (1977). Task oriented team development. New York: McGraw-Hill; 9Austin, J. R., “Transactive memory in organizational groups: The effects of content, consensus, specialization, and accuracy on group performance,” Journal of Applied Psychology, 88 (2003): 866-878.

10 McKinsey and Company. 2009. “How do I drive effective collaboration to deliver real business impact?”; Cross, R., Rbele, R., Grant, A., “Collaborative Overload” Harvard Business Review Jan/Feb 2016; Andersson, K., “Motivational Dilemmas in Collaborative Learning Activities: The Case of the Swedish International Development Cooperation Agency (SIDA),” Public Administration and Development, 29, 5, (2003): 341-351.

11 Frequently cited team models: De Meuse, K. P., Tang, K. Y., & Dai, G., “Construct validation of Lominger T7 Model of team effectiveness” Minneapolis, MN: Lominger International: A Korn/Ferry Company (2009); Hackman, J. R. “Leading teams: Setting the stage for great performances,” Boston: Harvard Business School Press (2002); Katzenbach, J. R., & Smith, D. K. The wisdom of teams: Creating the high-performance organization. Boston: Harvard Business School Press (1993); Rubin, I. M., Plovnick, M. S., & Fry, R. E. “Task oriented team development,” New York: McGraw-Hill (1977); LaFasto, F., and Larson, C. “When teams work best: 6000 team members and leaders tell what it takes to succeed,” Thousand Oaks, CA: Sage (2001); Lencioni, P. “The five dysfunctions of a team: A leadership fable,” San Francisco: Jossey-Bass (2002).

12 Faustino, J. and Booth, D., “Development entrepreneurship: How donors and leaders can foster institutional change,” Working Politically in Practice Series, Case Study No. 2. London: The Asia Foundation and ODI (2014); ODI Research reports and Studies, “Politically smart support to economic development,” March 2016; Booth, D., “Thinking and Working Politically,” GSDRC Professional Development Reading Pack No. 13. Birmingham, UK: University of Birmingham (2015); Drew, R., “Learning in Partnership: What constitutes learning in the context of south-north partnerships?” BOND Discussion Paper. London: British Overseas NGOs for Development (2002).

6

models and requirements onto local partners. Instead, the literature emphasizes the need for locally-led approaches that are embedded in the local context, and are locally negotiated and delivered. This alternate approach can lead to more effective development.

● Communities of practice: The literature found that organic learning communities were the most effective, but organizational support is needed to realize their benefits. This support includes resources (time, support for scheduling, organizing occasional meetings), recognition (rewards), and other support systems. Example: Inter-American Development Bank “thematic communities.”13

● Learning culture (leaders are heavily emphasized):14 The literature discusses the importance of a learning culture as the foundation for learning organizations. The literature discusses how organizations that encourage honest discourse and debate, and provide an open and safe space for communication tend to perform better and be more innovative. Leaders are central to defining culture, and “learning leaders” are generally those that encourage non-hierarchical organizations where ideas can flow freely. (Implication 2 & 10)

○ Selected leadership features that promote CLA: Willingness to provide a safe space for innovation (and failure); ability to understand and work within a changing and complex environment; ability to enable non-hierarchical relationships and constantly counters rigid hierarchical decision making; ability to contain anxiety; ability to manage change processes.

WHAT ARE THE KEY THEMATIC AREAS EMERGING FROM THE LITERATURE ABOUT COLLABORATING, LEARNING, AND ADAPTING?

● Power relations, funding, and competition:15 ○ External Collaboration: The literature mainly focuses on North-South dynamics in

development, and there is agreement that local knowledge needs to be folded into development activities. Unequal power relations between Southern and Northern institutions usually result in “teaching” North-South relations where learning is

13 Moreno, A., “Enhancing knowledge exchange through communities of practice at the Inter-American Development Bank Aslib Proceedings,” 53.8 (2001).

14 Faustino, J. and Booth, D., “Development entrepreneurship: How donors and leaders can foster institutional change,” Working Politically in Practice Series, Case Study No. 2. London: The Asia Foundation and ODI (2014); Hailey, J. and James, R. “Learning leaders: the key to learning organisations,” Development in Practice, 12(3) (2002): 398–408; Su-Chao, C. Ming-Shing, L., “A study on relationship among leadership, organizational culture, the operation of learning organization and employees' job satisfaction,” The Learning Organization 14.2 (2007); LaFasto, F., & Larson, C. “When teams work best: 6000 team members and leaders tell what it takes to succeed,” Thousand Oaks, CA: Sage (2001); Lencioni, P. “The five dysfunctions of a team: A leadership fable,” San Francisco: Jossey-Bass 2002); McKinsey and Company, “How do I drive effective collaboration to deliver real business impact?” http://www.mckinsey.com/~/media/mckinsey%20offices/canada/latest%20thinking/pdfs/how_do_i_drive_effective_collaboration_to_drive_real_business_impact.ashx; Blanchard, K. and Waghorn, T., “Make Sure your employees trust you - or else” Forbes http://www.forbes.com/2009/03/23/trust-respect-employees-leadership-managing-blanchard.html

15 Drew, R., “Learning in Partnership: What constitutes learning in the context of south-north partnerships?” BOND Discussion Paper. London: British Overseas NGOs for Development (2002); Vincent, R., Byrne, A., “Enhancing learning in development partnerships” Development in Practice, 08/2006, Volume 16, Issue 5; UNICEF strategic framework for partnerships and collaborative relationships (2009) http://www.unicef.org/about/execboard/files/N0928210.pdf; Booth, D., and Unsworth, S., “Politically smart, locally led development” ODI Research Reports and Studies, (2014) https://www.odi.org/publications/8800-politically-smart-locally-led

7

transferred from Northern organizations to the Southern ones. Instead, it would be beneficial to focus more on mutual learning partnerships.

■ Structural inequality in aid and development systems.16 ● As Southern NGOs are usually dependent on Northern donors for

limited funding, the Southern NGOs usually end up competing against each other, and this hampers collaboration and learning among these organizations. The literature did not present a solution for this.

○ Organizational culture/internal organizational structure:17 organizations with rigid hierarchical decision-making may hamper learning. Learning is more likely to take place in organizations that empower their workers, and where critical thinking, analysis, and creativity is encouraged and rewarded. A foundational culture of investigation, debate, and agility needs to be supported and reinforced by a broad set of tools (both technical and managerial), processes (such as recruitment) and systems (such as finance, procurement and M&E).

● Culture is key; and leaders shape culture:18 ○ Requirements for a learning culture include: decentralized/non-hierarchical decision-

making processes; availability of slack resources (including time); communities of practice; strong and enabling leadership; a risk-taking culture (experimentation); and knowledge management and sharing systems.

○ Leaders: Southwest, Netflix and others were successful because their leaders created a culture that was conducive to collaboration, learning, accountability, and adaptability.

○ “The heart of a learning organization is a learning leader who enables non-hierarchical relations. Leaders are, of course, particularly influential members of an organization, and their opinions and moods are quickly picked up by other members. Their views therefore permeate most organizational processes… The most important characteristic of the learning leaders...was their ability to understand and work within a changing and complex environment. Indeed, this ability was far more important than the specific learning strategies that they advocated, which seemed to be of less importance. Some of the learning leaders emphasized formal learning, others emphasized informal processes, while yet others focused on learning from new technologies and applied research. However, the result they were able to produce was similar in all cases: their

16 Takahashi, C., “Partnerships, learning, and development: a case study from Ghana,” Development in Practice, Volume 16, Issue 1 (2006).

17 Su-Chao, C., Ming-Shing, L., “A study on relationship among leadership, organizational culture, the operation of learning organization and employees' job satisfaction,” The Learning Organization 14.2 (2007): 155-185; McGregor, L., and Doshi, N.,. “How company culture shapes motivation,” Harvard Business Review https://hbr.org/2015/11/how-company-culture-shapes-employee-motivation

18 Schein, E., “The Learning Leader as Culture Manager,” Organizational Culture and Leadership. San Francisco: Jossey-Bass Publishers (1992); de Wet, J.P., and Schoots, J., “The Learning Organisation: Conditions of Possibility in a Feminist NGO,” Development in Practice, 26:1, (2016): 64-76.

8

organizations were able to respond to changing circumstances in order to carry forward their vision.”19

● Knowledge acquisition versus knowledge creation:20 There is a tension between cognitive learning (unobservable) and behavioral learning (observable) or between knowledge as an object that can be passed from person to person versus knowledge as something that is created. Essentially, there is a tendency to reduce learning down to observable behaviors precipitated by new systems/requirements, but not enough focus on knowledge being created.

○ Procedures set up in NGOs and development organizations to promote organizational learning often consider knowledge more as an object that can be transferred from one person to another rather than something that is created in interaction. The organizations have difficulty moving from cognitive information management to people-centered learning processes. A recent study of NGOs concludes that the “widespread and tangible outputs of knowledge and learning work tend, thus far, to be based on improved information systems, rather than improved processes or changed behaviors” and that, as a consequence, their learning structures are “more supply-led than demand-driven.” A tendency was noted among these organizations to “point to information systems as the ‘’end product’’ rather than specific processes for knowledge and learning.” 21

○ Solution/recommendation: Focus on more reflective activities where interactions lead to knowledge creation, rather than a sole focus on KM systems and information strategies.

● Organic learning via Communities of Practice (CoPs)/thematic groups/networks:22 ○ The literature recommends that for learning to take place, interactions should be

emphasized and all individuals should learn from each other. Limiting learning to downward flows of knowledge does not seem to be effective. CoPs are more effective as tools for reflection and learning when they form organically. Management/leaders, however, need to facilitate these organically formed learning groups bring them out of

19 Hailey, J., and James, R., “Learning Leaders: The Key to Learning Organisations,” Development in Practice 12(3/4) (2002): 398–408; Hovland, Ingie, H., “Knowledge Management and Organisational Learning: An International Development Perspective,” Overseas Development Institute, August (2003).

20 Huber, G. P., “Organizational Learning: The Contributing Processes and the Literatures,” Organization Science, Volume 2, Issue 1 (1991); Chen, A., and Edgington, T., "Assessing Value in Organizational Knowledge Creation: Considerations for Knowledge Workers," MIS Quarterly 29.2 (2005): 279-309; King, K. and McGrath, S. “Knowledge Sharing in Development Agencies: Lessons from Four Cases” OED Working Paper. Washington, D.C., World Bank (2003).

21 Ramalingam, B., “Implementing Knowledge Strategies: Lessons from international development agencies,” Overseas Development Institute Working Paper 244: 31 (2005).

22 Ayas, K., and Zeniuk, N., “Project-based learning: Building communities of reflective practitioners” Management Learning, 32, 61–76 (2001); Moreno, A., “Enhancing knowledge exchange through communities of practice at the Inter-American Development Bank,” Aslib Proceedings 53.8 (Sep 2001); DFID, “Doing the Knowledge: How DFID Compares with Best Practice in Knowledge Management,” London: Department for International Development. (available on www.dfid.gov.uk) (2000); Wesley, P.W., and Buysse, V., “Communities of practice: Expanding professional roles to promote reflection and shared inquiry,” Topics in Early Childhood Special Education, (2001).

9

silos and support them, and disseminate their knowledge across the rest of the organization and other organizations too.

○ Example: Inter-American Development Bank (IDB) Bank Networks (CoPs) emerged organically around different themes/sectors. These groups are self-organized, set their own objectives, and membership is largely voluntary and self-selected. These CoPs offer a space for dialogue among those working on similar issues and there is a general belief among network participants that fostering these Communities will result in more rapid organizational learning, more effective decision making use of lessons learned, and more rapid and effective problem solving.23

● Monitoring & Evaluation and single & double-loop learning:24 ○ Almost all organizations that work with donor funding are required to carry out M&E in

some form. On the face of it, M&E seems like a useful tool for information gathering, reflection and adaptation but it does not always translate into learning. In other words, M&E as it is currently designed primarily encourages single-loop learning, addressing specific problems/symptoms rather than trying to understand why the problems came up in the first place. It does not lead to for double-loop learning, which focuses on why symptoms are appearing, how underlying models can be improved, and questions assumptions and the broader system. There appear to be three main reasons for this: ■ First, M&E procedures were created by donors for accountability purposes

rather than developed to fit the learning needs of donor or recipient organizations. There is also some tension between learning and accountability. Recipient organizations hesitate to include weaknesses or program changes–highly relevant information for learning–into their monitoring and reporting for fear of losing future funding.

■ A second reason why M&E does not work well for learning is the emphasis of currently dominant M&E procedures on the quantitative measuring of “impact:” “donors want deliverables.” Increasingly, people question this approach since it is difficult to quantitatively measure the results of many development activities.

■ M&E plans are usually made at the beginning of a project, maybe even at the proposal phase, and usually remain unchanged through the project cycle. In many cases, even if organizations reflect and realize that their monitoring tools are not ideal, they avoid changing the M&E plan once a program has started.

23 Moreno, A., “Enhancing knowledge exchange through communities of practice at the Inter-American Development Bank Aslib Proceedings,” 53.8 (2001).

24 Raimondo, E., “What difference does good monitoring & evaluation make to world bank project performance?” World Bank Group Independent Evaluation Group, June 2016; Pritchett, et. al., “It’s all about MEE: Using structured experiential learning (“e”) to crawl the design space,” CGDEV Working Paper 322 (April 2013); Solomon, M., and Chowdhury, A.M., “‘Knowledge to action: evaluation for learning in a multi organisational global partnership,” Development in Practice 12(3): (2002): 346-354; Willemijn, V., “Learning by Southern Peace NGOs,” The Journal of Development Studies, 04/2010, Volume 46, Issue 4; Wallace, T. and Chapman, J., “Some Realities Behind the Rhetoric of Downward Accountability,” paper presented at INTRAC' s 5th Evaluation Conference: Measurement, Management and Accountability, KDK Conference Centre, Soesterberg, 31 March-4 April 2003; “When Will We Ever Learn? Improving Lives through Impact Evaluation,” Report of the CGD Evaluation Gap Working Group, May 2006.

10

■ The literature also identifies organizational assessments/evaluations/reviews, especially by external organizations, as a pivotal tool for learning. For example, a devastating external review of ActionAid led to the development and launch of their successful Accountability, Learning, and Planning System (ALPS) in 2000.25

○ A June 2016 World Bank study26 quantitatively analyzed the correlation between the quality of M&E and project outcomes. It found that good quality M&E is positively and significantly associated with project outcomes. Some simple factors that can improve M&E quality include ensuring: ■ M&E is incorporated into project management and not viewed as a separate

activity. ■ M&E is used for learning that informs decisions and enables adapting when

necessary. ■ M&E design is not overly complex and is aligned with existing

management information systems. ■ Data collected are controlled for quality to ensure credibility and ultimately its

usability for performance management. ■ M&E is not an operational afterthought but supported by a clear division of

labor between the World Bank team, clients, and implementing teams.

● Theories of change (TOC):27 TOC rarely unfold as predicted; they have to be adapted and reworked as new information emerges. Moving beyond single to double-loop learning should be a key element of TOC. Double-loop learning will not take place if underlying assumptions and theories are not revisited regularly and critically. While one of the biggest benefits TOC may bring is of greater organizational learning, this requires commitment to a broader model of adaptive and reflective practice.

○ The tendency to view a TOC as predominantly an upward accountability mechanism considerably constrains attempts to learn from the process. TOC should be seen as a tool of communication and learning, rather than a method of securing funding.

○ “A TOC approach needs to focus on process rather than product, uncertainty rather than results, iterative development of hypotheses rather than static theories, and learning rather than accountability.”28

25 Scott-Villiers, P., “The struggle for organizational change: how the ActionAid accountability, learning and planning system emerged,” Development in Practice, 12 (3&4) (2002): 424–35.

26 Raimondo, E., “What difference does good monitoring & evaluation make to world bank project performance?” World Bank Group Independent Evaluation Group, June 2016.

27 Vogel, I., “Review of the use of ‘Theory of Change’ in international development,” DFID review report (2012); Valters, C., “Theories of Change in International Development: Communication, Learning, or Accountability?” Asia Foundation, The Justice and Security Research Programme Paper 17 (2014); Valters, C., et. Al., “Putting Learning at the Centre: Adaptive development in practice” ODI Research Reports and Studies (2016).

28 Valters, C., “Theories of Change in International Development: Communication, Learning, or Accountability?” Asia Foundation, The Justice and Security Research Programme, (2014) Paper 17.

11

● Personality traits29 ○ An individual's cognitive skills/traits (e.g. attitudes towards using evidence and intrinsic

learning motivation) affect a person’s willingness and ability to learn. Some individuals may get defensive and closed to the idea of change when presented with reflection and learning opportunities.

● Adaptive management:30 Adaptive management requires an agile and enabling culture that helps organizations use rapid feedback loops to continuously and efficiently process and build on new information to achieve overall goals.

WHERE, WITHIN THE CLA FRAMEWORK, IS THERE NOT MUCH EVIDENCE?

● Resources for CLA: There is some literature on staffing for learning, particularly on how rotating staff can benefit from learning.31 This literature, however, is also related to internal collaboration. While there may not be a heavy focus on resources, given that the literature does emphasize the importance of CLA in general and specific aspects of CLA in particular, one can infer that the resources required to make CLA happen are also important.

● Game changers and scenario planning:32 Most of the evidence is in the private sector, and many of the articles are by consulting firms or businesses. The most cited example is of when Royal Dutch/Shell used scenario planning to anticipate the drop in oil prices in 1986. Scenario planning is also used for urban and public policy, but there is little evidence/research on scenario planning and game changers in development. Further research in the private sector, however, may demonstrate the value add of this approach to organizational effectiveness outside of the development sector.

WHAT METHODOLOGIES HAVE BEEN USED TO STUDY WHETHER COLLABORATING, LEARNING, AND ADAPTING MAKES A DIFFERENCE?

● Primary methodology: Case studies that review specific activities within organizations, or specific projects and collaborations across organizations.

29 Estabrooks, C., Thompson, D., Jacque, J., Lovely, E., and Hofmeyer, A., “A guide to knowledge translation theory,” Journal of Continuing Education in the Health Professions (2006) 26(1); Vreugdenhill (2010) Pilot Projects in Water Management: Practicing Change and Changing Practice; Van Mierlo, B., “Convergent and divergent learning in photovoltaic pilot projects and subsequent niche development” Sustainability: Science, Practice, & Policy (2012) 8(2).

30 Managing Complexity: Adaptive management at Mercy Corps https://d2zyf8ayvg1369.cloudfront.net/sites/default/files/Adaptive%20management%20paper_external.pdf; Valters, C., Cummings, C., and Nixon, H., “Putting learning at the centre: Adaptive development programming in practice,” ODI report (2016); Allan, C., and Curtis, A., "Nipped in the Bud: Why Regional Scale Adaptive Management Is Not Blooming,” Environmental Management, Volume 36, Issue 3 (09/2005); Jones, l., “Towards a Holistic Conceptualization of Adaptive Capacity at the Local Level: Insights from the Local Adaptive Capacity Framework,” ODI, March 2011.

31 Bourgeon, L. “Team Building and Organizational Learning in New Product Development Projects.” Organizational Learning and Knowledge 5th International Conference (2003).

32 Schwartz, P., The Art of the Long View. New York, NY: Crown Business, 2012 (originally Doubleday Currency, 1991); Grieco, J., “Moving Toward Improved strategic planning in U.S. Foreign Policy,” The Simson Center 2015; Diffenbach, J, Corporate Environmental Analysis in Large US Corporations, Long Range Planning Vol 16 No 3 (1983): 107-116; Wilkinson, A., & Kupers, R., “Living in the Futures,” Harvard Business Review, 91(5): (2013):119-127.

12

● Organizational surveys: Quantitatively, some researchers have used propensity score matching and employed organizational surveys to conduct multivariate analysis and develop statistical modeling systems (e.g. using structural equation modeling (SEM)). These measures have been used to determine if continuous improvement systems affect organizational learning, and whether these two factors (independently and jointly) affect organizational performance.

● Ethnographic research: Some has been done, specifically with regard to communities of practice (COPs) and social and knowledge networks.

● Action research: This type of research, in which the researcher takes an active part in the process that s/he studying, has been used to reflect on the experiences of development agencies.33

WHERE ARE PEOPLE CALLING FOR MORE RESEARCH?

● First and foremost, there is a need to expand the evidence base on the effect, impact, and/or contribution of CLA practices to organizational effectiveness and development outcomes. Specifically identified areas of research included:

○ Evaluation and impact assessment on knowledge management ○ Empirical examinations of the impact of organizational learning ○ How a learning culture affects job satisfaction and performance-related outcomes

● Within CLA as a technical area, the additional areas for research include: ○ Who controls and drives learning? Why? For Whom? ○ There is a Northern bias in the dominant knowledge management for development

discourse; more research from other regions is needed. ○ What does knowledge management in different constellations of development

organizations look like? ○ How are knowledge and learning being conceptualized in a given situation? This pertains

to the knowledge as a behavioral or cognitive activity. ○ Resources for CLA and game changers and scenario planning, given more limited

research in these areas.

33 White, N., Cardone R., and de Moor, A., “Learning 3.0: collaborating for impact in large development organizations,” Knowledge Management for Development Journal 10(3), (2004): 21-37; Reynolds, M., and Vince, R.,, “Critical Management Education and Action-Based Learning: Synergies and Contradictions,” Academy of Management Learning and Education 3 (4) (2004): 442–456.