MLD-610

MLD-610Generating and Using Evidence to Improve the Management of your Organization

Fall 2016Meets Tuesdays and Thursdays 8:45-10:00 AM

HKS 1 Brattle St. Room 401

Julie Boatright Wilson Faculty Assistant: Gina AbbadessaTaubman - 440 Taubman 485Julie_Wilson@Harvard.edu gina_abbadessa@hks.harvard.edu

Nearly every organization is under pressure to produce evidence that the programs it runs are effective. And most organizations are under pressure to demonstrate that they are making good use of public funds or public trust. But how can you as a manager use the same tools and strategies that external evaluators employ to improve the performance of your own organization? What information on current performance and operations would you need? How would you gather that information? How would you analyze it? And how would you feed it back to those in the organization so that they could work more effectively to improve overall performance? Finally, what are the risks to you, as a manager, of undertaking such an initiative?

Gathering and using data to more effectively manage your own organization presents some unique opportunities as well as unique challenges. In this course students will develop a framework for thinking about evidence and apply this framework through case discussions and group projects to a range of organizations and smaller programs. Students will learn about different strategies for gathering information and employ them in their group project.

Audience: This course is designed for one-year and two-year students who are interested in developing the skills and intellectual frameworks for generating and using evidence to improve the performance of their organizations. Students will learn to gather information and insights through individual interviews, focus groups, surveys and observation, and to analyze data on organizational strategy and performance.

Course Outline: The course begins by laying out a framework for thinking about evidence. We will then work through each part of the framework in depth: question zero, theories of change, logic models, output and outcome measurement, performance measurement, implementation analysis and management, and measures of effectiveness. Threaded through discussions of the framework will be classes in which students will learn how to conduct in-depth interviews and focus groups, undertake systematic observation, and construct and administer surveys. We will discuss sampling strategies and students will apply them to their group projects. Finally, we will spend time on the ethics of research and data gathering as well as strategies for managing organizational change based on the evidence generated.

Class Meetings: This course will meet Tuesdays and Thursdays from 8:45 to 10:00 AM in HKS 1 Brattle St. Room 401. Class sessions will include lectures, discussions, and presentations. We may supplement the class session with optional outside speakers.

Text and Readings: All readings are on line or on the Canvas website.

MLD-610 Page 1

MLD-610

Specific Requirements: Students will be responsible for completing the assigned readings and for coming to class prepared to discuss the materials. Because a major focus of this course is skills development, we will also be practicing many of the techniques in class and discussing what lessons we draw from our experiences.

Students will work in small groups for a unit within an organization, applying the class materials to assess the unit’s performance. Students will have two types of individual assignments: (1) four short reflection pieces describing what they learned from implementing each of the research methodologies1, and (2) two memos applying the materials discussed in class to specific situations or issues. Students will also be responsible for working in small groups to prepare a short report for their “client” organization and present their findings in class. Copies of some of the reports and presentations from last year’s students will be available as examples.

A number of units in the Kennedy School have volunteered to have students help them develop a plan for measuring performance and effectiveness. The advantage of using internal HKS units is that the turn-around time for getting to know the organization, setting up interviews and focus groups, observing activities and measuring performance will be very short. I am happy to entertain the idea of using organizations external to the Kennedy School, but we should discuss that in advance. I am also happy to talk with you about using your project for this class to delve more deeply into your PAE or SYPA project.

Required assignments include:

Individual assignments: Personal reflections on what you learned from 4 of the following 6 exercises:

o Conducting in-depth interviewso Conducting focus groupso Clarifying a question zero and theory of changeo Using administrative datao Observation researcho Implementing a survey

Complete the CITI or NIH human subjects training if you have not already done so. Once you have completed the training, you are certified for three years. If you have already completed the training, just submit your certificate.

Two individual memos: The first memo will be due September 20th. The topic will be provided one week in advance. The second memo will be due November 17th and, again, the topic will be provided one week in advance.

Group assignmentso Final group presentation o Final group paper for your client group

1 If your group project will not involve using at least four of the methodologies discussed in class, you must talk to Julie Wilson in advance to discuss options for developing the skills and reflecting on them.

MLD-610 Page 2

MLD-610

Class Participation: Class sessions will involve discussions of cases and readings, primarily focused on applying the lessons from the readings to the cases. Class sessions will also provide time for individual and group work and opportunities for individual and group reflection on what we are learning.

Determination of Grade: The final grade will be determined as follows: Participation in class discussion – 25% Individual reflections on exercises – 20% in total; 5% per reflection piece Memos – 20% (10% for each memo) Group project – 35% (20% for group memo/report and 15% for the presentation)

Expectations of Professionalism:

You are expected to abide by the University policies on academic honesty and integrity as explained in the Student Handbook. Violations of these policies will not be tolerated and are subject to severe sanctions up to and including expulsion from the university.

While study groups are encouraged, their proper purpose is not to do the homework assignments, but to help you learn the material. Each student is responsible for writing and submitting assignments that are labeled “individual.” While you may find it valuable to discuss the materials in your work group or a study group – and I encourage you to do this – assignments identified as “individual” should be written by you and provide your views on the issue.

* * * * *

Class 1: Introduction: Challenges for Managers: How to Generate and Use Evidence Sept. 1

This class will lay out the goals and structure of the course. In particular, we will raise the issues of (1) strategies by which management can question the effectiveness of their organization’s operations and gather data to assess it, (2) designing and implementing strategies to effectively gather information on performance and feed it back into organizational operations, and (3) strategies for changing the culture of an organization around gathering and using data, information and evidence.

Reading: Case: NYPD New Gladwell, Malcolm. 2006. “Million-Dollar Murray: Why problems like homelessness may be

easier to solve than to manage.” The New Yorker, February 13, 96-107.

Study Questions:1. What is the situation Bratton inherited when he became Police Commissioner? What are

the biggest problems the Police Department is facing? How would you characterize the organizational culture of the police department? The resources available to it? The way it gathers and uses data?

2. Bratton held strong views about the nature of crime and criminals as well as about police and their role in crime prevention. What were Bratton’s theories about crime? Criminals?

MLD-610 Page 3

MLD-610

Effective strategies for fighting crime? The role of citizens and police? To what extent are Bratton’s views consistent with those of Malcolm Gladwell?

3. What are the most important actions that Bratton took? In particular, how did Bratton generate and use evidence of police performance? Why were these actions important? What did they accomplish? To what extent and how did Bratton’s actions produce – directly or indirectly – a change in performance? What is the evidence for your assessment? What are the strengths of these changes? The weaknesses? Be as specific as you can about laying out the causal linkages between specific actions and specific accomplishments.

4. What was the impact of Bratton’s actions on rank and file police? On precinct commanders? On crime? What evidence do you have from the case to support your analysis?

5. What should Bratton do to sustain and build on the progress he has made? What are the new problems that have emerged from the changes he has made? From his management style? What do your answers to these questions suggest about what Bratton should do going forward?

Class 2: Laying out the Framework for Thinking About Evidence Sept. 6

We will use this class to begin developing our basic framework for thinking about data and other forms of evidence.

Reading: Our Piece of the Pie: From Data to Decision-Making:

http://www.bridgespan.org/getdoc/a0e779be-3725-44de-b330-2e92f9978047/Our-Piece-of-the-Pie-From-Data-to-Decision-Mak.aspx#.VIXBTsmAMmQ

Jeri Eckhart-Queenan and Matt Forti, “Measurement as Learning: What Nonprofit CEOs, Board Members, and Philanthropists Need to Know to Keep Improving,” Bridgespan http://www.bridgespan.org/Publications-and-Tools/Performance-Measurement/Measurement-as-Learning-What-Nonprofit-CEOs,-Board.aspx#.VkoHQL-VnhU

Demographics of Hartford: Hartford, Connecticut is an old New England industrial town located between Boston and New York City that, over the last several decades, has become the “headquarters” for many major insurance firms. The city’s population of 125,000 is 43% Hispanic, 35% black, 16% white and 6% “other.” Three-quarters of the Hispanics were born in Puerto Rico, which is a territory of the United States. One-third of the adults have less than a high school education and another 30 percent have a high school education only. Only 14% are college graduates. About half the households have an annual income of less than $25,000. The United States government sets the poverty level for a family of four at $23,850, which means that Hartford can be described as a poor city. The city is surrounded by higher income suburbs.

Situation: Many organizations are focused on strategies to help youth navigate the path through adolescence to adulthood, particularly youth who are poor and living in low-income neighborhoods. Our Piece of the Pie is an organization that has changed its focus from one of general support for everyone living in poor neighborhoods in Hartford to focusing solely on adolescents and young adults.

MLD-610 Page 4

MLD-610

The issues the organization addresses include how to keep youth on track, how to develop and implement effective education and workforce skills development programs and supports, deal effectively with personal and family challenges, and help youth avoid delinquent or criminal activities.

Challenge: We are only now beginning to learn about adolescent brain development and its implications for education and labor force policies. Our Piece of the Pie, struggling with its mission to support these young people, is trying to figure out what works for adolescents in Hartford and how their organization can make sure that it is always performing at its best. We will use their organization and efforts to lay out a framework for how you can lead your organization to better generate and produce evidence about its effectiveness.

Study questions – Our Piece of the Pie:

1. What is Our Piece of the Pie trying to accomplish? What is its strategy for achieving these goals?

2. How effective a program do you think Our Piece of the Pie is now? What are the criteria by which you made your decision about its effectiveness? Along what dimensions do you think effectiveness should be measured?

3. What should the relationship be between an organization’s theory of change and performance measurement and between performance measurement and program evaluation?

4. What are the challenges to collecting, analyzing and presenting data in ways that inform decisions from the front line to the CEO? To what extent and in what ways does collecting data and analyzing performance enhance and detract from a service provider’s efforts to pursue the organization’s mission?

5. What strategies did Our Piece of the Pie employ to try to change the culture of the organization around the generation and use of data? What are the particular challenges they faced in trying to do this?

Class 3: Clarifying Question Zero and Building a Theory of Change Sept. 8

Many organizations operate under complicated mission statements or legislation, often changed or amended in response to emerging issues or crises. Over time, these mission statements or legislation can become very complex and leave an organization unfocused. This is why it is important for managers to step back and clarify what their organization is trying to achieve and convey this message to those who work for them.

Question Zero is a concept developed to help leadership clarify an organization’s primary focus by stating in as close as possible to ten words or less what it is that they are trying to accomplish. For example, the Kennedy School’s Question Zero is: “train enlightened public leaders and generate ideas that provide solutions to challenging public problems. “ OK, that is 14 words, not 10. But this crystallization makes it clear what the Kennedy School should be held accountable for accomplishing. A theory of change is an organization’s road map for achieving its question zero. It is important to be systematic and thorough in laying out your theory of change.

Readings: Measured Approach: TEGV Assesses its Performance and Impact

MLD-610 Page 5

MLD-610

Sharon M. Oster, Strategic Management for Nonprofit Organizations: Theory and Cases, Chapter 2: The Mission of the Nonprofit Organization, pp. 17-28.

Study Questions:1. What is TEGV’s Question Zero? Does it have more than one Question Zero? If so, what are

the other Questions Zero? Does each hold equal weight or is one more important than the other(s)?

2. What are the implications of having more than one Question Zero? It might be useful to think about this question from the perspective of top management, front-line management, and those who used the services of your organization. It might also be useful to think about this question from the perspective of a broader set of stakeholders.

3. How would you go about figuring out what various people in your organization think your Question Zero is? Or your Questions Zero are?

4. What is the implication of having more than one Question Zero for your Theory/ies of Change?

Note: We will spend some time in the first three classes identifying and selecting a group project. By class 4 you should have organized your group and selected a project. Almost all assignments will relate to this project.

Class 4: In-Depth Interviews Sept. 13**Individual Memo topic given.

One way to begin to get an understanding of what an organization’s Question Zero and Theory of Change are and the extent to which definitions of each are shared among top management is to conduct a series of in-depth interviews. In this class we will spend time thinking about how you might phrase the questions to get a sense of what your respondent thinks the goals of his/her organization are and what they believe their theory of change is.

Readings: Weiss, Robert S., Learning from Strangers: The Art and Method of Qualitative Interview

Studies. New York: Free Press, 1995. Chapter 4: Interviewing (pp. 61-119)

Study Questions:1. When you identify the project you will work on for this course over the semester, you will

begin by conducting individual or small group interviews. We will practice these in class. You should come to class prepared to interview a fellow classmate about his/her last (or current) job. What is the organization/unit’s Question Zero? What is the Theory of Change for achieving Question Zero? Come to class prepared to be both the interviewer and the interviewee. Students will be randomly assigned to one role or the other in class.

Class 5: Building a Theory of Change Sept. 15

Social Impact Bonds or Pay-for-Success Contracts are one of the “latest” sets of initiatives in public policy. But what are they? While the idea behind them seems simple, in fact working out the details of how to design and implement a SIB initiative is complex. What is the theory of change behind Social Impact Bonds?

MLD-610 Page 6

MLD-610

You should leave the class with an idea of how you want to begin gaining an understanding of the theory of change for the group you are working for.

Readings: Massachusetts Pay-for-Success Contracts: Reducing Juvenile and Young Adult Recidivism Weiss, Carol Hirschon, “Nothing as Practical as Good Theory: Exploring Theory-Based

Evaluation for Comprehensive Community Initiatives for Children and Families,” New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James P. Connell, et al, The Aspen Institute, 1995. While this article is now two decades old, it is still one of the better and more accessible pieces. And the ideas in this article are just now coming back into fashion. You should read pages 65 to 73 before class. The remainder of the chapter, which is optional, is a careful application of the idea of a theory of change to the complex initiatives that go by the name Comprehensive Community Initiatives for Children and Families.

Study Questions:1. What is the Question Zero for ROCA? What is ROCA’s theory of change? 2. What is the Question Zero of Social Impact Bonds? What is the theory of change for Social

Impact Bonds?3. Are there some situations or challenges that would be inappropriate for a SIB initiative? If

so, why? What does this suggest for your understanding of the Theory of Change for SIBs?4. At the end of this class, we will select a topic for the next class on focus groups. Your

assignment for that class will be to develop a set of focus group questions.

Pre-class comment: Consider suggesting a topic for Tuesday’s focus groups and posting it on Canvas for your

classmates to consider.

Class 6: Focus Groups Sept. 20**Individual memo due.

Sometimes it is most productive to interview a small group of people together so that they build on one another’s comments and push one another to think through the specific issue of concern to you. If you want to run a focus group, how do you select the group members? How do you structure the conversation? What do you do with the information they provide?

Readings:

Understanding Focus Groups with Author Thomas Greenbaum, http://www.i5publishing.com/understanding-focus-groups-with-author-thomas-l-greenbaum/

Thomas Greenbaum, 10 Tips for Running Successful Focus Groups, http://www.groupsplus.com/pages/mn091498.htm

U. S. AID Center for Development Information and Evaluation, Performance Monitoring and Evaluation TIPS: Conducting Focus Group Interviews

MLD-610 Page 7

MLD-610

Study Questions:1. What are the advantages and disadvantages of focus groups as a strategy for gathering

information about the effectiveness of an organization’s operations?2. Based on the topic selected in our last class, write a set of focus group questions. In doing

so, think about your opening questions, your strategies for getting focus group participants to build on one another’s comments, how you will switch from question to question, how you will conclude the discussion, and how you would take notes on the discussion if this were part of an ongoing project. Some of you will be selected at the beginning of class to lead focus groups of your fellow students.

Class 7: Sampling in Qualitative Interviewing Sept. 22

We hear a lot about random sampling, but sometimes random samples of respondents are difficult, if not impossible, to obtain. We only need to think about the demise of land lines and spread of cell phones to realize that it may be impossible to realize the “statistical ideal.” This is why major telephone pollsters like Pew are focused on getting representative samples. We will talk about different sampling strategies in class.

Readings: Case: A New Dean: Doug Elmendorf Joins the Kennedy School Kuzel, Anton J. “Sampling in Qualitative Inquiry,” Doing Qualitative Research. Crabtree,

Benjamin F. and William L. Miller, eds. Newbury Park, CA: Sage Publications, 1992. pp. 31-45.

Sanford Labovitz and Robert Hagedorn, “The Population and Sampling,” Fist-fights in the Kitchen: Manners and Methods in Social Research. 1975, pp. 87-96.

Study Questions:1. How would you describe the HKS student population? We are known as being the most

diverse student population at Harvard, but what are the dimensions of our diversity?2. If you were “sampling” the HKS student population to assess their views of the School and

where it should head, how would you construct your sample? What strategies would you use to gather information from various sub-groups of students?

3. How would you describe Dean Elmendorf’s strategy for getting to know the HKS student population and gaining an understanding of their views of various issues.

4. What is your strategy for sampling individuals to be interviewed on your project? How might your sampling strategy vary with the type of information you want to gather?

5. What is your strategy for selecting focus group respondents for your project? How might this sampling plan differ from that for gathering information from individuals one-on-one?

Class 8: Observation and Participant Observation Sept. 27

As that famous social scientist, Yogi Berra, once said, “You can observe a lot by just watching.” This insight is very useful for process/implementation evaluation. There are some processes that you can most accurately assess by observing them happen. And, there is nothing quite like being on site to get an understanding of what it feels like to participate in a program. But there is nothing casual about unobtrusive observation or participant observation. You need to be clear about your questions and you need to think hard about sampling.

MLD-610 Page 8

MLD-610

Readings: We will draw on cases previously discussed, particularly TEGV, Our Piece of the Pie, and

Dean Elmendorf joins the Kennedy School, as well as your projects, as the basis for discussion.

Eugene Webb, et al, Unobtrusive Measures. Thousand Oaks, CA: SAGE Publications, 2000. Chapter 5: “Simple Observation,” pp. 113-126.

Family Health International, Qualitative Research Methods: A Data Collector’s Field Guide. Module 2: Participant Observation, pp. 14-28.

Study Questions:1. If you wanted to observe the activities described in the Our Piece of the Pie and TEGV cases,

how would you go about it? How would you sample times and sites? How would you observe? What information would you be gathering? How would you record and analyze it?

2. If you were to observe Kennedy School students, you would be a participant observer. How does that influence your sampling strategy? Your activities? Your capacity to gather information and insights? Your analysis? (Non-Kennedy School students should answer these questions based on observing at their own school.)

3. What do your answers to the above questions suggest about how you – as a manager – might use observation and participant observation to gain a deeper understanding of your organization and its operations?

Class 9: Logic Models Sept. 29

Our theories of change should lead us into the development of our logic models. And our logic models will feed into our performance management system, so we need to “get it right.” This class will focus on logic models and the way they change as an organization changes and matures.

Readings: Playgrounds and Performance: Results Management at KaBOOM! (A) and (B) Part C will be

handed out in class. Carolina Buitrago, “Framing Program Evaluation: Tinkering with Theories of Change and

Logic Models,” November, 2015. W. K. Kellogg Foundation, Logic Model Development Guide. (This is somewhat long, so you

should feel free to skim it for class. But it will be a very useful “how to” guide for your project.)

Study Questions:1. What is Question Zero for KaBOOM! Do you think they are clear about their Question Zero?2. KaBOOM! has changed its model over time. What was its initial theory of change? How did

that evolve and why? What is the current theory of change for KaBOOM!?3. What are the components of KaBOOM!’s current logic model? How has the logic model

changed over time?4. How should KaBOOM! measure its outputs? Outcomes? Impacts? What performance

metrics should it be tracking?

MLD-610 Page 9

MLD-610

Class 10: Implementing a Program: Structuring Pre-tests of Programs Oct. 4 to Gather Evidence on Theory of Change

Getting a program or initiative up and running is normally a great deal of work. But when you can be systematic about implementation, you can use it to your advantage to test your theory of change and identify the systematic challenges in implementation. This may allow you to make changes along the way to improve the program’s overall effectiveness. One only needs to think about the roll-out of the Affordable Care Act (Obamacare) to understand the benefits of taking advantage of pretesting a roll-out to ensure that your program will operate effectively.

Readings: Deworming Kenya Note: a reading may be added depending on the issues raised in class to date and the

issues emerging from your work on your projects

Study Questions:1. What are the pieces of evidence on which the Deworming Kenya program is based?

How strong is this evidence? Note that there is evidence from a number of different sources and fields.

2. Although the proposed intervention is quite simple as far as interventions go, actually getting the initiative up and running is very challenging. What challenges did the implementers face and what types of resources did they need to implement the “test run”?

3. Those implementing the intervention took advantage of the challenges they thought they would face to design their implementation in a way that would generate even more evidence about the program’s effectiveness. How did they do this? What lessons do you draw from their experiences?

Class 11: Process Evaluation Oct. 6

A program with a robust theory of change will not be effective if it is not implemented with fidelity. We often start our implementation or process analysis by mapping – mapping how clients will flow through the program, how information will flow through the program and how funds will flow through the program. Sometimes this process is quite straight-forward and sometimes it is complex. In the case of the Jamaica PATH Program, it is complex.

Readings: Jamaica PATH case: Part A Peter Rossi, et al, Evaluation: A Systematic Approach, Seventh Edition, SAGE Publications,

2004. Chapter 6: Assessing and Monitoring Program Process, pp. 169-201

Study Questions:1. The Jamaica PATH Program has a lot of moving parts. It is dependent on a lot of different

people who report through different chains of command doing their part. Try to sketch out how clients, information and money flow through the process.

MLD-610 Page 10

MLD-610

2. Where do you anticipate this information to flow smoothly? Where do you think the greatest challenges to accurate and efficient movement of data are likely to be? Why? What problems do you anticipate?

3. What do your responses to questions 1 and 2 suggest for how you want to organize your process/implementation assessment? What questions do you want to answer? Which research methods would you use to gather reliable information to answer those questions?

Class 12: Gathering Data: Developing and Implementing Surveys Oct. 11

Often the most reliable way to get information from individuals who participated in an activity is to ask them directly. We have all completed surveys – or refused to complete surveys. Sometimes the surveys seem thorough and make us think. Other times they are sloppy, incomplete, or have a clear political agenda. In this class we will discuss the basics of writing a good survey.

Readings: Floyd Fowler, Jr., Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA:

SAGE Publications, 1995. Chapter 4: Some General Rules for Designing Good Survey Instruments, pp. 78-103.

Joseph Wholey, et al, Handbook of Practical Program Evaluation. San Francisco: Josey-Bass Publishers, 1994. Chapter 11: Designing and Conducting Surveys, Thomas I. Miller, pp. 271-292

John D. McCarthy and M. Barbara McCarthy, “Power and Purpose in Survey Research (If You Got the Money, Honey, I Got the Time), in George H. Lewis, ed., Fist-fights in the Kitchen: Manners and Methods in Social Research. Santa Monica, CA: Goodyear Publishing Company, Inc., 1975.

Study Questions:1. You should be thinking about the information you want to gather through a survey for your

group project. Come to class with a list of the general areas of information you would like to address and specific pieces of information you would like to gather.

2. One important set of categories to consider are the following: What background behavioral information do you want to gather? What attitudes or orientations would you like to measure? Do you want to ask questions about anticipated future behavior?

Class 13: Using Surveys as a Management Assessment Tool Oct. 13

Michael Bloomberg focused attention on three particular populations while he was mayor of New York City: Early childhood, youth, and working adults in the low-wage labor market. His goal was to implement a number of different initiatives, rigorously assess their effectiveness, and fund those that produced results while eliminating those that did not.

Teen Action is one of the programs Bloomberg’s team assessed. This case details the program, its implementation, their assessment strategy and the initial results. Managers can take many lessons from this experience, but we want to focus on the data gathering techniques they used – particularly the survey. That said, it is worth thinking about how you, as a manager, might handle the internal politics of a data gathering initiative gone awry in your organization.

MLD-610 Page 11

MLD-610

Readings: Case: New York City Center for Economic Opportunity: Teen ACTION Program Della Volpe, John,

Study Questions: What was the Teen ACTION Program trying to accomplish? What was its Question Zero? Its

Theory of Change? What evidence did the Committee on Economic Opportunity (CEO) staff have that this theory of change made sense? What information were they hoping to gather from their assessment?

How would you describe CEO’s plan for assessing the effectiveness of the Teen ACTION Program? What were its strengths? Weaknesses? What would you have done differently?

Implementation of their assessment was fraught with problems. How might they have modified the design in the midst of implementing the study in order to gather the greatest amount of information? What information that they hoped to gather could not be gathered? What information could they have gathered given the constraints of the situation?

If you were advising CEO, how would you suggest they structure their next assessment? In what ways might on-line strategies like those used by della Volpe be useful for CEO with

this particular group?

Class 14: Using Information from Implementation Analyses to Oct. 18 Redesign Programs and Change Processes

We have been focusing on identifying the important questions about our organization’s performance and developing strategies for gathering that information. But what do you do when you learn that your initiative is not functioning as well as it might? How do you use information you have gathered to make changes?

Readings: Jamaica PATH: Part B TBA

Study Questions:1. How did those responsible for implementing the PATH Program assess the implementation?

What are the strengths and weaknesses of their strategy? What did they learn? What questions still remained?

2. Think back to our initial discussion of the Jamaica PATH Program and our suggested strategies for monitoring its implementation. In what ways was their assessment process similar to what we proposed? Different?

3. What does this suggest to you about the importance of systematically assessing program implementation? The costs of doing so? The benefits?

Class 15: Testing a Theory of Change Oct. 20

MLD-610 Page 12

MLD-610

An organization’s theory of change is important. As we have learned, many organizations have not thought carefully about their theory of change so don’t know if they are explicitly following. More important, they don’t know if their theory of change is “right.” The question is, how do you assess and test the validity of your theory of change?

Readings: Scared Straight: Freeport City Council Takes On Juvenile Delinquency: Part A Parts B and C will be handed out in class Ron Haskins and Greg Margolis, Show Me the Evidence: Obama’s Fight for Rigor and

Results in Social Policy. Washington, D.C.: Brookings, 2015. Chapter 1:Chapter 1: The Obama Strategy for Attacking Social Problems, pp 1-30. (Note: This chapter was written by Ron Haskins, a Republican, on the Obama strategy for funding evidence-based interventions.)

If you want to get a better understanding of the Scared Straight Program, you may want to watch part of all of this video: http://www.bing.com/videos/search?q=scared+straight+programs&FORM=VIRE2#view=detail&mid=014C3B8036A0D9FD4931014C3B8036A0D9FD4931

Study Questions: What is the Question Zero for the Scared Straight Program? What is the Theory of Change for the Scared Straight Program? If you were on the city council, how would you vote? What are the ways you can assess an organization’s theory of change?

Class 16: Outcome Measures and Benchmarking Oct. 25

If your theory of change is correct and you have implemented your program with fidelity, you should expect to produce your expected outcomes. But, since there is never enough time and there are never enough resources, you can’t run RCTs for all your initiatives. You need to have additional strategies. Carefully measuring outcomes and benchmarking are strategies you should consider. But you need to be parsimonious in selecting the outcomes to measure and assess, and you need to be careful and accurate in measuring them.

While we are learning a lot about adolescent brain development and now understand that there is a lot we don’t know, we do know that youth in our society need to meet certain milestones in order to move successfully to productive, law-abiding adulthood. As you reread this case, think about the key benchmarks the youth they serve need to meet.

Readings:• Our Piece of the Pie: From Data to Decision-Making:

http://www.bridgespan.org/getdoc/a0e779be-3725-44de-b330-2e92f9978047/Our-Piece-of-the-Pie-From-Data-to-Decision-Mak.aspx#.VIXBTsmAMmQ

• Robert Behn, “Why Measure Performance? Different Purposes Require Different Measures,” Public Administration Review, Vol. 63, No. 5, September/October 2004

Study Questions:

MLD-610 Page 13

MLD-610

1. If you were the director of OPP, what are the key measures you would want to track to assess the effectiveness of your programs? You might consider two or three for each of the pathways.

2. How would you describe the culture of Our Piece of the Pie? What are the likely characteristics of the people who work there? What is their training likely to have been? What aspects of their performance are particularly highly valued? What does this suggest about how you would want to design your benchmark measurement and use it for management purposes?

3. How did Our Piece of the Pie use performance data? What were the questions they wanted to address? How did they present the benchmarking data? What is your assessment of their strategy?

Class 17: Ethical and Legal Issues to Consider when Gathering Data Oct. 27

Institutional review boards are important, though they have a reputation in some circles for making research difficult. How do you know when you should be getting institutional review board approval for your work?

Readings:• Each student should complete the CITI on-line ethical training for the IRB if you have not

already done so. Once you complete this, you will be certified for three years.• Mary Roach, Stiff: The Curious Lives of Human Cadavers. New York: W. W. Norton &

Company, 2003. “Crimes of Anatomy,” pp. 39-57.• Herbert Kelman, “The Rights of the Subject in Social Research: An Analysis in Terms of

Relative Power and Legitimacy,” The American Psychologist, November 1972, pp. 989 – 1005.

Study Questions:1. As a manager, what are some of the guidelines you should abide by in collecting data on the

performance of your organization’s employees? Those your organization serves?2. What do these guidelines suggest for strategies about sharing information inside your

organization? With other organizations?3. What do these guidelines suggest about restrictions you might place on outside researchers

who want access to data your organization generates?

Class 18: Reviewing the Evidence Model and Update on Projects Nov. 1

The next several classes will focus on how you as a manager can use the information you gather to change organizational practice and culture. We will use this class to review the model and test its usefulness to your projects. Come to class prepared to talk about the project your group is working on – what you are learning and what the challenges are.

It is very important that we all come into this discussion not only prepared but also willing to be open and questioning about what we think we are finding in our research. This means that the discussion of this particular class session is confidential. To paraphrase an American aphorism, what happens in MLD-610 stays in MLD-610.

MLD-610 Page 14

MLD-610

Class 19: Using Data to Change Organizational Culture and Practice Nov. 3

It’s one thing to have the top management team or a few managers focus on data and performance. But how do you change the culture of your entire organization so that everyone begins thinking differently about how to assess current processes and performance and taking on responsibility for trying new strategies and measuring their effectiveness?

Readings: Cincinnati Children’s Hospital Amy Edmondson, “Strategies for Learning from Failure,” Harvard Business Review, April

2011, pp. 49-55. Edgar Schein, Organizational Culture and Leadership, 1992. Chapter 1:Defining

Organizational Culture and Chapter 2:Uncovering the levels of culture, pp 1-27.

Study Questions:1. Uma Kotagal and her team are trying to get physicians, nurses and other health care

providers to both use evidence-based techniques and generate evidence about what leads to and supports higher quality health care. In what way is the first part of this challenge – getting front line health care providers and their managers to use evidence based techniques particularly difficult in health care? In what way is this challenge similar to and different from the challenge of getting front line staff and their managers to use evidence-based practices in other fields?

2. The Cincinnati Children’s Hospital team developed a “theory of change” for improving the quality of care it provided and the quality of its overall operation. How would you describe this theory of change? What were its strengths in this particular situation? Its weaknesses?

3. What were the key domains of resistance to change? How did the Hospital’s Leadership and the Quality Improvement team address this resistance? Do you think they were effective? What would you have done differently?

4. Changing the culture of an organization takes both leadership and followership. What does the Cincinnati Children’s Hospital case tell us about both leaders and followers? What does it suggest leaders consider as they initiate and try to sustain the type of cultural change that totally infuses and organization?

5. What enables Cincinnati Children’s Hospital to sustain the change? If you were Jim Anderson’s replacement, what would you be concerned about?

Class 20: Management Strategies for Sharing Data to Solve Problems: Nov. 8 Wraparound Milwaukee

Often as part of the process of accomplishing our mission, we need to share information with other organizations about an individual, a family, a program or a situation. This turns out to be extremely difficult for a number of reasons. As you read the Wraparound Milwaukee case, think carefully about the data that are being shared and the safeguards in place to prevent misuse of the data.

MLD-610 Page 15

MLD-610

Readings: Wraparound Milwaukee case Stephen Goldsmith, et al, The Power of Social Innovation: How Civic Entrepreneurs Ignite

Community Networks for Good, San Francisco: Jossey-Bass, 2010. Chapter 6: Turning Risk into Reward. (On Canvas website)

Study Questions:1. What is the Wraparound Milwaukee model? What is its goal? How does it work? 2. A key factor in the Wraparound Milwaukee model is the capacity to share data about one

child and his or her family across a range of service providers. Why is this advantageous? What are the concerns about sharing data of this sort?

3. What lessons could other organizations draw from Wraparound Milwaukee about strategies for sharing data or information on specific individuals or situations?

Class 21: Generating and Using Data across Organizational Boundaries Nov. 10 To Achieve a Broad Mission

**Individual Memo topic given.

Some organizations that filled an important role at one point gradually become less relevant. In other words, the value they are producing may no longer be in demand. If this is the situation you find your organization in, how can you use data on your organization and the changing environment to reposition yourself? To form collaborations with others?

Readings:• Case: Guide Dogs for the Blind• Herman B. Leonard, “A Short Note on Strategy-Building in the Social Sector,” Harvard

University, May 2002.

Study Questions:1. What is Guide Dogs for the Blind’s Question Zero? Theory of Change? What are the

strengths of the organization? The challenges it faces?2. How has the situation for blind and visually impaired individuals changed since the

organization was founded? What other organizations and services are available for this group of individuals? What are the questions zero, strengths and challenges of these other organizations?

3. How did Geraldine Peacock use data and performance measurement to gain a better understanding of the strengths and weaknesses of her organization? To identify partners for collaboration? To persuade other organizations to start collaborating?

4. Going forward, what types of information should each of the participating organizations be gathering and sharing with one another? As a group, what information should they be collecting and assessing?

5. Clearly moving forward will take good political skills and leadership. But data and reliable information are also important. What does this case suggest about how leaders could use data to foster collaboration in pursuit of a goal?

Class 22: Collective Efficacy Nov. 15

MLD-610 Page 16

MLD-610

Most of the issues our organizations are trying to address are complex. Often this means that we are focused on only a small part of the overall issue or problem and are contributing a small, though often important, part to the overall solution of the problem. How might we gather and use evidence to show that we are

Readings:• Strive Together: Reinventing the Local Education Ecosystem• John Kania, et al, “Strategic Philanthropy for a Complex World,” Stanford Social Innovation

Review, Summer 2014, pp 26-37• John Kania, et al, “Embracing Emergence: How Collective Impact Addresses Complexity,

Stanford Social Innovation Review, 2013, pp. 1-7

Study Questions:1. STRIVE is structured as a “backbone” organization. What does that suggest for the roles it

plays? What responsibilities does it have to the other organizations contributing to the educational ecosystem?

2. How effective was STRIVE?3. How did they use evidence to assess effectiveness? To set goals? To encourage behavior on

the part of others?

Class 23: What do we mean by Evidence? Nov. 17**Individual memo due.

We began the course by laying out a model for evidence and spent our time figuring out how we as managers would use that model to better understand and improve our organization’s performance as well as change the culture of our organization around the generation and use of data for management. Each of you has applied the model to a specific program or organizational unit, which has given you a chance to practice using the techniques and skills the course covered. This is our chance as a class to assess the model.

Class 24: Presentations Nov. 22

Thanksgiving – no class

Class 25: Presentations Nov. 29

Class 26: Wrap Up Dec. 1

MLD-610 Page 17