MLD-610

MLD-610Generating and Using Evidence to Improve the Management of your Organization

Spring 2016Meets Tuesdays and Thursdays 11:45am-1:00pm

HKS 1 Brattle St. Room 401

Julie Boatright Wilson Assistant: Nicole GrenierTaubman – 440 Taubman – 459AJulie_Wilson@harvard.edu Nicole_Grenier@harvard.edu

Nearly every organization is under pressure to produce evidence that the programs it runs are effective. And many organizations are under pressure to demonstrate that they are making good use of public funds or public trust. But how can you as a manager use the same tools and strategies that external evaluators employ to improve the performance of your organization? What information on current performance and operations would you need? How would you gather that information? How would you assess it? And how would you feed it back to those in the organization so that they could work more effectively to improve overall performance? Finally, what are the risks to you, as a manager, of undertaking such an initiative?

Gathering and using data to more effectively manage an organization presents some unique challenges. In this course students will develop a framework for thinking about evidence and apply this framework through case discussions and group projects to a range of organizations and smaller programs. Students will learn about different strategies for gathering information and employ them in their group project.

Audience: This course is designed for one-year and two-year students who are interested in developing the skills and intellectual frameworks for generating and using evidence to improve the performance of their organizations. Students will develop the skills for gathering and analyzing data on organizational strategy and performance.

Course Outline: The course begins by laying out a framework for thinking about evidence. We will then work through each part of the framework in depth: question zero, theories of change, logic models, output and outcome measurement, performance measurement, implementation analysis and management and measures of effectiveness. In the process, students will learn how to conduct in-depth interviews and focus groups, undertake systematic observation, and construct and administer surveys. We will discuss sampling strategies and students will apply them to their group projects. Finally, we will spend time on the ethics of research and data gathering as well as strategies for managing organizational change based on the evidence generated.

Class Meetings: This course will meet Tuesdays and Thursdays from 11:45am to 1:00pm in HKS 1 Brattle St. Room 401. Class sessions will include lectures, discussions, and presentations. We may supplement the class session with optional outside speakers.

Text and Readings: All readings are on line or on the Canvas website.

Specific Requirements: Students will be responsible for completing the assigned readings before class and coming to class prepared to discuss the materials. Because a major focus of this course is skills development, we will also be practicing many of the techniques in class and discussing what lessons we draw from our experiences.

MLD-610 Page 1

MLD-610

Students will work in small groups for a unit within an organization, applying the class materials to assess the unit’s performance. Students will have both individual assignments, primarily reflecting on experiences applying materials, and group assignments reporting on the progress in getting to understand and make recommendations for your organizational unit.

A number of units in the Kennedy School have volunteered to have students help them develop a plan for measuring performance and effectiveness. The advantage of using internal HKS units is that the turn-around time for getting to know the organization, setting up interviews and focus groups, observing activities and measuring performance will be very quick. I am happy to entertain the idea of using organizations external to the Kennedy School, but we should discuss that in advance.

Required assignments include:

Individual assignments: Personal reflections on what you learned from each of the following exercises:

o Conducting in-depth interviewso Conducting focus groupso Clarifying a question zero and theory of changeo Using administrative datao Observation researcho Implementing a surveyo Complete the CITI on-line training in research ethics

Group assignmentso Memo laying out question zero and theory of change for your project.o Memo laying out implementation assessment, including sampling strategyo Memo laying out performance metricso Final group presentation o Final group paper for your client group

Class Participation: Class sessions will involve discussions of cases and readings, primarily focused on applying the lessons from the readings to the cases. Class sessions will also provide time for group work and opportunities for groups to report on what they are learning.

Determination of Grade: The final grade will be determined as follows: Participation in class discussion – 25% Individual memos reflecting on exercises – 25% Group project – 50%

Expectations of Professionalism:

You are expected to abide by the University policies on academic honesty and integrity as given in the Student Handbook. Violations of these policies will not be tolerated and are subject to severe sanctions up to and including expulsion from the university.

MLD-610 Page 2

MLD-610

While study groups are encouraged, their proper purpose is not to do the homework assignments, but to help you learn the material. Each student is responsible for writing up and submitting assignments. Separate copies of a group-constructed assignment are not acceptable.

* * * * *

Class 1: Introduction: Challenges for Managers: How to Generate and Use Evidence Jan. 26

This class will lay out the goals and structure of the course. In particular, we will raise the issues of (1) strategies by which management can question the effectiveness of their organization’s operations and demand data to assess it, (2) designing and implementing strategies to effectively gather information on performance and feed it back into organizational operations, and (3) strategies for changing the culture of an organizations around gathering and using data, information and evidence.

Reading: Nadler, David, “Confessions of a Trusted Counselor,” Harvard Business Review, September,

2005, pp. 68-77. Case: Measured Approach: TEGV Assesses its Performance and Impact

Questions:1. What is TEGV trying to accomplish? How did its goals change over time?2. What questions were the managers/leaders of TEGV asking about the organization? In your

view, were these the right questions? What additional questions would you have wanted to ask?

3. There are many ways we might define an organization’s performance. What are some of the ways TEGV might frame the concept of performance?

4. What are the advantages of hiring outside experts to assess your organization’s performance? The disadvantages? What are the advantages of relying on internal assessments of your organization’s performance? The disadvantages?

Class 2: Laying out the Model Jan. 28

We will use this class to begin developing our basic framework for thinking about data and other forms of evidence.

Reading: Our Piece of the Pie: From Data to Decision-Making:

http://www.bridgespan.org/getdoc/a0e779be-3725-44de-b330-2e92f9978047/Our-Piece-of-the-Pie-From-Data-to-Decision-Mak.aspx#.VIXBTsmAMmQ

Jeri Eckhart-Queenan and Matt Forti, “Measurement as Learning: What Nonprofit CEOs, Board Members, and Philanthropists Need to Know to Keep Improving,” Bridgespan http://www.bridgespan.org/Publications-and-Tools/Performance-Measurement/Measurement-as-Learning-What-Nonprofit-CEOs,-Board.aspx#.VkoHQL-VnhU

Demographics of Hartford: Hartford, Connecticut is an old New England industrial town between Boston and New York City that over the last several decades has become the “headquarters” for many major insurance firms. The city’s population of 125,000 is 43% Hispanic, 35% black, 16% white and 6% “other.”

MLD-610 Page 3

MLD-610

Three-quarters of the Hispanics were born in Puerto Rico, which is a territory of the United States. One-third of the adults have less than a high school education and another 30 percent have a high school education only. Only 14% are college graduates. About half the households have an annual income of less than $25,000. The United States government sets the poverty level for a family of four at $23,850, which means that Hartford is a poor city. The city is surrounded by higher income suburbs.

Situation: Many organizations are focused on strategies help youth navigate, particularly youth who are poor and living in low income neighborhoods. Our Piece of the Pie is an organization that has changed its focus from one of general support for everyone living in poor neighborhoods in Hartford to focusing solely on adolescents and their passage to adulthood. The issues they face include how to keep youth on track, develop education and workforce skills, deal effectively with personal and family challenges, and avoid delinquent or criminal activities.

Challenge: We are only now beginning to learn about adolescent brain development and its implications for education and labor force policies. Our Piece of the Pie, struggling with its mission to support these young people, is trying to figure out what works for adolescents in Hartford and how their organization can make sure that it is always performing at its best. We will use their organization and efforts to lay out a framework for how you can lead your organization to better generate and produce evidence about its effectiveness.

Study questions – Our Piece of the Pie: We will cover these questions over two class sessions, but thinking about how you would answer them all is useful in preparing for the first class discussion. How effective a program do you think Our Piece of the Pie is now? What are the criteria by which you made your decision about its effectiveness? Along what dimensions do you think effectiveness should be measured?

1. What should the relationship be between an organization’s theory of change and performance measurement and between performance measurement and program evaluation?

2. What are the challenges to collecting, analyzing and presenting data in ways that inform decisions from the front line to the CEO? To what extent and in what ways does collecting data and analyzing performance enhance and detract from pursuing of a service provider’s mission?

3. What does the experience of Our Piece of the Pie suggest for how you might write contracts? What performance measures you might request? What data you might provide and what other supports might be useful?

4. Should contract renewals be based on performance? If so, what measures of performance do you care most about? How might an organization like Our Piece of the Pie “game” your assessment system?

Class 3: Laying out the Model -- continued Feb. 2

See materials for class 2. We will continue the discussion of Our Piece of the Pie, focusing in particular on the following questions.

1. What strategies did Our Piece of the Pie employ to try to change the culture of the organization around the generation and use of data? What are the particular challenges they faced in trying to do this?

2. What are the general challenges organizations are likely to face when they try to change the culture of an organization to become one more focused on the types of performance measurement Our Piece of the Pie focused on?

MLD-610 Page 4

MLD-610

Class 4: Clarifying Question Zero: In-Depth Interviews Feb. 4

Many organizations operate under complicated mission statements or legislation, often added to in response to emerging issues or crises but never really thought through. This is why clarifying what an organizations Question Zero is is so important. For what do they want to be – or should be – held accountable? Is there agreement on this?

One way to begin to get an understanding of what an organization’s Question Zero is and the extent to which it is shared among top management is to conduct a series of in-depth interviews. In this class we will spend time thinking about how you might phrase the questions to get a sense of what your respondent thinks the goal of the organization is and

Readings: Measured Approach: TEGV Assesses its Performance and Impact Weiss, Robert S., Learning from Strangers: The Art and Method of Qualitative Interview

Studies. New York: Free Press, 1995. Chapter 4: Interviewing (pp. 61-119)

Study Questions:1. What is TEGV’s Question Zero? Does it have more than one Question Zero? If so, what are

the other Questions Zero?2. What are the implications of having more than one Question Zero? It might be useful to

think about this question from the perspective of top management, front-line management, and those who used the services of your organization. It might also be useful to think about this question from the perspective of a broader set of stakeholders.

3. How would you go about figuring out what various people in your organization think your Question Zero is? Or your Questions Zero are?

Note: We will spend some time in the first three classes on identifying and selecting a group project. By class 4 you should have organized your group and selected a project. All assignments will relate to this project.

Class 5: Building a Theory of Change Feb. 9

Most organizations or units within organizations do not take the time to think through their theory of change. Doing so turns out to be hard work – but also very important. We will use this class to revisit TEGV and develop a strategy for thinking about how to identify a theory of change. You should leave the class with an idea of how you want to begin gaining an understanding of the theory of change for the group you are working for.

Readings: Measured Approach: TEGV Assesses its Performance and Impact Weiss, Carol Hirschon, “Nothing as Practical as Good Theory: Exploring Theory-Based

Evaluation for Comprehensive Community Initiatives for Children and Families,” New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James P. Connell, et al, The Aspen Institute, 1995

Questions:

MLD-610 Page 5

MLD-610

1. What is the theory of change for TEGV? Does it have more than one theory of change? If so, what are the additional theories of change?

2. What strategies might you use to develop a good understanding of the theory of change for an organization?

3. At the end of this class, we will select a topic for the next class on focus groups. Your assignment for that class will be to develop a set of focus group questions.

Class 6: Focus Groups Feb. 11

Sometimes it is most productive to interview a small group of people together so that they build on one another’s comments and push one another to think through the specific issue of concern to you. If you want to run a focus group, how do you select the group members? How do you structure the conversation? What do you do with the information they provide?

Readings:

Understanding Focus Groups with Author Thomas Greenbaum, http://www.i5publishing.com/understanding-focus-groups-with-author-thomas-l-greenbaum/

Thomas Greenbaum, 10 Tips for Running Successful Focus Groups, http://www.groupsplus.com/pages/mn091498.htm

U. S. AID Center for Development Information and Evaluation, Performance Monitoring and Evaluation TIPS: Conducting Focus Group Interviews

Questions:1. What are the advantages and disadvantages of focus groups as a strategy for gathering

information about the effectiveness of an organization’s operations?2. Based on the topic selected in our last class, write a set of focus group questions. In doing

so, think about your opening questions, your strategies for getting focus group participants to build on one another’s comments, how you will switch from question to question, how you will conclude the discussion, and how you would take notes on the discussion if this were part of an ongoing project.

Class 7: Sampling in Qualitative Interviewing Feb. 16

We hear a lot about random sampling, but sometimes random samples of respondents are difficult, if not impossible, to obtain. We only need to think about the demise of land lines and spread of cell phones to realize that it may be impossible to realize the “statistical ideal.” This is why major telephone pollsters like Pew are focused on getting representative samples. We will talk about different sampling strategies in class.

Readings: Kuzel, Anton J. “Sampling in Qualitative Inquiry,” Doing Qualitative Research. Crabtree,

Benjamin F. and William L. Miller, eds. Newbury Park, CA: Sage Publications, 1992. pp. 31-45.

MLD-610 Page 6

MLD-610

Sanford Labovitz and Robert Hagedorn, “The Population and Sampling,” Fist-fights in the Kitchen: Manners and Methods in Social Research. 1975, pp. 87-93.

Questions:1. What is your strategy for sampling individuals to be interviewed on your project? How does

your sampling strategy vary with the type of information you want to gather?2. What is your strategy for selecting focus group respondents for your project? How might

this sampling plan differ from that for gathering information from individuals?

Class 8: Testing a Theory of Change Feb. 18

An organization’s theory of change is important. Most organizations have not thought carefully about their theory of change so don’t know if they are explicitly following one? More important, they don’t know if their theory of change is “right.”

Readings: Scared Straight: Freeport City Council Takes On Juvenile Delinquency: Part A Parts B and C will be handed out in class Ron Haskins and Greg Margolis, Show Me the Evidence: Obama’s Fight for Rigor and

Results in Social Policy. Washington, D.C.: Brookings, 2015. Chapter 1:Chapter 1: The Obama Strategy for Attacking Social Problems, pp 1-30. (Note: This chapter was written by Ron Haskins, a Republican, on the Obama strategy for funding evidence-based interventions.)

If you want to get a better understanding of the Scared Straight Program, you may want to watch part of all of this video: http://www.bing.com/videos/search?q=scared+straight+programs&FORM=VIRE2#view=detail&mid=014C3B8036A0D9FD4931014C3B8036A0D9FD4931

Questions: What is the Question Zero for the Scared Straight Program? What is the Theory of Change for the Scared Straight Program? If you were on the city council, how would you vote? What are the ways you can assess an organization’s theory of change?

Class 9: Logic Models Feb. 23

Our theories of change should lead us into the development of our logic models. And our logic models will feed into our performance management system, so we need to “get it right.”

Readings: Measured Approach: TEGV Assesses its Performance and Impact Carolina Buiitrago, “Framing Program Evaluation: Tinkering with Theories of Change and

Logic Models,” November, 2015. W. K. Kellogg Foundation, Logic Model Development Guide. (This is somewhat long, so you

should feel free to skim it for class. But it will be a very useful “how to” guide for your project.)

Questions:

MLD-610 Page 7

MLD-610

1. What is the logic model for TEGV?2. Does TEGV have two logic models? One for recruiting and sustaining volunteers and

another for its after-school initiatives?

Class 10: Implementing a Program: Structuring Implementation Feb. 25

to Gather Evidence on Theory of Change

Getting a program or initiative up and running is normally a great deal of work. But when you can be systematic about implementation, you can use it to your advantage to test your theory of change and identify the systematic challenges in implementation. This may allow you to make changes along the way to improve the program’s overall effectiveness. One only needs to think about the roll-out of federal health care reform to understand the benefits of taking advantage of a roll-out to ensure that your program will operate effectively.

Readings: Deworming Kenya Note: a readying may be added depending on the issues raised in class to date and the

issues emerging from your work on your projects

Questions:1. What are the pieces of evidence on which the Deworming Kenya program is based?

How strong is this evidence? Note that there is evidence from a number of different sources and fields.

2. Although the proposed intervention is quite simple as far as interventions go, actually getting the initiative up and running is very challenging. What are the challenges and what types of resources do the protagonists need to muster to implement the program?

3. Those implementing the intervention used some of the challenges to design their implementation in a way that would generate even more evidence about the program’s effectiveness. How did they do this? What lessons do you draw from their experiences?

Class 11: Process Evaluation Mar. 1

A program with a robust theory of change will not be effective if it is not implemented with fidelity. We often start our implementation or process analysis by mapping – mapping how clients will flow through the program, how information will flow through the program and how funds will flow through the program. Sometimes this process is quite straight-forward and sometimes it is complex. In the case of the Jamaica PATH Program, it is complex.

Readings: Jamaica PATH case Peter Rossi, et al, Evaluation: A Systematic Approach, Seventh Edition, SAGE Publications,

2004. Chapter 6: Assessing and Monitoring Program Process, pp. 169-201Questions:

MLD-610 Page 8

MLD-610

1. The Jamaica PATH Program has a lot of moving parts. It is dependent on a lot of different people who report through different chains of command doing their part. Try to sketch out how clients, information and money flow through the process.

2. Where do you anticipate this information to flow smoothly? Where do you think the greatest challenges to accurate and efficient movement of data are likely to be? Why? What problems do you anticipate?

3. What do your responses to questions 1 and 2 suggest for how you want to organize your process/implementation assessment?

Class 12: Process Evaluation: Using Administrative Data Mar. 3

Administrative data that are collected routinely in administering a program are very useful for assessing program operations. But these data are not without their problems. First, organizations have many different reasons for collecting data, some of which are dictated by outside organizations, confidentiality restrictions, agreements with other agencies, or historic practice. Second, those capturing and recording these data as part of their job are often overworked and, in some cases, under trained. This sometimes results in inaccurate or incomplete data collection. Often only those specific pieces of data necessary for eligibility determination are reliably captured. Thus, there are many challenges to using administrative data to assess program performance.

Readings: Jamaica PATH case Joseph Wholey, et al, Handbook of Practical Program Evaluation. San Francisco: Josey-Bass

Publishers, 1994. Chapter 16: “Collecting Data from Agency Records, pp. 374-385. David Pitt, Using Historical Sources in Anthropology and Sociology. New York: Holt, Rinehart

and Winston, Inc., 1972. Chapter 4: The Critical Analysis of Documentary Evidence, pp. 46-62.

Questions:1. As you think about your assessment of the implementation of the Jamaica PATH Program,

what administrative data would you want to collect and routinely review? What questions do you want these data to answer?

2. Where do you think you will be able to obtain the most reliable data? The least reliable data?

3. If you wanted to assess the reliability of the data collected, how might you do this? What would you be looking for?

Class 13: Process Evaluation: Observation/Participant Observation Mar. 8

As that famous social scientist, Yogi Berra, once said, “You can observe a lot by just watching.” This insight is very useful for process/implementation evaluation. There are some processes that you can most accurately assess by observing them happen. And, there is nothing quite like being on site to get an understanding of what it feels like to participate in a program. But there is nothing casual about unobtrusive observation or participant observation. You need to be clear about your questions and you need to think hard about sampling.

Readings: Jamaica PATH case

MLD-610 Page 9

MLD-610

Eugene Webb, et al, Unobtrusive Measures. Thousand Oaks, CA: SAGE Publications, 2000. Chapter 5: “Simple Observation,” pp. 113-126.

Family Health International, Qualitative Research Methods: A Data Collector’s Field Guide. Module 2: Participant Observation, pp. 14-28.

Questions:1. What questions about the implementation of the Jamaica PATH Program could best be

answered – or only be answered – by observing day-to-day activities?2. Given your response to question 1, in what ways would observation and participant

observation lead to insights you might not gain through the use of administrative data, interviews or focus groups?

3. How would you structure your observation? Your participant observation? When would you go to each site? How long would you stay? What would you do? What data/information would you be able to collect?

Class 14: Measuring Program Outcomes Mar. 10 If your theory of change is correct and you have implemented your program with fidelity, you should expect to produce your expected outcomes. But, since there is never enough time and there are never enough resources, you need to be parsimonious in selecting the outcomes to measure and in measuring them.

Readings: Measured Approach: TEGV Assesses its Performance and Impact Peter Rossi, et al, Evaluation: A Systematic Approach, Seventh Edition, SAGE Publications,

2004. Chapter 7: Assessing and Monitoring Program Outcomes, pp. 203-232.

Questions:1. Think back to our discussion of TEGV’s Theory of Change. What outcomes should TEGV be

measuring? This would include short term and long-term outcomes.2. Since TEGV offers students a number of different programs, how should they think about

the effectiveness of any individual afterschool program? The combination of afterschool activities in which students are involved?

3. What time period should they consider?4. What strategies would you consider using to measure these outcomes?

Spring Break – No Class Mar. 15

Spring Break – No Class Mar. 17

Class 15: Gathering Data on Outcomes: Developing Surveys Mar. 22

Often the most reliable way to get information from individuals who participated in an activity is asking them directly. We have all completed surveys – or refused to complete surveys. Sometimes the surveys seem thorough and make us think. Other times they are sloppy, incomplete, or have a clear political agenda. In this class we will discuss the basics of writing a good survey.

MLD-610 Page 10

MLD-610

Readings: Floyd Fowler, Jr., Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA:

SAGE Publications, 1995. Chapter 4: Some General Rules for Designing Good Survey Instruments, pp. 78-103.

Joseph Wholey, et al, Handbook of Practical Program Evaluation. San Francisco: Josey-Bass Publishers, 1994. Chapter 11: Designing and Conducting Surveys, Thomas I. Miller, pp. 271-292

John D. McCarthy and M. Barbara McCarthy, “Power and Purpose in Survey Research (If You Got the Money, Honey, I Got the Time), in George H. Lewis, ed., Fist-fights in the Kitchen: Manners and Methods in Social Research. Santa Monica, CA: Goodyear Publishing Company, Inc., 1975.

Questions:1. You should be thinking about the information you want to gather through a survey for your

group project. Come to class with a list of the general areas of information you would like to address and specific pieces of information you would like to gather.

2. One important set of categories to consider are the following: What background behavioral information do you want to gather? What attitudes or orientations would you like to measure? Do you want to ask questions about anticipated future behavior?

Class 16: Gathering Data on Outcomes: Using Surveys Mar. 24

Survey research, particularly rapid-turnaround political polling by telephone, is becoming increasingly difficult to connect. And while the web provides opportunities to reach out to a wide audience, it too has challenges.

Readings:• Mick P. Couper, “Web Surveys: A Review of Issues and Approaches,” Public Opinion

Quarterly, Winter 2000 (Vol. 64, No. 4), pp. 464-494.• Pew Research Center, “Advances in Telephone Survey Sampling: Balancing Efficiency and

Coverage Using Several New Approaches,” http://www.pewresearch.org/2015/11/18/advances-in-telephone-survey-sampling/

• Check the Pew site for the most updated information on new strategies for insuring valid sampling and coverage in surveys: http://www.pewresearch.org/methods/

• For an example of a web-based survey that produced data that had previously been unavailable, you may want to skim the following PAE: Helena Pylvainen, “Women Veterans Transitioning to Civilian Life: Assessing Needs and Priorities.” On canvas web site.

Questions:1. In our last class we discussed various aspects of survey construction and developed a sense

of what constitutes a high quality survey. But we should also be worried about who answers our survey questions and who does not. What are the key questions we should ask about the sample of individuals or groups who receive a survey? About the response rate?

2. What are some of the strategies you might employ to improve your sampling strategy? Your response rate?

MLD-610 Page 11

MLD-610

Class 17: Changing Organizational Practice Culture with Data Mar. 29

Measuring performance is useful, but what do you do with the data when you have it?

Readings:• Our Piece of the Pie• Robert Behn, “Why Measure Performance? Different Purposes Require Different

Measures,” Public Administration Review, Vol. 63, No. 5, September/October 2004

Questions:1. How would you describe the culture of Our Piece of the Pie? What are the likely

characteristics of the people who work there? What is their training likely to have been? What aspects of their performance are particularly highly valued?

2. How did Our Piece of the Pie want to use data? What were the questions they wanted to address?

3. What was the management strategy Our Piece of the Pie used to begin thinking about using data

Class 18: Performance Management: Measuring Outcomes and Mapping Back Mar. 31

Readings: Case: NYPD New Robert Behn, The PerformanceStat Potential: A Leadership Strategy for Producing Results,

Washington, D.C.: The Brookings Institution Press, 2014, Chapter 19, “Analyzing and Learning from the Data,” pp. 145-171.

Questions: When Bratton became commissioner, what was the public perception of the major

problems facing the Police Department? Bratton took a new approach to assessing the problems facing the NYPD. How did he go

about identifying the internal operating challenges? What did he find? How did he go about identifying the crime challenges? What did he find?

What lessons can we take from this case for managers interested in developing a strategy for gathering and using data to assess and change organizational performance?

How did Bratton use the data to change the culture of the NYPD? What are the potential pitfalls of Bratton’s strategy?

Class 19: Changing the Organizational Culture around Data April 5

It’s one thing to have the top management team or a few managers focus on data and performance. But how do you change the culture of your entire data so that everyone begins thinking differently about how to assess current processes and performance and take on responsibility for trying new strategies and measuring their effectiveness?

Readings:

MLD-610 Page 12

MLD-610

Cincinnati Children’s Hospital Amy Edmondson, “Strategies for Learning from Failure,” Harvard Business Review, April

2011, pp. 49-55. Edgar Schein, Organizational Culture and Leadership, 1992. Chapter 1:Defining

Organizational Culture and Chapter 2:Uncovering the levels of culture, pp 1-27.

Questions:1. Uma Kotagal and her team are trying to get physicians, nurses and other health care

providers to both use evidence-based techniques and generate evidence about what leads to and supports higher quality health care. In what way is the first part of this challenge – getting front line health care providers and their managers to use evidence based techniques particularly difficult in health care? In what was is this challenge similar to and different from the challenge of getting front line staff and their managers to use evidence-based practices in other fields?

2. The Cincinnati Children’s Hospital team developed a “theory of change” for improving the quality of care it provided and the quality of its overall operation. How would you describe this theory of change? What were its strengths in this particular situation? Its weaknesses?

3. What were the key domains of resistance to change? How did the Hospital’s Leadership and the Quality Improvement team address this resistance? Do you think they were effective? What would you have done differently?

4. Changing the culture of an organization takes both leadership and followership. What does the Cincinnati Children’s Hospital case tell us about both leaders and followers? What does it suggest leaders consider as they initiate and try to sustain the type of cultural change that totally infuses and organization?

5. What enables Cincinnati Children’s Hospital to sustain the change? If you were Jim Anderson’s replacement, what would you be concerned about?

Class 20: Management Strategies for Sharing Data to Solve Problems April 7

Often as part of the process of accomplishing our mission, we need to share information with other organizations about an individual, a family, a program or a situation. This turns out to be extremely difficult for a number of reasons.

Readings: Wraparound Milwaukee case Stephen Goldsmith, et al, The Power of Social Innovation: How Civic Entrepreneurs Ignite

Community Networks for Good, San Francisco: Jossey-Bass, 2010. Chapter 6: Turning Risk into Reward. (On Canvas website)

Questions:1. What is the Wraparound Milwaukee model? What is its goal? How does it work? 2. A key factor in the Wraparound Milwaukee model is the capacity to share data about one

child and his or her family across a range of service providers. Why is this advantageous? What are the concerns about sharing data of this sort?

MLD-610 Page 13

MLD-610

3. What lessons could other organizations draw from Wraparound Milwaukee about strategies for sharing data or information on specific individuals or situations?

Class 21: Generating and Using Data across Organizational Boundaries: April 12

Collective Efficacy

Most of the issues our organizations are trying to address are complex. Often this means that we are focused on only a small part of the overall issue or problem and are contributing a small, though often important, part to the overall solution of the problem. How might we gather and use evidence to show that we are

Readings:• Strive Together: Reinventing the Local Education Ecosystem• John Kania, et al, “Strategic Philanthropy for a Complex World,” Stanford Social Innovation

Review, Summer 2014, pp 26-37• John Kania, et al, “Embracing Emergence: How Collective Impact Addresses Complexity,

Stanford Social Innovation Review, 2013, pp. 1-7

Questions:1. STRIVE is structured as a “backbone” organization. What does that suggest for the roles it

plays? What responsibilities does it have to the other organizations contributing to the educational ecosystem?

2. How effective was STRIVE?3. How did they use evidence to assess effectiveness? To set goals? To encourage behavior on

the part of others?

Class 22: Ethical and Legal Issues around Gathering, Sharing and Using Data April 14

Institutional review boards are important, though they have a reputation in some circles for making research difficult. How do you know when you should be getting institutional review board approval for your work?

Readings:• Each student should complete the CITI on-line ethical training for the IRB if you have not

already done so. Once you complete this, you will be certified for three years.• Mary Roach, Stiff: The Curious Lives of Human Cadavers. New York: W. W. Norton &

Company, 2003. “Crimes of Anatomy,” pp. 39-57.• Herbert Kelman, “The Rights of the Subject in Social Research: An Analysis in Terms of

Relative Power and Legitimacy,” The American Psychologist, November 1972, pp. 989 – 1005.

Questions:1. As a manager, what are some of the guidelines you should abide by in collecting data on the

performance of your organization’s employees? Those your organization serves?2. What do these guidelines suggest for strategies about sharing information inside your

organization? With other organizations?

MLD-610 Page 14

MLD-610

3. What do these guidelines suggest about restrictions you might place on outside researchers who want access to data your organization generates?

Class 23: What do we mean by Evidence? April 19

We began the course by laying out a model for evidence and spent our time figuring out how we as managers would use that model to better understand and improve our organization’s performance as well as change the culture of our organization around the generation and use of data for management. Each of you has applied the model to a specific program or organizational unit, which has given you a chance to practice using the techniques and skills the course covered. This is our chance as a class to assess the model.

Class 24: Presentations April 21

Class 25: Presentations April 26

Class 26: Wrap Up April 28

MLD-610 Page 15