Evaluation in support of Strategic Learning

Principles of Practice

Tanya BeerCenter for Evaluation Innovation

Strategic Learning ≠ Evaluation

Strategic learning is the use of data and insights from a variety of information-gathering approaches—including evaluation—to inform decisions about strategy.

What counts as “strategy” and how is it created, executed, and adapted?

Challenge 1

Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.

What kind of data and intelligence are meaningful for informing strategic decisions?

Challenge 2

Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.

Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.

How do organizational and group culture, structures, and process support or hinder reflection and the use of data?

Challenge 3

Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.

What counts as “strategy” and how is it created, executed, and adapted

What kind of data and intelligence are meaningful for informing strategic decisions?

How do organizational and group culture, structures, and process support or hinder reflection and the use of data?

How do we design and practice evaluation to support strategic learning?

Principles of Evaluation For Strategic Learning

PRINCIPLE 1: Evaluation is positioned as a support for strategy and is seated at the strategy table

PRINCIPLE 2: Evaluation is integrated into initiative development and management and conducted in partnership.

Understanding Strategy

PRINCIPLE 3: Data to inform strategy can come from a wide variety of sources and methods.

PRINCIPLE 4: Evaluation emphasizes context.

PRINCIPLE 5: Evaluation is flexible and timely, and ready for the unexpected.

Principles of Evaluation For Strategic LearningCollecting meaningful data at the right time

Principles of Evaluation For Strategic Learning

PRINCIPLE 6: Program implementers place a high value on evaluation use, and evaluators help to support it.

PRINCIPLE 7: Evaluation must take place within a culture that encourages risk taking, learning, and adaptation.

PRINCIPLE 8: Evaluation is constructivist.

Understanding the drivers of data use

“You don’t plan a strategy, you learn a strategy.”Henry Mintzberg

What will it take for us to make use of data and other intelligence in our strategic decision-making

throughout the lifecycle of a strategy or program?

Tell a story about a time when your team either failed or succeeded at incorporating data or other intelligence into a strategic decision.

Rules: Talk about actual past situations, events, or results. No abstract concepts, interpretations or editorializing,

just what happened!! Tell stories about both successes and failures Listen carefully for (but don’t comment on) similarities,

differences between stories.

20 minutes to tell at least 4 stories.

Groundtruths

Insights

What patterns do you see in the stories?

What insights can we generate about the relationship between past results and what caused them?

Rules: Link insights clearly to groundtruth data (stories)--no ungrounded

assertions!

One person writes insights on a sheet of paper, and another records key insights on stickies by color:

25 minutes to generate insights and record them on stickies

CULTURE/PROCESS

DATA/EVALUTION

STRATEGY/PROGRAMS

OTHER

BREAK 10 minutes

Bring your stickies to us BEFORE you break!

Beginning at 11am, you will have 10 minutes to browse the gallery wall of insights. Make note of

the most compelling ones!

Quickly review the insights you listed and add any others you noticed from the gallery wall.

Given these insights, what hypotheses do you have about what it’s going to take to make better use of data and other intelligence in our strategic decision-making throughout the lifecycle of a strategy or program?

Rules: Use this structure: “In [situation X], if we do [action Y], then

we will produce [result Z].” Avoid vague statements (e.g., “build trust”) and push each

other to instead offer specific actions. One person records hypotheses and the group picks top 3.

20 minutes to generate hypotheses and pick top 3 most compelling.

Hypotheses

Give us your best hypotheses

Most doableMost impactfulMost creative

What will it take for us to make use of data and other intelligence in our strategic decision-making

throughout the lifecycle of a strategy or program?

IN PAIRS, describe upcoming opportunities in your work where you will test 2 or 3 hypotheses.

What is the actual meeting or event where a decision will be made?

What kind of decision(s) will be on the table? What kind of data will the group need to consider? How will you test a hypotheses to improve the group’s

application of the data to the decision?

Rules: Opportunities must be real things on your calendar.Start with what exists already and add new events only when

absolutely necessary.Help your partner think through how to make it happen.

Opportunities

So what was that like?

Act

Observe

ReflectAdjust

Plan

We are pretty good at this

THE WEAK LINK

If we take time, we can do well at this,

too

Action Learning Cycle

From Fourth Quadrant Partners, LLC

Emergent Learning™ Table

Framing Question: What will it take to…? How can we…?

Ground Truth

Insights

Opportunities

Hypotheses

Translate past lessons into a

theory of success

Recallrelevant

experiences and examine data

Learn from experience &

data

Apply new theory to

opportunities

From Fourth Quadrant Partners, LLC

PAST FUTURE

DOING

THINKING

“The capacity to adapt is higher level and of greater value than any specific ‘lesson learned’.”

“Learning in the Thick of It,” HBR

Let’s Eat!

Teams of 4-6

18 minutes

Tallest structure

GOAL: Build the tallest freestanding structure (measured from the table top surface to the top of the marshmallow)

✦The ENTIRE MARSHMALLOW MUST BE ON TOP (no eating it)

✦Use as much or as little of the kit (spaghetti, tape, string, marshmallow

✦You cannot use the paper bag as part of your structure.

✦You can break the spaghetti, rip the tape and string to create new structures.

✦ You cannot hold on to the structure when the time runs out. Those touching or supporting the structure at the end of the exercise will be disqualified.

18 minutes

Where Cognitive Traps Trip Us Up:Removing the Road Blocks to Learning

A bat and ball cost a dollar and ten cents.

The bat costs a dollar more than the ball.

How much does the ball cost?

Source: Shane Frederick

Ball = Bat = 10 cents One dollar

Ball =

5 cents

Bat = + One dollar

5 cents

Why are we talking about this?

You deal with complexity and uncertainty.

How can we help metropolitan regions shape the opportunities available to low-income families for access to good quality jobs

and services, and for development of their assets, particularly quality housing?

How can we reverse the childhood

obesity epidemic by

2015?

How can we achieve high-quality preschool for all 3- and 4-year-olds in our

state??

And you try to reduce or manage it (I said try).

31

Strategies and theories of change help to manage

complexity and uncertainty, but they

tend to be more descriptive than

prescriptive.

32

So evaluation could be enlisted to help.

How can we take advantage of the evolving

political climate?

What is motivating the actors and how do we

reinforce desired behaviors?

How should assumptions about the true cost of

implementation be adjusted?

But unfortunately, we are not rational.

We are hardwired to use heuristics and biases— cognitive traps—when processing information.

They affect:

How we access information

What we pay attention to

What we learn

How we apply it

We need to finish the equation.

Ask the

right questions and get the right

data

Structure the work to enable

regular use of data

+

Effectively process and use the data

+

What are cognitive traps?

36

System 1: Rapid intuitive decisions based on associative memory, images, and emotional reactions

System 2: Monitors the output of System 1 and overrides it when the result conflicts with logic, probability, or some other decision-making rule

Our minds have two processing systems.

The two systems act differently.

System 1 System 2

System 2 is lazy and trips us up, especially when we are busy.

There are all sorts of cognitive traps.

We are focusing on a few relevant to philanthropy and evaluation.

Decision Making

• Anchoring/adjustment

• Conjunction fallacy

• Focusing effect

• Overconfidence effect

• Curse of knowledge

• Ease of recall

• Planning fallacy

Social

• Actor-observer bias

• False consensus effect

• Fundamental attribution error

• Halo effect

• In-group bias

• Projection bias

• Self-serving bias

Memory

• Primacy effect

• Recency effect

• Serial position effect

• Google effect

• Hindsight bias

• Illusory correlation

• Egocentric bias

The availability heuristic makes us overestimate what comes to mind the quickest.

Probability of a real threat?

Confirmation bias makes us favor information that confirms our pre-existing beliefs.

State polls showing consistent leads in key battleground states

Late-October national polls showing gains

Hurricane Sandy performance

More engaged and motivated voters

Large and enthusiastic crowds in swing states

Beliefs that national polls were skewed.

Internal polls showing leads in key states.

Nate Silver’s “Model” of all presidential polling

Escalation of commitment makes it difficult to revisit previous choices.

$2000

Tires

$300

Alternator$220

Water Pump

$150

The Texas sharpshooter fallacy causes us to mistake coincidence for meaningful pattern.

Use devil’s advocacy.

Which consulting firm is better?

Yes Man

All your ideas are

great!

Devil’s Advocate

Here are 10 reasons you

shouldn’t do that.

Remind yourself what you don’t know.

1. Tell me what you know.

2. Tell me what you don’t know.

3. Only then can you tell me what you think.

--Colin Powell

Compare and

contrast across silos.

45

Reason analogically.

Health Education Environment

Get an outsider’s view.

The outsider is more capable of

generalizing across situations, and makes better

estimates and decisions than

the insider.

Outsiders Insiders

47

Let’s put it all together and think about…

…the team meeting.

48

Who is at the table?

Devil’s Advocate

Outsider

49

What are you talking and thinking about?

Wait. What don’t we

know here?

Didn’t our other program area experience a

similar problem?

I disagree. I should probably

speak up.

I want to take over this meeting! But I must control

myself….

50

Flip-Chart Go-Round: Sidestepping Cognitive Traps

Chart 1: One person read aloud the definition of the cognitive trap.

In what kinds of situations or for what kinds of decisions are organizations are particularly vulnerable to the trap?

Chart 2: One person read aloud the definition of the cognitive trap AND the previous team’s responses to the question in round 1.

What concrete things can programmatic staff and leaders do to help sidestep this trap?

Chart 3: One person read aloud the definition of the cognitive trap AND the previous team’s responses to the question in round 1 & 2.

What concrete things can evaluators do to help sidestep this trap?

51

Flip-Chart Go-Round: Sidestepping Cognitive Traps

Chart 1: 10 minutes

1) One person read aloud the definition of the cognitive trap.

2) Discuss:

In what kinds of situations or for what kinds of decisions are organizations are particularly vulnerable to the trap?

3) Record your thoughts on a flipchart as concisely (and legibly) as possible.

52

Flip-Chart Go-Round: Sidestepping Cognitive Traps

Chart 2: 15 minutes

1) One person read aloud the definition of the cognitive trap AND the first group’s answers to the question in the last round.

2) Add additional insights to the first group’s list of types of situations or decisions where this trap is particularly relevant.

3) Discuss:

What concrete things can programmatic staff and leaders do to help sidestep this trap?

4) Record your thoughts on a flipchart as concisely (and legibly) as possible.

53

Flip-Chart Go-Round: Sidestepping Cognitive Traps

Chart 3: 20 minutes

1) One person read aloud the definition of the cognitive trap AND the previous group’s answers to the question in the last two rounds.

2) Add additional insights to the last group’s list of ideas about what concrete things programmatic staff and leaders can do to sidestep the trap.

3) Discuss:

What concrete things can evaluators do to help sidestep this trap?

4) BE READY TO REPORT OUT ON THE FULL CHART!

Where Cognitive Traps Trip Us Up:Removing the Road Blocks to Learning