turning data into action workshop june 26, 2014

Turning Data Into Action WorkshopJune 26, 2014

2 ©2014 U.S. Education Delivery Institute

Kentucky prioritized gap closing as a major focus for the upcoming academic year

Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.

(Associate degrees)Graduation rate gaps: Underrepresented MinoritiesGap between the graduation rate of African-American, Latino, American Indian and Hawaiian/Pacific Islander students and the graduation rate of White and Asian students .Data Source: CPE Comprehensive Database (KPEDS)

3

2009-10 2010-11 2011-12 2012-130.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

7.2% 7.3%7.7%

6.8%

13.0%14.1% 13.7% 13.5%

Underrepresented Minority

Not Underrep-resented Minor-ity


(Associate degrees)Graduation rate gaps: UnderpreparedGap between the graduation rate of students who did not meet statewide collegeReadiness benchmarks and those who did.Data Source: CPE Comprehensive Database (KPEDS)

4

2009-10 2010-11 2011-12 2012-130.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

7.5%

11.3%8.9% 8.6%

22.8%

26.3%23.7% 24.4%

UnderpreparedPrepared


(Associate degrees)Graduation rate gaps: Low Income Gap between the graduation rate of Pell grant recipients and non-recipients .Data Source: CPE Comprehensive Database (KPEDS)

5

2009-10 2010-11 2011-12 2012-130.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

18.0%

20.0%

10.8% 10.9% 11.7%10.4%

13.9%

16.7% 15.5%

18.2%

Low IncomeNot Low Income


(Bachelor degrees)Graduation rate gaps: Underrepresented MinoritiesGap between the graduation rate of African-American, Latino, American Indian and Hawaiian/Pacific Islander students and the graduation rate of White and Asian students (using the IPEDS six-year graduation rate). Data Source: CPE Comprehensive Database (KPEDS)

6

2009-10 2010-11 2011-12 2012-130.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

36.9% 36.0% 33.6% 33.3%

49.0% 49.7% 49.4% 51.0%

Underrepresented Minority

Non Underrep-resented Minor-ity


(Bachelor degrees)Graduation rate gaps: UnderpreparedGap between the graduation rate of students who did not meet statewide collegeReadiness benchmarks and those who did (using the IPEDS six-year graduation rate).Data Source: CPE Comprehensive Database (KPEDS)

7

2009-10 2010-11 2011-12 2012-130.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

28.0% 29.2% 28.6% 27.8%

54.7% 55.3% 54.5% 56.5%

UnderpreparedPrepared


(Bachelor degrees)Graduation rate gaps: Low Income Gap between the graduation rate of Pell grant recipients and non-recipients (using the IPEDS six-year graduation rate). Data Source: CPE Comprehensive Database (KPEDS)

8

2009-10 2010-11 2011-12 2012-130.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

46.2%

34.5% 34.5% 36.3%

56.7%52.9% 51.5% 52.0%

Low IncomeNot Low Income


Today, we will cover…

Turning Data Into Action Workshop Agenda

9:00 Welcome & Meeting Overview9:40 Understanding Our Capacity to Use

Data to Make Decisions on Campus10:45 Break11:00 Understanding Current Progress and

Identifying Institutional Factors11:45 Lunch12:30 Prioritizing and Focusing Our Efforts1:30 Identifying Relevant Metrics 2:15 Break2:30 Determining Who and How 3:30 Identifying Next Steps4:00 Meeting Close


The delivery approach produces results by focusing leaders on four fundamental questions

“delivery” (n.) is a systematic process through which leaders can drive progress and deliver results.

It involves asking the following questions consistently and rigorously:

1 What are we trying to do?

2 How are we planning to do it?

3 At any given moment, how will we know whether we are on track?

4 If not, what are we going to do about it?


EDI supports implementation of education reform in a number of states and systems

Current EDI Partner Systems K-12Higher EducationBoth

Our mission is to partner with K-12 and higher education systems with ambitious reform agendas and invest in their leaders' capacity to deliver results. By employing a proven approach, known as delivery, we help state leaders maintain the necessary focus to plan and drive reform.


Our Higher Education Delivery Network consists of a growing number of systems across the country

EDI focuses on supporting systems in their implementation efforts to: Increase the percent of students who graduate with a degree

on time Close equity gaps in postsecondary access Close equity gaps in postsecondary success

Current Higher Education Partnerships California State University System Colorado State University System University of Hawai’i System Kentucky Council on Postsecondary

Education Minnesota State Colleges and

Universities University of Missouri System

New Jersey Office of Higher Education Pennsylvania State System of Higher

Education Tennessee Board of Regents The University of Texas System University of Wisconsin System


We will use this workshop to focus on using data as an action strategy to close gaps

Assess campus capacity to use data to drive decision making Understand current progress on campus and in the Council, as it

relates to achievement gaps Discuss institutional factors contributing to outcomes Refine key campus initiatives intended to improve outcomes Identify metrics in order to monitor those initiatives Identify who will be involved in monitoring and making mid-

course corrections and how monitoring will occur


What is our

current status?

Why are we in this

position?

What are we doing about it?

How are we going

to monitor progress

?

To meet our objectives, we will tackle the gap challenges by answering four questions


There are simple steps you can take to make the most of the workshop!

Choose a team leader/facilitator

Suspend disbelief!

Ask yourself, “What can we do now, with what we have?”

Try not to get caught up in the minute details

Leverage the expertise at the table

Leverage the efforts underway at your campuses

LET’S GET STARTED!


We believe that these behaviors are necessary for campuses to use data effectively in decision making

EDI’s tenets for data useCampuses must:

Have clear targets against which to measure performance

Use timely data (both headline metrics and leading indicators) that are monitor progress on a regular basis

Disaggregate the data in order to isolate outcomes for specific groups of students

Establish routines that help teams come to a shared view of progress

Involve IR in relevant conversations and analyses to take advantage of existing expertise

Empower decision makers to seek and use relevant data to support students


As we begin to plan how we’ll use data to make decisions, we need to be clear about some terms

Description

Strategy A coherent set of initiatives that are designed to maximize impact on your target metrics

Initiative A specific change in the way a campus supports and educates its students

Metric An outcome used to measure progress on an initiative, strategy, or both

Term

Depending on what your team is prepared to discuss, you can choose to either focus on one discrete initiative or on a strategy comprised of multiple initiatives during this workshop.


Poll Everywhere: Data-informed Decisions and Your Campus

How well does your campus currently use data to drive/inform decision making?

We do it, and we do it well We do it well in a few areas across campus Analysis Paralysis – We have it, but we rarely move to action “Who needs data? We go with our gut!” Data? What data? This is a challenge area for our campus


Workshop Agenda

Understanding Our Capacity to Use Data to Make Decisions

Followed by a break


Our goal is to not only understand our status but to discuss why, and how we can ensure that we maximize our impact

Our graduation rate for African American students went up 4% this year!

Huh. Interesting.

Our graduation rate for African American students went up 4% this year!

That’s probably because the cohort of students that entered when we began the new multicultural office support program graduated this year.

Not sure though…we should find out how many students this program served and what the early indicators said.

Yep! Early indicators indicated that this program was improving retention and credits earned for African American students.

We should monitor and scale up that support program from now on, so we maximize the impact from this important strategy!


What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?

This workshop allows campuses to develop a plan to assess progress on key campus strategies by asking 4 questions

Teams will review their campus’ data to develop a shared understanding of student success


What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


Teams will discuss the institutional factors that need to be addressed or leveraged to improve student performance


What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


Teams will identify, clarify, and refine key campus strategies to address challenges


What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


Teams will develop a routine to regularly assess progress on selected strategies


What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?

How much capacity do we have on campus to

answer these

questions?

Before we can answer these questions, however, we must reflect on our current capacity to use data to drive action


EDI has created a rubric to help campuses think about their ability to use data to drive decisions


The self-assessment tool helps campuses understand current capacity to answer these questions

Rubric: Capacity to Use Data for Decision Making on CampusRubric category This category asks us to think about…

Access and efficiency

Monitoring

Problem solving

The ease and speed with which data can be accessed or retrieved from campus systems

Culture

Actions taken to understand progress on campus goals and metrics

The processes we use to turn what we learn from campus data into action around campus strategies

The attitudes and reactions of campus faculty and staff toward making decisions using campus data


Use the rubric to determine how you would rate your campus in each category

Category


Monitoring

Problem solving

Culture

Rating Rationale Summary

Capacity to use data to make

decision on campus

Red: Our campus must focus on improving in this area to use data effectively in decision makingOrange: Our campus must improve in this area, though in some aspects more than othersYellow: Our campus should continue in the aspects we find strong, but focus on weak pointsGreen: Our campus is successfully using our capacity in this area to use data to make decisions


Example: one campus realized that monitoring and problem solving needed improvement in order to use data effectively

Category


Monitoring

Problem solving

Culture

Rating Rationale SummaryUnified data system with IR support, but faculty and staff generally do not know how to make data requests or what questions to ask

Conversations around areas of improvement almost never focus on solutions; we do not have priority owners

Our strategic plan calls for regular monitoring, but meetings get cancelled; when they do happen, we don’t have helpful data available

We feel that faculty, staff, and campus leadership are generally confident in our data systems and want to use data to drive the work

Capacity to use data to make

decision on campus

Example of completed rubric with rationales

Red: Our campus must focus on improving in this area to use data effectively in decision makingOrange: Our campus must improve in this area, though in some aspects more than othersYellow: Our campus should continue in the aspects we find strong, but focus on weak pointsGreen: Our campus is successfully using our capacity in this area to use data to make decisions


Exercise: Capacity To Use Data on Campus Self-Assessment

▪Individually, read your rubric and consider what your “traffic light” rating would be in each of the four areas. Write your thoughts on the rubric.

▪As a team, discuss your ratings and determine where your campus is strongest, where it needs the most improvement, and why.

Questions to ask each other…What prevents us from being green?Do our ratings mean the same thing?Are we being too harsh? Too easy?


Your Role in Strengthening Capacity

How can you leverage your role to strengthen your campus’ capacity to use data to make

decisions?

Record your answer on a card Share your card with your team members Place card on brown paper


Workshop Agenda

Understanding Current Progress and Identifying Institutional Factors

Followed by Lunch

Break


First, we will determine the level of challenge we face in serving our students and then discuss contributing factors

What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


“Good data, while essential, is only a start—

you then have to use it!” – Instruction to Deliver


There are two steps to understand the drivers of performance and evaluate relevant campus strategies

Identify the drivers of your

student success patterns

Identify and evaluate campus

strategies related to drivers of

performance patterns

▪ Pinpoint the root causes, or drivers of performance patterns

▪ Identify “usual suspects” when possible

▪ Utilize formal problem-solving tools when drivers are not evident

Description Why important▪ A campus cannot know

where to focus unless it identifies drivers

▪ Creating new or revamping initiatives without understanding existing strategies wastes resources

▪ Identify what the campus is currently doing to try and address drivers of underperformance

▪ Evaluate whether current initiatives are sufficient and recommend changes where necessary

1

2


Traditionally, data show several factors that are likely to impact how groups of students progress and persist

Race & Ethnicity

Gender

Income or Pell Status

High School GPA

Academic Major

Courses Taken During First Year

Number of Credits Accrued at Various Points

Some of those factors include:


Florida State University used data to understand student success and which students were lost along the way

Six-Year Graduation Rates 2002-2010, Underrepresented Minorities and White Students

2002 2003 2004 2005 2006 2007 2008 2009 201050%

55%

60%

65%

70%

75%

80%

60.0

71.5

64.5

74.1

Underrepresented Minority White

Why were some students successful while others were not?

What could Florida State do to make a difference for them?

Source: Florida State University


In looking at attrition rates, the team considered a range of time points and student background characteristics

Yearly Attrition Rates by Cohort: White, Female, First-Time In-State Students

0.5%1.6%

0.9%3.3%

2.1%3.9%

6.7%9.8%

10.2%17.1%

MINMAX

1995-2005



Hispanic female Pell recipients exhibited very different attrition rate patterns, suggesting the need for continued support over time to avoid dropouts

Yearly Attrition Rates by Cohort: Hispanic, Female, Pell Recipient, First-Time, In-State

Students

6.3%17.6%

MINMAX

6.9%17.7%

-1.2%10.5%

0%4.1%

-1.4%3.9%


1995-2005


Black male Pell recipients exhibited different attrition rate patterns, suggesting the need for continued support over time to avoid dropouts

6.0%14.0%

MINMAX

3.3%15.3%

-0.9%13.6%

0%11.0%

0%11.1%


Yearly Attrition Rates by Cohort: Black, Male, Pell Recipient, First-Time In-State

Students1995-2005


A number of other institutional factors also create unclear pathways to graduation

▪ Insufficient academic guidance/planning

▪ Policies to pose as barriers

▪ Misunderstanding degree requirements

▪ Inappropriate preparation/enrollment in prerequisite courses

▪ General education requirement confusion

▪ Course withdrawal or failure

▪ Switching majors

▪ Inability to enroll in necessary coursesat the appropriate time

42

Graduation Rate in Major by Introductory Course

Grade

The Difference Between Satisfying a Requirement and Being on Track

STRENGTHENING ADVISEMENT


Let’s look at Kentucky Campuses Success Gap data to help review your campus’s student success patterns

Are our gaps shrinking? Growing?

What do we know about closing gaps in higher education attainment?

Colorado State University System | Kentucky Council of Postsecondary Education | Louisiana Board of Regents | Minnesota State Colleges and Universities | Mississippi Institutions of Higher Learning | Montana University System | New Jersey Higher Education | Pennsylvania State System of Higher Education | Rhode Island Board of Governors for Higher Education | State University of New York | State University System of Florida | Tennessee Board of Regents | The California State University System | University of Hawaii System | University of Louisiana System | University of Missouri System | University of North Carolina System | University of Texas System | University of Wisconsin System |

19 Systems, 300 Campuses, 3.5 Million Students18% of Public Undergrads | 866K URMs | 1 million Pell Recipients


Include all students, not just first-time, full-time

Include part-time and transfer students, many of whom are low-income and underrepresented minorities

Report retention and success rates for low-income and transfer students at institution and in system

A2S Metrics Count All Students


Are we increasing success rates for underrepresented students?

GAP: Are we increasing success rates for underrepresented students relative to peers?

Benchmarking Progress on Success


The data handouts answer

these and other questions

about student success on

your campus.

Benchmarking Progress on Success


What is the overall freshmen rate?

What is the rate for transfer students?

How have these rates changed over time?

Are we increasing overall graduation rates?

Change in the Freshmen graduation rate since 2005 + 12 pts + 11 ptsChange in the Transfer graduation rate since 2005

State University

Bachelor's Programs

SUCCESS: IS THE CAMPUS IMPROVING THE RATE AT WHICH UNDERREPRESENTED STUDENTS COMPLETE?

OVERALL GRADUATION RATES

Freshmen graduation rate in 2012 59% 77%Transfer graduation rate in 2012


Non-Pell graduation rate in 2012 (cohort)

60% (2,840)

Change in the Non-Pell graduation rate since 2005

+ 10 pts

Gap between Pell and Non-Pell graduation rates

2 ptsChange in Pell/Non-Pell graduation rate gap since 2005

Narrowed 5 pts

LOW-INCOME FRESHMEN

Pell graduation rate in 2012 (cohort)

58% (1,472)

Change in the Pell graduation rate since 2005

+ 15 pts

43% 44%

49% 50%

55%58%50%

53% 54% 56% 57%60%

35%

45%

55%

65%

1999/05 2002/08 2003/09 2004/10 2005/11 2006/12Pell Rate Non-Pell Rate

Are we increasing graduation rates for underrepresented students?

What is the grad rate of Pell freshmen?

What is the grad rate of non-Pell freshmen?

How do rates for these student groups differ?



60% (2,840)


+ 10 pts



Narrowed 5 pts

LOW-INCOME FRESHMEN


55% (1,472)


+ 15 pts

43% 44%

49% 50%

55%58%50%

53% 54% 56% 57%60%

35%

45%

55%

65%


How have graduation rates for Pell and non-Pell freshmen changed over time?




60% (2,840)


+ 10 pts



Narrowed 5 pts

LOW-INCOME FRESHMEN


55% (1,472)


+ 15 pts

43% 44%

49% 50%

55%58%50%

53% 54% 56% 57%60%

35%

45%

55%

65%


How has the Pell freshmen rate changed over time? How has the non-Pell freshmen rate changed? Has the graduation rate gap between these groups

narrowed or widened over time?



Discussion: What do your data tell you?

Review your campus’s progress using the Access to Success report card

We will then discuss the following: Where are you making the most

progress? Where is the largest improvement

needed? What other data do you have

and/or use to understand progress?

After reviewing your data, answer: What institutional factors

contribute to your current status?


Workshop Agenda

Prioritizing and Focusing Our Efforts

Lunch


We will determine which strategies are most likely to improve outcomes for our students

What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


In order to better serve your student group, teams should develop a concrete action plan

“A goal without a plan is just a wish.” -Anonymous


Thinking through a strategy profile helps define the strategy and expectations about its impact

Description of strategy

What is the strategy? What initiatives make it up?

Definition of success

If the strategy were successful, how will we know?

Theory of action

Why and how do we believe that the strategy will help us to achieve the desired outcome? Express as one or more “If… then…” statements.

RisksBased in the delivery chain, what are the potential risks associated with this strategy? How will you mitigate them?

MilestonesWhen will the strategy begin and when will it end, or become “business as usual?”

Leading indicators/ feedback loops

What interim metrics can you use to measure progress? How will you know that the actors along the delivery chain are doing the things they need to do?

Scale and efficacy

Scale: how many students will be affected by the strategy? Efficacy: of those affected, how many new students will achieve the outcome we desire?


We have already begun to address some of these questions today

Description of strategy

What is the strategy? What initiatives make it up?

Definition of success

If the strategy were successful, how will we know?

Theory of action

Why and how do we believe that the strategy will help us to achieve the desired outcome? Express as one or more “If… then…” statements.

RisksBased in the delivery chain, what are the potential risks associated with this strategy? How will you mitigate them?

MilestonesWhen will the strategy begin and when will it end, or become “business as usual?”

Leading indicators/ feedback loops

What interim metrics can you use to measure progress? How will you know that the actors along the delivery chain are doing the things they need to do?

Scale and efficacy

Scale: how many students will be affected by the strategy? Efficacy: of those affected, how many new students will achieve the outcome we desire?

The gap we intend to close and the students affected.

Focusing on key institutional factors will help close the gap.


We can narrow our focus from all initiatives to those that focus on the right factors affecting our students

Narrow the list to only the relevant initiatives

All campus

initiatives

Those that target the

right outcome

“We have a lot going on at our campuses!”

Graduation rates for underprepared students

Underprepared students are more likely to need support in registering for class and applying for financial aid

Those that target the

right factors

affecting our

students

Priority strategy


Once we understand the initiatives that comprise our strategy, we can clearly define success


For the next several exercises, we will use the strategy profile worksheet


Exercise: Defining Success

Describe your strategy with the objective/gap and target metric that you will focus on

− Example: This strategy will improve our success rates for our first-generation students by providing more individual support and expanding current programs that disproportionally affect this group of students.

Identify the initiatives included in your strategy and the students that you are counting on these initiatives to serve

Define what success looks like for each initiative by Spring 2015

− Example: We want this initiative to be implemented in all Natural Sciences departments by Fall 2015 and to see an increase in the retention of first-generation students in those departments by Spring 2015.

Identify which institutional factors these initiatives address

− Example: Our campus currently does not offer enough guidance to students who are unsure of the most important decisions required to plan for and earn an bachelor’s degree.


Workshop Agenda

Identifying Relevant Metrics

Followed by a break


Once we know our strategy, we can consider how we expect to impact our students and when

“Ultimately, it is up to your team and those responsible for implementing the intervention to make the best judgment they can—informed by as many facts as possible and backed by sound logic—of how specific interventions will work together to affect your target metric.”

—Taken from Deliverology 101


Determining expected impact is essential for monitoring progress

Consider the planned progression of your performance indicator from your current levels to your goal over time. We want to create targets that will we achieve en route to achieving the goal.

2010 2012 2014 2016 2018 2020 2022 2024 2026 2028 2030

Goal!

Stud

ent

Out

com

es


Considering impact and setting targets serves a number of important purposes

Purpose of estimating expected impact To monitor and predict performance rather than react Provide early identification of problems to enable

corrective action Allow leadership to see impact of initiatives on an ongoing

basis and learn from past performance To give confidence that planned initiatives are sufficient to

reach the target


To assess impact, we can estimate the timing, scale, and efficacy for our strategy

Scale of impact▪ Estimate the number of students who will

be reached by the strategy.

Efficacy of impact▪ Assess, for each 100 students who are

impacted by the strategy, the number who will achieve our intended outcome who would not have before.

Scale x Efficacy = IMPACT

Timing of impact▪ Consider when the initiatives within the

strategy will begin to have impact and the duration of that impact.

Assess your priority strategy on timing, scale, and efficacy


Here are some questions to consider when discussing impact and timing of the strategy

▪ When will the strategy be scaled up and when will it be as effective as we plan?

▪ Do we expect our strategy to contribute the same impact towards our goal each year?

▪ Will the implementation of our strategy improve over time? Until when?

Impact

Timing

▪ Does our strategy reach enough students to contribute substantially to our goal?

▪ Will we be able to reach our goal given the expected impact of our strategy (once we factor other initiatives into account)?


Exercise: Determining Impact

Answer the following for your strategy:

▪Scale: Determine the number of students who will be reached by your strategy by Spring 2015

▪Efficacy: Estimate the percent of students targeted by the strategy who will contribute to your target metric

▪Impact: What is the resulting estimated impact and when will it occur?


We will use intermediate metrics and feedback loops to monitor our priority initiatives

What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


We know what we want to improve, but how will we know if our high impact strategies are making a difference?

Questions to consider for student-related metrics:

▪What data do our initiatives impact most directly?

▪How do we typically predict movement in our target metric?

▪How long will it be until we expect to see movement in each of the metrics we have identified?

Questions to consider for process-related metrics:

▪How many people or departments will need to change their practice? Which ones?

▪How will we know whether business as usual has changed?

▪What resources will these people need in order to implement these important campus initiatives with fidelity?


Example Leading Indicators for College Completion

Course Participation: Begin remediation in first term Complete remediation in first year Complete college-level math or

English in the first year Complete a college success course

Course Performance: High rate of course completion

(80%) Complete 20-30 credits in first

year

Student Enrollment Patterns: Earn summer credits Enroll full-time Enroll continuously, without stop-

outs On-time registration for courses

What percentage of students reach

each of the leading indicators?

What is the impact of reaching each of

the leading indicators on

success rates?

Campuses can use leading indicator analyses to dig into their student data to help determine actions


Florida State evaluated its strategies against important progress metrics

Impact of Florida State’s Success Coaching Program

Program that provides support for students around these 7 “soft” factors that influence retention and graduation:

Commitment to graduation

Managing commitments Finances School community Academics Effectiveness Health & Support

Program Description Overall Impact



In addition to looking at traditional progress metrics, a team can track key actions to determine progress

Example of a campus’ action plan

Month Timeframe

Action Responsibility

January By end of January

Emails to students with 35 attempted hours who have not been accepted into a major

Individual Responsible

January Ongoing Update department Degree Audit reports Individual ResponsibleJanuary Ongoing Individual contact with students who have

been placed on probationAcademic Section

January Ongoing Individual contact with students who have been placed on warning

Academic Section

February 1st week Offer Workshop: Students Taking Exploratory Paths to Success

Advising First

February 1st week in the month

Email to all F coded students w/100+ hours inquiring about graduation plans; email to all H coded students w/100+ hours inquiring about finishing/graduation plans


February 6th week of term

New transfer—How are you doing— deadlines



We can also use a delivery chain to find important metrics and feedback loops

A delivery chain is the set of actors (people or organizations), and the relationships between them, through which a given system activity will be implemented.

A delivery chain has one question at its core: Starting from the policy intent of a leader in your system and ending with the front-line behaviors and practices that this policy is designed to influence, how – and through whom – does a system activity actually happen?


Example: Using a delivery chain to find metrics and feedback loops for mandatory first-year remediation

Provost Faculty Senate

Enrollment Managemen

t OfficeStudent Affairs

Academic AdvisorsRegistrar

Information Technology

Entering Students

Continuing Students

Orientation Advisors

Provost’s Office

Faculty Affairs

Vice Provost

StudentsAcademic Departments

Inform

Info

rm

Info

rm to

up

date

IT

syst

em

Inform and trainCreate policy Inform

and advise

Info

rm

Example metrics/feedback loops

▪Number of students requiring remediation

▪Average time to complete remediation▪Remedial course passing rates

▪Number of remedial courses offered▪Number of advisors trained in new

policy▪Number of project benchmarks hit


Exercise: Identifying Relevant Metrics

As a team, discuss which leading indicators and feedback loops you will use to track progress.

Answer the following: What are the most important leading indicators or

feedback loops to track (related to outcomes and to process)?

Based on these data, how will we know if we need to take action?

When we do, what action will we take? How frequently will these data be available? What do we need to do to make these data available

to us?


Record your discussion on leading indicators and feedback loops using your strategy profile worksheet


Workshop Agenda

Determining Who and How

Break


Finally, we will determine a routine for monitoring the impact of our key initiatives on our key metric

What is our

current status?

Why are we in this

position?


How are we going

to monitor progress

?


One of the main benefits of routines is their ability to maintain focus despite distractions and barriers

“Fires” and other crises!

Changes in Leadership!

Institutional inertia!

Initiatives lacking robustness!

Cynicism!IMPLEMENT TO

IMPROVE STUDENT OUTCOMES

Shifting agendas!

ROUTINES


Routines provide us with checkpoints where we can compare current progress with what we expected

▪ Regularly scheduled checkpoints to assess if implementation is on track

▪ A source of structure and discipline to keep the core goals on the agenda

▪ Structure to have honest and helpful conversations about where the work is and why and what’s next

What are routines?

▪ Monitor performance: Understand if campus is on track to deliver on its goals

▪ Diagnose problems: Surface issues that are inhibiting progress and analyze data to pinpoint causes

▪ Address problems: Provide a venue to discuss and decide how to overcome challenges

What purpose do routines serve?


Effective routines share similar characteristics

Criteria for effective routines

Agree on a common purpose

Arrive at a shared view of performance and progress

Identify and solve problems

Encourage learning and collaboration

Identify and commit to clear next

steps


The following characteristics should be present in any good routine

▪The right people– The person receiving the information (usually

president, provost or other campus leader)– The person responsible/accountable– Other key stakeholders

▪Relevant outcome-focused data– Leading indicators– Results showing progress on trajectories– Judgments on the likelihood of delivery

▪A discussion of challenges

▪Clear deliverables/next steps

▪Crisp, concise presentation of information

▪The right frequency and timing

▪The right format for those involved– Written note– Stocktake or in-person meeting– Formal report

Characteristics of a good routine:


A “textbook” campus routine calls for five roles

Roles and responsibilities: Graduation Rates Delivery Plan and StocktakeProvost“The Chief”▪Holds others accountable for results▪Asks tough questions that challenge and

support▪Actively engages in problem-solvingTeam Leader

Broker▪Prepares Commissioner for meeting▪Designs agenda, keeps meeting on track

Team MemberAnalyst▪Prepares data and evaluations▪Works with goal team to prepare

Director of Academic AdvisingSponsor▪Holds overall accountability for the plan’s

success▪Answers the Commissioner’s questions and

actively solves problems

VP for Academic AffairsWorkhorse▪Holds day-to-day accountability for the plan’s

success▪Manages the team to implement the plan▪Works with delivery unit and provides evidence

to arrive at current assessment of progress, in consultation with sponsor


When preparing for a routine, invest time to make relevant data easy to display and understand

Example: Enrollment of first time, full time students by semester and cohort

Fall 2003 Fall 2004 Fall 2005 Fall 2006

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

0

50

100

Percent of cohort enrolled

11.7

13.3 18.4

6.2

13.0

12.2

21.9

8.4

Pell and Non-Pell

Fall 2003 Fall 2004 Fall 2005 Fall 2006

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

1 Fall1 Spring

2 Fall2 Spring

3 Fall3 Spring

4 Fall4 Spring

5 Fall5 Spring

6 Fall6 Spring

0

50

100

Percent of cohort enrolled

13.9

11.3

13.5

10.3

15.3

9.4

20.9

7.6

URM and Non-URM

Non-Pell

Pell

Non-URM

URM


Exercise: Determining Who and How

Determine the key people who will be involved in monitoring and how monitoring and decision making will

take place.

Answer the following:– Who from our team will lead this work?– Who else needs to be involved?– When and how often will we meet to review the progress of

this strategy to improve our key outcome?– What are our next steps? Who will do these things?


Record your discussion on who and how using your strategy profile worksheet


Workshop Agenda

Identifying Next Steps


Our Next Steps…

Identify and write on cards your key takeaway and two

immediate next steps your team will take to actualize your

strategy profile

Thank Youwww.deliveryinstitute.o

rg

Ellyn [email protected]

Omari [email protected]

g

Duncan [email protected]

turning data into action workshop june 26, 2014

Documents

graduation rate of students

year graduation rate

graduation rate of white

graduation rate of african

close achievement gaps

data source

completion rates

highquality degree production