basic skills, outcomes & fostering a culture of evidence and inquiry

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP

Group

Basic Skills, Outcomes & Fostering a Culture of Evidence and InquiryRob Johnstone, Skyline College; Pam Deegan, Miracosta College; Ian Walton, Mission College / ASCCC

Sponsored by the William & Flora Hewlett FoundationResearch conducted by the RP Group


Group

Agenda

Opening Dialog: Why are we struggling?

Challenges identified through the Basic Skills Outcomes Capacity (BSOC) Study

Strategies for responding to these challenges

Closing Dialog: What would help colleges strengthen cultures of inquiry?

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group


Group

BSOC FindingThere is a perception gap between

administrators and faculty regarding the

availability and use of assessment data.

College administrators believe that information

is more widely available and integrated into

implementation planning than those who work

more closely with students.


Group

The Information Gap

Administration

Deans / Managers

Faculty/Staff

No data needs being met

All data needs being met

Information Gap

Information Gap

Gap


Group

Opening Dialog

Findings from the BSOC survey suggest we are struggling to get the right information to the right people at the right time.

What are your thoughts?



Group

BSOC Finding

Colleges are data rich and information poor.


Group

The Data We Collect

SLO / Assessment

Productivity / Efficiency

Enrollment Mgmt

Data Not Being Used

Data Frequently Being Used

Unmet

Unmet Data Needs

Unmet Data Needs


Group

The Action Gap“Our college is quite successful at

gathering data and putting reports together based on that data. Where we struggle is with executing action plans as a result of this data. I don’t really think we do a good job of ‘doing’ anything with the data, or implementing programmatic changes or initiatives based on the data.”


Group

BSOC Finding

There is a pattern of gaps in the use of data and evidence across community colleges – gaps that might be inhibiting institutional learning.


Group

Institutional Data Fills Two Important Gaps

Elevated Level of Organizational

Awareness

Improved Processes & Performance

Knowledge Gap Performance Gap

Strategic Planning Function

Institutional Effectiveness & Student Success Function

(Poor Planning) (Good Planning) (Good Performance)

Classroom and Service Area Assessment

Standard Level of Organizational

Awareness

Institutional Outcomes and Benchmarks


Group

BSOC Finding

Colleges may be focusing on the wrong data.


Group

Typical Twenty Year Trend for Common Institutional Outcome Measures

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Retention Rate

Success Rate

Consider the multitude of changes over this period:

- faculty/staff turnover- program successes/failures- changing student demographics - budget contractions/expansion- evolving state & accreditation mandates- leadership turnover

and yet these performance metrics remained amazingly stable.

What does that tell us about the usefulness of these metrics in setting institutional strategies?


Group

Assessment and the Organizational Learning Map

Desired Outcome(Pass transfer-level course)Program Level Assessment

• Assess at the end of the semester• Modifications tend to be structural • Change is slow & deliberate

Assessment within the Classroom • Assess both during and at the end of the semester• Modifications are both structural & procedural• Change is frequent & free form

Entry Point (Three levels

below transfer)

Did the learning community for Math 100 work?

Within Math 100, did a group assignment produce stronger learning than a lecture?

Learning Pathways under Three Assessment Frameworks


Group

BSOC Finding

Institutional research supports institutional metrics to a higher degree than classroom assessment.


Group

Level of Involvement in the Use of Data

17%

83%82%

18%

Chief Instructional Officer Classroom Faculty

High Level of Involvement Low Level or No Involvement


Group

BSOC Finding

Much of what is needed to support good decision making lies beyond the current grasp of our institutional databases.


Group

Students Progress in a Non-Linear Fashion


Group

BSOC Finding

Over the last decade, most of the progress in gathering and utilizing data has been at the institutional level.


Group

Degree to Which Evidence is Used in Decision Making

High

Low

Us

e o

f D

ata

fo

r In

sti

tuti

on

al M

etr

ics

Use of Data in Classroom/Service Area Assessment

Low High

Strong Sense of Direction but Poor Understanding of

Program Performance

Much Context-Specific Knowledge but Poor Integration

Poor Planning & Weak

Understanding of Programs

Closing the LoopRobust Planning &

High Levels of Innovation

Outward Looking

Inward Looking

Forward LookingCluster of

California Community

Colleges


Group

BSOC Recommendations

Research should be brought to the intervention level, at the classroom.

1


Group

Expanded Definition of Research Capacity

Production

Dissemination

InterpretationTranslation

Closing & Widening the

Loop


Group

BSOC Recommendations• Research offices should receive support that will enable them to address multiple domains of research capacity.

• Technical assistance should be provided to colleges that will help them to strengthen cultures of evidence and inquiry.

The Bridging Research, Information, and Culture (BRIC) Project

2


Group

The BRIC Project Developing a Better Understanding of How to Close the Performance Gap

Standard Level of Organizational

Awareness

Elevated Level of Organizational

Awareness

Improved Processes & Performance

Knowledge Gap Performance Gap

Strategic Planning Function

Institutional Effectiveness & Student Success Function

(Poor Planning) (Good Planning) (Good Performance)

Institutional Outcomes and Benchmarks Classroom and Service Area Assessment


Group

Closing Dialog

The BRIC project will allow us to provide tailored support at colleges.

How could an external team of faculty, administrators, and researchers help deepen the culture of inquiry at your institution?



Group

Contact & Information

RP Group Websitewww.rpgroup.org

BSOC/BRIC Project ContactsRobert Johnstone, Project [email protected]

Priyadarshini Chaplot, Project [email protected]


basic skills, outcomes & fostering a culture of evidence and inquiry

Documents

culture of evidence

rp groupthe data

use of data

rp group basic skills

assessment data

data rich

rp groupbasic skills

metall data needs