Andrew LaManque, Foothill-De Anza CCDAlice VanOmmeren, CCC Chancellor's OfficeRyan Fuller, CCC Chancellor's OfficeDaylene Meuschke, College of the CanyonsBarbara McNeice-Stallard, Mt. San Antonio CollegeTerrence Willett, RP Group

RP Regional Research Groups – Spring Convening

2Measuring Our Success, RP Conference, April 2013

A Brief Overview

- RP Report on Prerequisite Validation- Student Success Scorecard- Wage Tracker- Professional Development Opportunities- IRB Issues- RP Group Update

Update on the Scorecard (formerly know as ARCC)

and WageMart

Alice van OmmerenRyan Fuller

Developments• In 2012, the Scorecard Advisory Group and

TRIS developed metrics and reporting format• In January 2013, researchers had 45-days to

review report format and Scorecard metrics • April 9, the public release of the Scorecard• Last week, the Scorecard on the DataMart• System of the State this summer and the

advisory group reconvenes in the fall

College Responsibilities • ARCC- Alias List for Updates and Revisions

– Two changes already, ESL and Math Remedial Rate• Board of Trustee Interactions (due March 2014)• Use the 45-day review period in January for

questions and data resubmission (data on demand)• Check other Scorecard resources on webpage

– New peer groups and % of missing SSN report• Use scorecard@cccco.edu or arcc@cccco.edu for

questions and feedback

Most FAQs/Issues of Concern • Why do Prepared students have lower rates

than Unprepared (persistence)?• What are the N’s behind the metrics?

– To disclose, or not? When is the cell size too small? • Revise Data on Demand files to include flags• Why are some of my remedial courses not

qualifying for the cohort?– TOP code and CB04

Remedial ESL Rate (cohort) Cohort is first attempt of an ESL course in any levels below transfer:• CB03 COURSE-TOP-CODE = 4930.84, 4930.85,

4930.86, 4930.87 • SB01 STUDENT IDENTIFIER STATUS = S (only

students with SSN)• CB21 COURSE-PRIOR-TO-COLLEGE-LEVEL = A,

B, C, D, E, or F• CB04 COURSE-CREDIT-STATUS = C

Remedial Math Rate (cohort) Cohort is first attempt of a Math course in two to four levels below transfer:• CB03 COURSE-TOP-CODE = 1701* • SB01 STUDENT IDENTIFIER STATUS = S (only

students with SSN)• CB21 COURSE-PRIOR-TO-COLLEGE-LEVEL = B,

C, or D• CB04 COURSE-CREDIT-STATUS = C (not D)

We are working on…• Revisions for next years’ Scorecard

– Advisory Group in the fall, student-counselor ratio• Guidelines for Measuring the Achievement

Gap using the Scorecard metrics• ISP and OOS Report (modifying methodology)• Exploring uses of the Scorecard metrics for

other reporting requirements• WageTracker or WageMart (release May 29th)

New Wage Tracker Tools on Datamart

Methodology

• Two different methodologies– Systemwide wages for award completers – By campus wages for award completers

• Award completer data joined with EDDUI wage data.

• Both wage methodologies break out by award discipline (6 digit TOP code) and award type.

Systemwide Award Completer Wages

• Award completers in a given academic year, exclude: – Enrolled anywhere in higher education after award– 21 and under at time of award

• Join with EDDUI (Unemployment Insurance) wage file • Sum all quarters of wages to student annual wage• All wages inflation adjusted using California CPI-U• Calculate median wages by discipline and award type• Median calculated for students with wages 2 years

before award, 2 years after and 5 years after award

By College, Award Completer Wages

• Same methodology as statewide wages, except:– Series of 8 years of award completers (to increase

the number of students in the cohorts)– Median calculated for students in cohort with

wages 3 years after award• Calculate median wages by “college,”

discipline and award type

Systemwide vs. By College Methodology

Wages03-04 04-05 05-06 06-07 07-08 08-09 09-10

By College

Awards00-01 01-02 02-03 03-04 04-05 05-06 06-07 07-08

10-11

05-06Systemwide

03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11

Awards

Wages

Data Caveats• EDDUI Data contains wages of occupations

covered by CA Unemployment Insurance• Excludes military, federal government, self-

employed, out of state and unemployed • No way of determining part time or full time

wages. Hours worked not in the EDDUI data.• Only students with SSNs • Wages are adjusted for inflation to current

dollars using the California CPI-U

Analysis Caveats• Data reflects only the wage outcomes of award earners

who remained in California• Region/locale significantly impacts wages (i.e. cost of

living), caution in comparing earnings across the state.• Wages are not necessarily from employment associated

with a particular award discipline, no information on job• Short term wage outcomes should not be a sole measure

of institutional effectiveness or program quality• Other factors besides earnings motivate students to earn

an award in a specific discipline

Biology

Occupational Therapy Tech.

Psychiatric Techician

Electronics and Electric Tech.

Health Information Tech.

Fire Tech.

Dental Hygienist

Respiratory Therapy

Radiologic Tech.

Registrered Nursing

$55,264

$56,379

$57,831

$58,417

$64,445

$66,421

$68,015

$72,582

$81,573

$81,640

Associate Degrees with highest median wages 5 Years after Award

Fire Tech.

Computer Networking

Electrical

Radiolocial Tech.

Diagnostic Medical Sonography

Physicians Assistant

Registered Nursing

Water & Wastewater Tech.

Electrical Systems and Power Trans.

Paramedic

$67,888

$68,358

$69,422

$72,900

$74,075

$78,748

$78,770

$79,205

$107,466

$113,360

Certificates with highest median wages 5 Years after Award

Accessing the new tools

• Query modules using both the Systemwide and ‘By College’ methods will be available in datamart (System Wage Tracker & College Wage Tracker)

• Release on May 29th

• Median wages by program code will be available for colleges in Data on Demand.

Daylene Meuschke, College of the CanyonsBarbara McNeice-Stallard, Mt. San Antonio College

Proposed RP Professional Development Plan 2013-14

21Measuring Our Success, RP Conference, April 2013

A Brief Overview

•The following slides present the proposed professional development plan for the RP Group in 2013-14.•The first slide presents the an overview of the plan.•Slides 3-5 provide detail for each segment of the plan.•Slides 6-8 identify possible webinar topics for each of the three big training areas.

Conferences Webin

arsRegional Research Groups

WorkshopsCoach

es

Working Groups

Recognition

Dissemination

RP GroupProfessional Development

Conferences

• SSS (October)• RP (April)

Webinars

• 30 minutes• Monthly• Moderated• 3 main areas: planning, assessment/evaluation & resources/tools

Regional Research Groups

• Propose meeting 3x / year (fall, spring, summer)• Devote one hour to training via webinar with skilled facilitators at each RRG to lead activities & discussion• Allot 30 minutes for updates from CCCCO, RP, & National issues• Include agenda item on communicating and helping campuses use data to inform decision-making processes*

Workshops/Other In-Person Training

• Leading from the Middle• “Research Connects” (in-person workshops – Fee TBD)

Coaching Program

• Can be used to disseminate Summer Institute (S.I.) Content• Webinars can be used to supplement S.I. content and facilitate connection with S.I. leaders• Fee for S.I. course content TBD• Pair up researchers with a “coach” either as part of S.I. or outside of participation in S.I. (upon request)

Working Groups

• Reporting Efficiency • Planning• SLO Assessment

Recognition

• RP Awards• Power Awards• Leadership*

Dissemination

• RP Perspectives• Regional Research Groups• Listservs (RP, SLO Assessment, Regional Research Groups)• RP website

Planning

Big Picture (webinar)

Program Review & ACCJC Standards

Strategic Plans/Educ Facility Master Plans

Alternative Approaches to Facilitate Engagement*

Lessons Learned

Assessment / EvaluationPre-Req Validation

Alternative Placement

Segmentation Analysis

Disproportionate Impact

Student Equity & DI

Scorecard (additional analyses)

Resources / Tools

Statistical Techniques

Wage Tracker Analyses

Needs Assessments (Labor Market

analyses)

Hold for GIS – look at AEA

Cal-PASS/PLUS

Videos*

IRB

Andrew LaManque, Ph.D.President, RP Group

Regional Research Meetings

RP Group Prerequisite Guide Overview

RP Regional Meeting, May 3, 2013

This project was made possible through the efforts of many dedicated people within the California Community College system! Special thanks goes to:

Anne Danenberg, Sacramento City College Lan Hao, Citrus College Andrew LaManque, Foothill De Anza Community College District Katherine McLain, Cosumnes River College Daylene Meuschke, College of the Canyons Ken Meehan, formerly Fullerton College Steven Spurling, San Francisco City College Keith Wurtz, Crafton Hills Lijuan Zhai, Fresno City College

Thanks!

30

Pre-implementation Are students highly unlikely to succeed in the target course without the

knowledge, skills, and abilities learned in the prerequisite course? Do student success rates in the target course vary for those students

with or without the pre-requisite? Is the projected impact on course availability, enrollment and success

different depending on student demographics?

Post-implementation Did the student success rates of a target course increase after the

implementation of a pre-requisite? Is the actual impact on course enrollment and success different

depending on student demographics?

Using Research Questions to Frame the Discussion

31

The Prerequisite Research Framework (pre)

32

The Prerequisite Research Framework (post)

33

Will be seeking input in the coming weeks: Institutional Researchers Faculty Chancellor’s Office Instructional and Student Service Officers

Will be reviewing the Disproportionate

Impact section

34

Next Steps

35

Questions?Please submit any additional questions to your webinar moderator and they will get back to you with answers.

Find out more:Andrew LaManque, Foothill-De Anza CCDlamanqueandrew@fhda.edu

Terrence Willett, The RP GroupFor the latest on our projects visit:http://www.rpgroup.org/projects/current

RP Group Update