Slice and DiceUsing existing data to answer novel questions about student outcomes

February 7, 2012

Jill KrollOffice of Career and Technical Education

Michigan Department Of Education

Carol ClarkCTE TRAC Director & Program Coordinator

GASC Technology Center

• Recognize challenges in using existing data to answer questions

• Gain knowledge of resources and tools to meet challenges of information needs

2

Learning Objectives

• Two examples - Two solutions–Example 1: Using existing data

–Example 2: Overcoming limitations of existing data

• Resources

• Discussion and Questions

3

Overview

Example 1: Using Existing Data

Do secondary CTE students in articulated programs have higher

placement rates?

4

Example 1: Using Existing Data

• What data are available to answer this question?

• In light of the available data, how might you refine the question?

5

Example 1: Using Existing Data

Refine your question - “Placement”–Total placement - “In Employment or

Continuing Education or Military”

–Placement in Continuing Education

–Placement in Community College

–“Related” Placement

•Can you think of other available data?

6

Example 1: Using Existing Data Step 1: Download Follow Up Survey Data

• Log in to CTEIS.com

• Follow Up tabReport

• Export Building Survey Data

7

8

Example 1: Using Existing DataSave or Open in MSExcel

9

Example 1: Using Existing DataStep 2: CTEIS Ad Hoc Query Tool

• Log In to CTEIS

• Choose Reports tab,

Funding Reports, Ad hoc Query

• Choose agency, building

• 4301 (select year)

• Selection criteria

10

Example 1: Using Existing Data Student-Level Data From CTEIS

11

4301-2009-2010

Example 1: Using Existing Data Export Student Data to MSExcel

12

4301-2009-2010

Example 1: Using Existing DataAnalyzing the Data

• Data analysis software

–Relational database software

–Spreadsheets

–Statistical software

–Others

13

Example 1: Using Existing Data

Tech Prep Program?2011

Total Placement(Employment or Continuing Education or Military)

Not Placed Yes, Placed Total

Not Tech Prep 562 (5.9%) 9,029 (94.1%) 9,591

Yes, Tech Prep 290 (5.3%) 5,226 (94.7%) 5,516

Total 852 14,255 15,107

14

Example 1: Using Existing Data

15

Tech Prep Program?

2011Attending School

Not Attending School

Attending School Total

Not Tech Prep 2,244 (23.4%) 7,347 (76.6%) 9,591

Yes, Tech Prep 1,212 (22.0%) 4,304 (78.0%) 5,516

Total 3,456 11,651 15,107

Example 1: Using Existing Data

16

Tech Prep Program?

2011Attending A Community College

Attending Another School Type*

Attending a Community College

Total

Not Tech Prep 3,898 (53.6%) 3,381 (46.4%) 7,279

Yes, Tech Prep 2,274 (53.1%) 2,010 (46.9%) 4,284

Total 6,172 5,391 11,563

*Attending business or trade school, college or university, military training or other training

Example 1: Using Existing Data

17

Tech Prep Program?

2011Related Placement

in Continuing EducationUnrelated Placement Related Placement Total

Not Tech Prep 1,249 (24.4%) 3,866 (75.6%) 5,115

Yes, Tech Prep 653 (21.7%) 2,359 (78.3%) 3,012

Total 1,902 6,225 8,127

Example 2: Overcoming Limitations of Existing Data

What is the impact of integrated instruction on academic

achievement?

18

Example 2: Overcoming Limitations of Existing Data

• Local data:–Number of students signed up for each

course

–Number of students who took test for academic credit

–-Number of students who passed test

–Number of students who signed up for RAC

19

REQUIRED ACADEMIC CREDIT (RAC)

ELA-12 14

MATH 220

SCIENCE 76

Visual/Performing Arts (VPAA)

65

TOTAL 375

20

GASC 2010-11Signed up for RAC

ELA-12 12

MATH 213

SCIENCE 65

Visual/Performing Arts (VPAA)

62

TOTAL 352

GASC 2010-11 Total Students Achieving RAC

Example 2: Overcoming Limitations of Existing Data

  Count % Increase in 2011-12(compared to 2010-11)

ELA-12 15 7%

MATH 412 87%

SCIENCE 94 24%

VPAA 114 75%

TOTAL 635  

21

GASC 2011-12 Signed up for RAC

Example 2: Overcoming Limitations of Existing Data

CTE Subject Academic Subject

Type of Academic Integration

Method of Qualifying for Academic Credit

Law & Public Safety EnglishLanguage Arts

Instruction by Academic Teacher

Papers graded by English TeacherCTE-specific content, ELA tasks

Multiple Programs(2nd Year Students)

Math Collaborative Teaching Model 1

Pre-Test/Post-Test(Standard Math Test Questions)

Health Science Collaborative Teaching Model

(different for each Health program)

Integrated post-test: Contextual academic test questions specific

to the program

Multiple Programs Visual/Performing Arts

Collaborative Teaching Model

2

Teachers choose a minimum of one project per semester and

grade with a rubric

22

Example 2: Overcoming Limitations of Existing Data

23

Example 2: Overcoming Limitations of Existing Data

Solutions to limitations in existing data:–Modify/Simplify question

24

Example 2: Overcoming Limitations of Existing Data

• Collect supplemental data

25

CTE Subject Academic Subject

Type of Academic Integration

Method of Qualifying for Academic Credit

Law & Public Safety EnglishLanguage Arts

Instruction by Academic Teacher

Papers graded by English TeacherCTE-specific content, ELA tasks

(blind)

Law & Public Safety None None Papers graded by English TeacherCTE-specific content, ELA tasks

(blind)

Example 2: Overcoming Limitations of Existing Data

• Solutions to limitations in existing data–Control extraneous factors

• Choose one subject area

• Use three methods of integrating instruction

• Use one method of qualifying for academic credit (assessing achievement)

26

Example 2: Overcoming Limitations of Existing Data

27

CTE Subject

Academic Subject

Type of Academic Integration

Method of Qualifying for Academic Credit

Validation

Therapeutic Services

Math Academic Instructor Provides Instruction

• Pre-Test/Post-Test using Standard Math Test Questions

• Integrated post-test: Contextual Math test questions specific to the program

College Course

Grade from STARR

DataAccuplacer test results

Same Same Collaborative Teaching Model(CTE instructor

Provides Academic Instruction)

• Same • Same

Same Same Pull-Out Academic Instruction (Tutoring)

• Same • Same

Resources

• Look for more existing data–National

• National Center for Education Statistics (NCES) website

• Midwest Regional Education Laboratory (REL)

–Existing local and state data

–Peers

28

29

NCES: Data Collection Tools

http://nces.ed.gov/

NCES: Example Content Areas

Regional Education LaboratoriesFunded by: Institute of Education Sciences (ies)

• Purpose: Provide access to high-quality, scientifically valid education research

• Publications: http://ies.ed.gov/ncee/edlabs/projects/

• Ask A REL: A collaborative reference desk service

• Midwest REL: American Institutes for Research

http://www.learningpt.org/

31

Midwest REL: Resources/Services

32

Contact:

Jill KrollSupervisor

Office of Career and Technical EducationMichigan Department Of Education

(517) 241-4354

KrollJ1@Michigan.gov

Carol ClarkCTE TRAC Director & Health and Human Services Platform Facilitator

Genesee Area Skill Center Technology Center(810) 760-1444 ext. 176

clarkcar@gasc.flint.K12.mi.us

33