using idea for assessment, program review, and sacs university of alabama birmingham september 11,...

Using IDEA for Assessment, Program Review, and SACSUniversity of Alabama BirminghamSeptember 11, 2012

Shelley A. Chapman, PhD

Plan for this Session

• Program Evaluation & Assessment of Student Learning• Group Summary Reports• Aggregate Data File• Benchmarking Reports• Accreditation Guides

What makes IDEA unique?

1. Focus on Student Learning

2. Focus on Instructor’s Purpose

3. Adjustments for Extraneous Influences

4. Validity and Reliability

5. Comparison Data

6. Flexibility

Student Learning Model: 2 Assumptions

Assumption 1:

Types of learning must reflect the instructor’s purpose.

Student Diagnostic Form

Assumption 2:

Effectiveness determined by students’ progress on objectives stressed by instructor

Diagnostic Report Overview Page 1 – Big Picture

How did I do?

Page 3 – Diagnostic What can I do differently?

Page 2 – Learning Details What did students learn?

Page 4 – Statistical Detail Any additional

insights?

Your Average (5-point Scale)

Raw Adj.

A. Progress on Relevant Objectives1

Four objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.

The Big Picture

ProgressOnRelevantObjectives

4

4.3 + 4.34.14.23.6

5

Summary Evaluation: Five-Point Scale

Report Page 1

Your Average Score

(5-point scale)

Raw Adj.

A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

Overall Ratings B. Excellent Teacher 4.7 4.9

C. Excellent Course 4.1 4.4

D. Average of B & C 4.4 4.7

Summary Evaluation(Average of A & D) 4.3 4.5

50%

25%

25%

Individual Reports to Group Reports

The Group Summary Report

How did we do?

How might we improve?

Defining Group Summary Reports (GSRs)

• Institutional

• Departmental

• Service/Introductory Courses

• Major Field Courses

• General Education Program

GSRs Help Address Questions

• Longitudinal• Contextual• Curricular• Pedagogical• Student Learning-

focused

Adding Questions

Up to 20 Questions can be added

• Institutional

• Departmental

• Course-based

• All of the above

Local CodeUse this section

of the FIF to code types of data.

Defining Group Summary Reports• Local Code

• 8 possible fields• Example: Column one – Delivery Format

• 1=Self-paced• 2=Lecture• 3=Studio• 4=Lab• 5=Seminar• 6=Online

Example from Benedictine University

Example Using Local code

Assign Local Code• 1=Day, Tenured• 2=Evening, Tenured• 3=Day, Tenure Track• 4=Evening, Tenure

Track• 5=Day, Adjunct• 6=Evening, Adjunct

Request Reports• All Day Classes

• Local Code=1, 3, & 5

• All Evening Classes• Local Code=2, 4, & 6

• Courses Taught by Adjuncts• Local Code=5 & 6

Description of Courses Included in this Report

Number of Classes IncludedDiagnostic From 42

Short Form 27

Total 69

Number of Excluded Classes 0

Response RateClasses below 65% Response Rate 2

Average Response Rate 85%

Class SizeAverage Class Size 20

Page 1 of GSR

UAB Spring 2012

Page 1 of GSR

Assessment of Learning

What are our faculty emphasizing?

How do students rate their learning?

How do our courses compare with others?

How do our students compare with others (self-rated characteristics)?

What efforts can we make for improvement?

(How can we “close the loop”?)

UAB Core Competencies

Shared Vision for a UAB Graduate

Possible IDEA Learning Objectives

Communication 8, 11

Knowledge 1, 2, 4, 7

Problem Solving 3, 9, 11

Citizenship 10, 12, extra questions

Are we targeting “Core Competencies” in the Core Curriculum?

UAB Core Curriculum Courses

1 2 3 4 5 6 7 8 9 10 11 12

EH 101 English Composition X X X X

BY 101 Topics in Contemporary Biology

X X X X

MA 105 Pre-Calculus Algebra X X X

ARH 101 The Art Experience X X X

HY 101 Western Civilization I X X X X

PY 101 Introduction to Psychology

X X X X

IDEA Learning Objectives

Are we targeting “Core Competencies” in the Core Curriculum?

UAB Core Curriculum Courses

1 2 3 4 5 6 7 8 9 10 11 12

EH 101 English Composition X X X X

BY 101 Topics in Contemporary Biology

X X X X

MA 105 Pre-Calculus Algebra X X X

ARH 101 The Art Experience X X

HY 101 Western Civilization I X X X X

PY 101 Introduction to Psychology

X X X

IDEA Learning Objectives

What are We Emphasizing?Percent of Classes Selecting Obj. as

Important or Essential

This Group Institution IDEASystem

Objective 1 16% 70% 78%

Objective 2 13% 59% 75%

Objective 3 41% 58% 75%

Objective 4 32% 35% 55%

Objective 5 23% 19% 32%

Objective 6 32% 14% 25%

Objective 7 22% 27% 27%

Objective 8 78% 43% 47%

Objective 9 19% 23% 41%

Objective 10 7% 11% 23%

Objective 11 68% 42% 49%

Objective 12 20% 23% 41%

Average # of Obj. Selected 3.7 4.2 5.7

Page 2

UAB Spring 2012

What are We Emphasizing?

Page 9 Section B

NumberRating

Percent indicating amount required

None or Little

Some Much

Writing 66 2% 17% 82%

Oral Communication 66 6% 42% 52%

Computer Application 66 50% 44% 6%

Group Work 66 27% 59% 14%

Mathematics/Quantitative Work

65 97% 3% 0%

Critical Thinking 66 0% 30% 70%

Creative/Artistic/Design 66 61% 33% 6%

How Did Students Rate their Learning on Core Competencies?

Do Students’ report of learning meet our expectations?

Pages 5 and 6

Raw Average

Adj.Average

# ofClasses

ThisReport

3.9 3.9 11

Institution 4.2 4.2 3,963

IDEA System

4.0 4.0 31,991

Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends)

How do students rate their learning?

Page 3 Part 1: Distribution of Converted Scores Compared to the IDEA Database

Overall Progress Ratings (Courses)

Page 3 Percent of Classes at or Above the IDEA database Average

Overall Progress Ratings (Courses)

Part 3: Percent of Classes at or Above This Institution’s Average

Page 4

Which teaching methods might we use to improve learning?

Page 7

Teaching Methods and Styles

Stimulating Student Interest # Classes Av. s.d.

15. Inspired students to set and achieve goals which really challenged them

42 3.8 0.5

Assessing the QEP for UABUAB’s Core Competencies

IDEA Learning Goals IDEA Teaching Methods Associated with Progress on Learning Goal

Communication8 Developing skill in expressing myself orally or in writing11 Learning to analyze and critically evaluate ideas, arguments, and points of view

2, 7, 8, 15, 16, 18, 19

Knowledge1 Gaining factual knowledge (terminology, classifications, methods, trends)2 Learning fundamental principles, generalizations, or theories4 Developing specific skills, competencies, and points of view needed by professionals in the field most closely related to this field7 Gaining a broader understanding and appreciation of intellectual and cultural activity (music, science, literature, etc.)

1, 2, 4, 6, 7, 8, 10, 12, 13, 15, 19

Problem solving3 Learning to apply course material (to improve thinking, problem solving, and decisions) 9 Learning how to find and use resources for answering questions or solving problems11 Learning to analyze and critically evaluate ideas, arguments, and points of view

2, 7, 8, 9, 13, 14, 15, 18, 19

Citizenship10 Developing a clearer understanding of, and commitment to, personal values12 Acquiring an interest in learning more by asking my own questions and seeking answers

1, 2, 4, 6, 7, 8, 11,13, 15, 16, 18,

Relationship

of Learning Objectives to

Teaching

Methods

How do students view course work demands?

Page 8B Student Ratings of Course Characteristics

Diagnostic Form Item # & Item Average % Classes Below 3.0

% Classes 4.0 or Above

33. Amount of Reading

This Report 3.4 21% 24%

Institution 3.3 31% 19%

IDEA System 3.2 33% 15%

34. Amount of work in other (non-reading) assignments




35. Difficulty of subject matter




Aggregate Data File

Allows you to •Use Excel Spreadsheet•Use with SAS or SPSS•Ask other types of questions•Display data in different ways

A. Progress of Relevant Objectives-Converted Score Category

Expected Distribution

Fall 2010 Raw

Spring 2011 Raw

Fall 2011 Raw

Much Higher (63 or higher) 10% 6% 7% 7%Higher (56-62) 20% 32% 34% 31%Similar (45-55) 40% 46% 46% 44%Lower (38-44) 20% 11% 9% 10%Much Lower (37 or lower) 10% 5% 5% 8%

B. Excellence of Teacher-Converted Score CategoryExpected Distribution

Fall 2010 Raw

Spring 2011 Raw

Fall 2011 Raw


C. Excellence of Course-Converted Score CategoryExpected

Distribution

Fall 2010 Raw

Spring 2011 Raw

Fall 2011 Raw


Summary Evaluation (Average of A, B, C)-Converted Score Category

Expected Distribution

Fall 2010

Spring 2011

Fall 2011


Primary Instructional Approaches

Fall 2010

Primary

Spring 2011

Primary

Fall 2011

PrimaryLecture 61% 60% 61%Discussion/Recitation 7% 8% 10%Seminar 7% 9% 9%Skill/Activity 7% 6% 7%Laboratory 8% 9% 9%Field Experience 1% 1% 1%Studio 2% 3% 2%Multi−Media 0% 0% 0%Practicum/Clinic 0% 0% 0%Other/Not Indicated 6% 4% 2%

Number Classes Rating 787 677 859

Secondary Instructional Approaches

Fall 2010

Secon-dary

Spring 2011

Secon-dary

Fall 2011

Secon-dary

Lecture 16% 13% 16%Discussion/Recitation 30% 27% 30%Seminar 2% 3% 4%Skill/Activity 11% 13% 13%Laboratory 3% 6% 4%Field Experience 2% 3% 2%Studio 0% 0% 1%Multi−Media 2% 3% 3%Practicum/Clinic 1% 1% 1%Other/Not Indicated 34% 31% 28%

Number Classes Rating 787 677 859

Course Emphases: Writing

Fall 2010

Spring 2011

Fall 2011

None or Little 25% 26% 22%Some 46% 45% 44%Much 29% 30% 34%

0%

20%

40%

60%

80%

100%

Fall 2010 Spring 2011 Fall 2011

None or Little

Some

Much

Course Emphases: Oral communication

Fall 2010

Spring 2011

Fall 2011


0%

20%

40%

60%

80%

100%


None or Little

Some

Much

Instructors’ Reports on Course Emphases: Selected Pairings-Writing and Oral Communication

Course Emphases: Critical thinking

Fall 2010

Spring 2011

Fall 2011


0%

20%

40%

60%

80%

100%


None or Little

Some

Much

Course Emphases: Writing

Fall 2010

Spring 2011

Fall 2011


0%

20%

40%

60%

80%

100%


None or Little

Some

Much

Instructors’ Reports on Course Emphases: Selected Pairings-Critical Thinking & Writing

Highlights for Sample U• Remarkably similar profiles

across terms

• Overall response rates ranged from 66% to 80%

• 1st term in which administration was primarily online achieved a 75% response rate

• Transition from paper to online (fall 2009 to fall 2010) does not show major differences in profiles

• Sample U faculty focus on 4-5 outcomes as essential/important

• Over the last 3 terms, a significant increase on several objectives has been observed: application of course material, oral and written communication skills, & analysis and critical thinking skills (objectives 3, 8, & 11, respectively)

BenchmarkingInstitutional and Discipline Reports

Benchmarking Reports

Comparison to•6-10 Peers•Same Carnegie Classification•IDEA database

Comparison Groups

Your University ---------- ----------

Peer* ----------------------------

Carnegie ---------- ----------

National ---------- ----------

* Peer group is based on 6-10 institutions identified by your institution

Benchmarking Reports

The student, rather than the class, is the unit of analysis

Percentage of positive ratings is given rather than averages

Response Rates

Your University: Student participation is similar to that of each comparison group

Your University=79%

Peer=77%

Carnegie=79%

National=75%

Students’ Perceptions

Graduate Students

Instructional Objectives Selected by Instructors

Instructors’ Intentions/ focus Students’ Self-Reported Progress on Learning

Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)

% of total classes where instructor selected objective as “Essential” or “Important”

IDEA Objective 3

Learning to apply course material (to improve thinking, problem solving, and decisions)

IDEA Objective 8

Objective 8: Developing skill in expressing oneself orally or in writing

% of total classes where instructor selected objective as “Essential” or “Important”

Developing skill in expressing oneself orally or in writing

Teaching Methods and Styles Reported by Students

(Diagnostic Forms Only)

• Fostering Student Collaboration

• Encouraging Student Involvement

Benchmarking Look at “General Education”

Using Aggregate Data for Assessment

UAB

Core

Competencies

Program Learning

Outcomes

Program Learning

Outcomes

Program Learning

OutcomesGroup Summary

Report

Group SummaryReport,

Include Extra Questions

Benchmarking: One Year or 3-5 Year Trend Report

Benchmarking: Discipline Report

Course Learning Outcomes


Course Learning

Outcomes


Course Learning

Outcomes

Accreditation

Guides• SACS

• NCATE

• CACREP

Questions?

using idea for assessment, program review, and sacs university of alabama birmingham september 11,...

Documents

gsr slide

instructor slide

flexibility slide

phd slide

big picture slide

benedictine university

relevant objectives

group reports