“when learners are at a distance, the careful design of ... · pdf file(cs 3.3.1) coc...

“When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations on the assessment of online learners.”Oosterhof, et al. (2008) Assessing Learners Online

1

Donald Staub, Title III Director Carteret Community College

?

?

?

?

??

Assessing Distance Learning

2

http://ccctitle3.wordpress.com/3

The Lineup• DL @ CCC

• Why Worry about DL Assessment?

• How We’re Doing It• The DL Program

• DL in Programs & Courses

• Possibilities & Challenges4

DL @ CCC• 33 programs @ CCC

• FA10 enrollment ~1800

• DL is 10+ years old

• FA10 336 total sections 243 seated sections (72%) 64 Internet (20%) 29 Hybrid (8%) 58% FT / 42% PT 5

Why worry about assessing DL?

Student Success Persistence(btw: success vs. retention vs. persistence) “Good assessment guides good PD/ID”

@ the College & Program-Level:

6

@ the Course-Level:

Generalizability(c.f. Oosterhof, et al, 2008)

“…the degree to which learner performances that are observed generalize to performances that are not observed.” [i.e. are they learning what we think we’re teaching?]

7

So, how are we doing this?

Caveat: We’re not saying this is right or wrong…it’s just the way

that we’re currently approaching it.

8

“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”

(CS 3.3.1)

COC Distance Education Policy Statementhttp://www.sacscoc.org/pdf/Distance%20and%20correspondence%20policy%20final.pdf

9

The (former) Guidelines

10

11

2009-2012 Strategic Plan

We will create a culture where all institutional actions are focused on improving student learning and success and leading to student goal completion.

Q1: Is the distance education program reflected in the institution’s strategic (or master) plan?

PM 16—Success and Withdrawal Rates by Instructional modalities

12

Q2: Is there evidence that outcomes for the program have been identified?

DL-AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.

DL-AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.

13

AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.

Assessment: Quality Assessment Plan, Success/ Persistence Rates, Student Surveys/Evaluations.

AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.

Assessment: The number of internet/hybrid sections offered.

Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?

14

QAP

15

The QAP(Quality Assessment Plan)

• Developed in-house (using models)

• Peer-reviewed by content specialist

• Stipends (T3)

• All current online courses have been reviewed

• All new online courses must be reviewed/ certified

http://web.carteret.edu/keoughp/TitleIII/FinalQAP.xls16

The QAP Score Card

17

points is the total possible a teacher can earn - believe its 91

minimum is what they need to pass - I am happy if they get 63 or above (hybrids may do feedback in class or individually so we learned the way the QAP is now hybrid instructors may get lower scores due to no feedback to students even through they may do it during or after class - so we are modifying this for hybrids next go around

earned is what they actually got so Laurie Freshwater may have earned a 70 and Mary Walton a 67 - below 60 we have them come back to Pre-Ah to do a 2 hour meeting to fix what needs to be addressed.

The QAP Score Card: Learning

18

QAP

Success/Persistence

19

Success - Spring 2010

• 75 sections

• seated: 75% success (persist: 87.1%)

• internet: 66% success (persist: 81.7%)

• hybrid: 73% success (persist: 87.8%)

• FA09 internet: 62 sections; 68% success

• FA08 internet: 61 sections; 67% success

• FA07 internet: 59 sections; 66% success

• FA06 internet: 48 sections; 66% success20

QAP

Success/Persistence

Program/Course Evalua7ons

21

Surveys & Evaluations: DL Program

22

Surveys & Evaluations: DL Program

23

Surveys & Evaluations: Course-Level

24

Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?

Modality Comparison Grade Distributions Program Reviews ILLOs (Gen Ed outcomes)

25

Modality Comparison

26

Modality ComparisonInstitutional-level

27

Modality ComparisonProgram-level

28

Modality ComparisonCourse-level

29

Modality Comparison

Grade Distribu7ons

30

Grade Distributions institution-level

%s

31

Grade Distributions course-level

32

Modality Comparison

Grade Distribu7ons

Program Reviews

33

Evaluate Instructional Modalities

34

!

Modality Comparison

Grade Distribu7ons

Program Reviews

ILLOs

35

ILLOs (Gen Ed outcomes)

1. Communication

2. Computer Literacy

3. Critical Thinking

4. Humanities & Fine Arts

5. Information Literacy

6. Personal Growth & Responsibility 36

Computer Literacy - CIS 110Outcome: Students demonstrate proficiency in the use of computer technology including application software, system software, and internet browsers.

Assessment: Final Exam in CIS 110.

37

Computer Literacy - CIS 110 Section analysis

38

Computer Literacy - CIS 110Same instructor teaching one section of each

39

40

Personal Growth & Responsibility ACA 115Outcome: Students demonstrate the ability to understand and manage self effectively.

Results Spring 2009: 131 total projects submitted; 81% scored 80 or above.

Seated 115: 85% scored 80 or above Online 115: 81% scored 80 or above

Assessment: Advisement assignment in ACA 115 & 118 evaluated by rubric.

41

Q4: Has the institution developed student learning competencies for the courses/programs offered by distance education? If these are the same competencies for courses/programs offered by “traditional” methodologies, is assessment identified for distance learning students separate from students taking courses by “traditional” methodologies?

42

“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”

So, are they learning what we think we’re teaching?

43

Yes! … No?…maybe??• PLLOs• Course-Level Assessment

And the answer is…

44

PLLOs

45

PLLOs (SLOs)

Same instructor teaching one section of each (common assignment from each section)

Business Administration• ACC 120• BUS 110• ECO 251/252

46

PLLO: preparation of financial statements as part of mid-term

Assessed in: ACC 120

47

BUS 110

ECO 250

48

PLLOs

Course‐Level Assessment

49

Course-level assessment - CLLOs• ACC 120

• ART 111

• EDU 119

• ENG 111

• HIS 111

• PSY 150 & 241

• RCP 114

• SOC 210 50

Course-level assessment ~ CLLOs

ACC 120 … Students will be able to:

1. Illustrate the steps in the accounting cycle.

2. Compute cost of goods sold and ending inventory using various pricing methods.

3. Prepare a multiple step income statement, retained earnings statement and classified balance sheet.

51

“Students placed more value on the CMS as an effective learning tool when they perceived the CMS as supporting interactivity among student, instructor, peers, content.” (Chou, Peng, Chang, 2010)

[as quoted by Cook, G. & Boldt, S.]52

ART 111Grade Evaluation ( Percentages and Points )

AssignmentCategory

Number ofAssignments

% of ea.towardFinalGrade

Total pointsgenerated by ea.

individualassignment

Total % ofCategory inFinal Grade

Total Pointsof Category

in Final grade

WeeklyQuizzes 14 2.5% ea. 80 35% 1,120

DiscussionBoard 16 2.5% ea. 80 40% 1,280

Mid TermEssay 1 10% ea. 320 10% 320

Final Exam 1 15% ea. 480 15% 480Totals 32 100% 3200

53

A - (100 - 90): - You posted 3 or more times and you completed the Assignment with a well thought through response. (The more you post MEANINGFUL responses over 3 the higher your grade. Though welcome and expected, please understand that a One or Two sentence "feel good" post such as (... Great response Michaela!!) or (... I love how you put the meaning and then went down and talked about them. it made it easy to understand!!) will count for very little towards your grade. 54

55

HIS 111

Essays (online) vs. Research Paper (seated) … same rubric 56

RCP 114

57

Other considera7ons

• Outcome‐assessment alignment

• Downstream assessment (course or clinical)

• E‐ports

• Sequenced self‐assessment (peer review ‐ reflec7on ‐ conference)

• 3‐2‐1 Evalua7on (3 concepts learned, 2 ques7ons asked, 1 idea posed)

58

59

Challenges

“I got rid of [the discussion board]…it’s a pain in the ass.”

“I have to pull teeth to get them to post even when they get a score”

“Value of discussion board is suspect at best.”

60

“My first goal is to get the instructors to agree on common CLLOs. Then we’ll work on common assessments.”

61

Can you say Parasympathemimetic?

“I don’t know” [what texts and assessments the other instructors are using in this multi-section course]

Outcome-Assessment alignment

Donald Staub Title III Director

Carteret Community Collegestaubd@carteret.edu

252- 222-6010

http://ccctitle3.wordpress.com/www.carteret.edu

62