an empirical study of in-class labs on student learning of linear data structures sarah heckman...

An Empirical Study of In-Class Labs on Student Learning of Linear Data StructuresSarah HeckmanTeaching Associate ProfessorDepartment of Computer ScienceNorth Carolina State University

ICER 2015

Problem

ICER 2015

CSC116 CSC216 CSC316

• 7-8 sections• 33 students• 1 instructor• 2 TAs• Lecture/Lab

• 1-2 sections• 70-90 students• 1 instructor• 2-3 TAs• Lecture

• 1-2 sections• 70-90 students• 1 instructor• 2-3 TAs• Lecture

Transition!

Retention!

Lab? In-Class Labs?Do Nothing?

Research Goal

• To increase student learning and engagement through in-class laboratories on linear data structures

• Hypothesis: active learning practices that involve larger problems would increase student learning and engagement

ICER 2015

In-class Labs > Pair & Share

Research Questions

• Do in-class laboratories on linear data structures increase student learning on linear data structures exam questions when compared to active-learning lectures?

• Do in-class laboratories on linear data structures increase student engagement on linear data structures exam questions when compared to active-learning lectures?

ICER 2015

Active Learning in CSC216• “engaging the students in the process of learning

through activities and/or discussion in class, as opposed to passively listening to an expert” [Freeman, et al. 2014]

• Control: Active Learning Lectures – 2-5 pair & share exercises per class– Submitted through Google forms

• Treatment: In-class Labs– Lab activity for the entire lecture period– Pre-class videos introduced topic

ICER 2015

Study Participants

Metric Section 001 Section 002# Enrolled 85 102Participants (completed course)

49 60

Dropped/Withdrawn (consenting only)

3 4

Women 9 10Meeting Time TH 2:20-3:35p MW 2:20-3:35p

ICER 2015

• Self-selected into section during standard registration period• Populations were similar as measured by a survey on

experience with tooling and self-efficacy.

Methods• Quasi-Experimental

– Counter-balanced design– Learning measured through exams– Engagement measured through observations of class

meetings

ICER 2015

Lists

Array Array

Array Array Linked Linked

Linked Linked

Iterators

Exam 1

001

002

Replication Materials:http://people.engr.ncsu.edu/sesmith5/216-labs/csc216_labs.html

Replication Materials:http://people.engr.ncsu.edu/sesmith5/216-labs/csc216_labs.html

Observed Class Meetings

Student Learning – Exam 1

• Part 4: Method Tracing with ArrayLists• Part 5: Writing an ArrayList method

ICER 2015

Item Points S001 Mean

S001 SD

S002 Mean

S002 SD

p-value

E1 P4#8 5 3.63 1.56 4.35 1.45 < 0.010

E1 P4#9 5 4.18 1.07 4.57 1.09 0.016

E1 P4#10 5 2.63 2.40 3.45 2.18 0.149

E1 P4 15 10.45 3.97 12.37 3.74 < 0.010

E1 P5 20 17.76 4.0 18.25 4.09 0.233

Student Learning – Exam 2

• Part 3 – Linked Node Transformation• Part 5 – Writing a LinkedList Method

ICER 2015

Item Points S001 Mean

S001 SD

S002 Mean

S002 SD

p-value

E2 P3 16 9.43 5.85 11.80 6.41 < 0.010

E2 P5 20 11.80 4.14 12.58 4.21 0.412

Student Learning – Exam 3

• Comprehensive 3 hour final exam• Stack Using an ArrayList• Queue Using a LinkedList

ICER 2015

Item Points S001 Mean

S001 SD

S002 Mean

S002 SD

p-value

E3 Array 10 8.31 2.45 8.46 2.49 0.313

E3 Linked 10 8.36 2.53 881 2.45 0.221

E3 Score 105 85.02 29.17 87.23 28.92 0.372

Student Engagement

• Observations for ArrayList and LinkedList class meetings

• Observers were graduate students and a colleague participating in a Teaching and Learning seminar

• Counts of students off topic during lecture and exercise portions of the class

• Some inconsistent use of the observation protocol

ICER 2015

Student Engagement

ICER 2015

Observation Class Type

# Off Topic – Lecture

# Off Topic – Exercise

Questions of Teaching Staff

1 Lab 5 7 32

2 Lecture 62 49 12

3 Lab 10 43 50

4 Lecture 46 16 ---

5 Lecture --- --- ---

6 Lab 5 10 33

7 Lecture 52 54 2

8 Lab 16 5 ---

Lab Average 9 16.3 38.3

Lecture Average 53.3 39.7 7

Lecture / Lab 5.9 2.4 0.2

Threats to Validity

• External Validity– Two sections of the same course, taught by the same

instructor, in the same semester, and same time of day

– Replication needed in other contexts to generalize further

– Could provide additional data points in future meta-analyses

ICER 2015

Threats to Validity• Internal Validity

– Selection bias: students selected their own sections• Initial surveys shows groups were similar

– Confounding factors• Materials shared between groups• Effect size – only 6 in-class labs

– Differential Attrition Bias• Considered “soft-drops” in the study

– Experimenter Bias• Participants were not revealed until after the semester was over

ICER 2015

Threats to Validity

• Construct Validity– Exams as Measures of Learning

• Exam 1 and Exam 2 were similar, but not the same, between sections

• Exam 3 was common• Does exam really measure student learning?

– Survey• Wording may be confusing for prior tool experience• Efficacy questions not a validated instrument

– Observation Protocol as Measure of Engagement• Inconsistent use by observers

ICER 2015

Discussion• Did in-class labs increase student learning?

– No, at least not as measured by exam questions– Both control and intervention were active learning

• Maybe a simple active learning intervention is enough– Comparisons with earlier semesters may show more

• Did in-class labs increase student engagement?– Yes and No– The atmosphere in the classroom was fantastic– But many questions were technology and not concept

• Completion – 72% of students earned a C or higher– Not reaching the higher levels of completion we expect from active

learning literature

ICER 2015

Future Work• Additional Work on Fall 2014 Data

– Compare results on final exam with previous courses– Incorporate analysis of other measures of learning –

projects, exercises, etc.• Starting in Fall 2015

– Additional in-class labs → Lab-based course– Measure types of questions asked during in-class labs– Use labs as a way to encourage best practices

(frequent commits to version control, TDD)

ICER 2015

Replication Materials:http://people.engr.ncsu.edu/sesmith5/216-labs/csc216_labs.html

Replication Materials:http://people.engr.ncsu.edu/sesmith5/216-labs/csc216_labs.html

Thank You!

Questions?Comments?Concerns?

Suggestions?

ICER 2015