Adventures in CPSAdventures in CPSAaron BruckAaron Bruck

Towns Group MeetingTowns Group MeetingSeptember 25, 2007September 25, 2007

Goals for the summerGoals for the summer►Complete CPS projectsComplete CPS projects

2 different projects2 different projects►Quantification and categorization of CPS Quantification and categorization of CPS

questionsquestions►Linking CPS results to results on examsLinking CPS results to results on exams

►Look into new ideas for research (and Look into new ideas for research (and possibly an OP)possibly an OP) Scientific LiteracyScientific Literacy Assessment toolsAssessment tools

What is CPS?What is CPS?►Students use “clickers” to Students use “clickers” to

respond to instructor-respond to instructor-generated questionsgenerated questions

►These responses are stored These responses are stored in an online databasein an online database

►Students received grades Students received grades based on the number of based on the number of questions answered correctlyquestions answered correctly

CategorizationCategorization►We decided to categorize the questions We decided to categorize the questions

in the following ways:in the following ways: Solo vs. BuddySolo vs. Buddy Definition vs. Algorithmic vs. ConceptualDefinition vs. Algorithmic vs. Conceptual Using Bloom’s TaxonomyUsing Bloom’s Taxonomy Using Smith/Nakhleh/Bretz FrameworkUsing Smith/Nakhleh/Bretz Framework11

►We also compared our analyses with We also compared our analyses with those of Mazurthose of Mazur22 to make sure we were to make sure we were looking for the right things.looking for the right things.

1Smith, K. C., Nakhleh, M. B., Bretz, S. L. An Expanded Framework for Analyzing General Chemistry Exams. Journal of Chemical Education. In press.2Fagen, A. P., Crouch, C. H., Mazur, E. (2002) Peer Instruction: Results from a Range of Classrooms. The Physics Teacher. 40, 206-209.

Categorization, cont.Categorization, cont.► Here are the results from one of the sections Here are the results from one of the sections

(others followed a similar trend):(others followed a similar trend):

Bloom's Taxonomy # of questions # Definition # Algorithmic # Conceptual

Knowledge (1) 44 40 3 1

Comprehension (2) 44 14 11 19

Application (3) 27 4 21 2

Analysis (4) 3 0 1 2

Synthesis (5) 0 0 0 0

Evaluation (6) 0 0 0 0

total 118 58 36 24

Bloom's Taxonomy # of questions # solo # buddy

Knowledge (1) 44 41 3

Comprehension (2) 44 34 10

Application (3) 27 18 9

Analysis (4) 3 1 2

Synthesis (5) 0 0 0

Evaluation (6) 0 0 0

total 118 94 24

More categorizationMore categorizationSmith/Nakhleh Framework

Bloom's Taxonomy # questions # Definition # A-MaMi # A-MaD # A-MiS # A-Mu # C-E # C-P # C-I # C-O

Knowledge (1) 44 42 0 0 1 0 0 1 0 0

Comprehension (2) 44 15 1 0 7 0 1 16 4 0

Application (3) 27 4 7 4 6 3 0 1 2 0

Analysis (4) 3 0 0 0 0 1 0 2 0 0

Synthesis (5) 0 0 0 0 0 0 0 0 0 0

Evaluation (6) 0 0 0 0 0 0 0 0 0 0

total 118 61 8 4 14 4 1 20 6 0

Results by CategoryResults by Category► A 2 tailed t-test (solo/buddy) and one way A 2 tailed t-test (solo/buddy) and one way

ANOVAs (all others) were performed to test ANOVAs (all others) were performed to test for statistical differences in the datafor statistical differences in the data

► Analyses showed no significant differences Analyses showed no significant differences between any of the categories and how the between any of the categories and how the students performed on the questionsstudents performed on the questions

► The only exception were the solo-buddy The only exception were the solo-buddy questions for one professorquestions for one professor

Solo vs. Buddy Question Type (D/A/C)

Bloom’s Taxonomy

Smith/Nakhleh/Bretz Framework

t p F p F p F p

Professor A -3.189 0.002* 0.730 0.485 0.307 0.820 0.285 0.942

Professor B 0.049 0.962 1.301 0.277 1.102 0.352 1.102 0.429

Professor C -0.579 0.564 1.001 0.371 2.456 0.067 1.923 0.064

Solo/Buddy AnalysisSolo/Buddy Analysis► Prompted by the unusual results, we further Prompted by the unusual results, we further

investigated the solo/buddy analysisinvestigated the solo/buddy analysis► We also looked at pairs of solo/buddy We also looked at pairs of solo/buddy

questions asked one after the other:questions asked one after the other:

Solo/Buddy N Mean Std. DeviationStd. Error

MeanSolo/Buddy 1 8 45.5875 19.09786 6.75211

2 8 70.2525 17.42225 6.15970

T-test results: t=-2.699 p=0.017 (significant difference)

That’s great, but…That’s great, but…►We found a significant difference We found a significant difference

between solo and buddy questions…between solo and buddy questions…but is it worth anything?but is it worth anything?

►Our next step was to see if this Our next step was to see if this apparent difference in performance apparent difference in performance due to style of question translated into due to style of question translated into better test scores on the exams.better test scores on the exams.

Exam AnalysisExam Analysis►We compared exam questions with We compared exam questions with

questions asked in class using CPS.questions asked in class using CPS.►Surprisingly, we found very few Surprisingly, we found very few

questions on the exams that directly or questions on the exams that directly or indirectly corresponded to CPS indirectly corresponded to CPS questions.questions.

►Each exam was analyzed individually Each exam was analyzed individually before pooling all of the data to before pooling all of the data to determine any and all effects.determine any and all effects.

Exam AnalysisExam Analysis# solo # buddy # neither

Exam 1 17 0 40Exam 2 21 9 31Exam 3 10 10 36Exam 4 29 10 64Totals 77 29 171

Question Effects  F value p valueExam 1 3.508 0.066Exam 2 2.162 0.124Exam 3 2.718 0.075Final Exam 2.793 0.066Pooled Exams 1.632 0.197per instructor…  Professor A 1.032 0.361Professor B 0.341 0.712Professor C 1.468 0.236

% correct

solo

% correct Buddy

% correct Neither

Exam 1 68.119 n/a 57.5222

Exam 2 56.5675 62.1900 66.1966

Exam 3 68.1138 67.9493 54.5532

Exam 4 66.1699 50.3368 60.3920

Totals 64.2338 60.0887 59.5438

All analyses showed no significant differences at the p=0.05 confidence level.

Instructor EffectsInstructor Effects► We also ran an We also ran an

analysis to check for analysis to check for any instructor any instructor effects that could effects that could have possibly have possibly skewed the data. skewed the data.

► Results showed no Results showed no significant significant differences at the differences at the p=0.05 level:p=0.05 level:

Instructor Effects

  F value p value

Exam 1 0.54 0.586

Exam 2 0.484 0.619

Exam 3 0.108 0.898

Final Exam 1.255 0.289

Pooled Exams 0.987 0.374

Is CPS better than nothing?Is CPS better than nothing?►A final analysis was performed between A final analysis was performed between

questions that correlated to CPS questions that correlated to CPS questions and those that did not.questions and those that did not.

►Unfortunately, no significant differences Unfortunately, no significant differences were found, though the average score were found, though the average score was higher for CPS questions.was higher for CPS questions.

CPS vs. Nothing ResultsCPS vs. Nothing Results

Results of ANOVA:

Percent Correct Sum of Squares df Mean Square F Sig.

Between Groups 827.451 1 827.451 2.271 .133

Within Groups 100204.068 275 364.378

Total 101031.520 276

Descriptive Statistics:

Percent Correct N Mean Std. Deviation Std. Error

1 106 63.0998 17.25487 1.67594

2 171 59.5438 20.13811 1.54000

Total 277 60.9046 19.13260 1.14957

ConclusionsConclusions►CPS is an effective lecture tool that CPS is an effective lecture tool that

engages students interactively in their engages students interactively in their contentcontent

►Most CPS questions are low-level Most CPS questions are low-level questions in terms of Bloom’s Taxonomy questions in terms of Bloom’s Taxonomy and other categorization toolsand other categorization tools

►Students seem to learn content through Students seem to learn content through interaction with their peers when using interaction with their peers when using CPS, though this does not necessarily CPS, though this does not necessarily correlate to success on examscorrelate to success on exams

What else did I do?What else did I do?► Research QuestionsResearch Questions

In the event that I need to do a project other than In the event that I need to do a project other than the NSDL project, what avenues are available?the NSDL project, what avenues are available?

Could any of these ideas turn into a possible OP in Could any of these ideas turn into a possible OP in the following months?the following months?

► Ideas of interestIdeas of interest Scientific LiteracyScientific Literacy

► What is the value of a textbook?What is the value of a textbook?► Could other materials help?Could other materials help?

AssessmentAssessment► Immediate feedback assessment technique (IFAT)Immediate feedback assessment technique (IFAT)

Could it work in chemistry?Could it work in chemistry?